microsoft word jethe editor letter april 2019 issue.docx journal of effective teaching in higher education, vol. 2, no. 1 dear contributors, reviewers, and readers: we would like to thank our colleagues from around the world who served as reviewers on our second issue of the journal of effective teaching in higher education. the interest in publishing with jethe has been incredible, and we could not have published this second issue without your help. the primary goal of jethe is to deliver high quality, data driven articles contributing to the scholarship of teaching and learning (sotl) in higher education. we believe the nine pieces included in volume 2 issue 1 demonstrate this commitment and reflect the range of work we will continue to publish in future issues. the articles address a diverse range of topics relevant to teaching in higher education today, including universal design for learning (udl), gaming pedagogies and techniques, and queer pedagogy, among others. the authors represent an equally diverse set of institutions from across the us and canada, and they utilized a range of qualitative, quantitative, and mixed methods in their work. we hope you find this issue to contain interesting and engaging examples of the scholarship of teaching and learning. all of the pieces provide meaningful implications for the practice of teaching across disciplines—an important contribution that is emphasized in jethe. we invite you to engage with the pieces in our second issue and consider making your own contributions to this work. we also invite you to become a reviewer for jethe. if you are interested in reviewing for our journal, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. the deadline for submissions for consideration for the next issue of jethe is august 15, 2019. we look forward to receiving interesting and rigorous submissions and working with some of you as authors and reviewers. james devita and colleen reilly coeditors, jethe microsoft word jethe editor letter fall 2022.docx journal of effective teaching in higher education, vol. 5, no. 2 dear contributors, reviewers, and readers: in many ways, the fall 2022 issue of the journal of effective teaching in higher education (jethe) reflects the ongoing efforts of faculty in higher education to return to a “new normal” as a result of the covid-19 pandemic. we continue to experience delays with the review and publication process but remain grateful for the work completed by our authors and reviewers who continue to contribute to jethe. readers will observe a notable theme in the fall 2022 issue, namely the increased emphasis on online and hybrid approaches to teaching. in the first article in volume 5, issue 2, craig, bergstrom, brooks and bichard focus on “navigating the new normal: teaching in the time of covid.” the three articles that follow address other aspects of online teaching and learning: leibold and schwarz examine “student perceptions of teacher online feedback,” love and marshall focus on “video feedback and instructor social presence in an asynchronous online course,” and gullo discusses “supporting students with disabilities in online learning environments.” the two additional articles focus on perceptions and assessments of student learning. elhage, black-parker, and laoesebikan utilized survey methods to examine chicago state university alumni experiences in their piece focused on student perceptions of educational efficacy at a predominantly black institution (pbi). dandotkar, cruz, and britt detail the relationship they found between students’ beliefs about the speed of knowledge acquisition and their memory for critical argument elements. as always, these articles include implications for the practice of teaching across disciplines—a central feature of all articles in jethe. despite the ongoing struggles we encountered as we seek a “new normal” in our work—and in higher education more generally—we are honored to present the contributions of the authors included in the current issue. we hope to see our audience engage with the articles and thank you all for your continued interest in and support of our efforts at jethe. we are currently accepting submissions for our spring 2023 and future issues and invite you to contribute your own work to the journal. to submit an article and/or download full-text copies of current and past issues please visit http://jethe.org. finally, we would like to invite you to become a reviewer for jethe. if you are interested in reviewing for our journal, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. thank you for your continued support of jethe! james devita and colleen reilly coeditors, jethe microsoft word jethe editor letter november 2019 issue.docx journal of effective teaching in higher education, vol. 2, no. 2 dear contributors, reviewers, and readers: one year ago we published our first issue of the journal of effective teaching in higher education. we have been overwhelmed by the positive response to the journal and would like to thank our colleagues from around the world who have submitted articles for consideration, served as reviewers, and downloaded and read the articles published in our first two issues (november 2018 and april 2019). the interest in publishing with jethe has been incredible with over 80 submissions in our first year. similarly, the demand for access to articles published in jethe has been equally impressive with nearly 2,000 full-text downloads of articles published in our first two issues. your help and support have been invaluable as we continue to grow and develop the journal. as part of our efforts to develop, we have recruited a diverse and committed group of scholars to serve on our editorial board. these individuals have graciously agreed to work with us to help to continue to deliver high quality, data driven articles contributing to the scholarship of teaching and learning (sotl) in higher education, which is the primary goal of jethe. we want to recognize those individuals for their willingness to serve with us in this role: deb bennett, mount royal university, calgary, ab canada amy fisher, university of mississippi, oxford, ms usa rosemary fisher, swinburne university of technology, victoria, australia laura pipe, university of north carolina greensboro, greensboro, nc usa hillary steiner, kennesaw state university, kennesaw, ga usa kathryn wymer, north carolina central university, durham, nc usa we believe that our commitment to high quality, data driven scholarship is exemplified by the eight pieces included in volume 2 issue 2. these articles address a diverse range of topics relevant to teaching in higher education today and effectively utilize a range of qualitative, quantitative, and mixed methods. articles in this issue cover topics related to online learning, including online courseware and community building. additionally, authors in this issue explore teaching in a range of disciplines, including calculus, biology, and writing. the articles also explore teaching across institutional contexts, notably among us community colleges and at an institution in new zealand. finally, several articles examine the effectiveness of peer coaching models and strategies to enhance critical reflection. letter from the editors 2 journal of effective teaching in higher education, vol. 2, no. 2 we hope you find that this issue includes interesting and engaging examples of the scholarship of teaching and learning. all of the pieces provide meaningful implications for the practice of teaching across disciplines—an important contribution that is emphasized in jethe. we invite you to engage with the pieces in our november 2019 issue and consider making your own contributions to this work. we also invite you to become a reviewer for jethe. if you are interested in reviewing for our journal, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. our next issue will be published in april 2020, and we look forward to working with some of you as authors and reviewers in production of that and future issues of jethe. to submit an article or download full-text copies of current and past issues please visit http://jethe.org. james devita and colleen reilly coeditors, jethe microsoft word editors_final.docx journal of effective teaching in higher education, vol. 6, no. 1 https://doi.org/10.36021/jethe.v6i1.377 dear contributors, reviewers, and readers: in 2017, we were invited by the director of the center for teaching excellence (cte) at our home institution, the university of north carolina wilmington (uncw), to bring new leadership to a journal on teaching and learning that was housed in the unit. at that time, the journal was known as the journal of effective teaching (jet) and was published in print and online although the online archive remains incomplete. when we became co-editors of the journal, we saw the potential to shift the focus of the journal to its current emphasis on scholarship of teaching and learning (sotl) in higher education and relaunched it as the online, open-access journal of effective teaching in higher education (jethe). when we first launched jethe, we didn’t know what to expect in terms of interest from potential authors or audiences. we have been humbled by the ongoing support and interest in the journal over the last 6+ years, and it has been a joy to work collaboratively as co-editors in this effort. after publishing our spring 2023 issue (volume 6, issue 1), dr. colleen reilly will be stepping down as co-editor of jethe to pursue another leadership position at uncw. her contributions to the journal are immeasurable and without her support we simply would not have the journal we do today. while she will be sorely missed as co-editor, colleen will remain a part of the editorial board, and we are grateful for all her efforts on behalf of jethe past, present, and future. this summer, we will be posting a call for individuals interested in joining jethe as a co-editor, associate editor, and/or editorial board members. we will seek a diverse group of scholars from different disciplines, parts of the world, and methodological expertise—and we hope you will consider applying to serve in one of these roles and help us to shape the future of the journal. thanks again for your ongoing support, and we hope you enjoy the current issue of jethe. included in volume 6, issue 1 of jethe are six (6) articles focused on a range of sotl topics that relate to critical issues in higher education today. the first article by suart, cassidy-neumiller, and harvey explores “modalities of faculty engagement with the scholarship of teaching and learning” at a university in canada. in addition to providing a summary of findings from their research, suart et al. provide recommendations for other institutions seeking to support faculty engagement in sotl. letter from the editors ii journal of effective teaching in higher education, vol. 6, no. 1 the next two (2) articles in the issue examine experiences of specific populations in higher education and seek to identify strategies for supporting these students’ learning. ricci and french are “aiming for alignment: faculty and student perspectives on effectively teaching first-generation students,” while baker and tukhvatulina share “reflections of adult learners in asynchronous online degree programs.” both articles offer key insights into supporting the respective student populations in their research. the two (2) articles that follow focus on methods of course delivery across various modalities. akkaraju discusses their use of “the oral exam— learning for mastery and appreciating it” in an online course. thibault and schroeder discuss their research comparing “traditional, mixed-mode, and online instruction for microbial metabolism before and during covid-19”— and seek to answer a question faced by many instructors about the most effective modality for instruction. the last article included in the issue focuses on an emerging topic in higher education: trauma-informed approaches to education. wells shares findings from “creating trauma-informed higher education classrooms: exploring undergraduate and graduate student perceptions” and asks some important questions about how higher education is integrating trauma-informed approaches into teaching and learning. thank you for your continued interest in and support of our efforts at jethe. we are currently accepting submissions for our fall 2023 and future issues and invite you to contribute your own work to the journal. to submit an article and/or download full-text copies of current and past issues please visit http://jethe.org. we hope you find this issue engaging and meaningful to your own work in the classroom and as a scholar. finally, please stay on the lookout for our call for new editors and editorial board members for jethe and thank you for your continued support of the journal! james devita and colleen reilly coeditors, jethe microsoft word blanton.docx journal of effective teaching in higher education, vol. 4, no. 1 learning by design: rethinking curriculum alignment through collaborative inquiry morgan v. blanton, appalachian state university, blantonmv@appstate.edu linda c. pacifici, appalachian state university abstract. this article reports on an application of a sotl research process for two teacher education instructors in an undergraduate teacher education course, teaching and learning in the digital age. after engaging in content analysis of student work samples derived from a digital sabbath assignment, the instructors asked more questions about how well the assignment aligned to course objectives and intended learning outcomes. collaboratively, they determined how the digital sabbath assignment could be revised to better meet the stated course objective. the authors anticipate that assignment revisions could result in improved student learning outcomes for the teacher education students. insights from the student responses and course materials are discussed as well as implications for teaching and learning in a teacher education program. keywords: teacher education, curriculum, scholarship of teaching and learning (sotl), digital sabbath “changing the status of the problem in teaching from terminal remediation to ongoing investigation is precisely what the movement for a scholarship of teaching is all about” (mckinney, 2007, p. 29). we, as two university instructors, found ourselves at this very crossroad realizing that a course learning goal was not met through a specific learning activity. we analyzed student learning products from a digital sabbath assignment, which required preservice teachers to take a break from social technology for an extended period of time. upon reflection, we noted a discrepancy between an assignment’s learning outcomes and the course goals. the research question that guided our inquiry was, “to what extent is the digital sabbath assignment producing intended outcomes that lead to mastery of course objective 2?” in this article, the authors briefly describe the assignment results as the stimulus for examining it as a curriculum alignment issue. we used the scholarship of teaching and learning’s content analysis and wiggins and mctighe’s backward design as our primary methodological approaches (bishop-clark & dietzuhler, 2012; chick, 2019; mckinney, 2007; wiggins & mctighe, 2005). a problem emerges the initial purpose for our collaboration was rooted in our desire to learn about the potential differences across five years of student projects in our teaching and learning in the digital age course. we thought that five years and significant changes in social technology would have an impact on how students viewed learning and learners in the digital age. however, through collaborative reflection regarding our assignment results, we realized that the greater issue was not rethinking curriculum alignment through collaborative inquiry 2 journal of effective teaching in higher education, vol. 3, no. 2 necessarily that our students experienced the digital sabbath assignment differently between 2012 and 2017. the problem was more importantly about the discrepancy we noticed between the assignment outcomes and our intended learning goals. it was not until we analyzed student learning data collaboratively that we realized the learning outcomes of the digital sabbath assignment did not meet our expectations or assumptions of course objective 2. the lack of alignment between our assignment instructions and our course objectives became apparent. therefore, we refocused our attention to curriculum design instead of ending the study after analyzing the digital sabbath assignment data findings. conceptual framework scholarship of teaching and learning (sotl) is defined as “the study of teaching and learning and the communication of findings so that a body of knowledge can be established” (bishop-clark & dietz-uhler, 2012, pg. 1; chick, 2019). as educators, sotl provided us with a roadmap for inquiring and exploring questions we had about our teaching and our students’ learning in a more formalized and structured manner. further, fanghanel (2013) suggests that sotl “offers a space for critique and endorsement of practice” (p. 60). our use of the sotl research methodology enabled us to think more deeply about our learning activities. the backward design model provided a guide for us in which lesson planning begins at the end of an instructional cycle with the learning goals (wiggins & mctighe, 2005). learning activities are the vehicle to reach the learning goals and come after the learning goals are identified. we applied the backward design model to our course in order to determine when and how our learning outcomes and intended learning objectives became misaligned. figure 1 illustrates how we envision the teaching/instruction framework of learning goals, assessment evidence, and learning plan working together with the sotl process. the backward design steps are anchored with the teaching process in an iterative manner. we applied the model in three stages. stage one focused on examining the desired results, stage two considered the assessment evidence, and stage three scrutinized the learning plan. figure 1 our conceptual framework rethinking curriculum alignment through collaborative inquiry 3 journal of effective teaching in higher education, vol. 3, no. 2 stage one: desired results teaching and learning in the digital age is a teacher education undergraduate course that explores topics related to emerging technologies and teaching in a 21st century world. the course is required in the professional core curriculum in a teaching education program at a regional comprehensive university in the southeast united states. the digital sabbath assignment was intended to align with course objective 2, “express the changing nature of learners and learning in the digital age.” stage two: assessment evidence to determine how well our students met course objective 2, we asked students to keep an activity log detailing how they spent their time without digital technology. in addition, we gave them reflection prompts to guide their thinking after the digital sabbath experience was completed. their responses in the activity logs and in their reflections were used to determine the learning outcomes and how well they mastered course objective 2. stage three: learning plan our preservice teachers participated in a digital sabbath assignment during the first month of the semester. the purpose of this assignment was for students to experience and gain insights by comparing and contrasting life with and without technology. considering course objective 2, “express the changing nature of learners and learning in the digital age,” we wanted our students, as future teachers, to understand and articulate how this experience related to them and their future students. the hours without digital communication were required to be consecutive and students were encouraged to choose a normally busy time of day in which they were typically on their devices. methods if the digital sabbath assignment was created to align with course objective 2, “express the changing nature of learners and learning in the digital age,” then how did our students’ activity logs and reflections indicate how well they met this learning goal? though we believed the digital sabbath experience was valuable to our students as individuals in a digital society, did it put them on a trajectory to better express the changing nature of learners and learning in the digital age, especially as future classroom teachers? based on these uncertainties, we decided to engage in collaborative reflection that examined the digital sabbath assignment by analyzing the design and alignment to our course objective. therefore, the research question that guided our collaborative work was, “to what extent is the digital sabbath assignment producing intended outcomes that lead to mastery of course objective 2?” rethinking curriculum alignment through collaborative inquiry 4 journal of effective teaching in higher education, vol. 3, no. 2 research design chick (2019) describes two project designs used in sotl research: “what is” and “what works” projects. our research design best fits into chick’s description of a “what is” project because we engaged in a systematic process of investigating our course assignments as teacher scholars. the “what is” focus became the centerpiece of our conceptual framework, and backward design became the systematic process through which we examined the assignment and learning outcomes. in “what is” projects, scholars provide “thick descriptions, snapshots of learning dynamics, or other detailed analyses of what student learning looks like, or what’s happening during the moments of learning” (2019). shifting our focus to our own instructional practices, we examined the findings of our digital sabbath assignment through the lens of curriculum design (mckinney, 2007). by collecting additional data from our assignment instructions and course objectives, we hoped to determine how well the assignment resulted in intended learning outcomes. data collection using backward design as a guide, we collected and analyzed data from the student responses, course objectives, and assignment instructions to better understand the instructional alignment, or lack thereof. we organized student learning data from the reflection and activity logs and began comparing those learning outcomes to course objective 2, “express the changing nature of learners and learning in the digital age.” in addition to our original data from student responses, we also used data from the assignment itself. we examined the instructions to understand the context students worked in as they completed the assignment. data analysis to analyze our data, we reviewed the course objectives, assignment instructions, and the nature of the students’ written responses. we engaged in document analysis which included cross-walking course objectives with intended assignment outcomes. further, we coded for common themes in the students’ reflections and cross-walked those themes with course objective 2. findings in stage 1, we identified the learning goal, and moved ahead to stage 2 of the backward design model. in stage 2, the assignment learning outcomes were closely reviewed. the primary learning goal in the digital sabbath assignment was for students to understand the changing nature of learners and learning in the digital age. assignment instructions were written directing students to reflect on their experience without digital technology. as teacher educators, we hoped the students would express how the digital sabbath experience informed them as future teachers. when designing this assignment, we assumed that if students reflected rethinking curriculum alignment through collaborative inquiry 5 journal of effective teaching in higher education, vol. 3, no. 2 on their experience and extended their thinking to the classroom, then this course objective would be partially met by completing this assignment. course objectives and learning outcomes alignment “the sotl puts the focus of the academic enterprise on students’ learning and urges the instructor to investigate, document, and present the results” (mckinney, 2007, pg. 101). we collected data from students’ activity logs and reflections. we coded these data for themes (elliott, 2018). descriptions and reflections related to the social and emotional impact of a digital sabbath were the most prevalent learning outcomes. increased face-to-face interaction and improved mindfulness were noted as common themes in the data and within coded social reactions to the digital sabbath experience. socially, the participants indicated an increased time spent in face-to-face interaction and "mindfulness" (focusing on the moment) as a result of taking a break from their devices. the increased presence was described as an “in the moment” connectedness that had been previously missed because of excessive technology usage. often distracted by their phones, the participants learned that they were better able to attend to the present. they also learned that living in the digital society increased the distraction they experienced by technology and realized they were missing opportunities to interact with those around them. we labeled the themes of strategic planning, self-realization, and busyness as emotional learning outcomes. these themes emerged within the context of participants' feelings of anticipating or feeling bored during the digital sabbath assignment. in order to combat possible boredom, participants intentionally and proactively planned when they would complete the digital sabbath assignment additionally, these students planned where and what they would do during the hours without their technology. prior to completing the assignment, students did not realize how dependent they were on their cell phones for daily tasks and interactions until they were in the midst of the assignment or as they reflected upon their experience afterward. while these social and emotional responses to the assignment were valuable, we questioned how well these learning outcomes aligned to course objective 2. when analyzing their responses, we looked for instances in which they referenced their future students or themselves as classroom teachers. we discovered that the digital sabbath assignment outcomes did not indicate that students reflected on themselves as learners or future teachers as often as we expected or assumed. rather, their reflections indicated a comparison of personal use and nonuse of digital and communicative technologies. analysis of the 2012 student responses led us to realize that two of the twelve student responses mentioned classroom or teaching implications. however, both of these implications for classroom practice focused more on replicating the experience with their future students. one student noted, “i think it would be a good idea for teachers to encourage their students to take time off from technology,” and mentioned that some students “are so attached to technology that rethinking curriculum alignment through collaborative inquiry 6 journal of effective teaching in higher education, vol. 3, no. 2 they cannot even go without it for an hour and a half long class.” another student in 2012 reflected, “when i become a teacher, i will definitely teach my students how to use the technology necessary to succeed in the world, but i will also teach them how to distance themselves from technology. i will show them how to interact with nature instead of playing video games, or read books for the fun instead of watching television. learning to love life without technology is well worth the sacrifice.” both of these students predicted that their future students will use technology often and will need to be forced to take a break from their use of technology devices. however, they did not apply their thinking about their students as learners who also enjoy using technology. technology was villainized in both of these examples and the focus was on excessive use instead of learning and learners. the other ten students in the 2012 group did not mention their future classrooms or students at all in their reflections. in the 2017 class, only three of the seventeen students extended their responses to include future classroom connections. two students noted how often technology today is used for instruction or for students’ engagement with instruction. for example, one student reflected, “i realized while i was trying to do my homework in the first hour almost everything requires me to use the internet to gather information. so, going into a classroom, i need to be able to show my students how to gather information.” another student realized, “we also use technology as a resource for children in schools. instead of just normal pencil and paper, it is also a great tool to help them learn different approaches of learning instead of just the normal techniques that we use.” both of these students recognized how much they use technology as learning tools as students in the digital age and suggested that their future students will also need to know how to use it for educational purposes as well. another student realized after engaging in the digital sabbath that communication is being negatively affected by technology. she asserted, “we depend so much on our technology that we don't know how to communicate with each other face-to-face. i think that's one of our biggest problems, and we need to try to fix it. as teachers we need to learn how to communicate with parents, other teachers in our schools, and even our principal, to solve problems. we need to talk to each other about how to improve our schools and classrooms. i think disconnecting ourselves from technology could help that especially for our generation because we’ve grown up with it.” although she did not mention students or instruction, her reflection was relevant to herself as a future educator and to the profession as a whole. because face-to-face rethinking curriculum alignment through collaborative inquiry 7 journal of effective teaching in higher education, vol. 3, no. 2 communication was one of the stronger themes of the original study, her reflection is also related to other students’ responses, although they did not specifically mention communication in education in their reflections. through a learning outcomes analysis, we acknowledge that a few students in each of our courses reflected on teaching and learning. yet the majority of the students did not make the teaching and learning connection as part of their reflections. we wondered why this assignment was not resulting in the expected learning outcomes intended for this course for future educators. we continued our research and looked closer at the assignment provided. analyzing the learning experience stage 3 of the backward design model enabled us to go deeper with our curricular analysis. we began with an examination of the assignment instructions that were provided to both the 2012 and 2017 courses. while the 2012 and 2017 assignments were quite similar, there were two main differences in the instructions: time and response platform. the 2012 class observed a twenty-four-hour digital sabbath which allowed for 6 hour increments and did not have to be consecutive time away from technology. in contrast, the 2017 class engaged in the digital sabbath for only five consecutive hours total. in figure 2, we have represented this difference by using the following notation: [xx-hour period]. further, in 2012, students were asked to produce “visual, textual, or photographic representation of your experience using a minimum of 10 photographs/text entries with written text for each.” the 2017 class was instructed to “write a blog post and create a video reflection where you reflect deeply on your experiences. use a minimum of 5 original photographs, each with written captions and embed your video in your blog.” the remaining instructions and reflection prompts we used to guide students’ work on this assignment are provided in figure 2. the differences between 2012 and 2017 instructions are identified using brackets and a contrasting font. rethinking curriculum alignment through collaborative inquiry 8 journal of effective teaching in higher education, vol. 3, no. 2 figure 2 integrated assignment instructions through a review of the assignment instructions, we learned that most of our reflection prompts focused on their personal responses to the digital sabbath experience. we provided only one prompt that related to instructional implications, and it was at the end of the suggested list. further, the list of reflection prompts were simply suggestions, not prescribed components of the assignment. having just a list of suggested reflection prompts gave students significant latitude regarding how they chose to reflect. however, when given loose parameters, many students chose to reflect on the personal implications instead of education-related implications. unlike the personal response prompts, we did not provide multiple reflection prompts to guide them to that end. discussion based on research findings, we discovered that our assignment did not align as successfully with our course objectives as we originally assumed. the collaborative process of sotl along with using the backward design model to guide data analysis produced useful results. we realized that some of our expectations of this assignment were left unspoken. this reflection process revealed assumptions we rethinking curriculum alignment through collaborative inquiry 9 journal of effective teaching in higher education, vol. 3, no. 2 had about the assignment. we uncovered our assumption that students would intuitively extend their thinking to their future profession with one reflection prompt. yet, after examining our assignment instructions and student responses, we now realize the limitations of our instructions. explicit instructions written towards the course objective goal could have operationalized our assignment expectations. ensuring our written directions matched our expectations could have improved the assignment outcomes and would have better aligned with our course objective. we believe we lost sight of the need for explicit instructions with this assignment. explicit instruction is a common pedagogical practice (marin & halpern, 2011); however, when teaching adults, it is important to remember that adults benefit from explicit instructions, too. furthermore, to better align with course objective 2, we learned to be more intentional about aligning our assignment instructions. improved intentionality leading students to focus on the changing nature of learners and learning could have improved overall alignment and learning outcomes. developing personal awareness regarding the impact of technology use was important and perhaps a first step. yet our students did not extend or articulate their thinking to consider how their relationship with their device related to the changing nature of learners and learning in their future classrooms. to improve the assignment, we believe we need to revise our instructions so students have the opportunity to examine their personal responses separately from their professional responses. these students needed time and space to sort through their social and emotional use of technology before trying to think outside of themselves to their future students. in retrospect, perhaps a more effective way to frame the reflection prompts would have been to categorize them so students chose prompts to help them reflect both personally and professionally. to improve clarity, instructions for the assignment could be organized into two parts. the first reflection section could focus on personal impact. personal impact includes but is not limited to their social, emotional, and physical reactions when taking a break from their devices. a second section could guide students to consider implications for themselves as teachers, and their future students as learners in a digital age. after the personal response, we could have provided them with multiple prompts to guide them to consider implications for themselves and their future students. by dividing the reflection into two sections, this revision to our assignment could support reflection and insights in a more intentional and substantial manner. being more explicit about the expectation to extend their thinking as teachers in their future classrooms is imperative for this assignment to align with course objective 2. we learned students may not engage in this type of thinking on their own. students in the course need specific questions to guide their reflection. our original directions provided one prompt related to education, and data analysis led us to understand that students needed more guidance. the following prompts could be used in the assignment. these prompts would lead students to more thoughtfully and intentionally consider classroom implications. we could include prompts such as the following: rethinking curriculum alignment through collaborative inquiry 10 journal of effective teaching in higher education, vol. 3, no. 2 ● how does this experience help you better understand learners in this digital society? ● knowing what you now know about your relationship with your digital device(s), how do you think learners’ needs have changed as a result of the wide-spread use of digital technology? ● what implications does your experience have on your future teaching methods? further, since the activity log was part of the assignment, we believe it caused confusion. perhaps this activity log enabled students to focus their reflections more on what they did with their time instead of extending their thinking to classroom implications. the resulting misalignment with course objective 2 could be in large part because of how we organized the instructions and listed required artifacts. students believed the main purpose of the assignment was digital self-awareness and reflection upon their activities. the organizational structure of the assignment was insufficient in guiding students toward our learning goal due to the assignment’s organizational structure. being more explicit with prompts that guide students to extend their thinking beyond the activity log to the root issues would have been a better way to organize the assignment. conclusion providing explicit instructions is a best practice (marin & halpern, 2011); however, as instructors in higher education, we sometimes take for granted that adults benefit from explicit instructions as well. taking time to ensure the written instructions match our unspoken expectations of the assignment is a reflective step that should not be overlooked. the sotl process, which focuses on instructor critical analysis, theorized reflections, and thoughtful enactment of student learning, supported our desire to rethink and review our digital sabbath assignment (bishop-clark & diez-uhler, 2012). this type of research activity guided our efforts to learn more from the student learning outcomes and the questions that remained. it is important for instructors at all levels of education, especially those in pre-service teacher education, to engage in curriculum review and revision on an ongoing basis. through a continuous improvement mindset, we need to scrutinize our own instructional practices in a way that improves our instruction. a continuous improvement mindset also serves as a model for our pre-service teachers. having conversations with our students about the effectiveness of our instruction, in addition to the course learning goals we have from the throughout the course, sets the stage for them to become reflective practitioners as well. curriculum design, review, and revision does not have to occur in isolation. working with course-alike colleagues to collaborate in the process is beneficial. collaboration was key for this endeavor because we drew on each other’s background knowledge and experience with curriculum design to improve our professional practice. rethinking curriculum alignment through collaborative inquiry 11 journal of effective teaching in higher education, vol. 3, no. 2 engaging in sotl with a colleague deepened our reflective practices in a number of ways. we were able to collaboratively analyze student work, encourage curriculum critiques, and learn from each other’s knowledge base and prior experience. this collaborative work enabled us to thoughtfully process student learning outcomes and used them to inform our collaborative curriculum revisions. the research from this inquiry project reinforces the value for ongoing curriculum review in higher education. close examination of course goals in alignment with course assignments can occur in course redesign. this review process affirmed the need for explicit assignment instructions in our study. ongoing, more frequent curriculum review and redesign prevents the error of engaging students over a fiveyear period in an assignment that was not adequately aligned to the intended course goals. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references bishop-clark, c., & diez-uhler, b. (2012) engaging in the scholarship of teaching and learning: a guide to the process, and how to develop a project from start to finish. stylus publishing. chick, n. (2019). scholarship of teaching and learning (sotl). https://my.vanderbilt.edu/sotl/ elliott, v. (2018). thinking about the coding process in qualitative data analysis. the qualitative report, 23(11), 2850–2861. https://nsuworks.nova.edu/tqr/vol23/iss11/14 fanghanel, j. (2013). going public with pedagogical inquiries: sotl as a methodology for faculty professional development. teaching & learning inquiry, 1(1). 59–70. https://www.jstor.org/stable/10.2979/teachlearninqu.1.1.59?seq=1#metada ta_info_tab_contents marin, l. m., & halpern, d. f. (2011). pedagogy for developing critical thinking in adolescents: explicit instruction produces greatest gains. thinking skills and creativity, 6(1), 1–13. mckinney, k. (2007). enhancing learning through the scholarship of teaching and learning: the challenges and joys of juggling. jossey-bass. wiggins, g. p., & mctighe, j. (2005). understanding by design. expanded 2nd ed. ascd. microsoft word sherr_final.docx journal of effective teaching in higher education, vol. 2, no. 2 nudging students to succeed in a flipped format gateway biology course goldie sherr, bronx community college of cuny, goldie.sherr@bcc.cuny.edu shylaja akkaraju, bronx community college of cuny seher atamturktur, bronx community college of cuny abstract. the intensive nature of gateway biology courses often results in poor student performance. this issue can be resolved with the usage of the flipped classroom method. our application of this model to a human anatomy and physiology course increased the retention and passing rates. however, the beneficial effects of flipping a classroom can be substantially weakened due to student disposition. to target this, we used the flipped classroom model in conjunction with a series of nudges in the form of self-monitoring logs and reminder notifications. students were responsible for tracking their attendance, punctuality, preparedness, and grades via a self-monitoring log that they had to complete each week. reminder notifications were sent to students via text or email messages. while the flipped classroom method accompanied by both self-monitoring logs and reminder techniques can be a powerful learning model leading to higher student success rates, we found that consistent nudging via text messages appears to be pivotal in ensuring student success. keywords: flipped learning, nudging, reminders, self-monitoring, student disposition, student success. gateway science courses are some of the most demanding classes that colleges offer. content is typically complex with topics building upon one another, so that new information cannot be mastered until the earlier information is fully understood. the cognitive load is generally high, making this very difficult for the students. moreover, a student population that is lacking good study habits and a science background can present significant challenges to the instructor as well. the flipped learning method, by including regular formative assessments, has been shown to be a suitable option to resolve these issues within a community college setting (akkaraju, 2016). the flipped classroom model is a learning methodology that veers away from standard textbook-style learning and revolves around a more student-centered learning process. the premise behind the flipped classroom method is that the instructor provides the students with the fundamental basics of a lesson prior to coming to class so that class time can be spent more effectively (bergmann & sams, 2014). before coming to class, students are required to go over the information, which is presented to them via a learning module prepared by the instructor. precious classroom time can now be used to go over deeper and more profound learning concepts as well as for group activities, discussions, and problem solving (bergmann & sams, 2012). in order to guarantee that students review the material, a formative assessment is recommended to be given at the beginning of each class. these low stakes quizzes can not only lead to better nudging students to succeed 58 journal of effective teaching in higher education, vol. 2, no. 2 preparation, but can also increase student enthusiasm, confidence, and motivation (warnock, 2004). flipped learning lends itself nicely to gateway science courses, which often contain a vast amount of material that must be covered in a short period of time. concepts are usually complex and are likely unfamiliar to students. the flipped classroom method thus allows students to enter the science classroom with a baseline of information that enables them to tackle the more difficult material in a less overwhelming way. previous studies have shown that the application of the flipped learning method in anatomy and physiology ii, a difficult stem course with low retention and passing rates, greatly increased student performance, morale, and retention (akkaraju, 2016). however, while the flipped classroom method has many benefits, it does have its limitations. if students display a poor disposition, then the flipped learning method can become unsuccessful. it is also important for the instructor using the flipped learning method to fully embrace the practice of formative assessments (moss & brookhart, 2009) by communicating regularly and predictably with the students. inconsistent communication may hinder student progress. in order to counteract these potential obstacles, we propose that the flipped classroom model be performed in conjunction with a series of nudges in the form of reminders and self-assessments. a nudge is a suggestion that has the potential to alter the behavior of the recipient and influence their decision. in the field of behavioral economics, nudging is used in a variety of ways. one powerful way of nudging is via reminders (sunstein, 2014). nudges via text messages have been shown to be highly effective in positively influencing the behavior and decision-making process of students (castleman & page, 2015). studies have also shown that students greatly benefit from both reminders and educational support given through a mobile platform of their choosing (carmean & frankfort, 2013; narasimhan, 2016). results showed that students who had nudging messages sent to their devices directly had greater academic performance than students who did not. additionally, the study also revealed that the students in mathematics and economics courses who received nudges had overall better final grades, better passing rates, and higher completion rates (carmean & frankfort, 2013). thus, these results indicate that a simple gesture, such as a reminder via email or education platform can have profound effects on learning. another powerful way to positively impact study behavior would be to nudge students to self-monitor their progress throughout the semester. self-monitoring is a key aspect of self-regulation, and it can potentially improve both student disposition and academic performance (harris, friedlander, saddler, frizzelle, & graham, 2005). self-regulation is a set of processes that promote student learning and success (zimmerman, 2002). ideally, self-regulation involves a forethought phase during which the student engages in goal setting and strategizing, followed by a performance phase during which the student focuses on tasks and selfmonitors progress, and finally a self-reflection phase during which the student engages in self-assessment and makes decisions about future strategies (zimmerman & capillo, 2003). students who perform the self-evaluation show an nudging students to succeed 59 journal of effective teaching in higher education, vol. 2, no. 2 increase in motivation and self-efficacy, thus not only boosting performance, but also enhancing the belief in their own abilities to succeed (schunk, 2003). according to assessment experts, the three main aspects of a dynamic student selfassessment process include self-monitoring, self-assessing, and self-correcting (mcmillan and hearn, 2008). self-monitoring can help students to raise awareness toward goals, self-assessing helps keep them focused on a specific target or benchmark, and self-correcting involves the corrective actions they take to overcome setbacks and celebrate achievement. some authors believe that selfmonitoring itself comprises two aspects—self-recording and self-assessment (graham, harris, & reid, 1992). the flipped learning method combined with a weekly self-monitoring log can aid students in their self-regulation process. formative assessments provide students the means for continual awareness of progress (nicol & marcfarlane-dick, 2006). as formative assessments are an integral part of the flipped classroom model and provide immediate feedback, the self-assessment aspect of self-regulation fits nicely into the framework of a flipped format course. we decided to make use of a self-monitoring log in our anatomy and physiology i course in order for students to keep track of their performance in formative assessments while targeting important areas of student disposition, such as regular attendance, punctuality, and sufficient preparation for each class session. these areas are particularly important to monitor as a lack in any one of them can render flipped learning, which relies heavily on proper learning habits, ineffective. additionally, we supplemented this self-monitoring log with consistent reminders to students, either via email or through a mobile messaging application called remind. we believed that nudges would be specifically useful to first year students who may not yet have proper learning habits. in this article, we detail the success of using the flipped learning format in conjunction with nudging via self-monitoring logs and reminder notifications. methods we implemented the flipped classroom method in a gateway human anatomy and physiology course at bronx community college of the city university of new york where more than 95% of the student body is made up of ethnic minorities and 90% of the students need developmental instruction in one or more basic skill areas. more than half the student population is either at or below the poverty line (bronx community college of cuny, 2019). our study involved seven sections taught by three different instructors over a period of four consecutive semesters. in six of the sections (sections a-f) students were nudged to complete a self-assessment log every week. they were also nudged to prepare for upcoming quizzes and exams either via email announcement using the blackboard learning management system or via text messages using a mobile messaging application called remind. in the seventh section (section g), the students were only nudged via text messages and were not involved in the practice of weekly self-monitoring. a total of 162 students nudging students to succeed 60 journal of effective teaching in higher education, vol. 2, no. 2 were enrolled across the seven sections with each section consisting of 21-26 students (table 1). table 1 overall set up of all seven sections and student performance data sections use of selfassessment log use of nudging via email use of nudging via (remind) instructor number of students c minus and above f grade w grade a consistent none consistent x 26 24 0 1 b consistent none consistent y 21 19 0 2 c consistent consistent none z 21 11 1 7 d consistent none inconsistent x 24 13 0 7 e consistent none consistent x 26 19 1 2 f consistent consistent none z 23 12 1 10 g none none consistent y 21 17 1 3 the flipped classroom format the lecture portion of the human anatomy and physiology i course lasts for about 3 hours and usually includes one or two short breaks. in our flipped classroom lectures, class always begins with a formative assessment in the form of a prelecture quiz. quizzes are typically given for the first 20–25 minutes of class and are relatively straightforward. these quizzes assess proficiency in the lesson’s fundamental information. quizzes are based on information found in the learning modules which students can access through the blackboard learning management system. quizzes are 20 points each and serve as a main source of motivation for students to come to class prepared, as mastery of factual knowledge is key to further understanding of increasingly difficult concepts. each of these low stakes quizzes is designed as a desirable difficulty (bjork, 2014), a strategy that challenges the student without overwhelming them. while preparation for a quiz requires some time and effort, it should not be so difficult as to elicit avoidance behavior on the part of the student. the quizzes are made up of 20 multiple choice, short answer, matching, or true/false questions and generally act as confidence boosters at the beginning of each lecture session. students are continually made aware that the benchmark grade for these low stakes formative assessments is 80% and that this benchmark must be met by the student in order to do well on higher stakes exams. because these quizzes are given at the beginning of each class, it is crucial that students show up to class on time in order to take them. once the quizzes are completed, class time is spent addressing conceptual and procedural (or application) knowledge. the higher order cognitive processing was addressed through a variety of learning opportunities offered by the three instructors: acting out concepts (bealer & bealer, 1996), interactive notetaking (akkaraju, 2018), quiz games (simkin, 2013), and problem solving in small groups (laal & ghodsi, 2012). it must be mentioned here that apart from the exposure to pre-training learning modules and pre-lecture quizzes nudging students to succeed 61 journal of effective teaching in higher education, vol. 2, no. 2 the rest of the lecture sessions were not uniform with each instructor adopting a different set of active learning exercises. nudging via a self-monitoring log the self-assessment component of the course is conducted after the completion of the formative assessment. students are handed their own self-assessment log (table 2), which they are required to personally fill out at the end of each class. areas of disposition that are addressed on the log include those of attendance, punctuality, preparedness, attentiveness, and cell phone usage. the monitoring log also includes a section for formative assessment grades and whether or not those grades meet the benchmark. this fulfils the self-assessing aspect of the selfmonitoring process. at the end of each class students are required to return their completed log to the instructor. this ensures that self-monitoring logs do not get misplaced or forgotten so that they can be handed back to the students during the following class. students are responsible for filling out their log during every class, allowing them to continually track their learning behavior (self-monitoring), evaluate their own performance on quizzes (self-assessing), and reflect on their learning behavior and study strategies to improve performance (self-correcting). students who are absent will have the date for that specific class crossed out on the self-monitoring log by the instructor, indicating that they were not present. table 2 example of the student self-monitoring log nudging via reminder notifications throughout the semester, instructors also supply a series of reminders, or nudges, to raise student awareness for important information regarding the course. these nudges primarily came in two forms: (a) email announcements via the blackboard learning management system or (b) a mobile messaging app called remind (remind, 2014) that can send announcements in the form of a text message to the students. using text messages is more convenient. the instructor creates the course within the remind platform; students then download the free remind app directly to their phone and sign up for their specific course without having to exchange any personal nudging students to succeed 62 journal of effective teaching in higher education, vol. 2, no. 2 information with their instructor. the benefit of sending reminders through this application, as opposed to using traditional email, is that students are much more likely to receive the message. since most students keep their phones with them throughout the day, the likelihood of their seeing the notification (which appears on the phone similar to the way a text message does) is much greater. these nudges serve as an additional aid for students, particularly those in their first year, to remind them of important class dates and information. additionally, students can privately contact the instructor via email or directly via remind, which has the option for the student to respond to their instructor also via a text message. students can send back questions regarding the material, ask for confirmation of exam/assignment dates, or inquire about additional course information. these nudges, therefore, not only serve as reminders for students to maintain a good disposition but also act as an open door for students to communicate with their instructor when needed. in four (sections a, b, e, and g) out of the seven sections, instructors used remind messages to nudge students about upcoming quizzes, exams, scheduling information, and any other important course reminders that the instructors wanted the students to know. the relatively informal medium of the remind platform allowed instructors to send friendly and encouraging messages to students (table 3). in an additional section (section d), the instructor also used remind but only in a sporadic and unpredictable manner. table 3 examples of the general tone and context of text messages sent to students reminders encouragement helpful information hello class! lecture quiz 3 tomorrow! please be on time. cheers! hello class! good luck with your preparation for tomorrow's lecture exam. don't forget to bring your take-home exam. get some sleep! tomorrow we will be doing chapter 11 muscles. the powerpoint slides for lecture are also posted on blackboard if you want to print them. hi everyone, this is a reminder: your final exam is tomorrow at 10am in me226. also, i will be having an office hour today from 12– 1pm. hello class! the lecture practice exam ii is live on blackboard. please practice until you get 100% good luck! happy easter! hello class! here is a quick guide to the practical portion of the lab exam including data for graphing. (accompanied by attachment) hope you had a nice spring break. reminder: you have a pre-lecture quiz on the skeletal system tomorrow. your final grade is up on blackboard. congratulations! you all passed! i am immensely proud of each one of you! have a great summer! i will be having my office hour today…it will be from 11:45– 12:45pm. nudging students to succeed 63 journal of effective teaching in higher education, vol. 2, no. 2 results student performance and completion rates were collected and analyzed from all seven course sections. we established a c minus as the benchmark grade for this gateway science course because students scoring less than a 70% in this course usually struggle in subsequent biology courses. the impact of nudging via reminder notifications our analysis showed that the seven sections fell into two groups based on whether or not they received nudging via reminder notifications in the form of text messages. figure 1. the impact of regular nudges via text message on student performance overall, our data showed that students enrolled in flipped learning sections combined with the practice of consistent nudges via text messages using remind as in sections a, b, e, and g fared significantly better (p=.002) than students enrolled in flipped learning sections that either did not use text messaging to nudge students, as in sections c and f, or only used it sporadically, as in section d (table 1 and figure 1). among the students that were gently nudged via remind on a regular basis, about 84% passed with at least a c minus in the course; whereas among the students that were nudged via email or inconsistently via remind, only 52% passed with at least a c minus in the course. 84.19020147 52.90717736 0 10 20 30 40 50 60 70 80 90 100 consistent usage of remind inconsistent/no usage of remind success rate p=0.002 nudging students to succeed 64 journal of effective teaching in higher education, vol. 2, no. 2 figure 2. the impact of regular nudges via text message on student retention we also found that gentle nudging via text message also significantly increased (p=.0018) student retention rates in this course (figure 2). figure 3. comparison of success rates by section to the semester course mean examining each section individually, we found the following: (a) nudging via email resulted in success and retention rates around the overall course mean (see sections c and f in figures 3 and 4); (b) inconsistent nudging via text message also resulted in success and retention rates at the overall course mean (see section nudging students to succeed 65 journal of effective teaching in higher education, vol. 2, no. 2 d in figures 3 and 4); and (c) consistent practice of nudging via text message resulted in success and retention rates that were significantly above the overall course mean (see sections a, b, e, & g in figures 3 and 4). this clear difference in performance, based on the method of nudging, indicates that the type of medium that instructors choose to nudge students and how often the nudge is employed can greatly impact overall success. we found this to be true even when the same instructor applied nudges consistently in one section and inconsistently in a different section thereby eliminating instructor bias to some degree. with the progression of technology, there are a plethora of new educational tools readily available to both students and faculty on mobile devices. the usage of such mobile apps enables instructors to communicate with students more effectively and allows for the most reliable way to remind students of course information. figure 4. comparison of retention rates by section to the semester course mean nudging via self-monitoring logs we found no direct evidence that self-monitoring logs had a significant impact on success and retention rates. however, when students (n=22) from one of the sections (section a) were surveyed regarding the usefulness of the self-monitoring practice, their response was resoundingly positive. participation in the survey was voluntary and anonymous. questions ranged from whether the self-monitoring log helped students keep track of their progress and meet their benchmark grades to whether it was useful for punctuality. most students found the self-monitoring log nudging students to succeed 66 journal of effective teaching in higher education, vol. 2, no. 2 to be either “strongly effective” or “effective” in the various disposition areas targeted (figure 5). figure 5. student perceptions of nudging via self-monitoring the impact of formative assessments on student performance in those sections with significantly high pass rates (sections a, b, e, and g), the students also met the benchmark of 80% for the weekly pre-lecture quizzes (figure 6) with students displaying positive behavioral traits in terms of preparedness, regular attendance, and punctuality. these sections also enjoyed 87% attendance and punctuality throughout the semester (figure 6). figure 6. the impact of nudging via text messages on preparedness, attendance, and punctuality 79.75 87.7705 0 20 40 60 80 100 students with at least 80% average on quizzes students with at least 80% attendance record pe rc et na ge average quiz grades and attendance (80% or higher) for sections that nudges with texts 7 11 8 11 8 3 7 5 3 3 2 1 3 4 5 5 1 1 0 0 t r a c k i n g p r o g r e s s m e e t i n g b e n c h m a r k s p u n c t u a l i t y s t a y i n g u n t i l t h e e n d o f c l a s s strongly effective effective somewhat effective ineffective not sure usefulness of the self-monitoring log (n=22) nudging students to succeed 67 journal of effective teaching in higher education, vol. 2, no. 2 discussion we found that the flipped classroom format generally has a strong positive impact on students enrolled in a gateway science course largely due to its emphasis on weekly formative assessments (see figure 6). students who do well in these weekly quizzes are better able to handle deeper concepts and apply this knowledge to real world scenarios. student performance in these formative assessments is therefore a strong indicator of overall performance in the course. however, the success of the flipped classroom format hinges on student disposition traits such as preparedness for each lecture session, punctuality, and attendance. our main finding is that employing nudges via text messages one day ahead of each quiz has a powerful impact on student disposition and performance (see figure 1). we believe that in order to be effective, the nudges must be predictable, consistent, timely, and in the form of text messages. even though the self-monitoring log allowed students to reflect on their own disposition, it did not appear to have a direct impact on student performance (see figure 4). we believe that self-monitoring was a nudge that may have been applied too late and after the fact, as it were. according to one of the proponents of the nudge theory, timing is critical when applying a nudge because it gives the recipient an opportunity to act in response to the nudge (sunstein, 2014). in our model, selfmonitoring did not help students with the forethought phase of self-regulation (zimmerman, 2002). another limitation of this study is that the self-monitoring log was not designed to address metacognitive thinking on the part of the students. in future semesters, we hope to elicit student input to improve the design of the selfmonitoring log to make this practice both meaningful and effective in enhancing student self-regulation. based on our results, we believe that the inclusion of a nudging component (in the form of text messages) within the instructional design of a flipped gateway science course can have a strong positive impact on student success. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references akkaraju, s. (2016). the role of flipped learning in managing the cognitive load of a threshold concept in physiology. journal of effective teaching, 16(3), 28–43. akkaraju, s. (2018). handwriting to learn: embedding a crucial study skill in a gateway science course. journal of effective teaching in higher education, 1(1), 55–72. nudging students to succeed 68 journal of effective teaching in higher education, vol. 2, no. 2 bealer, j. and bealer, v. (1996). acting out immunity: a simulation of a complicating concept, american biology teacher, 58(6), 360–362. bergmann, j., & sams, a. (2014). flipped learning: gateway to student engagement. eugene, or: international society for technology in education. bergmann, j., & sams, a. (2012). flip your classroom: reach every student in every class every day. arlington, va: international society for technology in education. bjork, r.a. (1994). institutional impediments to effective training. in r.a. bjork and d. druckman (eds.), learning, remembering, believing: enhancing human performance (pp. 295–307). washington, dc: national academy press. bronx community college of cuny (2019). self-study for the middle states commission on higher education, february 2019, p.ii. retrieved from http://www.bcc.cuny.edu/wp-content/uploads/2019/02/bcc-msche-selfstudy-2-7-19-1.pdf carmean, c., & frankfort, j. (2013, december 6). mobility, connection, support: nudging learners to better results. educause review online. retrieved from https://er.educause.edu/articles/2013/12/mobility-connection-supportnudging-learners-to-better-results castleman, b. & page, l.c. (2015). summer nudging: can personalized text messages and peer mentor outreach increase college going among lowincome high school graduates? journal of economic behavior and organization, 115(c), 144–160. graham, s., harris, k.r., & reid, r. (1992). developing self-regulated learners. focus on exceptional children, 24, 1–16. harris, k., friedlander, b. d., saddler, b., frizzelle, r., & graham, s. (2005). selfmonitoring of attention versus self-monitoring of academic performance: effects among students with adhd in the general education classroom. journal of special education, 39(3), 145–156. laal, m. & ghodsi, s.m. (2012). benefits of collaborative learning, procedia – social and behavioral sciences, 31(2012), 486–490. mcmillan, j. h., & hearn, j. (2008). student self-assessment: the key to stronger student motivation and higher achievement. educational horizons, 87(1), 40–49. moss, c.m. & brookhart, s.m. (2009). advancing formative assessment in every classroom: a guide for instructional leaders. alexandria, va: ascd. nudging students to succeed 69 journal of effective teaching in higher education, vol. 2, no. 2 narasimhan, r. (2016, august 16). small nudges=large impact on student success [blog post]. retrieved from https://www.pearsoned.com/large-impactstudent-success/ nicol, d.j. & marcfarlane-dick, d. (2006). formative assessment and self-regulated learning: a model and seven principles of good feedback practice. studies in higher education, 31(2), 199–218. remind. (2014). remind school communication [mobile application software]. retrieved from http://remind.com schunk, d. h. (2003). self-efficacy for reading and writing: influence of modeling, goal setting, and self-evaluation. reading & writing quarterly, 19(2), 159– 172. simkin, m.g. (2013). playing jeopardy in the classroom: an empirical study. journal of information systems education, 24(6), 203–210. sunstein, c. (2014). nudging: a very short guide. 37 j. consumer pol'y, 583, 1–7. http://dx.doi.org/10.2139/ssrn.2499658 warnock, s. (2004). quizzes boost comprehension, confidence. the teaching professor, 5. zimmerman, b. (2002). becoming a self-regulated learner: an overview. theory into practice, 41(2), 64–70. zimmerman, b., & campillo, m. (2003). motivating self-regulated problem solvers. in j. davidson & r. sternberg (eds.), the psychology of problem solving (pp. 233–262). cambridge: cambridge university press. doi:10.1017/cbo9780511615771.009 microsoft word akkaraju_oral_exam_final.docx journal of effective teaching in higher education, vol. 6, no. 1 https://doi.org/10.36021/jethe.v6i1.354 the oral exam—learning for mastery and appreciating it shylaja akkaraju, cuny bronx community college, shylaja.akkaraju@bcc.cuny.edu abstract. to reduce academic dishonesty and strengthen learning outcomes, i adopted in-depth oral examinations as my benchmark and summative assessments in a human anatomy & physiology course taught in an online asynchronous setting. this decision led my students and me down the transformative path of mastery learning. this was a threshold experience for my students who were learning how to think and express themselves as physiologists. this was also a threshold experience for me as i explored the scope of the oral examination in promoting skill acquisition while nurturing a relationship-rich learning environment. by employing deliberate practice principles including basic drills, one-on-one weekly check-ins, and small group recitation sessions, students exceeded benchmarks for conceptual understanding, mastery of fundamentals, and application of concepts to clinical scenarios. students consistently reported that they were happy within this learning environment. with meticulous planning, it is possible to motivate students to learn for mastery and acquire expertise by employing oral exams as the pivotal assessment strategy in an online course thereby also making academic dishonesty almost irrelevant. keywords: oral exams; mastery learning; threshold concepts; deliberate practice; pedagogy of care most human anatomy & physiology instructors believe that academic integrity is compromised in online courses (carrasco, 2022). however, it has been shown that oral exams can hold the key to avoiding academic dishonesty while promoting excellence (schurgers & qui, 2022). in fact, oral exams can deepen engagement in a way that traditional assessments cannot (theobold, 2021). i decided to adopt the oral exam for both benchmark (monthly) and summative (end of semester) assessments not just to reduce academic dishonesty issues but also as a potentially powerful vehicle to assess mastery of the subject matter. when instruction is designed with mastery learning in mind, the oral examination can serve as a highly effective assessment vehicle to evaluate mastery learning (carrasco, 2022; joughin, 2010) because it provides a wealth of information about student understanding (theobold, 2021). mastery learning requires a great deal of scaffolding, allowing learners to engage in deliberate practice, which culminates in the demonstration of expert performance (ericsson, 2008). the principles of deliberate practice that have been established in the training of elite athletes, musicians, dancers, and chess grandmasters have a lot in common (coyle, 2009; ericsson, 2008; ericsson & harwell, 2019). they all need unlimited trials that provide opportunities to focus on fundamentals, form mental representations, engage in repeated performance, practice slow learning, receive immediate feedback, and learn from errors (ericsson & pool, 2017). the oral exam journal of effective teaching in higher education, vol. 6, no. 1 67 for the instructor, deliberate practice requires meticulous planning and streamlining of content, making mastery learning design a threshold concept . a threshold concept is a doorway that leads to an experience that is both troublesome and transformational resulting in an entirely new way of thinking about something (meyer & land, 2003). a threshold concept is troublesome because at the beginning of the learning period, it may appear to be counterintuitive. it is transformational because by the end of the learning period, there is a major shift in thinking. mastery learning and expert performance can appear to be counterintuitive for most instructors because we do not normally expect introductory biology students to “think and talk like physiologists.” we expect such behavior from college seniors or graduate students; for this reason, mastery learning is a troublesome concept. if we were to demonstrate that college students taking an introductory course can acquire expertise, then we are entering the zone of transformation. the threshold experience itself is not without struggle, a sometimes-messy process which is played out within the learning environment known as the liminal space. those who are yet to engage in the threshold experience are said to be pre-liminal; whereas, those that have exited the liminal space fully transformed are said to post-liminal. all others are liminal (land et al., 2005). for students, mastery learning via oral exams is troublesome because it is unfamiliar, challenging, and demands deliberate practice to succeed. it is also transformational because students gain confidence in their ability to “speak physiology” not only during oral exams and practice sessions but also in everyday life with their family and friends. typically, within the liminal space, students struggle to achieve success as they take excursive (bold and exploratory) and recursive (timid and retreating) journeys during the threshold experience (figure 1). therefore, it is imperative that the instructor create a supportive liminal environment in which the students can engage and flourish (cousin, 2006). figure 1 learning for mastery is a threshold experience for the instructor and student the oral exam journal of effective teaching in higher education, vol. 6, no. 1 68 in the context of our collective trauma of the pandemic, the need for a highly supportive liminal environment and concepts such as pedagogical love, pedagogy of kindness, and relationship-rich education have become increasingly relevant (akkaraju et al., 2019; cho, 2005; lambert & felten, 2020; loreman, 2011; maatta & uusiautti, 2012; wilkinson & kaukko, 2020). it has been shown that student success is attributable to a relationship-rich learning environment (lambert & felten, 2020). care in pedagogy is classified as aesthetic care and authentic care (valenzuela, 1999). aesthetic care is superficial and limited to caring about student performance and disposition. authentic care is the deeper version that emphasizes a reciprocal relationship between a student and teacher in which the student feels cherished. to fulfil the conditions of aesthetic care, the tone, frequency, ease, and mode of communication becomes a crucial part of the instructional design (pritts, 2020). the use of a mobile messaging application can be a powerful way to impact student success (sherr et al., 2019) by making the students feel cared for while gently nudging them into completing all assignments on time. combined with the use of laughter, smile, thumbs up, and other positive emojis, text messaging allows for more relaxed communication with the students. in this article i address the following: (a) scaffolding formative assessments that are congruent with oral exam outcomes; (b) assessing for mastery; (c) structuring deliberate practice to result in mastery; (d) viewing mastery learning as a threshold experience to help create and maintain a nurturing liminal environment; and (e) creating a nurturing liminal environment which is central to mastery learning. methods i studied the effectiveness of oral exams in promoting mastery learning for a period of three consecutive semesters in an online, asynchronous section of human anatomy & physiology course at bronx community college of the city university of new york (cuny). each section consisted of 17–20 students with a total of 57 students. student performance data on oral exams was collected for all benchmark and summative assessments. student perceptions of weekly formative assessments, oral exams, and weekly check-ins were collected using an anonymous survey at the end of the semester. the project gained approval from the cuny institutional review board prior to the start of this project. scaffolding formative assessments to match oral exam outcomes a major learning outcome for the oral exam was that the student would demonstrate conceptual clarity. as a reuslt, weekly formative assessments were geared towards enabling the student to develop knowledge and skills that would gamely meet this outcome. content was streamlined which helped to focus on the fundamentals, deepen understanding of core concepts, and enhance the learner’s ability to readily apply physiological concepts to clinical scenarios. to achieve conceptual clarity, students were expected to engage in a series of scaffolded learning experiences that helped them to connect with the material, master the basics, dig deeper into the concepts, amplify learning through sketching and writing by hand, and teach concepts assigned for the week (figure 2). the oral exam journal of effective teaching in higher education, vol. 6, no. 1 69 in step 1, students were expected to take notes from mini-lecture videos that i had created and uploaded to my youtube channel (akkaraju, n.d.). in step 2, students answered multiple choice or matching questions at specific checkpoints of the minilecture video. in step 3, students practiced vocabulary and basic conceptual information by taking a quiz that was designed with the purpose of drilling the fundamentals. in step 4, they re-wrote notes to make sure that their information was accurate. these handwritten notes were provided by me and were congruent with the questions on the oral exam. in step 5, they took a comprehensive quiz addressing deeper conceptual and application questions. i held a weekly recitation session to review the quiz that students attended on a voluntary basis. recordings from these recitations were made available to the entire class. in step 6, students picked a time slot of their convenience to meet with me individually for 15 minutes to explain the concept of the week and to track their progress. figure 2 weekly formative assessments congruent with oral exam outcomes the formative assessments were scaffolded to maximize congruence. a core concept such as blood pressure travels through several learning experiences (formative and benchmark) by the time a student achieves conceptual clarity (table 1). the oral exam journal of effective teaching in higher education, vol. 6, no. 1 70 table 1 concept tracking to maximize congruence learning opportunity assessment type software used description of activity learning outcome (skill) learning outcome (knowledge) notetaking formative explain everything basics & youtube watch minilecture video & take detailed notes including sketches transcription fluency spelling sketching to learn factual check understanding formative playposit answer questions at various checkpoints retrieval practice factual vocabulary gamify learning the basics formative quizlet practice with flashcards & selftesting retrieval practice vocabulary factual conceptual & contextual learning formative goodnotes practice redrawing & sketching to engage with the concepts retrieval practice flow charting sketching conceptual procedural teaching to learn formative zoom engage in weekly check-in session to orally explain concepts of the week oral communication conceptual clarity digging deeper formative google form zoom complete a worksheet with deeper conceptual & contextual questions retrieval practice factual conceptual procedural recitation session formative zoom appear on zoom voluntarily to practice for the oral exam recorded sessions shared with the class oral communication retrieval practice factual conceptual procedural oral exam objective test benchmark (monthly) summative (end-ofterm) zoom google form complete online objective test & appear for a 20minute oral exam oral communication retrieval practice problem solving conceptual clarity factual conceptual procedural use of software to develop learning opportunities i experimented with a variety of software applications to develop and deploy learning opportunities: the oral exam journal of effective teaching in higher education, vol. 6, no. 1 71 • to develop explainer videos in a sketch-note style, i used a software application called explain everything basics (explain everything, 2020) which allows you to create animated videos and upload them directly to youtube. • to create real-time questions that appear at certain checkpoints of the youtube video, i uploaded the video to a software application called playposit (playposit, 2013) and used this software to create objective type questions to check basic understanding. • to gamify learning vocabulary, labeling, and basic facts, i created quizzes in a software application called quizlet (quizlet, 2005) and added it to a class that i had created. students sign into this quizlet class at the beginning of the semester to access their homework. • to demonstrate how sketch-notes can help create mental representations in biology and provide students with an open educational resource, i created a set of notes for the major concepts and clinical scenarios on a software application called goodnotes (goodnotes, 2011), which makes it easy to share with students. • to create a quiz with deeper conceptual and contextual questions, i used google forms (google, 2008). • to conduct small group recitation sessions, i used the zoom videoconferencing platform (zoom, 2020). designing deliberate practice i incorporated deliberate practice principles into the weekly formative assessments by doing the following: (a) allowing students unlimited trials on all formative assessments; (b) providing immediate feedback wherever appropriate via apps; (c) promoting mental representations by writing to learn (notetaking and making sketches); and (d) practicing oral skills by “teaching” concepts. for all six weekly assessments, students were only given credit if they demonstrated proficiency, allowing for slow learning. assessing for mastery students were provided with the oral exam questions, notes with answers, minilecture videos, and recitation sessions prior to each oral exam. there was full transparency. i used a score of 90% and above on the oral exam as an indicator of mastery and at least 80% on the oral exam as an indicator of proficiency. my benchmark for students achieving mastery in this course was set at 70% meaning that i expected at least 70% of the students to achieve mastery. for achieving proficiency, i set the benchmark at 80% meaning that at least 80% of the students would achieve proficiency. each oral exam session, which lasted 20–30 minutes was scored using a rubric (table 2). the students were required to have their camera on and were not allowed to refer to notes during the oral exam. each student received a completed rubric within a few hours of taking the oral exam. the oral exam journal of effective teaching in higher education, vol. 6, no. 1 72 maintaining a nurturing learning environment i designed a caring learning environment that focused on three aspects: communication, connection, and intervention. to keep communication channels open, i used the mobile messaging application called remind (remind, 2014). to forge a personal connection with each student, i included 15-minute one-on-one weekly check-in on zoom. students were allowed to pick a time slot that suited their schedule from a wide range of choices that included evenings and weekends. these sessions were useful in tracking progress, clarifying course information, evaluating conceptual clarity via the “teach it” exercise, and getting to know each other. throughout this process, i was guided by the threshold concept theory, which allowed me to observe the student learning experience with empathy as they journeyed from pre-liminal to liminal to post-liminal stages (see figure 1). following along the rules of mastery learning, students were allowed multiple attempts on the oral exam. if a student’s performance on weekly check-ins and oral exam sessions indicated that they needed intervention, additional online practice sessions were scheduled. two weeks prior to each oral exam, students were provided with a detailed study guide that was broken up into three parts: part a consisted of a list of “essay” type questions; part b consisted of a collection of figures and graphs; and part c consisted of a long list of questions dealing with the basics reminiscent of the oral exam format (see table 2). as a way of practicing for the oral exam, several 30minute small group online recitation sessions were arranged during the days leading up to the exam that students attended on a voluntary basis. during these sessions, students took turns to practice answering each question with their peers helping them when they got stuck. recordings from these sessions were made available to the entire class. table 2 sample rubric for oral exam criteria concept/question unfamiliar (60 and below) familiar (61-79) proficient (80-89) master (90100) comfort level (ease of explanation) nodal/non-nodal action potentials blood pressure cardiac output circulatory shock edema completeness (right amount of detail to support big picture) nodal/non-nodal action potentials blood pressure cardiac output circulatory shock edema nodal/non-nodal action potentials the oral exam journal of effective teaching in higher education, vol. 6, no. 1 73 criteria concept/question unfamiliar (60 and below) familiar (61-79) proficient (80-89) master (90100) conceptual understanding (cause and effect relationships evident) blood pressure cardiac output circulatory shock edema logical reasoning (can think on feet) walk me through the graphs part b basic knowledge (strong fundamentals) let’s check your basics part c results a major discovery of this study was that it is possible to assess for mastery using the oral exam as the assessment vehicle. the oral exam made thinking visible, allowing me to observe mental flow, fluency, and conceptual clarity. it was also possible to easily assess if the student had prepared for familiarity, proficiency, or mastery (see rubric in table 2). it turned out that 74% of the students (n = 57) demonstrated mastery and 16% achieved proficiency in the summative oral exam (figure 3). altogether 90% of the students achieved proficiency, which was well above the benchmark of 80% for proficiency. figure 3 majority of students learned for mastery 74% 16% 10% achieved mastery achieved proficiency the oral exam journal of effective teaching in higher education, vol. 6, no. 1 74 as for mastery, the students scored above the benchmark, which was set at 70%. student perceptions of the oral exam was favorable in general and specifically to the following statements (figure 4): (s.1) preparing for and taking the oral exam was a powerful learning experience; (s.2) taking oral exams helped me realize that i need to focus on understanding concepts rather than memorizing them; (s.3) over the course of the semester i became more comfortable speaking about physiology; and (s.4) i like this way of testing. figure 4 students who chose “strongly agree” or “agree” to statements 1–4 students were generally positive about the oral exam format, process, and outcomes (figure 5) when responding to the following statements: (s.5) oral exams helped me to become more aware of my strengths and weaknesses in my approach to learning; (s.6) the oral exam felt like a conversation about physiology with my instructor; (s.7) i left each oral exam knowing my strengths and areas for improvement; (s.8) i felt nervous taking oral exams; and (s.9) the oral exam format was clear to me. students unanimously strongly agreed to statements about the efficacy of the weekly check-ins and small group sessions (figure 6). the students responded positively to the following statements: (s.10) the weekly check-ins helped to keep me on track; (s.11) the weekly check-ins helped to know my instructor better and discuss any issues i was having with the class; (s.12) video recordings of the small group sessions were helpful to me in preparing for oral exams; (s.13) the “teach it” exercise helped me a lot to master concepts and apply them to clinical scenarios; and (s.14) i am happy that i took this class. 90% 86% 79% 76% 65% 70% 75% 80% 85% 90% 95% statement 1 statement 2 statement 3 statement 4 the oral exam journal of effective teaching in higher education, vol. 6, no. 1 75 figure 5 students who chose “strongly agree” or “agree” to statements 5–9 figure 6 students unanimously favored the weekly check-ins and small group recitations more than 95% of the students responded favorably to the weekly formative assessments. when asked to respond to the statements, “what study tips would you give to your peers on how to approach this exercise for the best results? describe your unique method of approaching this exercise,” many of them were eager to share their advice with their peers (table 3). 100% 100% 100% 100% 100% 0% 20% 40% 60% 80% 100% 120% statement 10 statement 11 statement 12 statement 13 statement 14 77% 77% 82% 86% 82% 72% 74% 76% 78% 80% 82% 84% 86% 88% statement 5 statement 6 statement 7 statement 8 statement 9 the oral exam journal of effective teaching in higher education, vol. 6, no. 1 76 discussion the students responded favorably to the organization of the weekly formative assessments, and all students agreed that the repetition of information in weekly assignments and weekly check-ins were very helpful with the oral exams. the amalgamation of intentional content, deliberate practice, and a highly supportive liminal environment had the desired effect on mastery learning (see figure 2). more importantly, it became apparent that it is possible to assess whether a student is learning for familiarity, proficiency, or mastery based on their oral exam performance (see table 2 and figure 3), which is difficult to discern in a more traditional written or objective type exam. the mastery design guided by deliberate practice principles helped student engagement to the extent that students were able to think and speak like physiologists. table 3 study tips to peers on how to approach an exercise formative assessment level of difficulty study tips to peers on how to approach this exercise write/sketch low • good to keep specially for any oral exams • have fun drawing and writing down the notes while you learn them! play posit quiz moderate • watch the video at least twice • focus on the videos first, bypass the questions until u understand. take many notes, small details can be important for the next step. repetition is your friend. quizlet quiz moderate • take detailed notes of the material provided and go through the flash cards first to learn the material • do all of them. it helps a lot! google quiz difficult • don’t get discouraged if you do not get everything right on the first try. use what you learned from playposit as well as quizlet to complete this. keep trying until you have a better understanding of what is being presented to you • the google quizzes helped a lot with testing my knowledge on all the information i learned with the quizlet, and playposits. i think they were really beneficial using your notes from playposit and understanding the concepts will help you complete your google quiz’s successfully teach it! difficult • i love “teach it” as it applies to real life scenarios, and you need to understand the material in order to teach it thoroughly which can give clarity on the cause and effects and allows you to start recognizing patterns or even curiosity to solve different clinical scenarios that it might relate to. viewing mastery learning as a threshold experience was reassuring to me when students would become stuck or discouraged with the process. students found the process of mastery learning to be challenging, time-consuming, and even nerve the oral exam journal of effective teaching in higher education, vol. 6, no. 1 77 wrecking, thereby providing support for the troublesome nature of a threshold experience (land et al., 2005). most students who successfully emerged from this experience were also transformed by it (see figure 3). for example, there was the realization that they needed to understand and not just memorize physiological concepts. nearly all the students (90%) felt that preparing and taking oral exams was a powerful learning experience. many were surprised that they were able to “talk physiology” to their friends and family members, which was perhaps the most transformational change, making mastery learning a threshold concept (meyer & land, 2003). about 90% of the students entered the liminal space. they engaged in deliberate practice hoping to achieve conceptual clarity and demonstrate the ability to think and speak like a physiologist. the majority of the students (74%) stepped over the threshold and achieved mastery, thereby showing that oral exams can motivate students to learn for mastery without raising any red flags regarding academic dishonesty—something that has also been observed by others (schurgers & qui, 2022). the construction and maintenance of a warmly supportive learning environment focusing on both esthetic and authentic care (valenzuela, 1999) paid off with 100% of the students responding favorably to the pedagogy of care model (see figure 5). they all felt that the weekly check-ins and small group recitation sessions were especially useful. this level of support may have spilled over to their learning experience because even if the deliberate practice was sometimes onerous, the students remained eager to engage at a deeper level and help their peers (table 3). conclusion from an instructor’s perspective, creating and maintaining optimum conditions for mastery learning to occur was both challenging and exhilarating. this was indeed a threshold experience. it was troublesome because of the extensive planning involved before the semester began and the laborious progress tracking that occurred throughout the semester. however, to observe student growth over the course of the semester during the weekly check-ins, small group recitations, monthly oral exams and the final oral exam was truly a powerful experience. with the rapid improvements in educational software, it should be possible to streamline progress tracking and simplify the workflow for both the student and the instructor, thereby making mastery learning via oral exams even more attractive. on the flip side, fears of plagiarism arising from the abuse of artificial intelligence can be allayed by a well-planned oral examination. the oral exam, when designed to humanize the learning experience, can create a learning sanctuary for students in an online environment. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. the oral exam journal of effective teaching in higher education, vol. 6, no. 1 78 references akkaraju, s. (2022). pedagogical love during the age of the pandemic: formative assessments to the rescue. the assessment review, 3(2). https://assessatcuny.commons.gc.cuny.edu/author/shylaja-akkaraju/ akkaraju, s. (n.d.). professor akkaraju [youtube]. https://www.youtube.com/channel/uc5uduww5zyx7nu5s_ckn_ew akkaraju, s., atamturktur, s., broughton, l., & frazer, t. (2019). ensuring student success: using formative assessment as the key to communication and compassion among faculty, students, and staff. new directions for community colleges, 2019 (186): 71–79. doi:10.1002/cc.20358 carrasco, m. (2022, january 28). concerns about online cheating decline. inside higher ed. cho, d. (2005). lessons of love: psychoanalysis and teacher/student love. educational theory, 55(1), 79–95. cousin, g. (2006). an introduction to threshold concepts. planet (17), 4–5. coyle, d. (2009). the talent code: greatness isn't born, it's grown, here's how. bantam books. ericsson, a. (2008). deliberate practice and acquisition of expert performance: a general overview. academy of emergency medicine, 15(11): 988–994. ericsson, k., & harwell, k. (2019). deliberate practice and proposed limits on the effects of practice on the acquisition of expert performance: why the original definition matters and recommendations for future research. frontiers in psychology, 10. https://doi.org/10.3389/fpsyg.2019.02396 ericsson, a., & pool, r. (2017). peak: secrets from the new science of expertise. first mariner books edition. mariner books/houghton mifflin harcourt. explain everything. (2020). explain everything basics [mobile explainer video application software]. https://explaineverything.com/explain-everythingbasics/ goodnotes. (2011). digital note-taking software for ipad [mobile application software]. https://www.goodnotes.com/ google forms. (2008). google forms [mobile application software]. https://www.google.com/forms/about/ joughin, g. (2010). a short guide to oral assessment. leeds metropolitan university/university of wollongong. https://teachlearn.leedsbeckett.ac.uk//media/files/clt/a-short-guide-to-oral-assessment-by-gordon-joughin.pdf the oral exam journal of effective teaching in higher education, vol. 6, no. 1 79 lambert, l. m., & felten, p. (2020). relationship-rich education: how human connections drive success in college. johns hopkins university press. land, r., cousin, g., meyer, j. h. f., & davies, p. (2005). threshold concepts and troublesome knowledge: linkages to ways of thinking and practicing: implications for course design and evaluation. in c. rust (ed.), improving student learning—diversity and inclusivity (pp. 53–64). proceedings of the 12th improving student learning conference. oxford centre for staff and learning development (ocsld),. loreman, t. (2011). love as pedagogy. sense publishers. maatta, k., & uusiautti, s. (2012). pedagogical authority and pedagogical love: connected or incompatible? international journal of whole schooling, 8(1), 21–39. meyer, j. h. f., & land, r. (2003). threshold concepts and troublesome knowledge: linkages to ways of thinking and practicing, in c. rust (ed.), improving student learning—theory and practice ten years on (pp. 412-424). oxford centre for staff and learning development (ocsld). playposit. (2013). interactive video vlatform [computer software]. https://go.playposit.com/ pritts, n. (2020, june 1). using announcements to give narrative shape to your online course. faculty focus. https://www.facultyfocus.com/articles/onlineeducation/online-course-design-and-preparation/using-announcements-togive-narrative-shape-to-your-online-course/ quizlet. (2005). study tool for quiz and flashcard creation [mobile application software]. https://quizlet.com/ remind. (2014). remind school communication [mobile application software]. http://remind.com schurgers, c., & qui, h. (2022, december 11). oral exams in support of academic integrity: from catching dishonesty to promoting excellence. integrity matters. international center for academic integrity. https://academicintegrity.org/resources/blog/106-2022/december2022/417-oral-exams-in-support-of-academic-integrity-from-catchingdishonesty-to-promoting-excellence sherr, g., akkaraju, s., & atamturktur, s. (2019). nudging students to succeed in a flipped format gateway biology course. journal of effective teaching in higher education, 2(2), 57–69. the oral exam journal of effective teaching in higher education, vol. 6, no. 1 80 theobold, a. (2021). oral exams: a more meaningful assessment of students’ understanding. journal of statistics and data science education, 29, 156– 159. doi: 10.1080/26939169.2021.1914527 torres, c. (2019). assessment as an act of love. ascd education update, 61(2). valenzuela, a. (1999). subtractive schooling: u.s.-mexican youth and the politics of caring. state university of new york press. wilkinson, j., & kaukko, m. (2020) educational leading as pedagogical love: the case for refugee education. international journal of leadership in education, 23(1), 70–85. doi: 10.1080/13603124.2019.1629492 zoom (2011). zoom video conferencing [mobile application software]. https://zoom.us/ microsoft word wells_final.docx journal of effective teaching in higher education, vol. 6, no. 1 https://doi.org/10.36021/jethe.v6i1.336 creating trauma-informed higher education classrooms: exploring undergraduate and graduate student perceptions tommy wells, bellarmine university, twells@bellarmine.edu abstract. colleges and universities may desire to become more trauma-informed in light of our growing understanding of the impact that trauma can have on student outcomes. however, there has been little research on what strategies and practices colleges and universities should implement to become more trauma-informed. one approach to this work could be to start in the college classroom to determine which trauma-informed strategies and practices are most beneficial to students with an understanding that undergraduate and graduate students may have different needs. in this study, i surveyed 60 school of education students, including both undergraduate and graduate students, to evaluate their perceptions of the importance of specific trauma-informed strategies and practices in the classroom, as well as their perceptions of how trauma-informed the host university is overall. both undergraduate and graduate students believed the host university is moderately trauma-informed. however, there was a perceived significant difference between undergraduate and graduate students as to how important specific trauma-informed classroom strategies and practices were. i discuss the importance of faculty reflecting on various trauma-informed strategies and practices they could implement in the classroom. keywords: trauma-informed practices; classroom environment; student perspectives the covid-19 pandemic has highlighted the need to develop trauma-informed environments and offer specific trauma-informed services in higher education (carello & thompson, 2021). exploring the need for trauma-informed colleges and universities is important, as research indicates that between 66% to 85% of children and adolescents have been exposed to a traumatic event before they reach college age (frazier et al., 2009; read et al., 2011; smyth et al., 2008), and students who have been impacted by trauma are more likely to have difficulty adjusting to college (banyard & cantor, 2004). trauma and other adversities also increase the risk that college students may develop posttraumatic stress disorder (ptsd), depression, substance use disorders, among others (anders et al., 2012; copeland et al., 2007; read et al., 2011; turner & butler, 2003). moreover, as much as 50% of students are exposed to a traumatic event in their first year of college (galatzer-levy et al., 2012). trauma exposure can impact students in different areas of their college experience, including retention and academic performance, which is related to their experiences in the college classroom. college students exposed to trauma face a higher likelihood of experiencing academic failure, which can impact retention (boyraz et al., 2013; duncan, 2000; harrison et al., 2020). a correlation between college persistence and ptsd symptomatology has been demonstrated (boyraz et al., 2013). for example, boyraz et al. (2013) studied first semester african american students who were creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 98 exposed to trauma and how ptsd symptomatology affected retention rates into their second year. notable differences were also observed by gender in that academic achievement and persistence were not affected in men, but women with higher levels of ptsd symptomatology had a higher dropout rate. boyraz et al. (2016) also found that the prevalence of ptsd among participants who were exposed to trauma was 12.4%. because of the various effects that trauma can have on college students, it is important to understand how colleges and universities support them. historically, higher education has taken a reactive approach in responding to trauma (new england board of higher education [nejhe], 2020). if a college student exposed to trauma needed support, services were provided individually to that student. student services commonly recommended included mental health services and counseling. in this approach, services are reactive and provided after a problem is indicated. in considering how k-12 schools have become more proactive and systematic in their approach to supporting students who have experienced trauma (avery et al., 2021), colleges and universities could consider how their responses could also become more proactive. establishing trauma-informed campuses with support services is important, but an even greater shift could occur when the college classroom becomes trauma-informed as well. as gross states, “higher education professors need to learn about trauma, its symptomology and its impact on current and prospective students. then, they need to change what they are doing with their students in and out of class” (nejhe, 2020, para. 19). before further exploring how colleges and universities can become more trauma-informed, it is important to offer conceptualizations of trauma and trauma-informed care. conceptualizing trauma although the impact of trauma on college students has been discussed, it is critical to offer a definition of trauma. for example, the substance abuse and mental health services administration (2014) holds the following definition of trauma: trauma results from an event, series of events, or set of circumstances that is experienced by an individual as physically or emotionally harmful or threatening and that has lasting adverse effects on the individual’s functioning and physical, social, emotional, or spiritual well-being. substance abuse and mental health services administration [samhsa], 2014, p. 7 additionally, according to the diagnostic and statistical manual of mental disorders, 5th edition, text revision (dsm-5-tr), trauma is defined in the context of posttraumatic stress disorder, and ptsd occurs when an individual has “exposure to actual or threatened death, serious injury, or sexual violence” (american psychiatric association [apa], 2022, p. 302). these competing definitions of how narrow or expansive to make the definition of trauma help to highlight how there is no single understanding of trauma and its impact. consequently, approaches to addressing trauma and creating trauma-informed environments also vary. creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 99 trauma-informed care and higher education trauma-informed care can be defined as approach that “recognizes the intersection of trauma with many health and social problems for which people seek services and treatment, aiming to sensitively address trauma along with an individual’s issues” (bowen & murshid, 2016, p. 223). trauma-informed care is also grounded in six core principles outlined by the substance abuse and mental health services administration (2014): 1. safety (e.g., students need to feel safe, and they use their schema to determine who is safe. they also determine if the location they are in is not safe.) 2. trustworthiness & transparency (e.g., educators need to “make a conscious effort to form a relationship with each child as a precursor to teaching and learning” (erdman & colker, 2020, p. 87). 3. peer support (e.g., peer support and mutual self-help are key vehicles for establishing safety and hope, building trust, enhancing collaboration, and utilizing their stories and lived experiences to promote recovery and healing” (substance abuse and mental health services administration, 2014, p. 11). 4. collaboration & mutuality (e.g., all members of a school community play a key role in helping children with trauma heal. relationships are likely the most important factor in whether a child impacted with trauma can overcome the trauma and thrive.) 5. empowerment & choice (e.g., each intervention should be chosen because it is uniquely suited for that student) 6. cultural, historical & gender issues (e.g., educators should “seek to understand and apply knowledge of cultural, historical, and gender issues in interactions with children and families, be culturally responsive in policies and classroom choices” (erdman & colker, 2020, p. 88). despite an anchoring in six core principles, there is not an agreement on the use or clear operationalization of the terms “trauma-informed approach,” “trauma sensitive,” or “trauma-informed system” (hanson & lang, 2016; maynard et al., 2017). different systems of care, such as education, may have applied the six core principles of trauma-informed care (substance abuse and mental health services administration, 2014) in varying ways. for example, the national child traumatic stress network, schools committee (2017) operationalizes trauma-informed practices in k-12 settings as a place that “promotes a safe and welcoming climate; seeks to create a structured and predictable learning environment that minimizes unnecessary trauma and loss reminders; focuses on building positive and attuned relationships between teachers and students, and among school staff” (p. 4). despite varying approaches to trauma-informed practices in k-12 schools (thomas et al., 2019), fewer resources and frameworks are available in higher education settings. however, davidson (2017) offers that colleges and universities can also be creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 100 systems of care and describes how creating trauma-informed higher education institutions requires “the entire campus community—faculty members, administrators, staff members, counselors, and clinicians—to deepen its shared understanding of trauma’s impacts on learning and agree to a campuswide approach,” to promote “the physical, social, emotional, and academic safety of every student” (p. 14). specifically, college and university faculty can take steps to ensure success for all learners within the classroom setting (davidson, 2017), as “the foundation for effective trauma-informed classroom practice is the educator’s grasp of how trauma impacts students’ behavior, development, relationships, and survival strategies” (p. 17). in a review of work by carello and butler (2014), downey (2013), health federation of philadelphia (2010), wolpow et al. (2009), and hoch et al. (2015), davidson (2017) offers one of the first available specific set of trauma-informed strategies and practices that faculty could implement in the classroom, which include checking in with students, preparing for significant anniversaries, being sensitive to family structures, identifying mentors and other support systems, expressing unconditional positive regard, maintaining high expectations, and maintaining appropriate boundaries, among others. additionally, knight (2015) emphasizes on how educators should neither ignore nor dwell on students’ past trauma and instead validate and normalize students’ experiences, help students to recognize how their past influences the present, and empower students. researchers have studied trauma-informed practices from the lens of k-12 students (e.g., acevedo & hernandez-wolfe, 2014; anderson et al., 2015; berger et al., 2007; ijadi-maghsoodi et al., 2017; west et al., 2014). these studies highlighted students’ voice through both focus groups and/or employing students as participant-researchers. therefore, to better support college students, it is imperative that the body of knowledge developed to help k-12 students impacted by trauma extend to colleges and universities, particularly through learning from college students’ perspectives on establishing and maintaining trauma-informed environments. to explore trauma-informed practices in higher education classrooms, this study seeks to understand college students’ perspectives on trauma-informed practices, including an expansion of trauma-informed practices or strategies that faculty could implement in the classroom. present study student perspectives on trauma-informed practices and strategies can help college and university faculty, staff, and administrators better understand how to support all students who may have been impacted by trauma. student perspectives are helpful in identifying the extent to which students believe that their higher education institution is trauma-informed and can indicate how faculty, staff, and administrators could evolve in their support. this study takes a specific focus on the nature of the college classroom and how faculty could create more trauma-informed environments for both undergraduate and graduate students. therefore, the following research questions are addressed in this study: what are college students’ beliefs on how to create a trauma-informed classroom environment? to what extent creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 101 does undergraduate or graduate status impact perceptions of trauma-informed support? because of the void in the extant literature on perceptions of trauma-informed colleges and universities between undergraduate and graduate students, i hypothesize the following: h1: undergraduate and graduate students will hold similar perceptions and beliefs of what encompasses a trauma-informed classroom environment. h2: undergraduate and graduate students will hold similar perceptions and beliefs of how trauma-informed the host institution is. methods a cross-sectional survey design was employed because this is a descriptive study aimed to examine college students’ perceptions and beliefs of trauma-informed practices at one point in time (creswell & guetterman, 2019). measure participants completed the student perspectives on creating trauma-informed classrooms survey that was designed for this study based on processes outlined in fowler (2014) because there was not an existing survey that addressed traumainformed practices and strategies in higher education settings. i began by conducting a review of the extant literature of trauma-informed practices in higher education. then, i held a focus group with six current students enrolled in the school of education at the host university, one faculty member in the school of education at the host university, and one student services staff member at the host university who have all previously learned about trauma-informed care and/or trauma-informed practices in k-12 settings to generate a comprehensive list of trauma-informed classroom strategies or practices for the college classroom. based on feedback from the focus group, i drafted initial survey items followed by a field pretest with the eight members of the focus group to check for content and face validity. survey items were then revised or removed for clarity. the final survey included 20 items on a likert type scale and four demographic items. fourteen of the 20 likert type scale items comprised the variable of traumainformed strategies or practices in which participants rated the importance of each potential strategy or practice for in-person classroom environments. the remaining six of the 20 likert type scale items comprised the variable of aspects of campus in which participants rated the extent of how trauma-informed the host university is. demographic items included year in college (i.e., first-year, sophomore, junior, senior, post-baccalaureate, master’s student, ed.s. student, doctoral student, nondegree student, other, and prefer not to say), race (i.e., asian, black/african american, native american or american indian, native hawaiian or other pacific islander, white/caucasian, two or more races, or prefer not to say), ethnicity (i.e., spanish, hispanic, or latino or prefer not to say), and gender (i.e., woman, man, non-binary, prefer to self-describe, or prefer not to say). creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 102 recruitment and procedure this study received institutional review board approval from the host university, a small, private institution located in an urban city in the south. i employed purposeful sampling (creswell & guetterman, 2019) to collect responses from students enrolled at the host university with the eligibility criteria that participants must have been currently enrolled in the school of education. students from the school of education were identified as the population because they were more likely to have received training in how to create a healthy classroom environment in k-12 settings through their coursework and/or prior work experiences, allowing them to potentially have a better understanding of what might be possible in terms of trauma-informed strategies or practices in a higher education classroom. to recruit participants, i emailed all students enrolled in the school of education, including a description of the study and informed consent, as well as access to the survey. data were collected online via qualtrics from march 2022 through april 2022. participants were not compensated and could withdraw from the study at any time. participants there were 60 participants out of 301 students contacted for a response rate of 19.9%. for gender identity, 7 identified as men, 39 as women, one as non-binary, one preferred to self-describe (i.e., she/they), and 12 preferred not to say. for race, eight identified as black or african american, two as multiracial, 36 as white, and 14 preferred not to say. for ethnicity, three identified as having spanish, hispanic, or latino origin, 45 did not, and 13 preferred not to say. for year in school, six were in their first year, four were sophomores, eight were juniors, four were seniors, 14 were master’s students, 12 were doctoral students, and 12 preferred not to say. data analysis data were analyzed in microsoft excel. construct reliability, validity, and descriptive statistics were calculated for rating scale items, and t-tests were conducted to test research hypotheses. results are explored using the six core principles of traumainformed care (i.e., safety, trustworthiness & transparency, peer support, collaboration & mutuality, empowerment & choice, and cultural, historical & gender issues) (substance abuse and mental health services administration, 2014). results first, i assessed survey items to determine construct reliability and validity (table 1). construct composite reliabilities (cr) are 0.72 and 0.83, and cronbach’s alphas (α) are 0.90 and 0.97. both reliability measures surpass the recommended 0.70 threshold (nunnally & bernstein, 1994). i evaluated discriminant validity using the average variance extracted (ave), but neither ave exceeded the recommended 0.50 threshold (fornell & larcker, 1981). the factor loadings of each item and descriptive statistics are shown in table 2 and table 3. creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 103 table 1 reliability, validity and construct correlations variable cr α ave trauma-informed strategies or practices .72 .97 .20 aspects of campus .83 .90 .47 note: cr = composite reliability; α = cronbach’s alpha; ave = average variance extracted. rating scales items covered various trauma-informed classroom strategies or practices as well as perceptions of aspects on campus. table 2 displays aspects of creating a trauma-informed campus based on likert scale ratings from 1 (strongly disagree) to 5 (strongly agree). participants expressed the following strategies and practices as most important for faculty to implement: not tokenizing a student based on identity (m = 4.81, sd = 0.44), showing students compassion and empathy in the classroom (m = 4.80, sd = 0.44), focusing on building a healthy classroom environment (m = 4.71, sd = 0.56), knowing where to go if there is an issue with an instructor (m = 4.59, sd = 0.57), and giving students individualized, supportive feedback (m = 4.57, sd = 0.57). table 2 importance of trauma-informed classroom strategies or practices strategy or practice a m sd factor loading showing students compassion and empathy in the classroom. 4.80 0.44 .721 building one-on-one relationships with students. 4.41 0.79 .556 building positive relationships among peers in the classroom. 4.41 0.75 .600 focusing on building a healthy classroom environment. 4.71 0.56 .807 allowing for individualized plans for attending classes. 3.77 0.98 .261 allowing for individualized plans for submitting assignments. 3.76 1.01 .238 alerting students ahead of time if class topics could be triggering. 4.23 0.96 .205 promoting open dialogue between students and the instructor. 4.33 0.66 .146 taking an active role in facilitating difficult conversations in the classroom. 4.33 0.63 .121 giving students individualized, supportive feedback. 4.57 0.57 .649 creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 104 strategy or practice a m sd factor loading describing what resources are available to students outside the classroom. 4.31 0.71 .208 recognizing the signs and symptoms of trauma in their students. 4.48 0.76 .511 not tokenizing a student based on identity (e.g., race/ethnicity, gender identity, military affiliation) 4.81 0.44 .194 knowing where to go if there is an issue with an instructor 4.59 0.57 .184 a participants rated strategies and practices on a likert scale (1 = not at all important to 5 = extremely important). in considering how much the host university is trauma-informed (table 3), participants perceived staff members (m = 3.43, sd = 1.02) to be more traumainformed than professors/instructors (m = 3.06, sd = 1.09). they also perceived that the university had more resources available for students' mental health (m = 3.38, sd = 1.17) than for students experiencing trauma (m = 3.02, sd = 1.18), which may indicate that participants view mental health and trauma as issues requiring unique and separate resources. broadly, participants believed that the university was moderately trauma-informed (m = 3.28, sd = 1.00). table 3 perceptions of aspects of campus aspect of campus a m sd factor loading mental health is a priority on our campus. 3.66 1.20 .844 there are enough resources provided for student mental health. 3.38 1.17 .927 there are enough resources provided for students experiencing trauma. 3.02 1.18 .736 professors/instructors are trauma-informed. 3.06 1.09 .318 staff members (e.g., academic advisors, resident advisors) are trauma-informed. 3.43 1.02 .450 the host university is a trauma-informed institution. 3.28 1.00 .615 a participants rated aspects on a likert scale (1 = strongly disagree to 5 = strongly agree). i used t-tests (two-sample assuming unequal variances) to evaluate the hypotheses, comparing undergraduate (n = 22) to graduate (n = 26) students on the variables of trauma-informed classroom strategies or practices (14 items) and aspects of campus (6 items). while there was no statistically significant difference between groups for aspects of campus, there was a statistically significant creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 105 difference between groups at the .01 level for trauma-informed classroom strategies or practices (table 4). table 4 t-test between undergraduate and graduate student based on variable variable total sample undergradua te graduate hypothesis support m sd m sd m sd t(df) p h1: traumainformed classroom strategies or practices 61.19 5.82 63.50 4.99 59.23 5.83 2.73(46) .01* unsupported h2: aspects of campus 19.70 5.45 18.27 5.04 20.96 5.58 -1.73(45) .09 supported * p < .01 discussion the goal of this study was to understand college student perceptions on how to create trauma-informed classroom environments as well as to determine if there were differences between undergraduate and graduate students. the first hypothesis (h1) considers whether there is a difference between undergraduate and graduate students on what they consider to be important trauma-informed strategies or practices in the classroom. results indicate that this is not supported, as undergraduate students (m = 63.50, sd = 4.99) rated the trauma-informed classroom strategies and practices as more important overall than graduate students (m = 59.23, sd = 5.83). the reason for this difference may be because graduate students prefer other trauma-informed strategies or practices that were not listed, or they may need less of this type of support in the classroom setting than undergraduate students. for example, graduate students may desire more trauma-informed support from student services (e.g., advising, mental health services) that are outside of the classroom environment, or perhaps graduate students simply do not need as much support in the classroom as undergraduate students. further exploration of the differences between undergraduate and graduate students’ needs for trauma-informed support in and outside of the classroom environment is an important next step, particularly because the second hypothesis (h2) is supported. h2 considers the extent that both undergraduate and graduate students perceive the host institution as trauma-informed. there was no statistical difference between undergraduates (m = 18.27, sd = 5.04) and graduate students (m = 20.96, sd = 5.58). because both undergraduate and graduate students perceive the institution as similarly trauma-informed in h2, h1 may indicate that undergraduate students hold higher expectations of trauma-informed support from faculty in the classroom. more specifically, all participants perceived the following as the top five important trauma-informed strategies and practices to implement in the classroom: (a) not creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 106 tokenizing a student based on identity, (b) showing students compassion and empathy in the classroom, (c) focusing on building a healthy classroom environment, (d) giving students individualized, supportive feedback, and (e) recognizing the signs and symptoms of trauma in their student. these traumainformed strategies and practices, as well as the others within the survey, align with the six core principles of trauma-informed care. for instance, an example of not tokenizing a student based on identity could be the instructor not calling on a black or african american student in the classroom with the expectation that the student is representative of all voices of their community in a discussion. this practice of not tokenizing identity is related to cultural, historical & gender issues in that faculty consider biases, beliefs, and values that they may hold, as well as the backgrounds and lived experiences of their students as instructors make pedagogical decisions in the classroom. next, the strategies of showing students compassion and empathy in the classroom as well as focusing on building a healthy classroom environment both speak to the principle of safety, including whether students feel safe and welcomed in the classroom and how much they perceive faculty as being caring. giving students individualized, supportive feedback relates to trustworthiness & transparency as well as empowerment & choice, as individualized feedback can empower students to continue what they are doing well and work on areas of weakness. individualized feedback also can build trust between students and faculty when students believe that faculty are being open and honest in the feedback provided. finally, the practice of recognizing the signs and symptoms of trauma in their students speaks to safety and cultural, historical & gender issues in that faculty can increase students’ safety by recognizing when they are experiencing trauma, which may be influenced by students’ cultural backgrounds and lived experiences. some of the trauma-informed strategies and practices may be straightforward for faculty to implement in the classroom, while others may be more out of faculty’s control. for example, building one-on-one relationships with students is within faculty’s discretion more so than it is for faculty to allow for individualized plans for students to attend class. while participants evaluated the importance of a range of trauma-informed strategies and practice, participants were not asked whether these strategies and practices need to be implemented simultaneously to create a trauma-informed classroom. because results indicated that undergraduate and graduate students rated the importance of these strategies and practices differently, it is likely that only some of these strategies and practices may be needed to improve the classroom environment depending on the students enrolled in the class. one strategy to determine which trauma-informed strategies and practices to employ in the classroom is for faculty to directly ask students at the beginning of the semester on how to best support them (e.g., open discussion, anonymous survey). these conversations could also occur one-on-one with students, especially if faculty may have identified a particular student as needing extra support. creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 107 limitations and recommendations for further research when interpreting the results from this study, there are notable limitations that should be considered given the study’s scope. these include the sample being limited to one university with a smaller sample size, which decreases the generalizability of findings across other institutions. additionally, while face and content validity may have met through the focus group, discriminant validity of survey items was weak, as indicated by the low average variance extracted for both variables. however, because this is a newly constructed survey, i view this initial study as exploratory and indicative of future directions. i hope to replicate this study with an increased sample size as well as with increased specificity in survey items, particularly in the trauma-informed classroom strategies and practices. while this study provides an initial set of strategies and practices that help create a trauma-informed classroom environment, there are remaining considerations that require further exploration. for example, next steps may include determining the knowledge and skills that are necessary for faculty to develop, so that they are better prepared to employ trauma-informed strategies and practices. however, further research is needed to explore the training that may be required for faculty to develop the skills and knowledge needed. while k-12 schools may have more robust guidance on creating trauma-informed classrooms and schools (e.g., avery et al., 2021; cole et al., 2005; national child traumatic stress network, schools committee, 2017), there is not as much research regarding trauma-informed guidance for higher education institutions. moreover, in addition to training faculty, creating trauma-informed colleges and universities requires further research into the roles and responsibilities of higher education staff and administrators. the trauma-informed strategies and practices included in this study were for inperson classroom environments. further research could investigate the needs of students in hybrid, online synchronous, and/or online asynchronous learning environments. finally, the finding that participants rated staff as more traumainformed than instructors is an interesting point to be further explored, such as through identifying which strategies or practices staff may be implementing that participants believe make them more trauma-informed. conclusion from the student perspective, there are several key trauma-informed practices and strategies that could support all students, both at the undergraduate and graduate level. results indicated that undergraduate and graduate rated trauma-informed strategies and practices differently, so not all practices or strategies may be equally as important. faculty may have discretion as to which ones to employ in their classrooms, and faculty may benefit from directly asking students about which strategies and practices would best support them. future directions for this work include exploring implications for trauma-informed strategies and practices beyond the classroom level, including what institutional support may be required to train faculty, as well as staff and administrators, to become more trauma-informed. creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 108 conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references acevedo, v. e., & hernandez-wolfe, p. (2014). vicarious resilience: an exploration of teachers and children’s resilience in highly challenging social contexts. journal of aggression, maltreatment & trauma, 23(5), 473–493. https://doi.org/10.1080/10926771.2014.904468 american psychiatric association (2022). diagnostic and statistical manual of mental disorders. (5th text revision ed.) washington, dc: american psychiatric association. anders, s. l., frazier, p. a., & shallcross, s. l. (2012). prevalence and effects of life event exposure among undergraduate and community college students. journal of counseling psychology, 59(3), 449–457. https://doi.org/10.1037/a0027753 anderson, e. m., blitz, l. v., & saastamoinen, m. (2015). exploring a schooluniversity model for professional development with classroom staff: teaching trauma-informed approaches. school community journal, 25(2), 113–134. https://files.eric.ed.gov/fulltext/ej1085667.pdf avery, j. c., morris, h., galvin, e., misso, m., savaglio, m., & skouteris, h. (2021). systematic review of school-wide trauma-informed approaches. journal of child & adolescent trauma, 14(3), 381–397. https://doi.org/10.1007/s40653-020-00326-w banyard, v. l., & cantor, e. n. (2004). adjustment to college among trauma survivors: an exploratory study of resilience. journal of college student development, 45(2), 207–221. https://doi.org/10.1353/csd.2004.0017 berger, r., pat-horenczyk, r., & gelkopf, m. (2007). school-based intervention for prevention and treatment of elementary-students’ terror-related distress in israel: a quasi-randomized controlled trial. journal of traumatic stress, 20(4), 541–551. https://doi.org/10.1002/jts.20225 bowen, e.a., & murshid, n.s. (2016). trauma-informed social policy: a conceptual framework for policy analysis and advocacy. perspectives from the social sciences, 106(2), 223-229. https://doi.org/10.2105/ajph.2015.302970 boyraz, g., granda, r., baker, c. n., tidwell, l. l., & waits, j. b. (2016). posttraumatic stress, effort regulation, and academic outcomes among college students: a longitudinal study. journal of counseling psychology, 63(4), 475–486. https://doi.org/10.1037/cou0000102 creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 109 boyraz, g., horne, s. g., owens, a. c., & armstrong, a. p. (2013). academic achievement and college persistence of african american students with trauma exposure. journal of counseling psychology, 60(4), 582–592. https://doi.org/10.1037/a0033672 carello, j., & butler, l.d. (2014). potentially perilous pedagogies: teaching trauma is not the same as trauma-informed teaching. journal of trauma & dissociation, 15(2), 153-168. https://doi.org/10.1080/15299732.2014.867571 carello, j., & thompson, p. (2021). lessons from the pandemic: trauma-informed approaches to college, crisis, change. palgrave macmillan. cole, s. f., o’brien, j. g., gadd, m. g., ristuccia, j., wallace, d. l., & gregory, m. (2005). helping traumatized children learn. massachusetts advocates for children. copeland, w. e., keeler, g., angold, a., & costello, e. j. (2007). traumatic events and posttraumatic stress in childhood. archives of general psychiatry, 64(5), 577-584. https://doi.org/10.1001/archpsyc.64.5.577 creswell, j. w., & guetterman, t. c. (2019). educational research: planning, conducting, and evaluating quantitative and qualitative research (6th ed.). pearson. davidson, s. (2017). trauma-informed practices for postsecondary education: a guide. portland, or: education northwest. https://educationnorthwest.org/sites/default/files/resources/traumainformed-practices-postsecondary-508.pdf downey, l. (2013). calmer classrooms: a guide to working with traumatised children. melbourne, victoria, australia: state of victoria, child safety commissioner, & queensland, australia: state of queensland, department of education, training and employment. duncan, r. d. (2000). childhood maltreatment and college drop-out rates: implications for child abuse researchers. journal of interpersonal violence, 15(9), 987–995. https://doi.org/10.1177/088626000015009005 erdman, s., colker, l. j., & winter, e. c. (2020). trauma & young children: teaching strategies to support & empower. national association for the education of young children. fornell, c., & larcker, d. (1981). evaluating structural equation models with unobservable variables and measurement error. journal of marketing research, 18(1), 39–50. https://doi.org/10.2307/3151312 fowler, f. j. (2014). survey research methods (5th ed.). sage. creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 110 frazier, p., anders, s., perera, s., tomich, p., tennen, h., park, c., & tashiro, t. (2009). traumatic events among undergraduate students: prevalence and associated symptoms. journal of counseling psychology, 56(3), 450–460. https://doi.org/10.1037/a0016412 galatzer-levy, i. r., burton, c. l., & bonanno, g. a. (2012). coping flexibility, potentially traumatic life events, and resilience: a prospective study of college student adjustment. journal of social and clinical psychology, 31(6), 542–567. https://doi.org/10.1521/jscp.2012.31.6.542 hanson, r. f., & lang, j. m. (2016). a critical look at trauma-informed care among agencies and systems serving maltreated youth and their families. child maltreatmenti 21(2), 95–100. https://doi.org/10.1177/1077559516635274 harrison, n., burke, j., & clarke, i. (2020). risky teaching: developing a traumainformed pedagogy for higher education. teaching in higher education, 1-15. https://doi.org/10.1080/13562517.2020.1786046 health federation of philadelphia, multiplying connections. (2010). cappd: practical interventions to help children affected by trauma. http://www.multiplyingconnections.org/become-trauma-informed/cappdinterventions-guide hoch, a., stewart, d., webb, k., & wyandt-hiebert, m.a. (2015, may). traumainformed care on a college campus. presentation at the annual meeting of the american college health association, orlando, fl. ijadi-maghsoodi, r., marlotte, l., garcia, e., aralis, h., lester, p., escudero, p., & kataoka, s. (2017). adapting and implementing a school-based resiliencebuilding curriculum among low-income racial and ethnic minority students. contemporary school psychology, 21(3), 223–239. https://doi.org/10.1007/s40688-017-0134-1 knight, c. (2015). trauma-informed social work practice: practice considerations and challenges. clinical social work journal, 43(1), 25–37. https://doi.org/10.1007/s10615-014-0481-6 maynard, b., farina, n., & dell, n. (2017). effects of trauma-informed approaches in schools. https://campbellcollaboration.org/media/k2/attachments/ecg_maynard_trau ma-informed_approaches.pdf national child traumatic stress network, schools committee. (2017). creating, supporting, and sustaining trauma-informed schools: a system framework. los angeles, ca, and durham, nc: national center for child traumatic stress. creating trauma-informed higher education classrooms journal of effective teaching in higher education, vol. 6, no. 1 111 new england board of higher education. (2020, april 21). trauma in the time of coronavirus and beyond: a nejhe q&a with karen gross. https://nebhe.org/journal/trauma-in-the-time-of-coronavirus-and-beyond-anejhe-qa-with-karen-gross/ nunnally, j., & bernstein, i. (1994). psychometric theory. mcgraw-hill. read, j. p., ouimette, p., white, j., colder, c., & farrow, s. (2011). rates of dsm iv-tr trauma exposure and posttraumatic stress disorder among newly matriculated college students. psychological trauma: theory, research, practice, and policy, 3(2), 148–156. https://doi.org/10.1037/a0021260 smyth, j. m., hockemeyer, j. r., heron, k. e., wonderlich, s. a., & pennebaker, j. w. (2008). prevalence, type, disclosure, and severity of adverse life events in college students. journal of american college health, 57(1), 69–76. https://doi.org/10.3200/jach.57.1.69-76 substance abuse and mental health services administration. (2014). samhsa’s concept of trauma and guidance for a trauma-informed approach. hhs publication no. (sma) 14-4884. substance abuse and mental health services administration. thomas, m. s., crosby, s., & vanderhaar, j. (2019). trauma-informed practices in schools across two decades: an interdisciplinary review of research. review of research in education, 43(1), 422–452. https://doi.org/10.3102/0091732x18821123 turner, h. a., & butler, m. j. (2003). direct and indirect effects of childhood adversity on depressive symptoms in young adults. journal of youth and adolescence, 32(2), 89–103. https://doi.org/10.1023/a:1021853600645 west, s. d., day, a. g., somers, c. l., & baroni, b. a. (2014). student perspectives on how trauma experiences manifest in the classroom: engaging courtinvolved youth in the development of a trauma-informed teaching curriculum. children and youth services review, 38, 58–65. https://doi.org/10.1016/j.childyouth.2014.01.013 wolpow, r., johnson, m.m., hertel, r., & kincaid, s.o. (2009). the heart of learning and teaching: compassion, resiliency, and academic success. office of superintendent of public instruction, compassionate schools website: https://www.k12.wa.us/sites/default/files/public/compassionateschools/pubd ocs/theheartoflearningandteaching.pdf microsoft word fisher.docx journal of effective teaching in higher education, vol. 2, no. 1 supporting and challenging student cognitions in the classroom amy k. fisher, j.d., lcsw, university of mississippi, afisher@olemiss.edu chris simmons, ph.d., lcsw, university of south florida susan c. allen, ph.d., lcsw, university of mississippi abstract. this article reports results of a mixed methods exploration of the effects of reflective supervision on cognitive growth in undergraduate students. students were enrolled in two sections of an introductory skills practice course containing an experiential exercise designed to enhance skill development. one group received supervision; the other did not. pretest and posttest analyses of measures used to assess cognitive development showed no statistically significant gains for either group. however, the group receiving supervision made more gains, and analysis of qualitative data revealed evidence of cognitive growth for this group. the research supports further investigation of educational interventions that may promote cognitive development in undergraduate students in helping professions and allied health fields. keywords: cognitive development, experiential learning, mixed methods, perry scheme, supervision one purpose of undergraduate education in the united states is to help students develop the capacity and skills needed for complex decision-making. with the everchanging landscape of societies across the globe, graduates will encounter highly complex individuals, groups, and communities. they will need to be cognitively capable of understanding and analyzing multiple issues and making informed decisions even when confronted with conflicting information. students with a high degree of cognitive complexity can be more flexible, open to new information, use creative problem-solving (streufert & swezey, 1986) and use a larger number of intervention alternatives (duehn & procter, 1974) than people with low cognitive complexity, who may have problems with various aspects of helping such as empathy and nonjudgmental attitudes (lyons & hazler, 2002; stoltenberg & delworth, 1987). this is particularly important for professionals who interact with these individuals and communities on a daily basis, such as law enforcement and corrections (morgan, morgan, foster, & kolbert, 2000), teaching (smith, 2015), counseling (granello, 2010), social work (simmons & fisher, 2016), and nursing and allied health professions (mccomb & kirkpatrick, 2016). however, teaching discipline-specific procedural skills alone may not be sufficient to bring about cognitive change needed for effective practice (anastas, 2010). helping students develop cognitive skills that will facilitate complex interactions must be intentional and should not be left to chance (gibbons & gray, 2004; fong, borders, ethington, & pitts, 1997). supporting and challenging student cognitions in the classroom 121 journal of effective teaching in higher education, vol. 2, no. 1 relevant literature studies across disciplines have found that students in helping professions do not make significant gains in cognitive development until they are in their final practicum or internship experiences (brendel, kolbert, & foster, 2002; granello, 2002; simmons & fisher, 2016), perhaps because of real world experiences encountered in the internship accompanied by reflective supervision (blocher, 1983; fong et al., 1997). however, research suggests that the development of higher order cognitive skills early in a program may help students approach learning and actual practice at a more complex level in advanced courses and internships (duys & hedstrom, 2000). in order to develop complexity in thinking and behaviors, students need to develop confidence in their own abilities, which motivates them to move from solely receiving knowledge to trusting in their intuition to construct knowledge (belenky, clinchy, goldberger, & tarule, 1986; love & guthrie, 1999). if classroom instructors understand where students are developmentally, they may offer the right amount of support and challenge to enhance development (perry, 1970, 1999). they could, in effect, create an ideal environment for cognitive development. in this paper, cognitive developmental theory will be used to explain cognitive processes among students in general and to discuss what processes may be occurring in the academic environment. with this in mind, educators may be better able to assist students in making cognitive shifts. the perry scheme one method widely utilized for conceptualizing and intentionally addressing college student cognitive development is the perry scheme (1970, 1999), which focuses directly on “the voice of the students” (perry, 1999, p. xiv) and explores epistemological development. the perry scheme uses categories and positions within categories to describe development during the college years (perry, 1970). the categories are dualism, multiplicity, relativism, and commitment within relativism. a summary of the scheme is contained in table 1. commitment within relativism has to do with ethical and moral development, and is beyond the scope of this paper. table 1 learner characteristics of students and perry positions dualism early multiplicity late multiplicity contextual relativism view of knowledge all knowledge is known. there is a most knowledge is known. there is a right way to we know some things for sure, but in all knowledge is contextual. there is no supporting and challenging student cognitions in the classroom 122 journal of effective teaching in higher education, vol. 2, no. 1 dualism early multiplicity late multiplicity contextual relativism right and wrong answer for everything. find the right answers. some knowledge is “fuzzy.” most areas we do not know anything. all opinions can be equally valid. absolute truth, but right and wrong can exist within specific contexts. view of the role of the instructor source of knowledge. good instructor = absolute authority and knower of truth. source of right way to find knowledge. role is to model the way. source of the process of thinking. instructor can also be completely discounted. source of expertise. role of expertguideconsultant. authority earned through having expertise. view of the role of the student role is to receive information. role is to learn how to learn. role is to learn to think for oneself and to learn to use supportive evidence. independence of thought is valued. role is to exercise the use of the intellect and shift from context to context. view of peers in the learning process peers are not a legitimate source of knowledge. peers are more legitimate with respect to process. interest in peers’ opinions, but instructor still final authority. peers are quite legitimate, but opinions can be discounted, as well as everyone else’s. peers are legitimate sources of learning if they use appropriate context. seeking peers opinions is part of process. evaluation issues evaluation directly related to sense of self. evaluation is the primary issue: how are my answers independent evaluation by the student evaluation of work can be separated from supporting and challenging student cognitions in the classroom 123 journal of effective teaching in higher education, vol. 2, no. 1 dualism early multiplicity late multiplicity contextual relativism bad/wrong answer = bad/wrong person. judged? fairness is paramount. should get good grade. evaluation of self. good critique includes both positives and negatives. note. from l. knefelkamp in perry (1999). students in dualism are learning how to learn. they believe that the student’s role is to receive information from trusted sources of knowledge (perry, 1970, 1999). ambiguity, multiple perspectives, and disagreement among respected authorities are obstacles. they are much more comfortable with firm definitions of “right” and “wrong.” motivation for learning is extrinsic—driven by the grades they receive on assignments. the dualism category contains two positions, basic (position 1) and full (position 2) dualism. perry (1970) found that few students arrived at college in position 1 (basic dualism), as this category described an almost childlike concreteness of belief in absolute truth and authority. however, position 2 (full dualism) finds students beginning to be able to compare and contrast information and provide explanations for their answers, particularly if they are supported with a high degree of structure and careful sequencing of the introduction of diversity of thought. the second category, multiplicity, represents the student’s growing ability to incorporate uncertainty. this category of also contains two positions: early (position 3) and late (position 4) multiplicity. students in position 3 have come to accept that uncertainty and diversity of opinions are legitimate, but they consider the uncertainty temporary. although still believing that all answers can be found with time, students in position 3 see process instead of content for the first time. they need structure to support their development, but they are beginning to experience new freedom in learning (perry, 1970, 1999). the transition to position 4 depends on the realization that we may never have the answer in many areas (perry, 1970, 1999). motivation for learning is still extrinsic; however, students come to understand that the best results are dependent on showing a reasoning process, rather than the correct answer. as they progress through multiplicity, they may show signs of intrinsic motivation. a familiar phenomenon of this stage is that students will often argue that any answer can be legitimate (and thus receive high marks), as long as the correct thinking process was followed. students in position 4 enjoy diversity of thought and may balk at structure; they are looking for a class atmosphere that promotes independence. these students are adept at the intellectual “game” (cornfield & knefelkamp, 1979, as cited in perry, 1999). these are clear indications that students are becoming more autonomous and are attempting to produce knowledge. supporting and challenging student cognitions in the classroom 124 journal of effective teaching in higher education, vol. 2, no. 1 the third category, relativism, represents a fundamental shift in cognitive development. students who reach relativism are able to ground “correct thinking process” in specific contexts and move toward looking to inner resources for knowledge and motivation for learning. this category also contains two positions: contextual relativism, position 5, and commitment foreseen, position 6. students understand that right and wrong can exist within certain contexts, but there is no absolute truth. relativism requires the ability to tolerate uncertainty and ambiguity. although this position is what many professors try to instill in their students, studies suggest that few achieve full relativism prior to graduation (granello, 2002). this model is meant to be descriptive. nonetheless, it can provide guidance for supporting student cognitive development and challenging students to move to the next level. the model provides insights that could be leveraged to promote student development in a way to foster essential facilitative qualities and skills to promote interaction with complex individuals and systems. cognitive complexity studies researchers across disciplines have examined the relationship between cognitions and facilitative qualities. cognitive complexity has been associated with traits essential to helping relationships such as confidence and high psychological functioning (birk & mahalik, 1996; brendel et al., 2002), more effective clinical hypotheses (holloway & wolleat, 1980), and skills such as critical thinking (rapps, riegel, & glaser, 2001) and sophisticated descriptions of client characteristics (borders, 1989). it has been linked to that most essential helping attribute, empathy (benack, 1988; king jr, 2011; lovell, 1999) and multicultural competence (cannon, 2008; wendler & nilsson, 2009), an essential quality for students in helping professions (see, e.g., council on social work education, 2015). researchers have also investigated the question of what factors impact cognitive complexity across professions. several have shown that educational level is related to cognitive complexity (brendel et al., 2002; granello, 2002; harrison & atherton, 1990; simmons, 2014). brendel and colleagues (2002) tested students at the beginning, middle, and end of their training, and their findings suggested that most of the students did not show an increase in cognitive complexity until they had completed their training. granello (2002) used the perry scheme to conduct a cross-sectional analysis of counseling students who were at the beginning, middle, and end of their training, finding that students made more gains in cognitive complexity when they were at the end of their education, after completing internships, than they made from the beginning to the middle of their training. simmons & fisher (2016) clarified these findings by showing that although the internship experience was related to cognitive development, traditional classroom instruction was not. in the sample studied, the internship did not occur until coursework was completed, thus suggesting that the internship experience had unique properties contributing to cognitive development. the authors speculated that internship education provided the optimal environment for cognitive growth— supporting and challenging student cognitions in the classroom 125 journal of effective teaching in higher education, vol. 2, no. 1 perhaps the combination of interaction with real-world clients plus guided reflection through supervision. pre-internship educational interventions to enhance cognitive complexity remain relatively unstudied. using a pretest and posttest design, one study examined the cognitive complexity of graduate counseling students either taking an enhanced basic skills course or another introductory social work course (duys & hedstrom, 2000). in addition to lectures, the enhanced skills course included role play, structured small supervisory groups facilitated by doctoral students or faculty, and feedback and evaluation of skill development. they found students in the enhanced skills course made significantly higher gains in cognitive complexity than students completing the introductory course. the study suggested that role play and guided reflection might support cognitive development prior to the internship experience. the current study with these findings in mind, the authors of the current study sought to design and explore the effects of a pre-internship experiential educational exercise that included role play and guided reflection through supervision to see what effect, if any, it had on cognitive development in undergraduate students in a social work program. the details of the assignment are contained in table 2. table 2 assignment details assignment timing pre-self-assessment beginning of semester classroom discussions weekly throughout semester five recorded role play beginning at middle of semester and continuing biweekly sessions until complete five supervision sessions following each role play session case presentation end of semester post-self-assessment end of semester over the course of the semester, graduate students learning clinical supervision reviewed undergraduate student role-play recordings and provided 30-minute supervision sessions following each role-play session that included verbal feedback. the graduate student supervisors also completed written evaluations of the undergraduate student skills at the midpoint and end of the semester and reviewed these in session with the undergraduate student supervisees. this allowed the students to begin to integrate classroom instruction with actual skills practice. supporting and challenging student cognitions in the classroom 126 journal of effective teaching in higher education, vol. 2, no. 1 the undergraduate students conducted an end-of-the-semester case presentation to their classmates on their role play and received feedback from the instructor and the group. finally, they completed a post-self-assessment to encourage further reflection on their development. throughout the semester, students were supported and challenged in supervision sessions as well as in the classroom. the present study was based on the assumptions that internship plays a significant role in the cognitive development of undergraduate students and these students make the most gains in cognitive development when they are in internship. what is unknown is whether they can make significant gains in cognitive complexity prior to the internship experience, i.e., whether cognitive development can be accelerated using guided reflection without exposure to actual clients. the purpose of this study was to explore the effect on cognitive development of role play supplemented with guided reflection through supervision of undergraduate students after 15 weeks of a pre-internship skills course. a research question guided this research: do undergraduate social work students’ levels of cognitive complexity change after 15 weeks of a pre-internship course requiring a semester-long skills role play with supervision? because this study was exploratory in nature, we made no formal hypotheses. methods and results the study used a consecutive mixed methods approach. the sample contained students from an introductory skills course who participated in role plays and received supervision. students completed quantitative instruments testing their levels of cognitive complexity at the beginning of the semester and then 15-weeks later at the end of the semester. following analysis of the quantitative data, the researchers analyzed a subset of students’ post-self-assessments in an attempt to better understand the quantitative results. quantitative data participants. the total nonprobability sample consisted of 37 students. the demographic characteristics of the sample and test statistics are provided in table 3. table 3 demographic characteristics of quantitative sample variable n % gender male 4 5.5% female 33 45.2% age 18-24 years 27 37.0% supporting and challenging student cognitions in the classroom 127 journal of effective teaching in higher education, vol. 2, no. 1 variable n % 25-29 years 4 5.5% 30-34 years 1 1.4% 35-59 years 5 6.8% ethnicity european american 13 17.8% african american 22 30.1% other 2 2.7% instrument. the learning environment preferences scale (lep) is a 65-item questionnaire that measures cognitive development (i.e., dualism, multiplicity, contextual relativism) along five domains that contribute to student learning (i.e., view of knowledge/learning; role of the instructor; role of the students and peers; the classroom atmosphere/activities; and evaluation methods) according to the perry scheme (moore, 1990). each domain contained a stem with 13 statements that participants rate as important or significant. to reflect student learning in social work, “to learn social work” was added to the general stems. this is consistent with changes made in previous research using the lep (granello, 2002, 2010; simmons, 2014; simmons & fisher, 2016). students were asked to rank the three most important statements out of 13 statements in each domain. the rankings were then used to compute students’ cognitive complexity index (cci) scores, which is a single numerical score between 200 (dualism) and 500 (relativism). cci was reported as a reliable subscale, with the alpha coefficients for the cci—by position—reported as .72 to .84, and the test-retest reliability was .89 (moore, 1990). criterion, concurrent, and construct validity were explored and found to be acceptable (see moore, 1990, for a detailed discussion of the lep's psychometric properties). procedures. emails with a link to the informed consent and pretest survey were sent to all students in two sections of an introductory practice course (sw 335) before the start of the semester. at the end of the semester, an email link to the posttest was sent to the students. procedures complied with the university’s institutional review board requirements. the surveys were administered online. no identifying information was collected. in order to match a students’ pretest with their posttest, students were given a set of questions at the beginning of each survey, which produced a unique identifier. to address the research question, students completed the lep and demographic information at the beginning and end of the semester. analysis and results. a paired-samples t-test was conducted to determine whether there were significant differences between pretest and posttest scores for the students on mean change in cognitive complexity. table 4 shows cognitive complexity pretests and posttest means and standard deviations for cognitive complexity, and the results of the paired-samples t-test. the test revealed that although mean cci scores increased, there were no statistically significant changes in cognitive complexity from pretest to posttest. results showed that 57% of the supporting and challenging student cognitions in the classroom 128 journal of effective teaching in higher education, vol. 2, no. 1 students made gains in cognitive complexity, 2% made no gains, and 41% showed decreases. table 4 cci means and standard deviation for demographic variables cci pre cci post n m sd m sd t(df) p gender male 4 314.5 37.6 318.0 29.4 female 33 314.2 44.1 321.9 34.4 age 18-24 years 27 318.4 39.4 315.2 29.6 25-29 years 4 289.3 70.0 345.5 15.0 30-34 years 1 390.0 396.0 35-59 years 5 294.4 12.7 321.2 50.9 ethnicity european american 13 318.4 31.2 320.7 27.2 african american 22 308.0 48.0 320.8 34.9 multi-ethnic 2 356.5 47.4 333.0 89.1 total 37 314.3 43.0 321.4 34.5 -.97(36) .34 note. n = 37. cci =cognitive complexity index. qualitative data participants and instrument. twenty-five of the 37 students enrolled in the courses completed written post-self-assessments as part of their course requirements. the post-self-assessment was designed to help the students assess their progress over the course of the semester. the students provided a summary of strengths and growth areas as professional practitioners and discussed critical incidents from the semester, chosen models or theory of practice, knowledge of culture and the importance of relationships, a summary of what was learned over the semester, and future goals as a professional social worker. although not administered specifically to assess cognitive complexity, the researchers explored patterns in the qualitative data that might yield additional information regarding cognitive development. data analysis. in the first cycle of exploratory analysis, provisional themes derived from perry’s scheme were used to summarize the raw data into categories, segmenting the text into meaningful units, and assigning codes and labels (miles, huberman, & saldaña, 2014). the team adjusted themes and codes during the second cycle, looking for patterns and other ways to organize the data. the team then coded the data using the themes derived from this process using a continuing supporting and challenging student cognitions in the classroom 129 journal of effective teaching in higher education, vol. 2, no. 1 iterative process that reflected an interactive model of data analysis. the team engaged in ongoing dialogue and reflection, referring to the coding, original data, and the audit trail when needed to further refine themes and reach consensus. the researchers used triangulation of data and thick descriptions (lietz & zayas, 2010) to protect reliability. to further increase the reliability of the analysis and check for reactivity and bias, the researchers utilized a third qualitative analyst who was not involved with the present study, had not taught students, and did not know the identity of the students in the study. the external analyst did not have access to the manuscript or coding and themes as developed by the research team. the external analyst initially had only post-assessments, the research question, and a copy of perry’s scheme. the external analyst coded the text that described experiences in relationship to perry’s model using a matrix that ranked text along the continuum of development of cognitive complexity. the purpose of this ranking was to elicit information that illustrated where students were in terms of development of cognitive complexity. data was subsequently coded into themes representing similar concepts, such as development and use of specific skills, self-awareness and personal learning, contextualization of learning, and motivation to learn and improve skills. the results of the external analysis were compared with the research team’s analysis and further review and discussion indicated consistency. results. the analysis resulted in two broad themes related to cognitive complexity: indicators of position within the perry scheme and skill development with and without complex understanding. both themes were related to the research question in that they provided information about cognitive development. the data illustrated that development for the group with supervision was fluid, with some students showing both multiplistic (more advanced) and dualistic (less advanced) ways of thinking. as students assessed their overall progress, knowledge learned, and skill development, however, a view of their progress emerged. the first broad theme, position within the perry scheme, generally revealed suggestions of movement within the stage of multiplicity. as students move through multiplicity, they develop greater openness to new/other perspectives and information. their motivation to learn moves from being extrinsically to intrinsically motivated, and critical thinking develops and deepens. students indicated a deepening openness to other perspectives. as one student stated, “[t]aking this class has opened my eyes to the definition of culture. before this class i considered a person[’s] ethnicity the base of their culture but it is much more than just that.” additionally, they demonstrated increased openness, understanding, and application of social work practice perspectives: “when i first entered this course, i ha[d] always wanted to help people that [do] not have access to things that are available for them. now, i use the strengths perspective to empower a client.” another student reported, “i am in a better position to empathize with others.” supporting and challenging student cognitions in the classroom 130 journal of effective teaching in higher education, vol. 2, no. 1 many students reported new excitement and confidence in their ability to learn. one student stated, “[g]aining new knowledge about the field of social work has given me a drive to continue to gain knowledge about being a better social worker.” another student related, i have worked hard in this course and hope that my service learning paper and the final will give me an a. if it does not, then i will gladly take the grade that is given to me because i feel that the knowledge that i gained in this class is unmeasurable. to be sure, many students remained firmly entrenched in extrinsic motivation. as one stated, “the goals i set early this semester [were] to complete all assignments and make a b or greater in this class.” these students placed great emphasis on their grades when discussing whether their goals had been met. some students expressed an increase in confidence over the course of the semester, which also demonstrated growth: i found out that i have really good social work skills while doing the role plays and meeting with my supervisor these last three months. in the beginning, i wasn’t sure if social work was the right career choice for me but now i feel like it fits well. another student declared, “with each day that passed i felt more confident….” the final category showing movement within the positions was increasing capacity for complex thinking and problem solving. this was shown through increasing belief in the process, rather than the result: “‘you can’t save everyone’…helped me realize that at the end of the day when you have done your best as a trained professional that is really all you can do.” increasing tolerance for ambiguity was noted: “it can be stressful and overwhelming, but i think that this something that will just come in time with practice.” one student summed up her growth as follows: i have learned that social work is not black and white, there is room for the “what if’s” in social work practice. i have learned that social work is more than just what is in the book. i have grown in the aspect that i have questioned my thoughts and action[s] more and the client situations less. the second major theme that emerged was skill development. differences between development with and without complex understanding were illuminated. students at a less complex level reported their learning in a mechanistic fashion, without any evidence of the implicit cognitive processes that accompany the skills: “i have [grown] in many areas. i am now more comfortable asking questions and asking open-ended questions.” another reported, “i didn’t have any idea about the different stages in assessing a client.” other the hand, students who demonstrated cognitive growth were able to interweave complex thought processes that accompany the use of skills: supporting and challenging student cognitions in the classroom 131 journal of effective teaching in higher education, vol. 2, no. 1 i was not aware of all the planning that takes place before meeting with a client or the stresses of worrying if you have done right by your client. another explained, “i have learned how to build new theories from practice as well as old ones. i have learned how to think more critically about the different types of clients and their situation[s].” discussion the goal of the study was to explore undergraduate students’ cognitive development during an introductory practice course taught with supervision of role plays. the quantitative analysis shows students did not make statistically significant gains in cognitive complexity from beginning to end of the semester. this is inconsistent with the duys & hedstrom (2000) study finding that graduate students made significant gains while enrolled in a pre-internship skills course using guided reflection through supervision and other studies showing undergraduate students may make similar gains in cognitive complexity (simmons, 2014; simmons & fisher, 2016). the lack of statistical significance could be due to the small sample size reflecting an issue of power. it may also be that, as speculated in similar studies, the construct of cognitive complexity is so global that it hard to see significant changes over the course of just one semester (see, e.g., fong et al., 1997). indeed, morgan et al.’s (2000) study of law enforcement trainees and criminal justice students revealed no significant changes in cognitive development following an educational intervention involving guided reflection, even though scores moved in a positive direction. morgan et al. also speculated the small sample size and short duration of the study contributed to those findings. it is worth noting that the undergraduate social work students in this study who received supervision made higher numerical gains in cognitive complexity than the students without supervision (7 points versus 1 point). previous research using the same instrument (cci) showed that students’ growth measured over the course of the entire pre-internship program was relatively small (2 points) (granello, 2002), or even declined in complexity (-10 points) (simmons & fisher, 2016). the qualitative data supports the speculation that growth for students at this stage is highly individualized or perhaps too nuanced to detect with the instrument used. in the quantitative sample, 57% (21 students) made gains, 41% (15) showed decreases in complexity, and 2% (1) stayed the same. the qualitative data sheds light on this finding: even among those students whose cognitive functioning was assessed as primarily dualistic, in certain areas they demonstrated movement toward multiplicity. for example, growing diversity of thought was illustrated by the students’ reports of openness to and embracing new information concerning topics from culture to theory/perspective, growing ability to empathize, and movement toward intrinsic motivation to learn. on the other hand, entrenchment in external motivation by some illustrates the idea that while developmental models are helpful for explaining growth, individual students can vary widely at any given point in the program (granello, 2002). indeed, morgan et al. (2000) attributed the lack of significant change in their study to what piaget called décalage—the “way in which supporting and challenging student cognitions in the classroom 132 journal of effective teaching in higher education, vol. 2, no. 1 development can be haphazard or uneven and often occurs as a result of cognitive dissonance which accompanies psychological growth” (p. 215). the concept of décalage is mirrored in the perry scheme by the notion of disequilibrium: students faced with a novel learning task may experience a temporary regression or lag in cognitive development (perry, 1970; widick, 1977). it is possible that the students involved in the role play with supervision faced great disequilibrium due to the added unknown and potentially complicating factor of supervision. limitations the lack of significant results, of course, greatly limits the generalizability of the study, as does the use of qualitative data. in addition, this study used a small, nonrandom sample from a single social work program. implications educators can use the perry scheme to support and challenge students so that their cognitive development supports facility in addressing the complexities of professional practice, especially such skills as critical thinking and the ability to “go beyond prescriptive solutions to complex problems” (harrison & atherton, 1990, p. 87). educators should work to discover and create learning activities that advance students’ cognitive abilities prior to internship. this study suggests that experiential learning paired with opportunities for reflection may support cognitive development. however, one cautionary note is that researchers may not be able to rely on solely quantitative measures to explore cognitive development. qualitative measures may provide much more nuanced and instructive insights for educators. future iterations of the exercise should be developed, implemented, and studied to ascertain the effects of guided reflection through supervision on cognitive complexity. there is a need for studies across disciplines and professions with higher numbers of participants and more robust design to help with generalizability. furthermore, students participating in guided reflection through supervision may need additional support to maintain their equilibrium at the beginning of the exercise due to the added novelty of the supervision experience. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references belenky, m. f., clinchy, b. m., goldberger, n. r., & tarule, j. m. (1986). women's ways of knowing: the development of self, voice, and mind. new york: basic books. supporting and challenging student cognitions in the classroom 133 journal of effective teaching in higher education, vol. 2, no. 1 benack, s. (1988). relativistic thought: a cognitive basis for empathy in counseling. counselor education & supervision, 216-233. https://eric.ed.gov/?id=ej373069 birk, j. m., & mahalik, j. r. (1996). the influence of trainee conceptual level, trainee anxiety, and supervision evaluation on counselor developmental level. the clinical supervisor, 14(1), 123-137. http://dx.doi.org/10.1300/j001v14n01_09 blocher, d. h. (1983). toward a cognitive developmental-approach to counseling supervision. counseling psychologist, 11(1), 27-34. https://doi.org/10.1177/0011000083111006 borders, l. d. (1989). developmental cognitions of first practicum supervisees. journal of counseling psychology, 36, 163-169. http://dx.doi.org/10.1037/0022-0167.36.2.163 brendel, j. m., kolbert, j. b., & foster, v. a. (2002). promoting student cognitive development. journal of adult development, 9(3), 217-227. http://dx.doi.org/10.1023/a:1016056227344 cannon, e. p. (2008). promoting moral reasoning and multicultural competence during internship. journal of moral education, 37(4), 503-518. https://doi.org/10.1080/03057240802399384 council on social work education. (2015). educational policy and accreditation standards. alexandria, va: author. https://www.cswe.org/getattachment/accreditation/accreditationprocess/2015-epas/2015epas_web_final.pdf.aspx duehn, w. d., & procter, e. k. (1974). a study of cognitive complexity in the education for social work practice. journal of social work education, 10(2), 20-26. https://doi.org/10.1080/00220612.1974.10672305 duys, d. k., & hedstrom, s. m. (2000). basic counselor skills training and counselor cognitive complexity. counselor education & supervision, 40(1), 1-9. http://dx.doi.org/10.1002/j.1556-6978.2000.tb01795.x fong, m. l., borders, l. d., ethington, c. a., & pitts, j. h. (1997). becoming a counselor: a longitudinal study of student cognitive development. counselor education & supervision, 37(2), 100-114. http://dx.doi.org/10.1002/j.15566978.1997.tb00536.x gibbons, j., & gray, m. (2004). critical thinking as integral to social work practice. journal of teaching in social work, 24(1/2), 19-38. https://doi.org/10.1300/j067v24n01_02 granello, d. h. (2002). assessing the cognitive development of counseling supporting and challenging student cognitions in the classroom 134 journal of effective teaching in higher education, vol. 2, no. 1 students: changes in epistemological assumptions. counselor education & supervision 41, 279-293. granello, d. h. (2010). cognitive complexity among practicing counselors: how thinking changes with experience. journal of counseling & development, 88, 92-100. http://dx.doi.org/10.1002/j.1556-6678.2010.tb00155.x harrison, w. d., & atherton, c. r. (1990). cognitive development and the "one foundation" controversy. journal of social work education, 26(1), 87-95. https://doi.org/10.1080/10437797.1990.10672137 holloway, e. l., & wolleat, p. l. (1980). relationship of counselor conceptual level to clinical hypothesis formation. journal of counseling psychology, 27(6), 539-545. http://dx.doi.org/10.1037/0022-0167.27.6.539 king jr, s. h. (2011). the structure of empathy in social work practice. journal of human behavior in the social environment, 21(6), 679-695. https://doi.org/10.1080/10911359.2011.583516 lietz, c. a., & zayas, l. e. (2010). evaluating qualitative research for social work practitioners. advances in social work, 11(2), 188. http://journals.iupui.edu/index.php/advancesinsocialwork/article/view/589 love, p. g., & guthrie, v. l. (1999). perry's intellectual scheme. new directions for student services, 88, 5-15. lovell, c. (1999). empathic-cognitive development in students of counseling. journal of adult development, 6(4), 195-203. lyons, c., & hazler, r. j. (2002). the influence of student development level on improving counselor student empathy. counselor education & supervision 42, 119-130. http://dx.doi.org/10.1002/j.1556-6978.2002.tb01804.x mccomb, s. a., & kirkpatrick, j. m. (2016). article: impact of pedagogical approaches on cognitive complexity and motivation to learn: comparing nursing and engineering undergraduate students. nursing outlook, 64, 37– 48. miles, m. b., huberman, a. m., & saldaña, j. (2014). qualitative data analysis : a methods sourcebook (3rd ed.): thousand oaks, califorinia: sage publications, inc. moore, w. s. (1990). learning environment preferences instrument manual. olympia, wa: center for the study of intellectual development. morgan, b., morgan, f., foster, v., & kolbert, j. (2000). promoting the moral and conceptual development of law enforcement trainees: a deliberate supporting and challenging student cognitions in the classroom 135 journal of effective teaching in higher education, vol. 2, no. 1 psychological educational approach. journal of moral education, 29(2), 203218. https://doi.org/10.1080/03057240050051765 perry, w. g. (1970). forms of intellectual and ethical development in the college years: a scheme. new york: holt, rinehart, and winston. perry, w. g. (1999). forms of intellectual and ethical development in the college years: a scheme. san francisco: jossey-bass. rapps, j., riegel, b., & glaser, d. (2001). testing a predictive model of what makes a critical thinker. western journal of nursing research, 23(6), 610-626. https://doi.org/10.1177/01939450122045410 simmons, c. (2014). graduate social work education and cognitive complexity: does prior experience really matter? journal of teaching in social work, 34(3), 235-247. https://doi.org/10.1080/08841233.2014.908337 simmons, c., & fisher, a. (2016). promoting cognitive development through field education. journal of social work education. 4(52), 462-472. https://doi.org/10.1080/10437797.2016.1198290 smith, d. (2015). exploring leadership development with supervisory officers through case inquiry. reflective practice, 16(4), 559-574. https://doi.org/10.1080/14623943.2015.1064382 stoltenberg, c. d., & delworth, u. (1987). supervising counselors and therapists: a developmental approach. san francisco: jossey-bass. streufert, s., & swezey, r. w. (1986). complexity, managers, and organizations. new york: academic press. wendler, a. m., & nilsson, j. e. (2009). universal-diverse orientation, cognitive complexity, and sociopolitical advocacy in counselor trainees. journal of multicultural counseling & development, 37(1), 28-39. https://doi.org/10.1080/08841233.2014.908337 widick, c. (1977). the perry scheme: a foundation for developmental practice. the counseling psychologist, 6(4), 35-38. microsoft word gilliam&toliver_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi 10.36021/jethe.v4i2.167 black feminist wondaland: reckoning, celebrating, and reclaiming joy in higher education elizabeth gilliam, florida state university s.r. toliver, university of colorado, boulder, stephanie.toliver@colorado.edu abstract. janelle monae’s, dirty computer, tackles issues like feminism, racism, sexuality, black womanhood, self-assurance, and growth. each song on the album is presented from a first-person point of view, offering a unique insight into a story that shares an intimate portrait of what it means to embrace authentic black womanhood. monae’s lyrical storytelling brings to life stories of love, loss, fear, and celebration, offering an experience that cannot be ignored. still, the numerous ways black women experience joy and celebration are often overlooked in higher education. thus, in this article, we center monae’s album and offer the framework, black feminist wondaland (bfw), to account for how black women reckon with the misogynoir enacted against us, celebrate ourselves as an act of radical resistance, and reclaim our joy in a society bent on keeping us in a state of sorrow. keywords: afrofuturism, black feminist wondaland, black feminist/womanist storytelling, dirty computer, black women in her album, dirty computer (robinson et al., 2018), janelle monae presents intimate commentary centering what it means to embrace authentic black womanhood in a society that often ignores and erases black women. the album is loosely grouped into three parts: (1) reckoning, which examines monae’s acknowledgement of how society views her as a black queer woman, (2) celebration, which explores her acceptance of self despite the world’s oppressive views; and (3) reclamation, which situates love—love of country, love of others, and love of self—as the catalyst for social change. in thematically situating the album this way, monae argues that dirty computers, “people who are made to feel like integral parts of their beings are bugs and viruses—can band together, find love, and fight back” (setaro, 2018). in keeping with the metaphor of the album—as we will continue to do throughout this article—we believe black women in higher education are often positioned as dirty computers, viruses infiltrating the university system and causing it to malfunction. however, monae suggests this anti-black reality is one we are not obligated to accept. she commands attention with a simultaneous grace and vigor and uses her power and platform to let black women know she sees who we are, as we are. thus, in this article, we center monae’s album as a love letter to black women and use it to offer the framework, black feminist wondaland (bfw), to account for how black women reckon with the racialized and gendered violence enacted against us, celebrate ourselves as an act of radical resistance, and reclaim our joy in a society bent on keeping us in a state of sorrow. we situate ourselves “in the along” by dreaming of an otherwise educational space that grapples with the elusive project of black female mattering in higher education. black feminist wondaland 85 journal of effective teaching in higher education, vol. 4, no. 2 misogynoir in higher education from a young age, black women are consistently reminded of the perceived “bugs” in our system. we are reminded of how we are less likely than our female peers to graduate on time or to obtain a post-secondary degree (baxley & boston, 2010; smith-evans & george, 2014). we are reminded that we are 5.5 times more likely to be suspended than our white female peers and that we receive more suspensions than any other racial group (george, 2015; onyeka-crawford et al., 2017). we are forced to modify our culturally sanctioned ways of being and adopt standards of “racelessness” (evans-winters, 2010; fordham, 1988), or assimilation into white culture while distancing ourselves from blackness, in order to avoid ostracization. when we do achieve academic success, our accomplishments are belittled as some educators focus more on our social decorum rather than our academic triumphs (carter, 2006; fordham, 1993; morris, 2007). ultimately, we are positioned as corporeal sites of struggle who infiltrate and “dirty” academic spaces. these difficult reminders continue when we dare to enter higher education. although black women persist through stereotypes and earn degrees at high rates, our stories are often excluded from the success discourse (patton & croom, 2017). historically white institutions often see our strength as liabilities rather than signs of empowerment (williams et al., 2020). various groups find themselves intimidated by our existence because our behaviors are often at odds with the controlling images (e.g., loud, angry, hyper-sexual, ratchet, sassy, uneducated) encircling perceptions of black womanhood (collins, 2009; lewis & miller, 2018; neal-jackson, 2020). because of this, we are “forced to invalidate stereotypes and legitimize [our] competence, intelligence, and overall worth” (bertrand jones et al., 2013, p. 326). this positioning of black women as inferior, subservient, hypersexual, hostile, or overly masculine is a form of misogynoir or “anti-black racism and misogyny worked against black women” (bailey & trudy, 2018, p. 764). it forces us to defend our existence and punishes us for refusing to accept the stereotypical roles forced upon us. ultimately, our essence as black and woman entwined with our other identities is categorized as detrimental aspects of our american code. afrofuturism a hallmark of monae’s work is the use of afrofuturism to paint a picture of a world vivid in its imagery and spirit. afrofuturism uses various combinations of science fiction, fantasy, magical realism, horror, history, and imagination to critique oppressions and imagine unbound black subjectivities in future or alternate worlds (barr, 2008; womack, 2013). black people have harnessed the aesthetic to combat socially constructed realities converging black identities with calamity and to contest stories where imaginative spaces align the persistence of black lives with a disastrous future (yaszek, 2006). afrofuturism has also been used as a foundation to disrupt modern ideologies restricting black imaginations and to provide a space for black people to envision the tools necessary to subvert oppressive paradigms black feminist wondaland 86 journal of effective teaching in higher education, vol. 4, no. 2 and create equitable futures (eshun, 2003). afrofuturism is a vehicle black people use to critique social injustice and position ourselves as agents of social change. black female afrofuturists often centralize black women and girls, redefining how our marginalized identities are depicted (barr, 2008; womack, 2013). through afrofuturism, black women create physical and symbolic spaces where we can explore our identities, digging behind societal ideas about what it means to inhabit a black female body, and creating room for other black women and girls to define their own existence in numerous and nuanced ways (morris, 2016). monae has consistently presented visions of possibility for black people, amplifying the voices of those who are positioned as viruses destined to destroy america. she previously used her android alter ego, cindi mayweather, to focus on the black female body, presenting herself in a “safe” package that was less a role-play than a glimpse into her truths (english & kim, 2013). monae offered this package because the public did not “know janelle monae, and [she] felt like [she] didn’t really have to be her because they were fine with cindi” (wortham, 2018). cindi was safer than janelle. this use of artistry calls upon bell hooks’ (2015) musings, where she asked appropriately if there is any surprise “people whose bodies have been perpetually used, exploited, and objectified should not seek to turn flesh into armor?” (p. 90). monae, like so many black women, is both hyper-visible and invisible (collins, 2009), so she created social armor, “safe” packaging that allowed her to hide her truths. through cindi, monae amplified visions of black possibility while also confining herself, but in dirty computer, she shed the armor and freed herself from captivity. in moving from body/flesh to android/armor and back again, we see monae’s work as the embodiment of black freedom dreaming. as kelley (2002) argued: progressive social movements do not simply produce statistics and narratives of oppression; rather, the best ones do what great poetry always does: transport us to another place, compel us to relive horrors, and, more importantly, enable us to imagine a new society (p. 9). through afrofuturist narratives, monae uses her body of work to ask listeners to understand and reckon with the horrors of black female existence in the u.s., but she also imagines and musically creates another place, a new society where we can reclaim our histories, our futures, our joys. she acknowledges the need for social armor but sings of worlds where we are not forced to constantly ready ourselves for battle. black feminist wondaland according to collins (2009), black women academics explore “the ‘inside’ ideas that allow black women to cope with and, in many cases, transcend the confines of intersecting oppressions of race, class, gender, and sexuality” (p. 98). this exploration is the foundation for black feminism. specifically, black feminist thought (collins, 2009; collins, 2016) centers black women’s lived experiences as credible sources of knowledge; focalizes the use of dialogue between community members as black feminist wondaland 87 journal of effective teaching in higher education, vol. 4, no. 2 a way to create new knowledge; and suggests emotion and care are essential to the research and knowledge validation process. as oppositional knowledge, black feminism believes “black women are inherently valuable, that our liberation is a necessity” and “the only people who care enough about us to work consistently for our liberation are us” (combahee river collective, 1977, para. 9). ultimately, black feminism provides us a location to reckon with, celebrate, and reclaim ourselves in ways traditional feminism does not. by exploring the “inside” ideas of dirty computer, we theorize black feminist wondaland (bfw), a critical theory blending the ideas of black feminism with the afrofuturistic ideals embedded within monae’s work. we explore the social and academic work of the album and discuss how it can be used to center how black women reckon with oppression, celebrate themselves, and reclaim joy. brittney cooper (2018) says “black feminism is about the world black women and girls can build, if all the haters would raise up and let us get to work” (p. 35). moving past the “haters” imposing their will against black women and girls, we acknowledge and speak truth to systems working together to maintain an oppressive state. we present this article as our contribution to the work, moving past the “haters” to tell our stories. reckoning “i don’t wanna live on my knees, i just gotta tell the truth.” monae is aware of her place in the world, not only as a commercialized “other,” but also as a black woman (english & kim, 2013). for instance, she presents to black women a very personal acknowledgment of the difficulties we face, reminding us we “are not america’s nightmare, [we are] the american dream.” lines like “but no matter where i was, i always stood out, black waldo dancing with the thick brows” and “but all i really ever felt was stressed out, kinda like my afro when it’s pressed out” speak frankly to the stereotypical images of black women forced upon us as well as how we are asked to erase ourselves and our histories to fit into white ideals (collins, 2009). additionally, she honors her pansexual identity and challenges the notion of mandatory heterosexuality in the music industry and black community (durham et al., 2013). she follows in the footsteps of fellow queer black artist meshell ndegeocello by defying what has been accepted in the realms of “sexuality, christianity, and the church…by bringing it into a queer context” (clay, 2008). this is shown when she raps, “i just wanna find a god, and i hope she loves me, too.” she acknowledges the oppressive heteronormative confinement of black women and uses her music to show solidarity. through reckoning, she contends with dominant narratives surrounding black female existence—we are the nightmares, the oddities, the bugs in the system—and uses these acknowledgements to prepare for a better future. black feminist wondaland 88 journal of effective teaching in higher education, vol. 4, no. 2 celebration “black girl magic! y’all can’t stand it, y’all can’t ban it—made out like a bandit! they been tryin’ hard just to make us all vanish—i suggest they put our flag on a whole ‘nother planet!” taking time to celebrate the self is an act of resistance, and celebrating others can be an act of revolution. monae not only takes the time to celebrate herself, but she honors all black women by reminding us not only of the power we hold within ourselves, but of the responsibility we have to face the larger society with power from within. for instance, “django jane” is an homage to the power of black women, a praisesong where we feel seen and heard, an anthem we could use whenever we got down or weary. when she proclaims, “jane bond, never jane doe, and i django, never sambo,” she introduces a feminized james bond, refuses gender norms, rejects black women’s erasure, and refuses racist stereotypes all within the space of ten words. in “i like that,” monae describes how she has come to a place of complete understanding and personal validation, at one point saying “i don’t care what i look like, but i feel good. better than amazing, and better than i could.” through these songs, monae celebrates all parts of black womanhood and commemorates black life and self-definition. she moves from reckoning with the world to the celebration of the self within that world. reclamation “hold on, don’t fight your war alone. hate all around you, don’t have to face it on your own. we will win this fight, let all souls be brave— we’ll find a way to heaven, we’ll find a way.” being black in america is a life of constant struggle. it is showing pride in your race and culture in a society that cares nothing for your survival. it is wanting to show pride in a country where your freedom was and is conditional. monae, however, provides space to consider what it would mean to reclaim our place in the fabric of america as citizens of a country that raised us, albeit with dysfunction. in one line, she says, “hands go up, men go down. try my luck? stand my ground. die in church, live in jail. say her name, twice in hell.” she speaks these lines, leaving no room for any misunderstanding of her words or intentions, as she pays tribute to trayvon martin, the victims of the 2015 charleston church shooting, and sandra bland. she says their names, remembers their stories and claims their histories as part of american history. the album ends with a proclamation: “this is not my america. but i tell you today that the devil is a liar, because it’s gon’ be my america before it’s all over.” here, monae says we are not resigned to the fate offered to us because we have the right to stand and fight for what is ours. we will collectively find a way to reclaim our freedom, even if we have to create a path that does not yet exist. black feminist-womanist storytelling black feminist wondaland 89 journal of effective teaching in higher education, vol. 4, no. 2 baker-bell (2017) described black feminist-womanist storytelling as a methodology located at the intersections of autoethnography, black women’s language and literacy practices, black feminist and womanist theories, and storytelling. she argued that the theoretical and methodological combination provides black women with a means to collect, write, analyze, and theorize our stories while we simultaneously heal from them. in centering autoethnography, black feminist/womanist storytelling allows us to center our personal experiences, note our relationships with others, engage in self-reflection, and balance our academic lives alongside our emotional and creative existences. by grounding itself in black feminist and womanist theories, the methodology makes space for us to consider the wholeness of our identities and center ourselves as people worthy of deep study. through the upliftment of black literacy traditions, black feminist/womanist storytelling gives us permission to rely upon our ancestor’s ways of knowing and being. it allows us to consider story as a vehicle to transmit our knowledges, our experiences, our struggles, and our triumphs to the world in hopes that these narratives promote self and communal healing as well as societal change. we call upon black feminist-womanist storytelling because we wish to highlight our stories as “sources of legitimate knowledge” (haddix, 2016, p. 22). we wish to write ourselves free (carey, 2016). rather than relying upon a conceptual focus or centering an analysis of external data, we center ourselves as black women currently located within higher education. we use storytelling to highlight how we have reckoned with misogynoir in higher education and center how we have learned to celebrate ourselves and reclaim our joy in spite of it. we use our stories as monae used her album: to provide clear access to our lives without pretense, costume, or citation. we follow monae’s poetic and narrative style as embodiment and example of the methodological work we foreground in this article. through story and song, we ask readers to contend with us, as we are, to relive our horrors, but more importantly, imagine a new higher education alongside us. in putting our stories in conversation, we bear witness to each other’s experiences, but we also provide a space for readers to see us, to reckon with the violence we have experienced, and to listen as we celebrate and reclaim our wondaland in higher education. our stories, our songs elizabeth’s song i reckon with my place in academia. i recognize at this point, it is part of my preparation for my time out in the world. i never wanted much to do with higher education. black women learn from a young age that education is the one true currency that will take us to the highest heights of our hopes and dreams. i resented that view. i wanted to do things i enjoyed, and i wanted to do them my way. i wanted to show black children the lives they wanted to live were at their black feminist wondaland 90 journal of effective teaching in higher education, vol. 4, no. 2 fingertips, and even if they couldn’t reach them, they had someone, at least one person, that believed in them. through twists and turns, i found myself in graduate school. after my time at an hbcu, moving to a pwi was a challenge. luckily, my master’s degree program was full of black teachers who brought invaluable field experience to our lessons. they allowed and encouraged a frankness i had been longing for. and of course, white teachers that mean well, but are truly without empathy for your passion, your purpose, your mission. the professor that worked in policy assured me the school-to-prison pipeline was more of a “trickle,” a series of unfortunate events no one could really attribute to schools. the professor that dismissed my work on black women and girls as just “fun” work that wouldn’t have “real value” in a policy world. they thought they were helping. “but all i ever really felt was stressed out, kind of like my afro when it’s pressed out.” i celebrate my place in academia. i recognize that to get to this point, i acknowledge the parts of me that i yield to it, and those i hold dear. i cried alone on a toronto subway, after participants at my first academic conference were content to overlook me during a research presentation. i had spent so long bending my work to those that told me they meant me well. i went to them for support during this lonely time and found dismissal. those tears were cleansing. they reminded me how i had gotten to where i was, and why i needed to stay there. why i needed to stay true to who i am, and not to the person they wanted me to be. black children still needed to know someone believes in them, and still believes in them. my charge wasn’t to those that would simply evaluate my work, but to the community whose lives could be changed by my contributions for years to come. i began to study and celebrate the work of black authors even more. ask more questions for more uncomfortable answers. i sought mentors that saw the value not just in my work, but in my existence as a person. i remembered my reason to celebrate. “i’m always left of center, and that’s right where i belong.” i reclaim my place in academia. they can prepare me, but they cannot undo me. in knowing black children deserve so much more than they are often given, i remind myself that i am a black child. i have an opportunity to make a mark with black feminist wondaland 91 journal of effective teaching in higher education, vol. 4, no. 2 my work. conforming to those that would seek to have me silenced, is not only a detriment to myself, but a disservice to the world that needs to hear more of what black women have to say. we don’t deserve to be told our work does not have real value. we don’t deserve to have our lived experiences questioned. we don’t deserve to have our words and ideas taken and credited to those that would see us under their feet before they would have us at the heads of their tables. we deserve to be the loudest ones in the room. for that volume to be from the sounds of our joy rather than our pain. we deserve to share our stories. to a world that may not deserve them, but that needs them. we deserve to empower not just each other, but generations of black thought leaders that will grow and develop with and after us, and for that empowerment to create a world of black joy that is incomparable. we are here to reclaim our peace, and our time. “love me, baby. love me for who i am.” stephanie’s story “i’m not that special, i’m broke inside, crashing slowly, the bugs are in me.” she froze. she’s done this before, every time that word spewed from a mouth it didn’t belong to. her face is still, poised so as not to alarm the person who embodies dominance through age, gender, position, race. on the inside, there’s an earthquake, a sudden violent shaking at the core of her being destroying her soul. although she’s breaking down internally, she knows her place. if she shouts in protest or rolls her eyes, the stereotypes will activate. no one will care about the why behind her reaction. they will only see black and woman and outrage. they won’t see how she crashes slowly toward despair, for they don’t wish to see. they dream her as nightmare. as he repeatedly says, “n****r,” not at her, but to her in the midst of conversation, she sits silently. every n a bullet. every hard -er a lingering wound. still, she sits, hoping she’ll be able to cauterize the wound with thoughts of black love and black dreams. this time, it doesn’t work, but she continues to try. she must. she knows how he sees her. this was clear when he said, “i don’t know what you do, but you’re publishing so it must be good work.” he had the opportunity to know her, but he chose not to. she has reckoned with this, with the knowledge that he has erased her personal history and replaced it with his constructed image. so, he continues to speak, and she continues to wilt. she is crashing slowly as he plants the “bugs” he believes already exist within her. black feminist wondaland 92 journal of effective teaching in higher education, vol. 4, no. 2 “now go on girl and use that sauce...if you don’t then that’s your loss.” she sits in an auditorium, waiting for the speakers to begin. in time, two black women and one black man take the stage. their voices sing a collective song of black celebration, as they highlight the knowledges of black and brown youth across the diaspora. as they sing, the room changes colors, vibrant hues of amber, coral, rose, and sapphire. joy resounds from their voices as they uplift and celebrate black life. they sing, “our stories matter,” “save our children,” and “center black life.” they lead the auditorium in appreciation and adoration of black existence. she hears their call and wonders how she can celebrate herself. her wounds haven’t healed. she presumes the presenters are still nursing wounds inflicted upon them by the academy. and yet, they sing. they celebrate black people and black culture in harmonizing melody, and although the song is meant for all to hear, it falls gracefully on her ears. she rises, joins the chorus and sings. her song, although it contains different words, adds to the cacophony of sound, the melodic discord of black praisesong. she sings, “my research is enough,” “black girls are enough,” “i am enough.” in this instance, she rejects society’s positioning of her and refuses racist, stereotypical images. she celebrates herself, her work, her existence. she defines herself on her terms, and smiles. “let’s introduce ourselves from a free point of view.” over time, she’s heard the joyous songs of black students, faculty, and staff who face oppression together. they refuse inferiority and reclaim the academic space their forebears fought for. they’re not alone. she’s not alone. she never was. this awareness of a communal and ancestral spirit allows her to persevere. but what does it mean to persevere in a place that refuses her? what does it mean to reclaim herself in the fabric of academic life? she thinks about these questions, but she can’t respond. the answers elude her. they will continue to evade her awareness until she realizes reclamation is formed through community. it’s upon this realization that she decides to hold tight to the black people who raised her, the black women who nurtured her, the black ancestors who paved a way for her. they aren’t welcomed in academia, for they are too country, too poor, too black. and yet, she must find a way to block the noise of dominance and salvage the resounding call of black joy and black life. she doesn’t accept her fate or submit to the call of dominance. she introduces herself freely and says her name, stephanie. she is no longer frozen. black feminist wondaland as resistance monae acknowledged how attending to present social issues, celebrating in spite of oppression, and reclaiming joy can be a space of vulnerability, especially for black women (wortham, 2018). similarly, we acknowledge that sharing our stories was vulnerable, painful work. still, we called upon bfw to excavate our personal experiences in higher education to help educators think about ways to cultivate black feminist wondaland 93 journal of effective teaching in higher education, vol. 4, no. 2 academic spaces where black women can reckon with how we are positioned in society, while simultaneously celebrating ourselves and reclaiming our own stories. our stories suggest that even though we both experienced pain in higher education, we found celebration and joy. as love (2019) argued, joy is resistance, and it comes from releasing pain, generating art, and loving the self and others. bfw allows us to do this work—to fight against tension, surrender our pain, tell our stories, and love ourselves. as we consider the elusive project of black female mattering, we look to our stories as learning tools. from our narratives, we see the need for leaders, professors, staff, and all who interact with black women in higher education to engage in the following: (1) center black women and black life unapologetically, without fear; (2) celebrate black women not because we exceed your expectations, but because we exist; (3) trust the pain, experiences, and words of black women, not just when social unrest requires a statement, but at all times. higher education stakeholders must reckon with black women’s social and academic experiences; celebrate black women, those who are in the classroom and those who are not; and find ways to help black women reclaim space in academia. as these recommendations are not field-specific, we offer several questions for educators to consider: 1. how are black women represented across your syllabus? are we relegated to one month or unit or are we included throughout? do the stories, articles, and books on the reading list showcase black women’s celebration and reclamation alongside oppression, pain, and/or struggle? 2. how do you celebrate black women in your classroom and in your everyday life? how do you make space for black women to be seen and heard? 3. what does your course content and your academic field make possible and impossible for the black women in your classrooms? 4. how are you ensuring that black women do not have to fight misogynoir alone? asking these questions is paramount. numerous scholars have identified ways to support black women in higher education, including adequate mentorship, safe and supportive university environments, peer connections, and proactive coping mechanisms (bertrand jones et al., 2013; bertrand jones et al., 2015; dortch, 2016; hotchkins, 2017). still, creating higher education spaces where black women can live empowered and fulfilled existences requires those within higher education to be cognizant of black women’s liminal academic lives, as we struggle to survive in a world that means us harm (cooper, 2020). creating spaces where black women’s lives matter requires an understanding of how we celebrate ourselves and reclaim joy in the midst of the pain. it requires higher education stakeholders to recognize the full spectrum of existence we bring with us into academia. as black women, we reckon with how we are positioned in society and in higher education. we see how the aesthetics of our joy are cast aside. we know what pain black feminist wondaland 94 journal of effective teaching in higher education, vol. 4, no. 2 and violence look like, but that is not the whole of our lives. our celebration and reclamation find themselves in lifting our voices, sharing our joys, and resting in community. they find themselves in classrooms where we do not have to consistently defend our existence. they find themselves in professors and teachers who allow us the space to be loud, to speak back, to sing praisesongs. this may be difficult and possibly uncomfortable for some because misogynoir runs rampant; however, if we are to mitigate the suffering of black women in higher education, black freedom, black joy, and black wondalands must be a top priority. conclusion we use wondaland in homage to monae’s record label, television and film production company, brand consulting firm, and activist hub (ringen, 2018). we chose this term rather than a word tied to the album to center the broadness of monae’s ideas. the album provides the basis for our theorizing, but wondaland is more than one album or one artist. it is a way to showcase how this collective movement starts with monae’s work and ends in the “next time” (baldwin, 1963, p. 89), in the future realm of possibility within and beyond the academy. in the same way, we hope our stories present examples of the broadness of black women’s higher education experiences even as we acknowledge that there are many more stories to uncover. it is our hope that in the “next time,” celebration and reclamation are a given, that our joy matters in these spaces, and that higher education sees us, not as america’s nightmare, but as america’s dream. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references baker-bell, a. (2017). for loretta: a black woman literacy scholar’s journey to prioritizing self-preservation and black feminist-womanist storytelling. journal of literacy research, 49(4), 526–543. https://doi.org/10.1177/1086296x17733092 baldwin, j. (1963). the fire next time. penguin books. bailey, m., & trudy. (2018). on misogynoir: citation, erasure, and plagiarism. feminist media studies, 18(4), 762–768. barr, m. (2008). afro-future females: black writers chart science fiction’s newest new-wave trajectory. the ohio state university press. baxley, t. p., & boston, g. h. (2010). classroom inequity and the literacy experiences of black adolescent girls. in j. zajda (ed.), globalization, education, and social justice (pp. 145–159). springer. https://doi.10.1007/978-90-481-3221-8_10 black feminist wondaland 95 journal of effective teaching in higher education, vol. 4, no. 2 bertrand jones, t., wilder, j., osborne-lampkin, l. (2013). employing a black feminist approach to doctoral advising: preparing black women for the professoriate. journal of negro education, 82(3), 326–338. https://doi.org/10.7709/jnegroeducation.82.3.0326 bertrand jones, t., wilder, j., patterson, s., & davis, d.j. (2015). creating a “safe and supportive environment”: mentoring and professional development for recent black women doctoral graduates. international journal of doctoral studies, 10, 483–499. https://doi.org/10.28945/2305 carey, t. (2016). rhetorical healing: the reeducation of contemporary black womanhood. state university of new york press. carter, s. p. (2006). “she would’ve still made that face expression”: the use of multiple literacies by two african american young women. theory into practice, 45(4), 352–358. clay, a. (2008). “like an old soul record”: black feminism, queer sexuality, and the hip-hop generation. meridians, 8(1), 53–73. collins, p. h. (2009). black feminist thought: knowledge, consciousness, and the politics of empowerment (2nd ed.). routledge. collins, p. h. (2016). black feminist thought as oppositional knowledge. departures in critical qualitative research, 5(3), 133–134. combahee river collective. (1977). the combahee river collective statement. https://www.blackpast.org/african-american-history/combahee-rivercollective-statement-1977/ cooper, b. c. (2018). eloquent rage: a black feminist discovers her superpowers. st. martin’s press. cooper, t. p. (2020). exploring the voices, experiences, and leadership career paths of women of color in higher education. [doctoral dissertation, abilene christian university]. https://digitalcommons.acu.edu/cgi/viewcontent.cgi?article=1290&context=e td dortch, d. (2016). the strength from within: a phenomenological study examining the academic self-efficacy of african american women in doctoral studies. the journal of negro education, 85(3), 350–364. https://doi.org/10.7709/jnegroeducation.85.3.0350 durham, a., cooper, b. c., & morris, s. m. (2013). the stage hip-hop feminism built: a new directions essay. signs: journal of women in culture and society, 38(3), 721–737. https://doi.org/10.1086/668843 black feminist wondaland 96 journal of effective teaching in higher education, vol. 4, no. 2 english, d. k., & kim, a. (2013). now we want our funk cut: janelle monáe’s neoafrofuturism. american studies, 52(4), 217–230. https://doi.org/10.1353/ams.2013.0116 eshun, k. (2003). further considerations on afrofuturism. the new centennial review, 3(2), 287–302. http://doi.org/10.1353/ncr.2003.0021 evans-winters, v., & esposito, j. (2010). other people’s daughters: critical race feminism and black girls’ education. journal of educational foundations, 24(1/2), 11–24. fordham, s. (1988). racelessness as a factor in black students’ school success: pragmatic strategy or pyrrhic victory? harvard educational review, 58(1), 54–84. https://doi.org/10.17763/haer.58.1.c5r77323145r7831 fordham, s. (1993). "those loud black girls": (black) women, silence, and gender “passing” in the academy. anthropology & education quarterly, 24(1), 3–32. george, j. a. (2015). stereotype and school pushout: race, gender, and discipline disparities. arkansas law review, 68(1), 101–129. haddix, m. (2016). cultivating racial and linguistic diversity in literacy teacher education: teachers like me. routledge. hooks, bell. (2015). sisters of the yam: black women and self-recovery (3rd ed.). routledge. hotchkins, b. (2017). black woman students at predominantly white universities: narratives of identity politics, well-being, and leadership mobility. naspa journal about women in higher education, 10(2), 144–155. https://doi.org/10.1080/19407882.2017.1326943 kelley, r. d. (2002). freedom dreams: the black radical imagination. beacon press. lewis, m. m., & miller, s. j. (2018). how does it feel to be a problem? a conversation between two feminist black queer femme chairs. feminist formations, 30(3), 79–90. love, b. (2019). we want to do more than survive: abolitionist teaching and the pursuit of education freedom. beacon press. morris, e. (2007). “ladies” or “loudies”?: perceptions and experiences of black girls in classrooms. youth and society, 38(4), 490–515. https://doi.org/10.1177/0044118x06296778 black feminist wondaland 97 journal of effective teaching in higher education, vol. 4, no. 2 morris, s. (2016). more than human: black feminisms of the future in jewelle gomez’s the gilda stories. the black scholar, 46(2), 33–45. https://doi.org/10.1080/00064246.2016.1147991 neal-jackson, a. (2020). “well, what did you expect?”: black women facing stereotype threat in collaborative academic spaces at a predominantly white institution. journal of college student development, 61(3), 317–332. https://doi.org/10.1353/csd.2020.0030 onyeka-crawford, a., patrick, k., & chaudhry, n. (2017). let her learn: stopping school pushout for girls of color. national women’s law center. https://nwlc.org/let-her-learn/ patton, l., & croom, n. (2017). critical perspectives on undergraduate black women (pp. 1–14). in l. patron & n. croom (eds.), critical perspectives on black women and college success. routledge. ringen, j. (2018, november 19). how singer-songwriter, actress-activist janelle monae gets so much done. fast company. https://www.fastcompany.com/90263428/how-singer-songwriter-actressactivist-janelle-monae-gets-so-much-done robinson, j. m., irvin, n., webb jr., j. m., joseph, c., parks, t., dean, j., irvin, r.g., tuffuor, n. k., brown, p. l., murray, r., parker j. k., wade, r., webb, j. m., & williams, p. (2018). dirty computer. wondaland. setaro, s. (2018, october 29). reckoning, celebration, and reclamation: janelle monae dissects dirty computer. complex. https://www.complex.com/music/2018/10/janelle-monae-dissects-dirtycomputer smith-evans, l., & george, l. (2014). unlocking opportunity for african american girls: a call to action for educational equity. naacp legal defense & educational fund. https://www.nwlc.org/sites/default/files/pdfs/unlocking_opportunity_for_afric an_american_girls_final.pdf williams, q., williams, b. m., & brown, l. c. (2020). exploring black girl magic: identity development of black first-gen college women. journal of diversity in higher education. advance online publication. https://doi.org/10.1037/dhe0000294 womack, y. (2013). afrofuturism: the world of black sci-fi and fantasy culture. chicago review press. black feminist wondaland 98 journal of effective teaching in higher education, vol. 4, no. 2 wortham, j. (2018, april 19). how janell monae found her voice. the new york times magazine. https://www.nytimes.com/2018/04/19/magazine/howjanelle-monae-found-her-voice.html yaszek, l. (2006). afrofuturism, science fiction, and the history of the future. socialism and democracy, 20(3), 41–60. https://doi.org/10.1080/08854300600950236 microsoft word virtue_final.docx journal of effective teaching in higher education, vol. 2, no. 2 using fiction to support identity development and transition in conditionally-enrolled students emily e. virtue, western carolina university, evirtue@wcu.edu abstract. the transition to college can be difficult for many first-year students. students who are conditionally enrolled may struggle more than their peers in terms of feeling comfortable in an academic setting. this case study explores how conditionally-enrolled students who participated in a summer bridge program read popular literature to explore their own struggles in the transition to college. findings call for faculty and staff who work with conditionally-enrolled students to support and accept identity exploration in the classroom, particularly during the first semester. keywords: identity, first-year transition, fiction, faculty as students transition to college, faculty and staff interact with young adults who struggle to understand themselves and their place in the world (specifically on their new campus). for over fifty years, researchers have noted this challenge is heightened for students who are labeled high-risk or at-risk based on previous academic achievement, first-generation status, and other social factors (blythe, darabi, simon kirkwood, & baden, 2009; egerton, 1968; gordon, 1975; kulik, kulik, & shwalb, 1983; lowe & cook; 2010; tinto & sherman, 1975). because students often bring evidence of their development into the classroom setting, it is crucial to foster an environment that allows identity development to be explored. students labeled “high-risk” are more likely to achieve when they see examples of successful students who share identity and personality traits similar to their own (bandura, 1977; johnson, taasoobshirazi, kestler, & cordova, 2015; stephens, hamedani, & destin, 2014). yet, there are not often opportunities for students to see exemplars in their first-year classroom settings. this study examined whether fiction about college students in transition can foster positive views of identity in conditionally-enrolled students. the research questions that guided this study were as follows: 1. how can fiction that is thematically focused on identity and college-aged student transitions aid in identity development and transition? 2. how can reading and writing assignments in a first-year course for conditionally-enrolled (often labeled high-risk) students help create a positive academic identity and ease the transition process? background and framework development in the college context while student development and identity have been studied, little is known about student identity as it relates to conditionally-enrolled students in their first using fiction 71 journal of effective teaching in higher education, vol. 2, no. 2 semester of study. arnett’s (1998, 2011) emerging adult theory argues that during the ages of 18-25 most people experience the freedom to explore the possibilities of adulthood and that the scope of this exploration is quite large. emerging adulthood brings with it greater opportunity for identity exploration than any other time in a person’s life. if students are given an environment open to exploration, they are more likely to make meaning of and conceptualize their identity. issues of alienation complicate the identity process. high-risk students often struggle to develop a personal identity that seems to fit into the world of higher education (mann, 2008). to combat these issues, mann (2001) argues that instructors must avoid passive teaching approaches (such as surface learning or rote memorization) and create deep learning experiences for students to truly engage with and be shaped by the content. mann (2001) also argues for the use of hospitality and safety in the classroom. she asserts that students will feel more secure in their academic identity if they are welcomed into the classroom and if they feel safe to explore their thoughts without fear of criticism or retribution. in this study, my pedagogical choices were informed by mann’s (2001, 2008) directives, particularly with respect to providing deep learning experiences and creating a welcoming environment in which students feel safe. practices such as interactive workshops, creative and visual projects, and class discussions also support tapp’s (2014) call for participatory pedagogy. a student’s sense of belonging on campus is impacted by their interactions with community members (faculty, staff, and students) at an institution. strayhorn (2012) defines the sense of belonging in college as follows: students’ perceived social support on campus, a feeling or sensation of connectedness, the experience of mattering or feeling cared about, accepted, respected, valued by, and important to the group (e.g. campus community) or others on campus (e.g., faculty, peers). (p. 3) additionally, strayhorn (2012) argues that students’ sense of belonging is linked to their social identities. writing, literature, and identity while some research exists on the relationship between writing and identity development, there is scant research on identity development related to reading fiction. alder (2016) noted there is a strong connection between learning and identity formation. both alder (2016) and george (2012) agree that identity formation is difficult because it is multiple, ever developing, and renegotiating. personal connection with the material and being given a space to think independently is vital to identity development. these goals can be accomplished through the everyday practices of being in an english class (alder, 2016). the ability for self-reflection and the capacity to understand oneself in a particular environment can be partially determined by the involvement of the instructor. kill (2006) provided evidence that the work of renegotiating classroom identity lies on the shoulders of the instructor just as much as it does the student. scanlon, rowling, and weber (2007) noted that students desire a deep connection with their using fiction 72 journal of effective teaching in higher education, vol. 2, no. 2 instructor, similar to the connections present in secondary education, but that often instructors are unavailable. scanlon et al. (2007) argued that a lack of support creates a constraint on student ability to move from knowledge about being a student at the university level to knowledge of being a student [author emphasis]. further, they argued that situated interaction (scanlon et al., 2007, p.237) is key to identity formation. likewise, lave and wenger (1991) posited that the phenomenon of learning is directly linked to identity; that is to say, identity changes as learning happens. yet another approach to understanding identity can be seen in the novel method (king, 2013). for the novel method to work, the student reader must be acutely aware of the chosen text. such awareness will allow the student to develop a framework for understanding their identity through characters, situations, and metaphors in the novel (king, 2013). this exercise, while useful, is not often used in literature classes where the focus is on the content, not the reader or student-aswriter. in the current study, students undertook the novel approach to writing about their own identity. the framework for this study utilizes king’s (2013) novel approach, mann’s (2001, 2008) directives, and tapp’s (2014) work on participatory pedagogy to guide an introductory seminar in literature aimed at assisting student identity development. daily class assignments were constructed with the intent to create a space for students to consider their identity and how it might change in their transition to college. their final formal assignment (discussed later in the article) directed the students to use the novel method (king, 2013) to explore their identity via the lives, situations, and circumstances of fictional characters. this framework put an emphasis on self-exploration and understanding of identity development while fostering an environment that allows new college students to be active participants in their learning. methods this case study focuses on the early transition of conditionally-enrolled students during a summer bridge program. the stories of their lives, explored and explained via works of fiction, highlight their reality as new students. such data may give instructors and student support staff a clearer understanding of the transition process that occurs in a condensed time-period common in bridge programs. context this study was conducted at a regional university with conditionally-enrolled students. conditionally-enrolled students must participate in the bridge program (bp). these students have an average act score of 18 and an unweighted high school gpa of 2.7. such numbers traditionally indicate that student academic success will be limited, labeling them as “high-risk” students. bp students are cohorted (20-25 students) and take three classes together in a five-week long summer program. the learning community structure allows for some content overlap between the three classes. students in this study were enrolled in a using fiction 73 journal of effective teaching in higher education, vol. 2, no. 2 learning community that consisted of a literature seminar (the subject of this article), introductory psychology, and a university transition course. students read various texts throughout the summer session but focused most of their time and attention on the novel fangirl (rowell, 2013), a fictional account of one student’s first year at a large, public university. this text was chosen because of its realistic depiction of the characters’ college transition processes, specifically how the main character struggled to understand herself and her identity in the college environment. after reading the texts, students explored their identities as it related to the characters and circumstances in the novel or other course readings. participant selection i recruited participants from two bp cohorts enrolled in my english literature seminar. students enrolled in the course were given the choice to opt out of the data collection though they remained in the course. the sample size for the document analysis portion was 41 (18 male, 23 female). in the sample, 33 students identified as white, 3 as black, 1 as native american, 2 as multiracial, and 2 as latinx. the sample, while convenient, was also purposeful in that it “highlight[s] what is typical, normal, and average” (patton, 2015, p. 268) in the bp population. all participants were asked to complete a follow-up interview the following fall semester and four participants (one male, three female) agreed. the demographics above help to describe the typical makeup of an bp cohort; however, i do not wish to suggest that demographics can clearly inform an understanding of the group. the participants’ lived experiences, social identities, and personal circumstances affected their identity development. data collection the data collection began with attribute coding (saldaña, 2009) for participants. participants were given pseudonyms. additional demographic information was collected on the participants based on university records from the registrar’s office: age, hometown, intended major, ethnicity, and gender. written responses to two prompts were downloaded from blackboard and re-identified with student pseudonyms. assignment one asked students to “describe how you identify yourself as a student and a person. what drives you? how do you make choices for yourself? what is most important to you? give specific examples from your life and make sure you fully explain your answer.” assignment two was completed during the last week of the semester: throughout the course of the semester we’ve read a number of pieces that have featured college-aged students and their attempts to understand themselves and their place in the world. choose a character with which you most identify—that is not to say this character is an accurate reflection of you but that there is something in the character that you can identify with (for example you may understand their demeanor, their background, their thought process, etc.). use examples from both the text and your own life to using fiction 74 journal of effective teaching in higher education, vol. 2, no. 2 describe who you are and how literature helps you discover this identity. you may choose to discuss multiple characters from the same text or use characters from different texts. however, if you use more than one example, you must be detailed and specific in each example. you must specifically reference from the text to demonstrate your understanding. data analysis during document analysis, i looked for trends in student interpretation of their identity. bowen (2009) explains that “documents provide a means of tracking change and development” (p. 31). by using the documents as the primary data source, i ascertained whether student identity development shifts occurred. thematic analysis protocol (bowen, 2009; fereday & muir-cochrane, 2006) was used to identify patterns in the student writing and interviews. writing analysis documents were coded by theme based on prompt. prior to the first reading of the documents, i identified four descriptive code themes i thought would be apparent in most of the responses: students’ belief in their intelligence or their ability to be a good student; being hardworking; being unfocused on school work or not seeing it as a top priority; and their confidence level related to identity. identifying themes prior to reading the documents, or a priori, can enable an analysis that directly answers the research question (saldaña, 2016). this approach allowed me to focus on the goal of answering the research questions: 1) how can fiction that is thematically focused on identity and college-aged student transitions aid in identity development and transition? and 2) how can reading and writing assignments in a first-year course for conditionally-enrolled (often labeled high-risk) students help create a positive academic identity and ease the transition process? as new themes presented themselves, they were recorded. analytic memos were written after each coding session. during the third (and final) coding phase, themes were further delineated by additional sub-code grouping and by applying magnitude coding. interviews after the interviews were completed, i transcribed each one. copies of the transcripts were emailed to the participants for their review. after corrections were made, transcripts were coded. i began with a priori coding because of the nature of the study. saldaña (2016) explained that because the concept of “identity” is so varied across disciplines and contexts, it is “most likely essential” to pre-establish codes (p. 72). in the first round, i looked for codes related to family influence on identity, perception of self as student, and confidence, to which i applied magnitude coding in order to “indicate intensity, frequency, direction” (saldaña, 2016, p. 86). during the first round, i made note of emergent codes and applied magnitude notations to these as well. in the second round of coding, i used pattern coding to help develop themes from the emergent codes. using fiction 75 journal of effective teaching in higher education, vol. 2, no. 2 trustworthiness as both the course instructor and the researcher, i used several strategies to increase trustworthiness. first, i obtained irb approval from the study site. use of multiple data points allowed me to corroborate findings to draw conclusions (bowen, 2009; merriam & tisdell, 2016). my role as instructor allowed me to fully understand the context of the classroom and aspects of the bp program that other researchers might not benefit from. four months passed between the completion of the class and the analysis of data. this lapse in time allowed for the analysis to be a distinct process from the initial grading that took place during the semester. findings five major categories presented themselves in analysis of the documents and interviews: the relationship between the literature content and “real life,” the importance of family and friendship, the prevalence of shyness and/or “social awkwardness,” the effect of college stereotypes, and the multiplicity of identity. reality reflected in writing many of the participants identified with the characters they encountered in literature. the characters enter college for the first time, and, therefore, much of what they endure was also experienced by the students enrolled in the course. while the accounts the students read were fictional, the literature helped students realize their feelings were not uncommon. students pointed to the relief they felt knowing that their struggles were not unique. linda stated: it is also nice to know that i can identify myself with someone else and that other girls experience the same feelings and struggles that i have or will experience. fangirl allows me to read and feel what they are going through as if it were my emotions. it is nice because some of the things that they go through, i can look back on my life, and i can see where i’ve dealt with a similar situation. […] fangirl gives you the opportunity to see the experiences and lessons that have been contributed and learned to the lives of wren and cath, and this allows me to understand and feel the book on a personal level. the comfort students found in reading about struggles similar to their own allowed students to understand that transition is hard for many, and, despite their strife, they can be successful college students. william echoed this sentiment when describing his reaction to the required reading: “in a way, you feel as if you’re not alone and that there’s someone else out there going through the same struggle as you.” students may worry that they are the only one having trouble with their transition as people are not prone to sharing their perceived weaknesses, yet william found relief in knowing he was not alone. seeing characters experience similar moments of hardship demonstrated that not only were the students not alone, they also saw characters change and ultimately experience the positive outcomes of being a college student. using fiction 76 journal of effective teaching in higher education, vol. 2, no. 2 the relief many students experienced also gave them confidence to seek out help and to persevere through the challenging aspects of their college transition, which echoes arnett’s (2011) criteria of emerging adulthood: accepting responsibility for one’s actions and making independent decisions. the students believed their success was possible because the characters were also successful. the various ways in which students used characters and situations to explain their identity as students demonstrates that using literature as a tool for self-exploration can be effective. further, it points to the value in using king’s (2013) novel approach to using a primary source as means to understand oneself. students clearly articulated the connection between their lives and course readings in cogent, distinct ways. these findings support alder’s (2016) assertions that identity development can take place in english assignments and lave and wenger’s (1991) findings that learning and identity formation are linked. family relationships many students noted both at the beginning of the term and at the end that the importance of family was key to their identity. students explained that family members gave them strength, encouragement, and increased self-esteem. interestingly, in nearly equal measure, a second group of students identified negative family relationships as markers of their identity. in both positive and negative relationships, students pointed to the influence family members had on their belief in their ability to succeed in college. chelsea highlights the impact family encouragement played on her ability to transition to college: when i first moved to college, i felt very alone and homesick like i had no one at all, i didn’t know anyone here so that made it worse. my family went out of their way to facetime me every day, send me care packages in the mail, and even surprise me with visits to make me feel loved and as if they were still there for me even if they were not physically with me every moment of the day. the encouragement chelsea received from her family helped her get through the transition in a manageable way. though she felt lonely at times, she knew she was not alone. she remarked that her family interactions were what drove her to succeed and they were the reason she “gives her all” at school. these findings reflect vygotsky’s (1978) theory of sociocultural cognitive development that suggests that development is advanced through social interaction. in chelsea’s example, her family’s scaffolding (vygotsky, 1978), or temporary support, gave her the confidence to engage in the new college setting while still feeling supported by those at home. while chelsea and many others found encouragement and strength to succeed in their family interactions, other students found strength through unsupportive or poor relationships. these relationships reminded the students of their goals and, in some cases, students worked to prove naysayers wrong. eric explains, “my parents have always told me that i don’t have the drive or willpower to succeed, and ever since high school, i have been trying to prove them wrong.” for eric and others, the negative feedback and disbelief in their abilities proved to be a form of using fiction 77 journal of effective teaching in higher education, vol. 2, no. 2 encouragement. rather than meet family expectations, the students chose to rise above adversity and believe in themselves when those closest to them did not. awkwardness abounds transitions are often difficult and as students struggle to adjust, discomfort can reign. for many, discomfort manifests in the form of shyness or feeling intensely awkward in social situations. yet, as students struggled with their awkwardness, they found comfort in the fact that others felt the same way. nearly two-thirds of the students highlighted their affinity for the character cather (cath) from fangirl who is particularly shy and struggles to connect to her peers. cath is an introvert and has few friends early in her college transition. many students explained that their own situation mirrored cath’s, even the students who did not identify as introverted. sarah describes her interactions with others as “painfully awkward” and “cringe-worthy” because she doesn’t often engage in social interactions. sarah’s ability to equate her general state of shyness and social awkwardness to that of a character gave her a bit of comfort because she knew what the character was going through. many students discussed being unprepared for college life in general and for social interactions in particular. tiffany explains that social situations in college were stressful: “[cath’s] fear of judgement in social situations is something i can definitely relate to […] cath and i both fear venturing outside of our comfort zones, making natural college stress multiply.” while learning a new environment and meeting new people are natural stressors for students in transition, the level of preoccupation students had with their awkwardness in social situations is telling. this may be due to less frequent genuine face-to-face contact (kim, 2017). it is encouraging that students are aware of their need to increase social skills as this indicates their self-awareness and desire for interaction. however, faculty may want to consider encouraging more face-to-face dialogue and social interactions to give the students practice with interacting with others and becoming adept at reading and understanding others. participatory pedagogy (tapp, 2014) would provide a space for students to interact one-on-one and may alleviate some of the social fears like those experienced by my participants. in this study, students noted that the multiple activities in class helped them connect with people and talk with them when otherwise they would not have interacted. the effects of college stereotypes in class discussions, interviews, and their writing, many students confirmed that the expectations of college are largely built on what they’ve seen in popular culture. they believed that nearly everyone would drink (and to do otherwise would be frowned upon), that college is a non-stop party, and that academics barely fit into their perceived image of college. brittney explains: i have my heart set on getting that infamous college experience—the one from the movies […] while wren is feeling the need to experiment, i do too, however, not on wren’s level […] my version of experimenting runs along the using fiction 78 journal of effective teaching in higher education, vol. 2, no. 2 lines of going to parties with my friends. i like to dress up and have a good time but still be safe. here, brittney highlights what she has been told about college. the fact that her impressions have nothing to do with academics stands out. her concern is only with her social life: academics appear as an afterthought in her summary of what college should be. similarly, students note that independence and freedom are a key component of college life. william wrote about the importance of freedom, “when i got here, i did my work and i still do it, but i went to parties and i turned up a lot because my mom isn’t here and i can do what i want.” yet, william also recognizes the limits of such an experience, “but in all reality, you don’t want to become the life of the party because it can bring you down without you even knowing it. you can see that in wren.” william identified his practices as being similar to those of characters he read about and applied the lessons to his own life. he recognized that many (including him) view college as an opportunity to party and experience freedom while at the same time noted how those practices were not ideal conditions for success. interestingly, students expressed how even parents fall into the belief that college is supposed to be like the movies. robert explains, “my parents told me that i needed to not be such a good kid and go out and party and have fun (as long as i passed my classes).” it is evident that popular culture influences both student and family expectations of college, and when those expectations are not grounded in academics, concerns about social opportunities can have significant weight. further, how students understood their own identity was built on the stereotypes they were familiar with—even with the acknowledgement that such stereotypes were not necessarily positive. the influence of stereotypes of college on students played a significant role in the choices students made when they arrived. many, however, learned from their mistakes just as the characters they read about did. michelle discussed the realities and downfalls of the college experience: she [wren] turns to alternative methods of coping such as partying every weekend and participating more in her social life than in her education. i can relate to this as well because i have been very involved in the social aspects of life these last five weeks. in the end, wren eventually realizes that she needs to focus on more important things like getting an education. the past week has really opened my eyes to the fact that i need to step it up if i want to remain a student at this university. i need to focus on school even if i miss out on the social parts of being a college student. after reading about the fictional character’s situation, michelle could articulate how her own expectations of college life had negatively impacted her. while some (like brittney, william, and michelle) looked forward to the “college experience,” others struggled to find their place in a world that seemed unwelcoming to them. sarah stated, “i know what it is like to be the kid who doesn’t want to make friends, party, and go crazy in college. it separates you from using fiction 79 journal of effective teaching in higher education, vol. 2, no. 2 a lot of people, and rowell depicts this well with cath in the novel.” jennifer confirms sarah’s reaction: how she [cath] was like in her room and she never wanted to go out and everything, i felt like i was going to be like that and so i related to that. and then i saw that i didn't need to be like that when i was like, reading the book. so, it made me realize i need to like go out and make friends and find out who i am and not just be scared. i think with fangirl it made me realize that i was spending too much time in my room and not really making friends because i was scared of not fitting in kind of deal. so, it made me realize, you know, i'm actually pretty okay. […] i mean there's no covering this up. bp was full of like, alcohol, drugs, and partying. everyone is just like, wild. and i was like, this isn't me! […] and then when i met my friend group i was like, oh, these people are a lot like me. they don't party and do drugs and this will be okay. jennifer highlighted that because many of her peers seemed concerned with partying, she felt as though she would not fit in. many students experience similar desires for freedom in their first few weeks on a college campus. this desire might have been heightened in the bp group as their first time on campus came as soon as two weeks after high-school graduation. by asking the students to consider aspects of their identity through fiction, students were provided with an avenue to explore their values and their identity as student. these findings support alder’s (2016) call for identity exploration as well as the importance of providing a safe place students to learn from and question themselves (mann, 2008). multiple identities very few students stated that they only identified as one character they encountered in their readings. while the characters they engaged with were layered and round, students clearly understood that their own identity could not be boiled down to any one aspect or example. michelle explained how she identified with more than one character and how those identities helped her make decisions. referring to the characters cath and wren, she stated: i see myself in both of them. they are two completely opposite individuals, and i feel like i am a mix of the two of them. reading this book was a very great experience, and it helped me see where my priorities should be though my time at [university]. i need to exhibit characteristics of both wren and cath. be social, but do not let it take over. focus mainly on your education, that is what you’re here for after all, but do not let it stop you from enjoying yourself. michelle and others demonstrated that their identity cannot be boiled down to one clear marker and that to do so would be a disservice to themselves. the students in this study began to recognize the layers of their identity and that different aspects of their identities can complement or complicate how they function in their new environment. such observations may suggest that their time spent in class thinking and reading about college student identity allowed them to understand the using fiction 80 journal of effective teaching in higher education, vol. 2, no. 2 multiplicity of identity, an observation supported by both alder (2016) and george (2012). implications the conditionally-enrolled students who participated in this study highlighted various identities that new students bring to their college experience. while their identities are diverse, their understanding of themselves and others in the college transition highlight commonalities that faculty and staff members should be aware of and consider addressing. faculty who teach conditionally-enrolled students should be aware of the various factors that impact student transition and how those transitions may contribute to classroom performance. for example, professors could engage in conversation about the transition process and what students’ expectations of college life are—such discussions should take place beyond the confines of a university studies/transition course. as the students in this study affirmed, the readings and subsequent class discussions allowed the students to explore not only character development but also their own development in a time of stress and transition. in the interviews, students noted how helpful it was to have such conversations in class. these frank conversations allowed for relationship building that can support a sense of belonging in students (strayhorn, 2012). acknowledgement of the emotional toll transition can take on a student may serve to alleviate some of the stress related to academic performance. this supports vygotsky’s (1978) theory that given appropriate support (scaffolding) from the teacher, new students (including those considered high-risk) can thrive in the college setting. though students in this study had an instructor who specifically focused on addressing these issues in class, most students do not experience such conversations in a typical first-semester classroom setting. students may feel more comfortable seeking advice or help from a professor if they know the professor is attuned to what students are going through emotionally and socially while they are trying to prove their worth in their new college environment. the findings in this study confirm scanlon et al.’s (2007) earlier assertions that first-year students seek relationships with their faculty members and look to them for support. while bridge programs for conditionally-enrolled students seek to prepare students for college-level work, failing to acknowledge the social and emotional development needed during this time is a terrible misstep. universities should expect faculty members to prepare students academically while at the same time support social development. administrators may need to revise or reconsider the scope of programs for conditionally-enrolled students by training faculty to have such conversations, even if those topics are tangentially or unrelated to the course material. students in this study affirmed arnett’s (1998) theory that early adulthood presents a great opportunity for identity exploration that is often manifested in one “making independent decisions” and “accepting responsibility for one’s actions” (p. 7). as students explore their new environment, they are bound to also experience moments of hesitation, confusion, and self-doubt. faculty who can provide using fiction 81 journal of effective teaching in higher education, vol. 2, no. 2 environments for exploring these decisions and feelings may build confidence and a sense of belonging in students, particularly with the use of fiction or non-fiction pieces that highlight what is common during a time of transition. this could be done in a number of college courses, such as transition courses and those in the fields of anthropology, communications, political science, psychology, sociology, and writing. finally, as the findings of this study highlight, family perceptions and interactions influence students in their attempts to be successful in their new college environment. not only should family members try to understand their role in student success, they too, might benefit from reading some of the fictional pieces the students read in class. i encourage parents to familiarize themselves with current issues new students face, particularly in relation to the results highlighted in the findings of this study: how family dynamics and college stereotypes influence students’ understanding of college and that students are entering a new phase in life in which they begin to understand themselves as multifaceted individuals. such preparation will allow parents to continue to support students even as they are expected to navigate this new phase of their lives more independently. limitations the majority of students who seemed to understand themselves better because of the literature were white, female students. this is likely because a majority of the characters from readings were also white, female students. while male characters were present in various readings, most were white. fictional representations of people of color in the transition to college are scant. various socio-economic and academic backgrounds were depicted in the required readings; however, racial differences were not prevalent. though students did not address the readings’ lack of diversity in their writing or interviews, it is possible students did not relate to characters because the characters seemed different from themselves. conclusion conditionally-enrolled students often struggle with the college transition and developing a positive student identity (mann, 2001). many of the participants pointed to aspects of a character’s life that echoed their own: struggling to fit in, knowing who they are, and finding friends in a new environment. students found comfort in reading about characters similar to themselves. the identification with successful students (albeit fictional) helped students feel empowered to continue in the bridge program and in college. many students indicated that they learned something about themselves based on the characters’ actions when they too experienced issues related to transition, social situations, or poor life-choices. these findings confirm alder’s (2016) and george’s (2012) previous findings related to identity exploration in the classroom. while this study did not seek to prove causation, it is worth noting that 95% of the participants continued their studies in the fall and 85% were still enrolled a year later (compared to the institutional retention rate of 79.9%). there were likely a number of factors that contributed to the success of these participants; however, using fiction 82 journal of effective teaching in higher education, vol. 2, no. 2 those in the study directly attributed their strong academic identity to their summer experience. use of fiction that focuses on the college transition may encourage students to accept their feelings about their transition and recognize that while transitions are difficult, they can succeed in their new environment. faculty and staff who work with students in transition must accept (and expect) that students will not leave their social selves outside of the walls of the classroom. to expect students to bring solely their academic identity with them to the classroom would be unwise and unsupportive. to prepare for students to work through their transitions in the classroom with the support of faculty and staff may provide the confidence and comfort students need to succeed. conflict of interest the author declares that there is no conflict of interest regarding the publication of this article. references alder, e. (2016). becoming a student of english: students’ experiences of transition into the first year. arts & humanities in higher education, 2016, 1–19. doi:10.1177/1474022216628303 arnett, j. j. (1998). learning to stand alone: the contemporary american transition to adulthood in cultural and historical context. human development, 41(5/6), 295–315. doi:10.1159/000022591 arnett, j. j. (2011). emerging adulthood. in m.e. wilson (ed.), college student development theory, (pp. 149–164). new york, ny: learning solutions. bandura, a. (1977). self-efficacy: toward a unifying theory of behavioral change. psychological review, 84(2), 191–215. http://dx.doi.org/10.1037/0033295x.84.2.191 blythe, s., darabi, r., simon kirkwood, b., & baden, w. (2009). exploring options for students at the boundaries of the “at risk” designation. writing program administration, 33(1/2), 9–28. bowen, g. a. (2009). document analysis as a qualitative research method. qualitative research journal, 9(2), 27–40. doi:10.3316/qrj0902027 egerton, j. (1968). high risk. southern education report. retrieved from https://files.eric.ed.gov/fulltext/ed020985.pdf fereday, j. & muir-cochrane, e. (2006). demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. international journal of qualitative methods, 5(1), 80–92. george, s. (2012). the performed self in college writing: from personal narratives to analytic research essays. pedagogy, 12(2), 319-341. doi:10.1215/15314200-1503613 using fiction 83 journal of effective teaching in higher education, vol. 2, no. 2 gordon, e. w. (1975). opportunity program for the disadvantaged in higher education. eric/higher education report, 6. washington, dc: george washington university. johnson, m.l., taasoobshirazi, g., kestler, j.l., & cordova, j.r. (2015). models and messengers of resilience: a theoretical model of college students' resilience, regulatory strategy use, and academic achievement. educational psychology, 35(7), 869–885. kill, m. (2006). acknowledging the rough edges of resistance: negotiation of identities for first year composition. college composition and communication, 58(2), 213–235. kim, j. h. (2017). smartphone-mediated communication vs. face-to-face interaction: two routes to social support and problematic use of smartphone. computers in human behavior, 67, 282–291. http://doi.org.10.1016/j.chb.2016.11.004 king, v. (2013). self-portrait with mortar board: a study of academic identity using the map, the novel and the grid. higher education research and development, 32(1), 96–108. doi:10.1080/0729460.2012.751525 kulik, c. c., kulik, j. a., & shwalb, b. j. (1983). college programs for high-risk and disadvantaged students: a meta-analysis of findings. review of educational research, 53(3), 397–414. lave, j. & wenger, e. (1991). situated learning: legitimate peripheral participation. new york, ny: cambridge university press. lowe, h. & cook, a. (2010). mind the gap: are students prepared for higher education? journal of further and higher education, 27(1), 53–76. https://doi.org/10.1080/03098770305629 mann, s. j. (2001). alternative perspectives on the student experience: alienation and engagement. studies in higher education, 26(1), 7–19. doi:10.1080/03075070020030689 mann, s. j. (2008). study, power, and the university. maidenhead, uk: open university press. merriam, s. b., & tisdell, e. j. (2016). qualitative research: a guide to design and implementation (4th ed.) san francisco, ca: jossey-bass. patton, m. q. (2015). qualitative research and evaluation methods (4th ed.). thousand oaks, ca: sage. rowell, r. (2013). fangirl. new york, ny: st. martin’s press. saldaña, j. (2009). a coding manual for qualitative researchers (2nd ed.). thousand oaks, ca: sage. saldaña, j. (2016). a coding manual for qualitative researchers (3rd ed.). thousand oaks, ca: sage. scanlon, l., rowling, l., & weber, z. (2007). ‘you don’t have like an identity…you are just lost in a crowd’: forming a student identity in the first-year using fiction 84 journal of effective teaching in higher education, vol. 2, no. 2 transition to college. journal of youth studies, 10(2), 223-241. doi:10.1080/13676260600983684 strayhorn, t. l. (2012). college students’ sense of belonging: a key to educational success for all students. new york: routledge. stephens, n. m., hamedani, m. g., & destin, m. (2014). closing the social-class achievement gap: a difference-education intervention improves firstgeneration students’ academic performance and all students’ college transition. psychological science, 25(4), 943–953. doi:10.1177/0956797613518349 tapp, j. (2014). “i actually listened, i am proud of myself”: the effects of participatory pedagogy on students’ construction of academic identities. teaching in higher education, 19(4), 323–335. doi:10.1080/13562517.2013.860108 tinto, v., & sherman, r. h. (1975). the effectiveness of secondary and higher education intervention programs: a critical review of literature. new york, ny: columbia teachers college. vygotsky, l. s. (1978). mind in society: the development of higher psychological processes. m. cole, v. john-steiner, s. scribner, & e. souberman, eds. cambridge, ma: harvard university press. microsoft word schererorourkesemanvarnerziegler_final2.docx journal of effective teaching in higher education, vol. 3, no. 1 coteaching in higher education: a case study of instructor learning hannah h. scherer, virginia tech, hscherer@vt.edu megan o’rourke, virginia tech rachel seman-varner, virginia tech peter ziegler, virginia tech abstract. tenure-track faculty at research intensive universities have competing responsibilities in their role as both researchers and teachers, leading to barriers to pedagogical change. this has been well-documented in the sciences. coteaching models of collaborative planning and teaching with an explicit goal of facilitating educator growth have been successful in increasing the use of research-based pedagogy in higher education settings. we documented our experiences coteaching in an undergraduate ecological agriculture course and, drawing on sociocultural views of faculty work as learning, identified opportunities for learning that occurred. we found that the collaborative teaching team structure provided timely access to pedagogical knowledge, the collaborative planning process was a mechanism for faculty learning, and observing members of the team teach provided opportunities for new insights. coteaching has had significant, lasting impacts for the instructors. we recommend that universities implement coteaching as a mechanism for supporting faculty use of student-centered pedagogy. keywords: coteaching, collaboration, faculty learning, higher education, pedagogical change tenure-track faculty at research intensive universities have competing responsibilities that are uniquely tied to their role as both researchers and teachers. science faculty face barriers to pedagogical change such as accountability structures (e.g., promotion and tenure review processes) that do not reward improvements to teaching, limited time, and lack of support (henderson et al., 2011). confounding the issue, faculty hired into these positions may have minimal preparation for teaching in higher education. brownell & tanner (2012) further argue that scientists’ training may have actually supported development of a professional identity that is antithetical to seeing themselves as a teacher. simultaneously, the knowledge base demonstrating the effectiveness of studentcentered teaching is steadily growing (national research council, 2012; weimer, 2013), offering new opportunities for faculty to employ evidence-based strategies. efforts to promote adoption of new pedagogy in undergraduate science, technology, engineering and math (stem) teaching, however, need to be sustained over time and include feedback (henderson et al., 2011). we addressed these challenges by employing a coteaching model of collaborative planning and teaching in a higher education setting. we had an explicit goal of utilizing a student-centered approach that included a variety of student engagement techniques and formative assessment (barkley, 2010). the specific coteaching model we used situates learning to teach alongside a more experienced teacher as a central purpose (roth, 2001). drawing on sociocultural views of faculty coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 16 work as learning (lattuca, 2002; lave, 1988; lave & wenger, 1991), we identified opportunities for learning that arose from coteaching. in this case study, we present an account of our experiences in order to illustrate the affordances of coteaching as a learning opportunity and allow others to explore potential applications of the model in their context. literature review higher education faculty development efforts reported in the literature cluster around six areas of focus: skill development, mastery of teaching methods, reflection, institution-level plans, disciplinary understanding of pedagogy, and action research or inquiry (amundsen & wilson, 2012). these activities, however, rarely explicitly address the “academic and social context in which faculty work and in which new knowledge must be embedded, practiced, and refined” (amundsen & wilson, 2012, p. 110). in undergraduate stem teaching, efforts external to the classroom, such as efforts to disseminate curricular materials to faculty, are not effective (henderson et al., 2011). learning through collaborative faculty work unlike pre-college teachers, it is less common for new higher education faculty members to receive formal preparation for teaching prior to hiring; much of this learning occurs instead through interactions with colleagues. in one case study, faculty often learned about teaching from colleagues through unofficial channels such as informal mentors (jawitz, 2007). other interactions can be highly structured, such as in faculty learning communities (flcs) where groups of faculty learn together with the goal of strengthening their teaching and learning efforts; flcs are highly effective in producing learning outcomes for participants (cox, 2004). additionally, faculty can learn from collaborations with education experts in efforts such as curriculum design teams (burrell et al., 2015) and postsecondary stem education improvement projects (bouwma-gearhart et al., 2014). education experts can serve as “brokers of education research and theory” (bouwma-gearhart et al., 2014, p. 43) who hold both procedural and tacit knowledge of pedagogical research and theory. these “brokers” can facilitate stem faculty learning about reformed teaching practices by translating ideas in a way that makes them accessible to stem faculty with little training. from the perspective of collaboration as a source of faculty learning (lattuca, 2002), collaborative teaching (also called coteaching or team teaching) can also be viewed as a mode for learning. while often motivated by content expertise and student learning outcomes as the primary drivers (wenger & hornyak, 1999), some studies have documented benefits of collaborative teaching with respect to faculty learning. learning from colleagues through the experience of collaborative planning has been found to be highly valued by faculty in teacher preparation programs (bacharach et al., 2008; graziano & navarrete, 2012). crow and smith (2005) describe their experience of teaching together in a health and social studies program as a powerful mechanism for enhancing their own learning through promoting deeper shared reflection on teaching practice. faculty within coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 17 collaborative teaching teams have reported learning from colleagues in the areas of disciplinary knowledge and pedagogical practice (culhane et al., 2016; ferguson & wilson, 2011; morelock et al., 2017). in one program, faculty also discussed the benefits of working with and learning from colleagues with more teaching experience (helms, 2014). benefits related to development of teaching skills and personal development as educators have also been reported for graduate students teaching as a team (burns & mintzberg, 2019; chanmugam & gerlach, 2013). these findings are consistent with outcomes reported for collaborative teaching as a mode of professional practice at the k-12 level (cook & friend, 1995). considerations for entering into teaching together a growing body of case studies, evaluation of institutional programs, and qualitative research studies provides guidance for faculty considering entering into a collaborative teaching arrangement. in terms of team member interactions, developing a good working relationship and committing to “open communication and continual reflection” (p. 8) are essential (holland et al., 2018). setting goals for these interactions in addition to teaching-related goals can provide a solid foundation for the relationship (holland et al., 2018; orlander et al., 2000). throughout the collaborative teaching experience, it is important to attend to the expertise that each faculty member contributes (ferguson & wilson, 2011; looft & myers, 2019), who holds structural power due to their position in the institution (morelock et al., 2017), and how ownership of the course and responsibility for decision making are shared (ferguson & wilson, 2011; morelock et al., 2017). beyond the classroom, institutional policies and culture can either support or hinder collaborative teaching arrangements and how credit is assigned. at a large research university, faculty expressed concern that their collaborative teaching efforts were not recognized by administrators and that budgetary or resource concerns were prioritized (bryant et al, 2014). in some instances, faculty report that coteaching with another person takes more than 50% (morelock et al., 2017), which is an additional concern when negotiating credit assigned versus effort. potential models for team teaching that work with traditional modes of assigning one instructor per course include having one team member receive full credit with the other donating their time, sharing major assignments between two courses taught by different instructors, and linking two courses through shared class time and assignments (ford & gray, 2011). coteaching: a model for purposeful faculty learning in praxis coteaching has emerged in recent years as a promising practice for faculty development. this version of coteaching, as described in detail by roth (2001), was originally developed for pre-service k-12 science teachers (e.g., roth et al., 1999). in this model, the pairing consists of a novice and an experienced teacher. they share responsibility for all aspects of teaching, allowing for “teachers to experience the classroom at the elbows of another practitioner” (roth, 2001, p. 15). the coteaching model addresses the theory-practice gap in science teacher preparation through “situating learning to teach science in the praxis of teaching science” (p. 12) and explicitly addresses the tacit dimensions inherent in teaching (roth et al., coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 18 1999). this model has successfully been employed in higher education settings (with various names) for graduate student training (davis-bundrage & medvedev, 2016; walters & misra, 2013), mentoring less experienced instructors (bryant et al., 2014) and faculty development to promote the use of evidence-based teaching practices in undergraduate science courses (henderson et al., 2009; holland et al., 2018). coteaching allows new instructors the opportunity to observe and experiment with new teaching strategies while learning from a more experienced colleague (bryant et al., 2014; holland et al., 2018). the coteaching model differs from traditional collaborative teaching in higher education in that learning to teach is an explicit goal for the novice teacher and the experienced teacher is identified for their expertise in research-based pedagogy within the disciplinary context of the course. theoretical framework sociocultural theories are a powerful way to understand collaborative faculty work as learning (lattuca, 2002), which can be described as the “personal and shared construction of knowledge” (lattuca & creamer, 2005, p. 4). from this perspective learning is viewed as mediated by culture (vygotsky, 1978) and constitutes an “integral part of generative social practice in the lived-in world” (lave & wenger, 1991, p. 35). this situated experience is foundational, not secondary to abstracted forms of knowledge (lave, 1988). if learning occurs through social practice, however, what is possible to learn may be limited by a person’s ability to observe others (hutchins, 1996). furthermore, “knowledge always undergoes construction and transformation in use” (lave, 1996, p. 8); in practice there are variations in how a particular activity unfolds in different situations, how knowledge is applied and constructed in practice, and what problems a person encounters in different settings (lave, 1988). in order to gain insights into faculty learning in our study, we specifically draw on lave’s theory of cognition in practice that calls for examination of the activities of persons-acting as they unfold in a particular setting (lave, 1988) and lattuca’s (2002) framing of interdisciplinary faculty work as learning. methods we employed an ethnographic approach in the design of this study in order to document opportunities for learning that arose in practice. the lead researcher, hannah, was a participant observer in the collaborative teaching team. data sources for the study included: • hannah’s jottings from meetings and classes, fieldnotes constructed from those jottings, and memos composed throughout the planning (pf) and teaching (tf) process; • individual reflections for each class period composed by members of the teaching team (r) following a standard set of questions, including “what did i learn today that will be useful to me as an instructor?”; coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 19 • semi-structured exit interviews conducted by hannah with team members (ei) with questions designed to promote reflection on what they learned about teaching throughout the coteaching experience; • and artifacts from the unit of interest (lesson plans, syllabus, handouts, etc.). hannah conducted the data analysis by first open coding ethnographic fieldnotes and developing initial categories following the constant comparative method (lincoln & guba, 1985). she then selected targeted categories that related to mechanisms for learning and developed focused codes within these categories following emerson, fretz, and shaw (2011). next, she conducted a second cycle of coding of fieldnotes, reflections, memos, and exit interview transcripts using these focused codes and developed themes, using artifacts for triangulation where relevant. she wrote integrative memos throughout analysis and conducted member checking through team debriefing sessions during and after analysis. members of the teaching team agreed upon the final themes reported here for their significance both theoretically and practically. course context and teaching team members our course, ecological agriculture, is situated within an experiential, interdisciplinary, undergraduate minor (civic agriculture and food systems, cafs) at a land-grant university in the united states. cafs is administered at the college level, with faculty members from multiple departments teaching courses and serving on the task force that oversees the minor. cafs is challenging resource allocation and models for collaborative teaching at our university and each course is led by a collaborative teaching team in which collaborative planning and team teaching are the norm (clark et al., 2013). in some cases, faculty instructors each receive credit for teaching, however this was not the case in our course. the characteristics of the teaching team for our case study and specific affordances of our unique situation that enabled us to implement the coteaching model were: • megan was a new lead instructor for the course with significant disciplinary expertise but little higher education teaching experience. she was instructor of record and received full teaching credit for the course. • hannah was an agricultural education faculty member with a science background. she had relevant content expertise and a research interest in innovative models for stem educator professional development. involvement in the course aligned with the research component of her appointment and she did not receive teaching credit. • the cafs minor graduate teaching assistant (gta) (rachel) was already assigned to the course. the gta was primarily responsible for grading student work, but operated as an active member of the team in all planning activities. • pete had an appointment with the academic programs office that oversees the minor and he participated in that capacity. he brought extensive prior experience with the course and cafs that ensured that the course remained aligned with the overall objectives for the minor. coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 20 our course content focused on fundamental principles of ecology as applied to the context of sustainable agriculture. learning experiences included: classroom and field experiences, service learning, debates, a farm design project, and critical reflections. the in-class portion of the course had been predominantly lecturebased in previous years and we aimed to significantly revise a portion of the course to be more student-centered. in our implementation of coteaching, subject matter expertise played a key role in who taught which topics. throughout the process, hannah was able to guide decisions related to the course in real-time before megan had invested time into planning the course on her own. findings collaborative teaching team structure provided access to pedagogical knowledge hannah has expertise in student-centered pedagogy developed from practice, facilitating professional development experiences, and training pre-service teachers and served as an “education broker” (bouwma-gearhart et al., 2014) for the team, someone who could help “brainstorm techniques to most effectively teach the content to students” (rachel ei). she was able to offer ideas from the education research literature that were new to the team members. for example, megan commented: there's a term that i never even used or heard before, but you talk about scaffolding a lot. and i guess just that being conscious of what is going to be new or hard for them and how to make the steps of getting to a better product easier and more feasible. (megan ei) hannah’s procedural and tacit knowledge of student-centered teaching became a resource for the other team members throughout the planning process and in the classroom. framing coteaching as a sociocultural learning process (lattuca, 2002) allows us to view these experiences, resources, and tools that hannah brought to the project as potential sources of knowledge to be transformed and applied in this new setting. collaborative planning was a mechanism for learning a key element of the coteaching model (roth, 2001) is collaborative planning that involves an experienced teacher and a novice, which allows for the complexities inherent in the planning process to be revealed. in our case, meetings with the entire team occurred (7 during summer, weekly during the semester) and group discussions during team meetings served as a major component of our planning process. additionally, team members did individual work in preparation for team meetings and to finalize the lessons prior to teaching class. during team meetings, the group discussion primarily centered on typical course planning topics, including the course schedule and flow of topics; lesson planning; course assignments and grading; and working with the learning management coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 21 system. megan guided the planning process and team members contributed by providing input and resources related to their expertise and prior experience with the course and/or topic. early in the planning process, hannah “volunteered to research activities for each class and bring ideas to the meetings” stating that she has “a ‘bag of tricks’ that [she] can easily draw from, so it isn’t too much work” (pf). there were also multiple instances during planning conversations where a team member sought input related to pedagogy and acted on it, such as in this exchange from the first planning meeting from the perspective of hannah, the “education broker”: megan asked me [hannah]: so, “what’s the best way to teach this stuff?” i explained my process for starting from learning objectives by asking myself “what do i want the students to remember in 6 months?”, deciding how i want them to be able to use the material, and then developing activities from there…megan then offered to take my suggestion of writing learning objectives for each topic and then we decided to look at them all together next week. (pf) in this moment, hannah introduces the concept of backwards design (wiggins & mctighe, 2005) in real-time as it naturally arises in the context of the planning process. megan’s question opens the door to this opportunity for hannah to describe her planning process and megan commits to following through on the suggestion. collaborative planning meetings also provided the opportunity to discuss pedagogy along with content when planning individual lessons and hannah frequently made suggestions regarding instructional approach and student engagement techniques. for example, in planning a 50-minute class about nitrogen, phosphorus, and carbon cycling: megan said she would normally just split the class up into thirds and lecture for 15 minutes on each. she asked what i would do and i suggested starting with the big picture (multiple forms of matter, not all available) and then having a group for each one and have them answer some questions and then teach the class. she could then tie it all together at the end. (pf) here megan acknowledges that she would have defaulted to didactic teaching and hannah was able to provide a suggestion for a student-centered approach that was tailored to the particular content that needed to be covered. megan followed through on this idea and developed and taught a version of this activity. planning meetings thus provided an opportunity for faculty members to have realtime guidance in making important decisions about the structure, content, and teaching strategies employed in the course. team members followed up with actions outside of the meetings and were collegial and accountable to each other throughout the process, which is an element of successful coteaching in higher education (henderson et al., 2009). planning the lessons together went far beyond splitting up the semester topics and allowed for generative discussions about coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 22 pedagogy to occur in concert with the learning objectives, content, and overall flow of the topics from class to class. through bringing together our collective expertise and making it accessible to other team members, we constructed new knowledge about how to teach this particular course. as megan stated: ...now that we've gone forward i feel like i can go backwards. so, seeing more what were the most valuable big outcomes that we had, now how do i go back and build to them to a more successful product. and so i think…some of the ideas that we introduced in this class about simplifying the teaching, having a main take-home message, scaffolding so that [students] build the skills to complete the main project. now i can go back and more consciously build those steps in to the farm project and the debates and those...bigger outcomes. (megan ei) the strategies described here and the intent to continue to improve on the course represents powerful learning outcomes for megan. these take-aways, learned through the practice of planning and teaching her first college level course, have the potential to impact her for an entire career. observing members of the team teach provided opportunities for new insights i would see you do something and then i would say “i want to try that” [because] just the content alone takes so much time that to do both the content and these new techniques that aren't at my fingertips and i've never thought about before, realistically i'm not going to do both at the same time. it was good, i just copied you! (megan ei) in our implementation of coteaching, there was typically just one instructor leading class on a given day. all members of the teaching team, however, were present in each class to observe and participate; this was highly beneficial in learning how to implement student-centered techniques. team members were able to observe student engagement techniques enacted, including the following. • stations (also called gallery walk): small groups of students respond to a prompt on a large sheet of paper hung around the room and then circulate to view and respond to what their classmates contributed to each prompt (barkley, 2010) • think-pair-share: in response to a question posed by the instructor, students first think on their own and then share with a partner before reporting out to the whole class (barkley, 2010) • jigsaws: small groups of students first develop expertise on a topic and then the groups are reconfigured so that the new groups have “experts” from each original group (barkley, 2010) • interactive lectures: lectures are punctuated by short student engagement techniques and opportunities for students to contribute their ideas (macdonald & teed, 2019) coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 23 seeing the “nuances” (pete ei), flow, and “managing the time” (rachel ei) in practice were particularly valuable. additionally, observation allowed for team members to see things not go well and this prompted reflection-in-action as the class unfolded. this was particularly important for pete, who explained: i could recognize things that i would say are attributable to last minute planning or...just things getting too busy, and [i thought], “oh, ok, this is what it looks like from the other side” and it sort of gave me that point of reflection...[t]hat's just one aspect. the rest of it is, again, watching in action everybody else and thinking about it and being cognizant of what you're trying to do in the classroom...it really helps reflect on your own practice when you're sitting there thinking. (pete ei) thus, being able to observe others enacting student-centered techniques provided opportunities to learn through modeling and personal reflection. finally, observation of individual personalities led to new insights about what is possible in science teaching. rachel in particular discussed the contrasts with past experiences in science, stating “i come from [a traditional]…lecture style background, that's what most of my coursework has been in…i get the professor…using powerpoint and [stating] ‘these are the facts’” (ei). seeing alternatives to this approach allowed rachel to build her confidence in not having to conform to this model, concluding “i can just be myself” (ei). our implementation of coteaching disrupted the pattern of reproduction of teacher-centered approaches for team members through providing access to alternative modes. in addition to the general benefits of observation, there were opportunities for learning that arose from the coteaching model. times when hannah intervened when someone else was teaching provided powerful learning opportunities: there were...a couple times i noticed you [hannah] jumped in and they...were perfect, i mean they were exactly what needed to be done to salvage whatever was going on and that was really valuable...to me, those were the biggest learning moments, i'm like “oh, alright, that's what it should have been.” (pete ei) in these instances, hannah was able to draw on her expertise and recognize in the moment what needed to happen. the infrequency of these moments likely arose from the lack of clarity on expectations for team member participation in class and “out of respect for the instructor and not wanting to interrupt the flow” (tf). further clarifying classroom roles and expectations for contributions in class could enhance opportunities for learning through coteaching. coteaching further allowed for iterative cycles of observation and practice. as megan stated, “it was the observation and then the re-observation because there were iterative time points. you did it, i tried it, you did it, i tried it, you did it...that was really useful” (ei). furthermore, because this happened in the context of the specific course that we were working with, it was particularly valuable: coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 24 it was different having you part of the teaching team compared to say if i just went and watched a random physics lecture using a technique because my head's not wrapped around...the nuances of the content and how difficult it is to present...seeing you simplify things that i know are complex…was really useful. (megan ei) coteaching allowed for the subtleties in teaching to come through in an authentic setting. learning in practice led to new insights that would not likely arise in a more abstract context (e.g., henderson et al., 2011) as each instance of problem-solving in practice is unique and allows for existing knowledge to be transformed (lave, 1988). conclusion and recommendations our implementation of the coteaching model went beyond traditional team teaching in higher education by making the goal of learning to teach explicit, thus affording team members a heightened opportunity to learn from collaboration, observation, and reflection in practice. similarly, it went beyond short-lived, decontextualized faculty development models by embedding learning to teach in an authentic context. from the perspective of practice theory, coteaching served as a vehicle for faculty learning that employed the lived experience of teaching together as a mechanism for learning a complex process. our case study was unique to our context, but allows for some recommendations for practice that may be of use to others interested in coteaching. target the course development phase and include gtas for the lead instructor, coteaching in the first semester of the course was particularly impactful. additionally, rachel gained a new perspective on how she could operate in a future faculty role. without training in student-centered pedagogy, the default strategies that megan and rachel described are consistent with examples of teaching that they had access to from previous experience, which is realistically all that an instructor can be expected to do (hutchins, 1996). with so many barriers to faculty pedagogical change (brownell & tanner, 2012), it can be difficult for instructors to go out of their way to access anything different. through this coteaching experience, we observed and participated in diverse teaching practices that elicited a high level of student engagement and enthusiasm that can serve as a benchmark for future iterations of the course and other courses. consistent with holland et al. (2018), we recommend targeting early career faculty, particularly as they take on new courses, for coteaching. be intentional in defining coteaching purpose and role of the “education broker” in our experience, a key element that led to meaningful learning outcomes for team members was hannah’s role as an “education broker,” someone with enough familiarity with course content to be able to teach and enough expertise to model student-centered course planning and teaching techniques. throughout the coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 25 experience, it was important for hannah to be purposeful in making her thought process explicit when discussing planning, provide constructive feedback and suggestions in real time, and intentionally model a variety of classroom engagement strategies when she was teaching. the purpose of coteaching as an opportunity to learn to teach was forefront for hannah, however, this could have been more explicit for other team members. we recommend more structure, such as a contract outlining respective roles and responsibilities, in forming new coteaching teams (see holland et al., 2018). embrace imperfection and reflection as learning opportunities student-centered teaching is a complex, messy process and seeing mistakes and the nuances of teaching lead to new insights. the lived experience of working with someone with more experience can lead to learning how to teach, including learning how to navigate challenges that arise in real time. engaging the coteaching team in reflective practice (both individually and as a team) allowed for a unique opportunity for deepening understanding and building capacity for employing student-centered teaching in the future. we recommend that coteaching teams actively work to build positive relationships amongst members so that the coteaching environment is one that supports experimentation and reflection. consider allocation of resources and feasibility in context a major consideration moving this work forward is the allocation of university resources to support a coteaching model, which will vary widely depending on institutional context. the focus of our case study was confined to our coteaching experience, so we have limited recommendations for practice at an institutional scale. in our case, all members of the teaching team had a valuable professional learning experience. the lasting benefits for the specific course, however, could likely have been achieved with the involvement of just the lead instructor and education broker. we recommend careful consideration of the available expertise, potential collaborators from other departments, and the timing of the investment when developing a coteaching program. the inclusion of a gta on the team allowed for the lead instructor to devote more of her time and attention to lesson planning, so we highly recommend this where feasible. coteaching was an efficient use of megan’s time as a new faculty member because learning to teach was embedded in her teaching responsibilities, not an additional burden. it was a highly productive semester with lasting benefits that efficiently allowed for sharing of expertise in practice. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 26 references amundsen, c., & wilson, m. (2012). are we asking the right questions?: a conceptual review of the educational development literature in higher education. review of educational research, 82(1), 90–126. https://doi.org/10.3102/0034654312438409 bacharach, n., heck, t. w., & dahlberg, k. (2008). co-teaching in higher education. journal of college teaching & learning, 5(3), 9. https://doi.org/10.19030/tlc.v5i3.1298 barkley, e. f. (2010). student engagement techniques: a handbook for college faculty. jossey-bass. bouwma-gearhart, j., perry, k. h., & presley, j. b. (2014). improving postsecondary stem education: strategies for successful interdisciplinary collaborations and brokering engagement with education research and theory. journal of college science teaching, 44(1), 40–47. https://www.nsta.org/store/product_detail.aspx?id=10.2505/4/jcst14_044_01_40 brownell, s. e., & tanner, k. d. (2012). barriers to faculty pedagogical change: lack of training, time, incentives, and tensions with professional identity? cbe life sciences education, 11(4), 339–346. https://doi.org/10.1187/cbe.12-09-0163 bryant, l. h., niewolny, k., clark, s., & watson, c. e. (2014). complicated spaces: negotiating collaborative teaching and interdisciplinarity in higher education. the journal of effective teaching, 14(2), 83–101. https://files.eric.ed.gov/fulltext/ej1060430.pdf burns, v. f., & mintzberg, s. (2019). co-teaching as teacher training: experiential accounts of two doctoral students. college teaching, 67(2), 94–99. https://doi.org/10.1080/87567555.2018.1558169 burrell, a. r., cavanagh, m., young, s., & carter, h. (2015). team-based curriculum design as an agent of change. teaching in higher education, 20(8), 753–766. https://doi.org/10.1080/13562517.2015.1085856 chanmugam, a., & gerlach, b. (2013). a co-teaching model for developing future educators' teaching effectiveness. international journal of teaching & learning in higher education, 25(1), 110–117. clark, s., c. byker, k. niewolny, and j helms. (2013). framing an undergraduate minor through the civic agriculture and food systems curriculum. north american colleges and teachers of agriculture, 57(2), 56–67. cook, l., & friend, m. (1995). co-teaching: guidelines for creating effective practices. focus on exceptional children, 28(3), 1–16. coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 27 cox, m. d. (2004). introduction to faculty learning communities. new directions for teaching & learning, 2004(97), 5–23. https://doi.org/10.1002/tl.129 crow, j., & smith, l. (2005). co-teaching in higher education: reflective conversation on shared experience as continued professional development for lecturers and health and social care students. reflective practice, 6(4), 491–506. https://doi.org/10.1080/14623940500300582 culhane, j. h., niewolny, k., clark, s., mcconnell, k., & friedel, c. (2016). learning through collaborative and interdisciplinary teaching: a case study of faculty work as learning in sustainable agriculture education. nacta journal, 60(2), 219–226. davis-bundrage, m., & medvedev, k. (2016). teacher training and student learning outcomes in family and consumer sciences: a mentoring and co-teaching case study. journal of human sciences and extension, 4(1), 34–50. emerson, r. m., fretz, r. i., & shaw, l. l. (2011). writing ethnographic fieldnotes (2nd ed.). university of chicago press. ferguson, j., & wilson, j. c. (2011). the co-teaching professorship: power and expertise in the co-taught higher education classroom. scholar-practitioner quarterly, 5(1), 52–68. ford, j., & gray, l. (2011). team teaching on a shoestring budget. honors in practice online archive, 7, 103–111. graziano, k. j., & navarrete, l. a. (2012). co-teaching in a teacher education classroom: collaboration, compromise, and creativity. issues in teacher education, 21(1), 109. helms, j. (2014). teaching and learning in sustainable agriculture curricula: a case study of faculty work as learning at a land grant university (doctoral dissertation). https://vtechworks.lib.vt.edu/handle/10919/47354 henderson, c., beach, a., & famiano, m. (2009). promoting instructional change via coteaching. american journal of physics, 77(3), 274–283. https://doi.org/10.1119/1.3033744 henderson, c., beach, a., & finkelstein, n. (2011). facilitating change in undergraduate stem instructional practices: an analytic review of the literature. journal of research in science teaching, 48(8), 952–984. https://doi.org/10.1002/tea.20439 holland, t., sherman, s. b., & harris, s. (2018). paired teaching: a professional development model for adopting evidence-based practices. college teaching, 66(3), 148–157. https://doi.org/10.1080/87567555.2018.1463505 hutchins, e. (1996). learning to navigate. in s. chaiklin & j. lave (eds.), understanding practice: perspectives on activity and context (pp. 3–34). cambridge university press. coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 28 jawitz, j. (2007). new academics negotiating communities of practice: learning to swim with the big fish. teaching in higher education, 12(2), 185–197. https://doi.org/10.1080/13562510701191943 lattuca, l. r. (2002). learning interdisciplinarity: sociocultural perspectives on academic work. the journal of higher education, 73(6), 711–739. lattuca, l. r., & creamer, e. g. (2005). learning as professional practice. new directions for teaching and learning, 2005(102), 3–11. https://doi.org/10.1002/tl.192 lave, j. (1988). cognition in practice: mind, mathematics, and culture in everyday life. cambridge university press. lave, j. (1996). the practice of learning. in s. chaiklin & j. lave (eds.), understanding practice: perspectives on activity and context (pp. 3–34). cambridge university press. lave, j., & wenger, e. (1991). situated learning: legitimate peripheral participation. cambridge university press. lincoln, y. s., & guba, e. g. (1985). naturalistic inquiry. beverly hills, calif: sage publications. looft, r., & myers, m. j. (2019). exploring faculty and student reflections on collaborative teaching in the honors seminar classroom. journal of interdisciplinary studies in education, 8(1), 140-157. macdonald, h., & teed, r. (2019). interactive lectures. starting point: teaching entry level geoscience. https://serc.carleton.edu/introgeo/interactive/index.html morelock, j. r., lester, m. m., klopfer, m. d., jardon, a. m., mullins, r. d., nicholas, e. l., & alfaydi, a. s. (2017). power, perceptions, and relationships: a model of coteaching in higher education. college teaching, 65(4), 182–191. https://doi.org/10.1080/87567555.2017.1336610 national research council. (2012). discipline-based education research: understanding and improving learning in undergraduate science and engineering. committee on the status, contributions, and future directions of discipline-based education research. board on science education, division of behavioral and social sciences and education. the national academies press. orlander, j. d., gupta, m., fincke, b. g., manning, m. e., & hershman, w. (2000). coteaching: a faculty development strategy. medical education, 34(4), 257–265. roth, w.-m. (2001). becoming-in-the-classroom: learning to teach in/as praxis. in d. r. lavoie & w.-m. roth (eds.), models of science teacher preparation: theory into practice (vol. 13; 13., pp. 11–30). dordrecht: kluwer academic publishers. coteaching in higher education: a case study of instructor learning journal of effective teaching in higher education, vol. 3, no. 1 29 roth, w.-m., masciotra, d., & boyd, n. (1999). becoming-in-the-classroom: a case study of teacher development through coteaching. teaching and teacher education, 15(7), 771–784. https://doi.org/10.1016/s0742-051x(99)00027-x vygotsky, l. s. (1978). mind in society: the development of higher psychological processes. harvard university press. walters, k., & misra, j. (2013). bringing collaborative teaching into doctoral programs: faculty and graduate student co-teaching as experiential training. the american sociologist, 44(3), 292–301. weimer, m. (2013). learner-centered teaching: five key changes to practice (2nd ed.). jossey-bass. wenger, m. s., & hornyak, m. j. (1999). team teaching for higher level learning: a framework of professional collaboration. journal of management education, 23(3), 311-327. https://doi.org/10.1177/105256299902300308 wiggins, g. p., & mctighe, j. (2005). understanding by design. alexandria, va: association for supervision and curriculum development. journal of effective teaching in higher education, vol. 1, no. 1 the interprofessional showcase: evaluating an event to increase professional understanding and collaboration kathleen a. gould, ed.d, r.d., l.d.n clinical associate professor department of health sciences towson university andrea barton, m.s. clinical assistant professor department of kinesiology towson university karen day, m.s. clinical associate professor department of audiology, speech-language pathology & deaf studies towson university abstract. traditional didactic instruction is not an effective means to provide the interprofessional education that health and human services professional students need. this paper describes a college event aimed at engaging undergraduate and graduate students from several academic departments in interprofessional collaboration. participants toured and interacted in academic spaces of other professional disciplines to gain an understanding of these professional roles. interprofessional collaboration was demonstrated and experienced in these spaces and in the group discussions that followed these activities. a series of microvignettes, questions, and road blocks provided problems for participants to solve in interprofessional groups. pre-post comparison surveys indicated that participants increased in their understanding of their professional roles and the roles of others in health care and community settings. an improved understanding of the benefits and challenges of interprofessional teams was achieved and participants felt more confident in their ability to work in these teams at job or internship sites. keywords: interprofessional education, health professional education, interprofessional collaboration, interdisciplinary. traditional didactic education of health care professionals occurs in settings where students learn with others in the same professional discipline. it has been argued that this form of instruction does not develop the skills needed to effectively collaborate with others in future professional roles (who, 2010). additionally, traditional uni-professional instruction provides limited opportunity for critical thinking and collaborative problem solving (reeves, tassone, parker, wagner, & simmons, 2012). in a review of interprofessional education efforts over the last 30 years, reeves et al. (2012) suggested that a lack of student preparation for collaboration and interprofessional communication resulted in negative implications for patient care and service. traditional uni-professional education is inadequate the interprofessional showcase 38 journal of effective teaching in higher education, vol. 1, no. 1 preparation for working in a healthcare system that according to the association of american medical colleges (2012) should emphasize higher-quality, lower-cost care models where health care providers practice in interprofessional teams. the preparation of health care professionals is not the only realm where interprofessional skills are needed for effective practice. johnson and graves (2017) indicated that curricula in the sciences need to reflect the collaborative and interdisciplinary environments of working scientists. self and baek (2017) advocated for and developed a plan for interdisciplinary instruction for design students. teacher preparation also needs to incorporate interprofessional training to best serve overall needs of future students (anderson, 2013; dobbs-oates & morris, 2016; tarr, whittle, wilson, & hall, 2013; wilson, mcneill, & gillon, 2017). additionally, arora and arora (2015) suggested the incorporation of interdisciplinary instruction to business education in the areas of supply chain management and marketing to handle complex problems that occur in these environments. lastly, lattuca, knight, ro, and novoselich (2017) emphasized the value of bringing engineering students together with those from other disciplines to develop interdisciplinary competence. to prepare future practitioners for the new environment in health care, medical schools and other health profession schools increased their focus on interprofessional education (ipe), in which students of different health professions learn together in preparation to practice team-based care (association of american medical colleges, 2016). the american interprofessional health collaborative (aihc) supports a team centered approach to healthcare and emphasizes the need to educate those entrusted with the health of individuals, communities, and populations to value and respect each other’s unique expertise and skills. the ability to work together is fundamental to care that is effective, safe, of high quality, and efficient in terms of cost, resources, and time (aihc, 2015). this paper describes the development, implementation, and evaluation of a college wide interprofessional showcase event (ipsce) that could serve as a model for interprofessional education in a variety of settings beyond health care professional education. the goal of this event was to provide students with exposure to the environments that other professionals work in and allow them to engage in problem solving teamwork required in the workplace. the ipsce took place in the college of health professions (chp) at a mid-atlantic public university. it involved participants rotating through interactive clinical/educational spaces on campus and discussion of several micro-vignette problems in interprofessional groups. evaluation of students’ reaction to the event was imperative to provide a better understanding of its effectiveness on attitude change regarding interprofessional practice. literature review interprofessional training is vital to developing students who are prepared to meet the needs of a diverse and challenging workplace. according to the world health organization framework for action on interprofessional education and collaborative the interprofessional showcase 39 journal of effective teaching in higher education, vol. 1, no. 1 practice (who, 2010), interprofessional education is defined as occurring when two or more professions learn about, from, and with each other to enable effective collaboration and improve health outcomes. this framework further defined a collaborative practice-ready health worker as someone who has learned how to work on and serve as a competent member of an interprofessional team (who, 2010). developing workers who demonstrate these skills is the goal of interprofessional education. the emphasis on the need for interprofessional education has not been confined to the training of competent health care professionals. the increasing complexity of workplace environments have necessitated the introduction of ipe into teacher training, business, science, design, and engineering disciplines (anderson, 2013; arora & arora, 2015; dobbs-oates & morris, 2016; johnson & graves, 2017; lattuca et al., 2017; self & baek, 2017; tarr et al., 2013; wilson, mcneill, & gillon, 2016, 2017). for example, tarr et al. (2013) discussed child protection education directed at trainee teachers that included interprofessional instruction involving education, health, police, and social work disciplines. the need for effective collaboration in the school environment between teachers and speech and language therapists (slts) was the emphasis for an interprofessional education program that included case-based instruction supplemented with discussion to share respective expertise with literacy instruction (wilson et al., 2016). these researchers also investigated the outcomes of shared professional practice placements for student teachers and student slts on their preparedness to effectively collaborate to support children’s learning and well-being (wilson et al., 2017). to simulate business interactions that occur in advertising industry service supply chains, arora and arora (2015) developed and evaluated an experiential lab game to engage supply chain management (scm) and marketing students in interdisciplinary problem solving. game participation increased student understanding of the complex interrelationships between scm and marketing in real world business environments. recognizing the importance of interdisciplinary competency in engineering practice, lattuca et al. (2017) sought to understand the role of student characteristics, college experiences, and faculty beliefs in competency development. their findings suggested that it was imperative for the engineering curriculum to promote interdisciplinary thinking to develop these competencies. additionally, co-curricular experiences that brought engineering students together with other majors were important to the development of interdisciplinary competence. to mimic the interrelationships in scientific research, johnson and graves (2017) developed and implemented a chemistry-genetics course collaborative to provide an interdisciplinary learning environment for chemistry and genetics students. their attempt to create an understanding of the interrelationships between the scientific disciplines and emphasize the collaboration that occurs between practicing scientists was met with challenges due to a disconnect between course content and student expectations. similarly, self and baek (2017) developed and implemented courses that were team taught by industrial design and human factors engineering instructors to provide students with interdisciplinary learning experiences. when compared to single instructor courses in the same topic areas, the team-taught classes provided students with a the interprofessional showcase 40 journal of effective teaching in higher education, vol. 1, no. 1 greater understanding of the relevance of different disciplines to the course subject. however, self and beck (2017) found that single instructor classes received more overall positive evaluations of course quality indicating the need to further refine the team-taught course design. determining effective teaching methods to engage students and impart interprofessional skills is a challenge in the academic setting that has been bound by traditional forms of instruction. abu-rish et al. (2012) surveyed the literature from 2005 to 2010 and found several models for ipe directed at pre-licensure health professionals. prevalent among these models were one-time workshop style events utilizing small group discussion, case-based learning, large group lectures, reflective exercise, clinical teaching or direct interaction with patients, simulation or community-based projects. the most popular formats, small group discussion and case-based learning, used in conjunction appeared more conducive to student interaction and team building as compared to didactic instruction (abu-rish et al., 2012). d’eon (2005) suggested that case based learning (cbl) and problem based learning (pbl) are important methods to fortify the success of ipe because these approaches utilize the skills required for effective teamwork. he argued that the two main components for the transfer of learning are complexity of the learning task and a cooperative-experiential approach to engage in the learning task. learners benefited from activities that promoted positive interdependence, face-toface interaction, individual accountability, interpersonal and small-group skills, and group processing. students should engage in a four-stage model of planning, doing, observing, and reflection for effective experiential learning. cbl and pbl formats have been utilized in various settings as a method to facilitate ipe (e.g. charles, bainbridge, & gilbert, 2010; curran, heath, kearney, & button, 2010; macdonnell, rege, misto, dollase, & george, 2012; villadsen, allain, bell, & hingley-jones, 2012; wilson et al., 2016). participants in these events included professionals, post-graduate students (curran et al., 2010), undergraduate health professional students (charles et al., 2010) as well as student teachers and speechlanguage therapy students (wilson et al., 2016). charles et al. (2010) described an undergraduate interprofessional workshop designed to expose students to interprofessional practice. this workshop was designed to assist students in acquiring an understanding and appreciation of the role that other health and human service professionals play in clinical situations. teams of students developed a collaborative plan for delivering patient services on a faculty-designed case while other students observed the team interactions. participation in the case and observation of the event allowed students to reflect on their understanding of interprofessional practice. curran et al. (2010) found that a one-day interprofessional workshop improved the attitudes of post-graduate residents toward healthcare teamwork and provided a satisfactory learning experience for nursing and allied health staff. physicians, nurses, social workers, occupational therapists, dietitians, and physiotherapists the interprofessional showcase 41 journal of effective teaching in higher education, vol. 1, no. 1 participating in an interprofessional shared decision-making (ip-sdm) workshop felt that their knowledge of ip-sdm improved (stacey et al., 2014). similarly, medical, midwifery, social work, and early years students valued the shared learning in a two-hour workshop designed to facilitate cooperation in providing care for a familybased problem scenario (villadsen et al., 2012). students’ positive attitudes toward ipe improved because of workshop participation. macdonnell et al. (2012) reported that medical, nursing, and pharmacy students participating in an ipe workshop showed an enhanced understanding of other professionals. their results indicated that students were able to effectively work together as a team without prior familiarity with each other or significant ipe experience. workshop participation helped establish trust and effective communication between the interprofessional participants (macdonnell et al., 2012). gould, day, and barton (2017) evaluated an interprofessional case study workshop with 289 participants from 13 different disciplines. they found that student participation increased awareness of other professional roles and the benefits and challenges of interprofessional teams. background the interprofessional showcase event the ipsce took place in the college of health professions (chp) of a public university with a total student enrollment of 4,528 in 23 graduate and undergraduate programs. the 209 participants were from athletic training; audiology; exercise science; family studies; health education; nursing; occupational therapy; and speech language pathology. most students at the event were junior or senior level undergraduate students (164; 78%) and the remainder (45; 22%) were graduate students from speech/language pathology (33; 16%) and audiology (12; 6%). participants had no previous formal academic exposure at this university to professional practice in college programs other than their own as chp did not offer interprofessional coursework at the time of this event. therefore, the ipsce was their first exposure to interprofessional collaboration in the current academic setting. the ipsce consisted of two major components: 1. an interprofessional showcase in which students toured and interacted with activities in professional workspaces/academic learning areas of various chp disciplines and, 2. interprofessional discussion where students were provided with micro-vignettes to engage in problem solving. the interprofessional showcase committee (ipscc), comprised of faculty from each chp department and the department of family studies in the college of liberal arts, developed four micro-vignettes for interprofessional problem solving. the ipsce objectives were provided to students at registration via an online link to prepare them for collaborative discussion. the objectives and expected learning outcomes were as follows: the interprofessional showcase 42 journal of effective teaching in higher education, vol. 1, no. 1 1. gain exposure to settings of clinical practice and/or academic learning other than those affiliated with your pursuit of study. 2. engage in collaborative, team-based learning involving prioritization and development of strategies which mutually address complex health care issues. 3. develop a broader understanding of challenges and benefits of working on an interprofessional team. 4. acquire an enhanced appreciation of your own role and the roles of other professionals on an interprofessional team. interprofessional showcase the first activity of the event, the interprofessional showcase, introduced students to settings of clinical practice and/or academic learning other than those affiliated with their pursuit of study. the showcase activities were designed to be interactive and hands-on to provide students with a broader view of the health professions and stimulate thought about their potential for collaborative interactions within each setting. prior to the event, the ipscc determined which clinical practice/academic learning spaces on campus would best highlight interprofessional collaborative practice. committee members solicited faculty showcase presenters and predetermined activities for each showcase space. these activities focused on the clinical/educational activities that occur in the space, health professionals that are involved, and interprofessional collaboration that occurs. students were preassigned to the showcase spaces not related to their major to maximize their exposure to a variety of professional settings. for example, a speech language pathology student was assigned to the nursing simulation lab for an experience while an occupational therapy student was assigned to the vestibular lab. showcase spaces and the activities that occurred there are enumerated in table 1. table 1 professional affiliations, activities, and showcase spaces professional affiliation activity showcase space athletic training helmet removal, spine boarding, and intubation athletic training lab audiology rotary chair/vestibular assessment vestibular lab audiology audiology assessment audiology booth exercise science vo2 max testing exercise science lab the interprofessional showcase 43 journal of effective teaching in higher education, vol. 1, no. 1 exercise science research and education programs university wellness center nursing adult patient simulation adult health simulation lab nursing obstetrics patient simulation family health simulation lab nursing pediatric patient simulation pediatric simulation lab occupational therapy visual deprivation demonstration occupational therapy kitchen occupational therapy autism independent living skills center for autism apartment occupational therapy pediatric assessment occupational therapy classroom occupational therapy psychosocial assessment occupational therapy classroom occupational therapy splinting demonstration occupational therapy classroom occupational therapy trunk, balance and motor control surf board room occupational therapy virtual voices occupational therapy classroom occupational therapy/ speech language pathology therapeutic nursery school activities children’s treatment center speech language pathology school aged child therapy speech treatment center the event began with students arriving at a central location where they were welcomed, introduced to the event activities, and received their showcase assignments. students were divided into groups to travel, accompanied by faculty committee members, to the showcase spaces on campus by walking or by charter bus. upon arrival, students were greeted by showcase faculty presenters who provided information about the space and introduced the predetermined activities. showcase presenters were encouraged to facilitate thinking and discussion related to interprofessional collaboration in each clinical/learning space and activity. students participated in two showcase demonstrations lasting approximately 20 minutes each and returned to the workshop venue on foot or by bus. their experiences in the showcases were referenced throughout the remainder of the event to encourage students to develop a broader understanding of their own professions, other professions, and interprofessional collaboration. the interprofessional showcase 44 journal of effective teaching in higher education, vol. 1, no. 1 interprofessional micro-vignette discussion the second part of the ipsce consisted of interprofessional discussion of three micro-vignettes. students reconvened at a conference-style venue on campus and were preassigned to two mixed discipline numbered roundtables of no more than 8 students per table. each student’s table assignments were pre-printed on their name tags. at each table, students were provided with a packet of information that contained a micro-vignette, two guiding questions designed to start discussion of the micro-vignette, and four individual envelopes labeled roadblock #1, roadblock #2, roadblock #3, and roadblock #4. the chp dean opened the event with a brief address about ipe, and the ipscc chair instructed students to introduce themselves and their major to the other students around the table. students were asked to identify a table leader to take notes and speak on behalf of the members of the table. students were allotted approximately 20 minutes to discuss the micro-vignette providing perspectives from their own disciplines and/or personal experiences. the micro-vignette and guiding questions initiated the interprofessional discussion and the event activities continued as illustrated in figure 1. micro-vignette examples an example of a micro-vignette and the associated guiding questions and roadblocks is as follows: scenario: mr. gonzales, a 68-year-old male with high blood pressure, is playing in a park with his grandchildren when he experiences symptoms of a stroke including slurred and slowed speech, weakness in his right arm, and facial asymmetry indicating an ischemic stroke. figure 1. step by step sequence of interprofessional roundtable activities the interprofessional showcase 45 journal of effective teaching in higher education, vol. 1, no. 1 guiding questions: 1) where would you start with the client? 2) what role would your discipline potentially play in the care of this client/patient? roadblock #1: mr. gonzales does not read and write english well and has difficulty understanding medical information since english is not his first language. roadblock #2: mr. gonzales lives alone. his daughter is a 28-year-old single mom who works two jobs, but she lives close by and can help in a limited capacity, mostly on the weekends. when he returns home, he will need assistance with medication management, daily hygiene tasks, and meal preparation. roadblock #3: after discharge from the rehabilitation facility, he will need to continue with physical therapy two days per week and join a local gym where he will exercise on his own three days per week. roadblock #4: he has been working with the speech language pathologist who suspects he also has an underlying hearing impairment. additional questions: 1) what would be a priority intervention that you could implement within your professional role? 2) what other health professions/disciplines would be involved in the care of this client/patient? 3) how would you work together for the best outcome for this client/patient? upon completion of the first interprofessional micro-vignette discussion, students were directed to re-package the table packets and roadblock envelopes for the next group of students assigned to the table. table assignments and student re-grouping were strategically planned to provide for different student composition and microvignettes for the second round of discussions. three micro-vignettes were utilized for the first and second table assignments insuring that each student would discuss a different micro-vignette at each of their table assignments. the two additional micro-vignettes used for interprofessional discussion involved a high school athlete with diabetes and a sixty-year-old marathon runner diagnosed with prostate cancer. wellness micro-vignette the wellness micro-vignette used for discussion was as follows: scenario: you and your colleagues are members of the wellness team that received a contract to work with employees and their families at a major corporation in your city. as part of this team your task is to identify the potential health risks in the clients and plan appropriate physical and mental health screening and assessments, potential direct and indirect interventions, and wellness education to help reduce the risks. the interprofessional showcase 46 journal of effective teaching in higher education, vol. 1, no. 1 guiding questions: 1) where would you start with the corporate wellness client? 2) what role would your discipline potentially play in the care of this client/patient? roadblock #1: many of the hourly employees work long hours and do not have access to onsite exercise facilities. roadblock #2: company headquarters is in a part of town where fast food restaurants predominate, and employees are allowed a 30-minute lunch break. roadblock #3: the onsite corporate nurse informs you that numerous employees have multiple chronic disease diagnoses including diabetes, hypertension, cardiovascular issues, elevated blood lipids, hearing loss, and arthritis. roadblock #4: numerous employees indicate regular use of tobacco products and alcohol. additional questions facilitator can pose to continue the discussion: 1) what would be a priority intervention that you could implement within your professional role? 2) what other health professions/disciplines would be involved in the care of this client/patient? 3) how would you work together for the best outcome for this client/patient? event wrap-up discussion the event concluded with an overview of the day’s activities facilitated by members of the ipscc. wireless microphones were used to enable students to share with the larger group what they observed during showcasing and interprofessional discussions. students also responded to several questions regarding interprofessional education and collaboration. discussion was driven by these questions but became open-ended and spontaneous as the wrap up progressed. at the conclusion, students completed an online event evaluation survey using their mobile devices or provided ipads. a survey completion confirmation page served as proof of participation and a means by which the student attendees were awarded a signed certificate of event attendance. methods the ipsce was designed to introduce students to other professions in chp and to engage them in interprofessional discussion. this pre-post comparison study investigated attitude change in the ipsce participants. the sample consisted of 182 students who registered for the event and provided matched pre-and post-survey responses. data collection two weeks prior to the showcase event, students completed an online registration survey and created a unique identifying number to enable their responses from the the interprofessional showcase 47 journal of effective teaching in higher education, vol. 1, no. 1 pre-survey to be linked anonymously with the post-event survey. a four point likert scale where 1 was strongly agree and 4 was strongly disagree was used to assess participant knowledge about other professions and attitudes toward interprofessional collaboration using the statements found in table 2. chronbach’s alpha for these questions was 0.906 suggesting a high degree of internal consistency between the survey questions in measuring attitudes related to interprofessional understanding. at the close of the showcase, participants completed a digital post-evaluation survey on site using personal devices or ipads provided at the event. using the unique identifier created at registration, participants responded to statements regarding attitude change as a result of participation on the same criteria screened in pre-event survey. chronbach’s alpha for these questions was 0.91 suggesting a similar degree of internal consistency in the post-survey as was demonstrated in the pre-event attitude survey. data analysis pre-post results that could not be paired were discarded and means were calculated for the 182 responses. a paired t-test using an alpha level of .05 was performed on these means using ibm spss statistics for windows, version 19.0. (armonk, ny). ethics approval was granted by the university institutional review board for the protection of human participants. the research was considered exempt from general human participant requirements with exemption number 15-x075. results matched data from 182 preand post-responses were analyzed from the following majors: athletic training (13, 7.1%); graduate audiology (11, 6.0%); exercise science (31, 17.0%); family studies (14, 7.7%); health education (14, 7.70%); nursing (46, 25.3%); occupational therapy (24, 13.2%); and graduate speech language pathology (29, 15.9%). a paired t test showed a statistically significant difference in six attitude questions. pre-post-means and paired t results for all survey questions are reported in table 2. a likert scale where 1=strongly agree, 2=agree, 3=disagree and 4=strongly disagree was utilized. therefore, a decline in mean values from pre-event to postevent suggests a positive shift in student attitude. table 2 pre-post attitude means and paired t for the 2016 interprofessional showcase event pre-post questions n mean (sd) t df p pre: my academic preparation has enabled me to confidently verbalize the role of my profession as it relates to other professions in a health care/community setting. 182 1.73 (.698) the interprofessional showcase 48 journal of effective teaching in higher education, vol. 1, no. 1 post: this event has enabled me to confidently verbalize the role of my profession as it relates to other professions in a healthcare/community setting. 182 1.42 (.615) 4.629 181 .000 pre: my academic preparation has enabled me to clearly explain to others the role of other professionals in a health care/community setting. 182 1.84 (.650) 5.010 181 .000 post: through participation in this event, i am able to clearly explain to others the role of other professionals in a healthcare/community setting. 182 1.55 (.571) pre: i would need guidance to effectively interact with other healthcare professionals to mutually address complex issues in a healthcare/community setting. 182 1.77 (.682) 5.284 181 .000 post: this event has prepared me to successfully interact with other healthcare professionals to mutually address complex issues in a healthcare/community setting. 182 1.43 (.607) pre: my academic experiences have prepared me to identify the benefits of working in interprofessional teams. 182 1.60 (.695) 4.073 181 .000 post: this event has prepared me to identify the benefits of working in interprofessional teams. 182 1.37 (.577) pre: my academic experiences have enabled me to identify the challenges of working in interprofessional teams. 182 1.85 (.669) 6.007 181 .000 post: this event has prepared me to identify the challenges of working in interprofessional teams. 182 1.49 (.592) pre: i have had the appropriate academic preparation to work as a member of an interprofessional team in a clinical rotation, internship or job. 182 1.81 (.714) 5.364 181 .000 post: this event provided the academic preparation to work as a member of an interprofessional team in a clinical rotation, internship, or job. 182 1.46 (.582) p < .05 the interprofessional showcase 49 journal of effective teaching in higher education, vol. 1, no. 1 likert scale 1=strongly agree, 2=agree, 3=disagree, and 4=strongly disagree the ipsce was successful in bringing together students across the chp and a group of family studies students from the college of liberal arts to participate in a variety of work spaces and engage in interprofessional discussion. activities in the showcase spaces introduced participants to practice areas not typically integrated into their previous academic experiences at the university. students actively learned about the roles other professionals have in the workplace and how their professions might be integrated into these settings. the micro-vignette discussions allowed participants to share their expertise with other health professional students. pre-post survey results showed that participants gained a better understanding of their professional roles as a result of the event. verbalizing their professional roles to others increased this understanding as evidenced by the change in attitude before particpation in the ipsce mean 1.73 (sd = 0.698), to the after participation mean 1.42 (sd = 0.615). these findings are consistent with turrentine et al. (2016) who found that nursing and medical students had increased knowledge of their professional roles and could communicate this information better after participation in an ipe activity. additionally, the sharing in the showcase spaces and interprofessional discussion increased participants’ understanding of other professional roles. students’ attitudes shifted more towards strongly agreeing, mean 1.55 (sd = 0.571), that they could clearly explain the role of other professions after ipsce participation than they could before the event, mean 1.84 (sd = 0.65). information gleaned from each showcase space was built upon in the interprofessional micro-vignette group discussions. observing other professionals in simulated work environments, participating in these environments, and solving the micro-vignette problems with other pre-professionals assisted students in gaining a broad understanding of other professional programs in the college. these findings agree with those of wilson et al. (2016, 2017) who found that student teachers and speech-language therapy students who participated in an ipe event and shared professional practice placements were better able to understand the other’s professional role and, consequently, increased their competencies for collaborative practice. prior to the event, participants agreed that they would need guidance, mean 1.77 (sd = 1.43), to interact with other professionals in the health care environment. after the event, students more strongly agreed that they were better prepared to work in interprofessional groups, mean 1.43 (sd = 0.607), because of participation in the ipsce. being able to express their professional roles more confidently and understand the roles of other health professionals made participants feel more prepared to interact with colleagues to address complex issues in healthcare and community settings. students enjoyed the opportunity to provide their professional expertise and benefited from what they learned about other professional roles. similarly, gould, lee, berkowitz and bronstein (2015) found that medical and social work students valued the opportunity to learn from one another in a collaborative ipe activity. their collaborative approach to problems provided an enhanced understanding of patient needs and resulted in improved care. the interprofessional showcase 50 journal of effective teaching in higher education, vol. 1, no. 1 as noted previously, improved outcomes are important in a variety of work settings in addition to health care. the benefits of interprofessional collaboration were noted by dobbs-oates and morris (2016) in improving prek-12 student outcomes particularly as it related to the education of pre-service special educators and school counselors. they noted that interprofessional collaboration among a variety of professionals is essential to school effectiveness and student well-being and emphasized the importance of ipe in the training of these professionals. additionally, the importance of ipe was emphasized by tarr et al. (2013) who developed face to face education in teacher training that incorporated interprofessional instruction involving education, health, police, and social work disciplines to ensure effective provision of child protection services. interprofessional discussion of the micro-vignettes brought out a variety of perspectives from individual professions and resulted in creative problem solving. differing opinions and perspectives did not dampen student enthusiasm regarding the benefits of interprofessional collaboration. instead they increased their understanding of these benefits from pre-event, mean 1.60 (sd = 0.695), to postevent mean, 1.37 (sd = 0.577). a decrease in these means indicated a stronger trend toward strongly agreeing that there are benefits to interprofessional collaboration. likewise, student attitudes regarding the challenges of interprofessional collaboration moved in a more positive direction from prior to the ipsce mean, 1.85 (sd = 0.669), and after the event mean, 1.49 (sd = 0.592), through the interprofessional discussions and activities provided in the ipsce. understanding both the benefits and challenges of interprofessional collaboration was an important goal of this event because it allowed students to acquire a better understanding of both the positive and potentially negative situations that could develop in collaborative team work settings. students felt that this event provided them with an introductory academic experience to enhance their ability to work as a member of an interprofessional team. their attitudes toward this preparation shifted in a more favorable direction from before the event mean, 1.81 (sd = 0.714), to the post-event result mean, 1.46 (sd = 0.582). as previously mentioned, the participants had no formal participation in an ipe event at this university. therefore, the changed attitude after event participation accomplished an intended outcome of increasing student self-reported confidence in their ability to work in interprofessional teams. reeves et al. (2012) suggested that effective ipe should provide an interactive learning approach. the ipsce provided a combination of observation, simulation, interaction, and interprofessional discussion and could serve as a model for ipe in other pre-professional programs. similarly, arora and arora (2015) engaged business students in an interactive, interdisciplinary problem-solving game combining the above characteristics, which resulted in increasing their understanding of the nuances of supply chain, marketing, and advertising disciplines. active participation in simulated events and problem solving as a team had similar benefits to students trying to understand complex business situations as health professionals trying to solve complex patient problems. the interactive and experiential elements of these learning events increased student confidence to work as a part of an interprofessional team in clinical rotations, internships, and the interprofessional showcase 51 journal of effective teaching in higher education, vol. 1, no. 1 professional positions. it is imperative that students have exposure to similar events to prepare them for future professional roles. limitations the results and observations obtained in this study are specific to the students who participated in the ipsce. it is possible that these students have similar characteristics to other health care professional students or other pre-professional students. however, these students represent a small subset of chp students and of health professional students in general and results may not be the same with different student populations. additionally, chp programs were not equally represented by participants in the ipsce. therefore, the results were influenced by the programs that had a larger representation. evaluation of attitudes related to interprofessional collaboration were taken immediately following the event. therefore, follow-up assessments would be necessary to determine if attitude change was maintained over time. conclusion the goal of ipe is to develop professionals that are prepared to participate in collaborative work environments. the ipsce provided chp students with an introduction to and simulation of this work environment. through event participation, students communicated effectively with their colleagues in other preprofessional programs. this communication and problem-solving increased confidence in their own role and their understanding of other professional roles. ipsce participation increased students’ understanding of interprofessional collaboration and improved their self-assessment of their ability to work as a member of an interprofessional team. as an introductory event, the ipsce was instrumental in setting the stage for future interprofessional training opportunities both inside and outside the classroom. the attitude change that resulted from ipsce participation was important because it allowed students to appreciate the uniqueness of their professional role and increased their understanding of other professional roles. the event also provided participants with an appreciation of the value of interprofessional collaboration in the workplace. in doing so, the ipsce provided an initiation event for ongoing development of student interprofessional competencies throughout chp. additional study is needed to determine the most effective methods to further develop and reinforce these competencies by incorporating ipe both inside and outside the academic classroom and professional placement settings at our institution. the need for interprofessional collaboration in health and other professions will only increase in the future. nandan and london (2013) suggested that the global economy and information age have increased the complex and dynamic nature of problems faced by communities and nations. they have, therefore, advocated for developing interprofessional competencies for students in both undergraduate and graduate education programs. nandon and london (2013) indicated that “most professionals (e.g. social work, accountants, engineers, economists, nurses, physicians, lawyers, and teachers) regularly work with individuals from different professional fields. the complexity of issues they address the interprofessional showcase 52 journal of effective teaching in higher education, vol. 1, no. 1 necessitates cooperation and collaboration with diverse professionals” (p. 819). therefore, it is vital that educators in all pre-professional programs identify, develop, and implement effective ipe in the academic setting to improve student understanding of other professional roles and enhance future interprofessional collaborative practice. conflicts of interest the author(s) declare(s) that there is no conflict of interest regarding the publication of this article. references abu-rish, e., kim, s., varpio, l., malik, e., craddick, k., blondon, k., & zierler, b. (2012). current trends in interprofessional education of health sciences students: a literature review. journal of interprofessional care, 26(6), 444– 451. doi:10.3109/13561820.2012.715604 american interprofessional health collaborative. (2015). american interprofessional health collaborative transcending boundaries: transforming learning. retrieved from https://aihcus.org/storage/documents/aihc%20case%20statement%2011%2030%201 1.pdf anderson, e. m. (2013). preparing the next generation of early childhood teachers: the emerging role of interprofessional education and collaboration in teacher education. journal of early childhood teacher education, 34(1), 23–35. doi:10.1080/10901027.2013.758535 arora, a., & arora, a. s. (2015). “supply chain—marketing shark tank” experiential lab game in interdisciplinary business education: qualitative and quantitative analyses. decision sciences journal of innovative education, 13(1), 21–43. doi:10.1111/dsji.12053 association of american medical colleges. (2012, september). changing delivery system increases focus on interprofessional education. retrieved from https://www.aamc.org/ association of american medical colleges (2016, september) interprofessional learning prepares students for team-based practice. retrieved from https://news.aamc.org/medical-education/article/interprofessional-learningprepares-team-practice/ charles, g., bainbridge, l., & gilbert, j. (2010). the university of british columbia model of interprofessional education. journal of interprofessional care, 24(1), 9–18. doi:10.3109/13561820903294549. curran, v. r., heath, o., kearney, a., & button, p. (2010). evaluation of an interprofessional collaboration workshop for post-graduate residents, nursing https://news.aamc.org/medical-education/article/interprofessional-learning-prepares-team-practice/ https://news.aamc.org/medical-education/article/interprofessional-learning-prepares-team-practice/ the interprofessional showcase 53 journal of effective teaching in higher education, vol. 1, no. 1 and allied health professionals. journal of interprofessional care, 24(3), 315– 318. doi:10.3109/13561820903163827. d'eon, m. (2005). a blueprint for interprofessional learning. journal of interprofessional care, 19(1), 49–59. doi:10.1080/13561820512331350227. dobbs-oates, j., & morris, a. w. (2016). the case for interprofessional education in teacher education and beyond. journal of effective teaching, 42(1), 50–65. doi:10.1080/02607476.2015.1131363 gould, k. day, k. h., barton, a. t. (2017) changing student attitudes through interaction: findings from an interprofessional workshop. journal of interprofessional care, 31(4), 540–542. doi:10.1080/13561820.2017.1287165 gould, p. r., lee, y., berkowitz, s., & bronstein, l. (2015). impact of a collaborative interprofessional learning experience upon medical and social work students in geriatric health care. journal of interprofessional care, 29(4), 372–373. doi:10.3109/13561820.2014.962128 johnson, a. f., & graves, c. w. (2017). what makes us who we are? investigating the chemistry behind genetics in an interdisciplinary course for undergraduate students. part i. journal of college science teaching, 46(3), 19–27. lattuca, l. r., knight, d. b., ro, h. k., & novoselich, b. j. (2017). supporting the development of engineers' interdisciplinary competence. journal of engineering education, 106(1), 71–97. doi:10.1002/jee.20155 macdonnell, c. p., rege, s.v., misto, k., dollase, r., & george, p. (2012). an introductory interprofessional exercise for healthcare students. american journal of pharmaceutical education, 76(8), 1–6. retrieved from http://www.ajpe.org/doi/full/10.5688/ajpe768154 nandan, m., & london, m. (2013) interdisciplinary professional education: training college students for collaborative social change. education + training, 55(8/9), 815–835. retrieved from https://doi.org/10.1108/et-06-2013-0078 reeves, s., tassone, m., parker, k., wagner, s. j., & simmons, b. (2012). interprofessional education: an overview of key developments in the past three decades. work, 41(3), 233–245. doi:10.3233/wor-2012-1298 self, j. a., & baek, j. s. (2017). interdisciplinarity in design education: understanding the undergraduate student experience. international journal of technology and design education, 27(3), 459–480. doi:10.1007/s10798-0169355-2 stacey, d., brière, n., robitaille, h., fraser, k., desroches, s., & légaré, f. (2014). a systematic process for creating and appraising clinical vignettes to illustrate interprofessional shared decision making. journal of http://www.ajpe.org/doi/full/10.5688/ajpe768154 https://doi.org/10.1108/et-06-2013-0078 the interprofessional showcase 54 journal of effective teaching in higher education, vol. 1, no. 1 interprofessional care, 28(5), 453–459. doi:10.3109/13561820.2014.911157. tarr, j., whittle, m., wilson, j., & hall, l. (2013). safeguarding children and child protection education for uk trainee teachers in higher education. child abuse review, 22(2), 108–115. doi:10.1002/car.2275 turrentine, f. e., rose, k. m., hanks, j. b., lorntz, b., owen, j. a., brashers, v. l., & ramsdale, e. e. (2016). interprofessional training enhances collaboration between nursing and medical students: a pilot study. nurse education today, 4033–4038. doi:10.1016/j.nedt.2016.01.024 villadsen, a., allain, l., bell, l., & hingley-jones, h. (2012). the use of role-play and drama in interprofessional education: an evaluation of a workshop with students of social work, midwifery, early years and medicine. social work education, 31(1), 75–89. doi:10.1080/02615479.2010.547186 wilson, l., mcneill, b., & gillon, g. t. (2016). a comparison of inter-professional education programs in preparing prospective teachers and speech and language pathologists for collaborative language–literacy instruction. reading and writing, 29(6), 1179–1201. doi:10.1007/s11145-016-9631-2 wilson, l., mcneill, b., & gillon, g. t. (2017). inter-professional education of prospective speech-language therapists and primary school teachers through shared professional practice placements. international journal of language & communication disorders, 52(4), 426–439. doi:10.1111/1460-6984.12281 world health organization. (2010). framework for action on interprofessional education and collaborative practice. retrieved from http://www.who.int/hrh/resources/framework_action/en/ conflicts of interest the author(s) declare(s) that there is no conflict of interest regarding the publication of this article. http://www.who.int/hrh/resources/framework_action/en/ microsoft word 66-article text-855-1-6-20210407.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.66 creating project-based learning for online art classrooms alice lai, suny empire state college, alice.lai@esc.edu abstract. project-based learning (pbl) is considered an engaging and promising pedagogy across diverse disciplines and student populations in the united states in the digital age. research on pbl in online environments and in the field of art has, however, been limited. thus, the purpose of the current study was to examine the theoretical grounds of pbl and analyze its pedagogical features and application. of specific interest is krajcik and shin’s (2014) pbl model, which includes six pedagogical features: asking driving questions, emphasizing learning goals, fostering authentic practice, enhancing collaboration, learning with technology, and creating artifacts. an account of a qualitative descriptive pilot study conducted in an undergraduate online art course further explicates the application of this pbl model. the pilot study accentuates ways to create an online pbl art classroom and illustrates students performing pbl through their arts-based research projects. reflection on pedagogical recommendations and the challenges of implementing pbl in the online art classroom concludes the study. keywords: project-based learning, online art class, arts-based research recent national surveys have corroborated the exponential growth of fully or blended online courses in u.s. higher education (ginder et al., 2018; seaman et al. 2018). college and university faculty are increasingly expected to teach courses in online environments. this development has motivated researchers and educators in online education to conduct theoretical and experiential inquiries to identify, implement, and evaluate online pedagogies (alexander et al., 2019; anderson, 2008). these researchers’ underlying aspirations have been to create a robust and engaging online learning environment supporting 21st-century students’ learning preferences and cognitive development. marc prensky (2001), best known for popularizing the terms “digital native” and “digital immigrant,” argued that digital native students’ learning preferences and knowledge construction processes differ from those of their digital immigrant professors. small and vorgan’s (2008) cognitive neuroscience and neuroimagingbased research further provided insights into digital natives’ mental functioning. the researchers speculated that daily absorption of and interaction with the new technologies from social media, virtual interactive entertainment, and the internet have stimulated neural circuitry in human brains resulting in certain kinds of social and intellectual abilities and patterns. by studying and comparing neural circuitry between the digital natives and digital immigrants and the effect of social media on digital natives’ neural pathways in the brain and cognitive processes, they discovered that digital natives simultaneously acquire and (re)construct knowledge socially, cognitively, and technologically. creating project-based learning for online art classrooms 95 journal of effective teaching in higher education, vol. 4, no. 1 educators understand that technology plays an essential role in facilitating 21stcentury students’ learning, yet prensky (2010) observed a mindset in which concerned educators tend to worry about their ability to catch up with new technologies and to believe they need additional training in how to use new technologies for meaningful teaching. he offered some advice, proposing that the challenge facing 21st-century teachers is to worry less about the details of new technology and instead become comfortable with a different and better kind of pedagogy that enables students to build educational partnerships with or through technology and to use technology to actively enhance their own learning. according to prensky, professors—especially those who teach nontechnology-related subjects—do not need to become technology “experts” in order to teach effectively. instead, they should endeavor to identify and implement a pedagogy that promotes students to partner with technology to accomplish pedagogical objectives. in recognition of prensky’s (2001, 2010) and small and vorgan’s (2008) assertions, the purpose of the current study was to introduce a pedagogy that corresponds to 21st-century digital learners’ cognitions, technological conditions, and ways of knowing while addressing a significant research gap in college online art education. this pedagogy, called project-based learning (pbl), is appreciated as a progressive educational model and a promising pedagogy in the digital age (condliffe et al., 2017). although numerous pbl researchers have considered technology indispensable in creating a pbl environment, research on pbl in online classrooms and at the postsecondary level is limited; moreover, in the field of art, research focusing on online teaching and pedagogy has been sparse. thus, conducting additional case studies on the implementation of pbl in online classrooms is necessary to assist professors in the arts, art history, and art education in designing robust online curriculum. consequently, the aim of this article was to present a theoretical synopsis and a pilot case study of pbl, emphasizing the following topics: (a) the theoretical foundations and six pedagogical features of pbl, (b) the application of pbl in an online art appreciation course illustrated by the student projects, and (c) lessons learned from the current study. theoretical grounds of project-based learning pbl has been continuously theorized, applied, and revised by a number of researchers and institutions across diverse disciplines and student populations (boss & krauss, 2014; boss & larmer, 2018; condliffe et al., 2017; dierker et al., 2012; hunter-doniger, 2018; krajcik et al., 1998; krajcik & shin, 2014; larmer et al., 2015; laur, 2013; shiraz & larsari, 2014; vaz et al., 2013). this has resulted in various versions of pbl, each presenting certain features corresponding to the needs of the discipline and the cognitive level of student population. nevertheless, according to pbl researchers and practitioners (e.g., hunter-doniger, 2018; krajcik & shin, 2014), pbl can be generally defined as a pedagogical praxis promoting a student-centered, inquiry-based, and systematic approach to learning, where students actively engage in the construction of their own knowledge by devoting considerable time to devising projects to probe and propose solutions to real-world problems relevant to students’ lives. the foundation of pbl derives from four learning sciences ideas: (a) active construction, (b) situated learning, (c) social creating project-based learning for online art classrooms 96 journal of effective teaching in higher education, vol. 4, no. 1 interaction, and (d) cognitive tools (krajcik & shin, 2014). as elucidated below, these ideas are embedded in the models of learning and developmental science theorized by john dewey, jean piaget, and lev vygotsky. dewey (1938, 2001) aligned his educational theories with a philosophy of pragmatism and a profound belief in democracy. the investigating and proposing solutions to real-world problems, situated learning, and student-centered projects emphasized in pbl reflect dewey’s educational theories. dewey argued that curriculum and classroom activities should be relevant to learners’ lives and lead to the development of practical skills and knowledge that one can use in real life. proposing an educational process of learning by doing, dewey believed that meaningful learning requires learners to take an active role and immerse themselves in the process of their own learning. he advocated classroom democracy, in which teachers assume the role of facilitator and guide to help create both a learning environment and process that enable students to achieve contextualized, experiential, real-world, and active learning. in dewey’s view, learning involved experiential, social, and interactive processes through which knowledge does not merely passively transmit from teacher or book to students but is actively and collaboratively constructed by students. in piaget’s (1968) theory of cognitive development, experience-based learning and the learner’s interaction with the external environment are recognized as significant components in a child’s acquisition of knowledge. this theory contributes to the real-world-based learning and the use of cognitive tools in the knowledge construction process as embraced by practitioners of pbl. a child’s cognitive development occurs through continuous internal mental coordination as a result of biological adaptation and maturation as well as external interaction with the environment (gallagher & reid, 1981/2002). to piaget, knowledge was not simply a piece of information to be delivered and received; knowledge is acquired through experience and a set of structures progressively constructed by interaction between the learner and the external environment. this constructivist approach suggests the learner’s active building up of knowledge and cognitive process. growth in knowledge can, moreover, be stimulated by a humanor technology-generated feedback process followed by questions, contradictions, and consequent mental adaptation and reorganization of existing units of knowledge into new and more complex intellectual structures (cockcroft, 2009). vygotsky’s (1978) theory of social constructivism, probing the social aspect of the learning process, provides a framework to appreciate the social interaction and collaboration—hallmarks of pbl—in the learner’s active knowledge construction process. vygotsky believed that cognitive development occurs first on a social level rather than at the individual’s internal level as proposed by piaget. thus, according to vygotsky’s social constructivist learning theory, meaningful learning that can stimulate greater cognitive growth or lead to lifelong retention of knowledge occurs when learners are engaged in social activities such as interaction and collaboration with other people (amineh & asl, 2015). vygotsky further recognized the zone of proximal development (zpd) as a key aspect in his learning theory: the zpd is “the distance between the actual developmental level as determined by independent creating project-based learning for online art classrooms 97 journal of effective teaching in higher education, vol. 4, no. 1 problem solving and the level of potential development as determined through problem solving under adult guidance or in collaboration with more capable peers” (p. 86). this indicates that meaningful learning occurs in a guided and collaborative environment where someone or something more knowledgeable or at a higher level of cognitive capacity regarding a particular task, process, or concept is present to help learners to reach the zpd (hunter-doniger, 2018; rogers & sun, 2018). knowledgeable guidance can come from a teacher, a peer, an electronic tutor, or an electronic learning program. features of project-based learning through extensive research on and constant refinement of project-based learning activities, krajcik and shin (2014) identified six key pedagogical features essential in project-based learning environments. meanwhile, condliffe et al. (2017) compiled additional activities recommended by various researchers in pbl. features and activities of pbl, as delineated in this section, can provide online art professors ways to create a holistic project-based learning environment. driving questions pbl begins with asking substantial and challenging questions that are (a) “feasible” in that students can practically design and perform investigations on the questions and find solutions to the questions; (b) “worthwhile” in that students can perceive the impact of their projects in the real world; (c) “contextualized” so that the questions are relevant to students’ lives and echo professional concerns in realworld situations; (d) “meaningful” so that students can feel invested and motivated to address the questions; and (e) “ethical” in that students adhere to the ethical standards of the discipline (five keywords drawn from krajcik & shin, 2014, p. 281; descriptions added). focusing on learning goals the purpose of pbl is to facilitate students’ achievement of the curricular objectives and standards set forth by the discipline. thus, the professor necessarily guides students in designing their projects, resulting in the acquisition of disciplinary knowledge, skills, and cognitive abilities reflecting the standards of the discipline, ranging from specific content knowledge to professional guidelines to academic skills. engaging in authentic and discipline-based practices krajcik and shin (2014) experimented with pbl in science classrooms and have identified a feature they called “engaging in scientific practices” (p. 276). because pbl has been successfully applied to a variety of disciplines, the name of this feature has been modified by the author of this article to capture both krajcik and shin’s principle and a principle promoted by other pbl researchers (condliffe et al., 2017; hunter-doniger, 2018; laur, 2013). this feature primarily emphasizes the authentic inquiry students should undertake by adhering to discipline-based or creating project-based learning for online art classrooms 98 journal of effective teaching in higher education, vol. 4, no. 1 professional processes, tools, and quality standards or by making a real-life impact through end products of their projects. hunter-doniger (2018) recommended several examples of authentic and arts-based inquiries ranging from “making scientific observations through painting” to “sketching animals and hypothesizing environmental issues” (p. 50). collaboration students collaborate with peers, instructors, or community members to generate ideas and find answers to the driving questions. collaboration enables students to be guided by others who are more knowledgeable in the subject matter or possess stronger cognitive capacity through which students can acquire additional ideas, methods, or competencies to respond to the driving questions. collaboration also resembles how professionals work together in real-world situations to solve problems. for example, art and art education professors acknowledging the technosocial realities of digital learners have employed such technologies as web 2.0 and the three-dimensional virtual world to sustain collaborative activities to enhance students’ arts-based research processes (sweeny, 2010). using technology tools to support learning while embarking on the inquiry process, students use technologies to assist and improve their learning. the professor and students should select technology primarily for educational—not only social, motivational, or entertainment— purposes. technology can include cognitive tools that provide information, facilitate students’ investigations of driving questions, or are required in the discipline. online art professors have documented various student projects, highlighting the promising aspects of learning with technologies. examples include students employing google earth to observe architectural details, using the internet to find information, perusing artworks in the prominent virtual museums, using free online art software to create digital art, or applying a teleconferencing tool to share data and work on projects with others at a distance (boss & krauss, 2014; holland, 2006; lai, 2020; sweeny, 2010). creation of artifacts students create tangible products as the solutions or representations of the solutions to the driving questions. this feature is particularly aligned with an artmaking requirement in the discipline of art. the products can be presentations, scientific models, artwork, short stories, digital games, or web sites, allowing students to publicly demonstrate their acquired knowledge, skills, or answers to the driving questions. through the artifact creation process, researchers in pbl (condliffe et al., 2017; hunter-doniger, 2018; lai & cooper, in press) also believe that students are able to enhance their cognitive abilities by following the principle of learning by doing and gain experiential knowledge of the materials and processes used in the discipline. creating project-based learning for online art classrooms 99 journal of effective teaching in higher education, vol. 4, no. 1 in a review of pbl literature, condliffe et al. (2017) uncovered additional, finer approaches to pbl: considering that students may be unfamiliar with pbl pedagogy and may be reluctant to propose driving questions, a warm-up “anchoring event” (krajcik & shin, 2014, p. 282) or an introduction is recommended to ease students into the pbl environment. student voice and choice grant learners some degree of autonomy and responsibility in deciding what driving questions, working processes, and products are meaningful, relevant, real, and appropriate to them (larmer et al., 2015). a comprehensive pbl program places projects as central, not peripheral, to the curriculum; hence, it requires sustained project cycles, including ongoing presentations of the project, self-assessment, reflection, feedback, and revision (parker et al., 2013). consequently, professors need to allocate ample time for students to complete project cycles. when feasible, students showcasing their end products to the targeted audience beyond the classroom is recommended; this is believed to strengthen students’ motivation and commitment to devise and complete realistic and professional projects. the practices of project cycles and public presentation have a long history in art classrooms reaching back to the 16thcentury sculpture and architecture schools (larmer et al., 2015) as well as in contemporary digital art studios (sweeny, 2010). finally, in designing a pbl environment, the professor assumes a role as a guide, facilitator, project sounding board, or project codesigner and endeavors to create opportunities for students to fully perform pbl processes. designing project-based learning in the online environment to experiment with pbl in the online environment, a qualitative descriptive pilot study was conducted in an undergraduate online art course. qualitative descriptive approaches have been adopted by clinical and classroom researchers to analyze and present data (lambert & lambert, 2012; nassaji, 2015). the goal of a qualitative descriptive study is not usually to generate a theory or prove a hypothesis but to explain phenomena. a descriptive study is “more concerned with what rather than how or why something has happened” (nassaji, 2015, p. 129); furthermore, such study should be mindfully selected “when a straightforward description of a phenomenon is desired” (lambert & lambert, 2012, p. 256). for example, a descriptive study can be highly relevant and useful when a need exists to explore and summarize what has occurred in a classroom and an educational process in a detailed manner and when the educational process involves instruction and instructional materials that are less known, new, or interesting to the targeted audience. to this end, a descriptive account of the pilot study has been explicated here with focus on the pedagogical features, learning activities, and course structure. the pilot study was implemented in an online undergraduate women’s art history course in fall 2017. it was a pioneer trial conducted in the author’s department where more than two dozen online art courses were offered annually. to shed light on ways to create an online pbl art course, a descriptive qualitative method was deemed necessary. the author was the professor and researcher, who had taught this online course for eight years. the course had been developed as a general education course and attracted a large number of students majoring in subjects creating project-based learning for online art classrooms 100 journal of effective teaching in higher education, vol. 4, no. 1 other than art. during the pilot study, the author modified the original curriculum and transformed it into a pbl model. the majority of 20 enrolled students had previously taken at least one online course, but they lacked experience with online pbl art courses. the women’s art history online course was delivered through a course management system called moodle. it comprised three content-oriented workshops surveying women’s art in three historical periods. instead of typically instructing the students to complete a range of assignments on different topics, the revised online course required each student to carry out only one arts-based research project throughout the entire 15-week long semester. arts-based research prompts students to examine or create artworks that “raise awareness and foster critical consciousness by presenting alternative, emotional, evocative, attention-grabbing and/or resistive stories, images, and performances [and communicate] information about the experiences associated with differences, diversity, and prejudice” (leavy, 2020, p. 25). informed by this principle, students were expected to devise and conduct their own arts-based research projects to investigate real-world problems impacting women’s lives. in pbl, instructors need to create an environment where students’ art projects are central to the course (hunter-doniger, 2018). thus, a workshop entitled “research project” devoted to students’ arts-based research projects was added to the online course. the title of and link to the workshop were placed on the course homepage so that students could immediately access the workshop and activities connecting to their and their peers’ projects. this workshop encompassed a series of learning forums, each dedicated to a specific set of related activities helpful in encouraging students to follow pbl features to complete their projects. for example, the first forum, “research project description,” provided a description of the requirements and timeline of the project. students were instructed to visit the forum and to ask questions about the requirements when the semester began. each of the subsequent forums led the students to participate in specific pbl activities: (a) brainstorming project ideas including driving questions, (b) identifying resources and compiling a bibliography, (c) peer reviewing and discussing work-in-progress, and (d) submitting the final project. in addition, a learning forum entitled “gallery talk” was created for the students to submit and discuss their creative products, which had contributed to their research projects. the “research project” workshop along with the associated pbl activities accentuated several key components noted in the literature review above. the configuration of the workshop was aimed at providing the students with a concrete and systematic way to accomplish project cycles consisting of ongoing peer reviews and discussions of the project, assessments, and revisions. because the students were expected to periodically visit and comment on project ideas, resources, and works-in-progress from beginning to end, the configuration made possible a collaborative learning and research community. the students’ learning process was scaffolded with technologies and tangibly manifested in multiple ways, including learning with the moodle classroom tools and employing the digital tools in the art disciplines to create their projects. for example, the students were able to use the creating project-based learning for online art classrooms 101 journal of effective teaching in higher education, vol. 4, no. 1 asynchronous and synchronous communication apparatus to view and discuss their projects, access the reliable image repositories via the college’s online library, and stroll around selected virtual museums via the hyperlinks embedded in the discussion forums. the students could create digital art, complying with the professional standard to upload their artwork-in-progress to the designated learning forums and receive peers’ and professor’s timely feedback. students performing project-based learning the six pbl features are used as a framework in this section in which the projectbased learning activities are enumerated and illustrated by two students’ examples of arts-based research projects examining the issues around (a) procreation and (b) employment inequality in the artworld and global farming practices, all from a critical gender standpoint. driving questions a set of overarching questions reflecting the course subject of women’s art history was initiated by the author-professor to guide the students in brainstorming their own questions: what was or is one gender-based issue in society that has impacted your life? how have artists addressed the issue through a creative means? class discussion of these questions served as a warm-up anchoring event to entice students to reflect on the real-world controversies they deemed relevant to their lives. for example, based on personal experience and concerns of gender norms in procreation, one female student majoring in nursing questioned the social, physical, and psychological consequences of infertility and explored how frida kahlo (1907– 1954), a renowned mexican painter in the u.s. art world, depicted her struggles with procreation in her paintings. another female student majoring in art and perceiving herself as a victim of patriarchal society was concerned about the prevalent issues of the gender inequality women in various professions and times have experienced. she investigated the lives of the renaissance artist artemisia gentileschi (1593–c. 1656) and 21st-century farmers, exploring such questions as the following: what equal employment challenges have women encountered in the history of art? what financial and agricultural challenges do female farmers, especially in third world countries, face today? how might artists help to shine a light on the issues of misogyny in the artworld and society? focusing on learning goals learning goals included several curricular requirements set forth by the school of arts and humanities, which obligate students to demonstrate an understanding of a selected art form and its creative process; study artistic practice in a variety of social and cultural contexts (e.g., gender-based, identity-based, and location-based contexts); exercise art-based research skills by using virtual libraries and art databases; and present a robust analysis of artwork created personally or by other artists. students were asked to engage in discussion with peers and to offer constructive feedback about how their projects might address the learning goals during the brainstorming phase. creating project-based learning for online art classrooms 102 journal of effective teaching in higher education, vol. 4, no. 1 engaging in authentic and discipline-based practices the students used textbooks and other reliable art and scholarly resources to learn about different ways contemporary artists confront, portray, or publicize genderbased issues through art, enabling them to expand their queries of professional art mediums, artworks, and creative processes. applying newly acquired knowledge, they created artistic products to demonstrate what they had learned and their responses to the driving questions. these activities fostered students’ commitment to professionalism and authentic arts-based research. for example, the two students who investigated the controversies associated with procreation and the employment inequality in the artworld and global farming communities felt a sense of genuineness and importance about their projects when they perceived that their end products—artworks and research reports—may have a practical impact in raising awareness of misogyny related to parenthood in real life or empowering women in various professions who may be reluctant to point out occurrences of gender inequality to speak up. collaboration as noted above, the online workshop facilitated students’ partaking in the required and continual collaboration with peers. students posted and discussed their workin-progress with peers throughout the semester. upon receiving feedback from the author-professor and peers, students had opportunities to improve their projects. because students’ driving questions were often derived from personal yet common experiences in contemporary society, some students tapped into people outside the classroom to generate data for their projects. for instance, the student looking into the matter of procreation had collected information from health professionals and other women who shared the concern. in contrast, the student investigating employment inequality in farming practices received helpful feedback from a peer in the course. this classmate had female friends in farming business and offered additional information comparing female and male farmers’ roles in the farming business and noting gender-based pay disparity. using technology tools to support learning the most frequently used components provided by moodle, the course management system, included the text-based asynchronous online discussion tools and large-size (50 mb) audio and visual file uploading and receiving capacity. these enabled the students to engage in discussions about their projects, view or listen to peers’ end products, and revisit comments made by self and others. students’ learning performance and progress were documented in the online course, which was helpful for them to continue reexamining and revising their projects. the students were strongly encouraged to use the college’s online library, the internet, and social media to find information, study current news, and gather ideas from people outside the course regarding art, social issues, and women’s lived experiences locally, nationally, and globally. moreover, the students researched prominent artists and artworks through arts databases online, including reputable virtual museums, image repositories, and open educational sources. ultimately, they creating project-based learning for online art classrooms 103 journal of effective teaching in higher education, vol. 4, no. 1 created end products employing a variety of digital tools, such as digital cameras, professional digital design software, and other image manipulation tools. these technologies not only served as learning tools to permit rigorous research and project cycles but are also the required tools professional artists use in the 21stcentury artworld. creation of artifacts students’ end products included a plethora of research reports and artworks demonstrating what they had learned and their responses to the driving questions. for example, the nursing student who undertook research on procreation and infertility created a digital photo collage, consisting of an infant, surrounded by a series of pictures of medical equipment, needles, and charts. she lamented: “the patriarchal norm associating women with fertility, along with the innate desire to conceive a child, cause women suffering from infertility to experience a multitude of all-consuming emotions and a sense of incomplete womanhood” (personal communication, december 11, 2017). her photo collage and research echoed the theme of frustrated motherhood in surrealist artist frida kahlo’s paintings and life. by incorporating realistic images and medical objects like an arm showing an intravenous line, medical tools, and a picture of infant in the photo collage, the student also exhibited an understanding of kahlo’s artistic methods and the feminist approaches embedded in her paintings. in peer discussions, several female students shared the sentiment revealing perpetual gender norms that have led them to continue to be burdened by the obligation to fulfill motherhood through biological childbearing. they further expressed that they have also resorted to discretely enduring the psychological and physical pain caused by the infertility treatments rather than following alternative ways of achieving motherhood. the art student who explored the gender-based challenges of equal employment opportunity in the artworld and farming communities created a poster-like graphic artwork. she learned that both worlds shared similar misogynistic practice and that artist artemisia gentileschi had employed art to display women’s strength in counteracting the patriarchy. she discovered that u.s. farm policy has resulted in barriers to women’s achievement of financial independence through farming and that male farmers earn substantially more money than female farmers in the u.s. in other countries female farmers work longer hours than men tending farms and completing domestic chores; however, women are still denied access to land rights that are essential for them to sustain a farming business and grow and produce food for their families. these problems can also be seen in the renaissance artworld where female artists were deprived of the necessary resources and the respect they needed to obtain financial and artistic independence. the background of her poster was occupied by three dark-skinned female farmers carrying baskets filled with green products of the harvest on their heads and walking in a field overgrown with tall brown plants. she brought the text “empower women, end world hunger” to the foreground to dominate two thirds of the poster. this poster seems to exude hope for female farmers, suggesting their role in ending world hunger. using the design software adobe photoshop, she manipulated the images and text in a way that resembles professionally designed graphic art. one student praised the quality of creating project-based learning for online art classrooms 104 journal of effective teaching in higher education, vol. 4, no. 1 the artwork and suggested that this end product can bring real-life impact, if “it could be used as a bumper sticker or on some promotional items to advocate the cause” (personal communication, december 14, 2017). two students had effectively performed pbl. their research reports and artistic products incorporated their research findings and responses to the driving questions; displayed a fundamental understanding and application of the elements of art, principles of design, and digital tools used by professional artists and graphic designers; demonstrated their studio art skills; and met the discipline-based learning goals. final reflection and recommendations the primary intent of the current article was to introduce project-based learning and elucidate ways to create a pbl environment in an online art class. aligned with the six features of pbl (krajcik & shin, 2014), this article enumerated pbl activities supporting student arts-based research projects. krajcik and shin’s (2014) pbl model emphasizing the use of technology to support learning, collaborative learning, and the creation of artifacts for demonstrating the learning outcomes appeared to coincide with the digital learners’ cognitive preferences, the online course setting supported by moodle, and the disciplinary and professional requirements in art. moreover, the two students’ performances of pbl, indicating deep learning of their chosen subject matter, corroborate other successful examples showcased in pbl literature (boss & krauss, 2014; condliffe et al., 2017; krajcik & shin, 2014; vaz et al., 2013). thus, this study can provide information that may benefit online art professors. the pilot study revealed that this online class was suitable for pbl. several recommendations can thus be made here. first, in online classrooms, a centralized project space can expediently enable students to share their projects-in-progress and participate in ongoing review and discussion of their own and their peers’ projects as frequently as they wish. professors can require students to do so routinely to help ensure a solid learning community, practice of project cycles, and steady progress. second, the pilot online classroom was configured to include textbased asynchronous discussion forums corresponding to the phases of pbl and large-size audio and visual file uploading and viewing capacity as well as access to online arts databases, social media, and virtual museums. these technologygenerated resources can significantly contribute to successful pbl in online art classrooms. third, to encourage students “partnering” with technology (prensky, 2010), the author-professor welcomed students’ selection of the technology appropriate for their learning projects and preferences. the projects of the nursing and art students revealed that they acquired a range of technologies to conduct research, view artworks online, and create a photo collage and digital art. this echoes pbl researchers’ assertion that project-based learning can strengthen student-centered inquiry processes and students’ ability to become independent learners (boss & krauss, 2014; boss & larmer, 2018; condliffe et al., 2017; laur, 2013; vaz et al., creating project-based learning for online art classrooms 105 journal of effective teaching in higher education, vol. 4, no. 1 2013). forth, online classrooms can offer certain advantages to practice pbl that may not be readily available because of the time and space constraints of face-toface learning environments; for example, online classrooms can be equipped with convenient access and continuous opportunities supporting robust peer collaboration that can turn technology into a core and “natural” learning partner rather than an extra tool or burden in the educational process. challenges pertaining to pbl in online art classrooms should also be recognized. in the pilot study the students were unable to observe each other’s artmaking face-toface and offer immediate feedback on art mediums, styles, and techniques. students who had limited artmaking experience found it intimidating to create artworks in isolation; thus, they tended to express modest self-assessment about the artistic aspect of their end products. in the long run, such a learning experience may diminish their interest and achievement in art. another challenge pointed toward the professor’s role in facilitating student online discussions. although the driving questions and self-selected artworks were impactful and meaningful to the students, their deep investment in personal and private questions at times provoked highly opinion-based and uncritical discussions of individual affairs and feelings. students could become offended, dejected, or silenced if their peers and professor did not express the expected empathy toward them or agree with their sentiment and interpretation of art. to prevent or remedy these online learning situations, using the moodle asynchronous communication tools, the author-professor had assisted individual students in choosing art medium suitable for their artmaking experiences and projects. encouraging students to visually document and discuss with peers their artmaking process was another strategy facilitating students’ artmaking experience. alternatively, professors can take advantage of online videoconferencing tools to arrange synchronous meetings to assist less experienced students in making art. professors can also create a prolonged or inclusive anchoring event, providing students with guidance to critique art, exercise constructive criticism, and undertake critical dialogue about personal and social issues. these instructional strategies may pave ways for more effective discussions of art and social issues in the online classrooms. after all, prensky (2010) pointed out that in the 21st-century digital age, “asking good questions, providing context, ensuring rigor, and evaluating the quality of students’ work” (p. 3) are still important teaching responsibilities. when planned carefully and systematically following the theoretical foundations and pedagogical principles, pbl can be a liberating and promising pedagogy for online art classes. pbl supports the kinds of cognitive inclinations, technology usage, and learning processes favored by 21st-century students. pbl can be effectively implemented in the online environment; however, as online courses continue to grow across different disciplines, large-scale research will be necessary to contribute to the long-term success of pbl online. creating project-based learning for online art classrooms 106 journal of effective teaching in higher education, vol. 4, no. 1 conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references alexander, b., ashford-rowe, k., barajas-murph, n., dobbin, g., knott, j., mccormack, m., pomerantz, j., seilhamer, r., & weber, n. (2019). horizon report: 2019 higher education edition. https://www.learntechlib.org/p/208644/ amineh, r. j., & asl, h. d. (2015). review of constructivism and social constructivism. journal of social sciences, literature, and languages, 1(1), 9‒16. anderson, t. (ed.). (2008). theory and practice of online learning (2nd ed.). athabasca university press. boss, s., & krauss, j. (2014). reinventing project-based learning: your field guide to real-world projects in the digital age (2nd ed.). international society for technology in education. boss, s., & larmer, j. (2018). project based teaching: how to create rigorous and engaging learning experiences. ascd. condliffe, b., quint, j., visher, m. g., bangser, m. r., drohojowska, s., saco, l., & nelson, e. (2017). project-based learning: a literature review. mdrc/lucas education research. https://eric.ed.gov/?id=ed578933 cockcroft, k. (2009). piaget’s constructivist theory. in j. watts, k. cockcroft, & n. duncan (eds.), developmental psychology (2nd ed., pp. 324‒343). uct press. dewey, j. (1938). experience and education. simon and schuster. dewey, j. (2001). democracy and education by john dewey: a penn state electronic classics series publication. pennsylvania state university. dierker, l., kaparakis, e., rose, j., selya, a., & beveridge, d. (2012). strength in numbers: a multidisciplinary, project-based course in introductory statistics. the journal of effective teaching, 12(2), 4‒14. https://uncw.edu/jet/articles/vol12_2/dierker.html gallagher, j. m., & reid, d. k. (2002). the learning theory of piaget and inhelder. brooks cole publishing. (original work published 1981.) creating project-based learning for online art classrooms 107 journal of effective teaching in higher education, vol. 4, no. 1 ginder, s. a., kelly-reid, j. e., & mann, f. b. (2018). enrollment and employees in postsecondary institutions, fall 2017; and financial statistics and academic libraries, fiscal year 2017: first look (provisional data; nces 2019-021rev). u.s. department of education. national center for education statistics. http://nces.ed.gov/pubsearch holland, m. c. g. o. (2006). the virtual school of architecture and design. wit transactions on the built environment, 90. https://www.witpress.com/elibrary/wit-transactions-on-the-builtenvironment/90/17145 hunter-doniger, t. (2018). project-based learning: utilizing artistic pedagogies for educational leadership. art education, 71(2), 46‒51. https://doi.org/10.1080/00043125.2018.1414542 krajcik, j., blumenfeld, p., marx, r. w., bass, k., fredricks, j. r., & soloway, e. (1998). inquiry in project-based science classrooms: initial attempts by middle school students. journal of the learning sciences, 7(3‒4), 313‒350. https://doi.org/10.1080/10508406.1998.9672057 krajcik, j. s., & shin, n. (2014). project-based learning. in r. k. sawyer (ed.), the cambridge handbook of the learning sciences (2nd ed., pp. 275‒297). cambridge university press. https://doi.org/10.1017/cbo9781139519526.018 lai, a. (2020). student-led online discussions: a feminist praxis to mitigate digital gender inequality. in l. wilton & c. brett (eds.), handbook of research on online discussion-based teaching methods (pp. 435‒457). igi global. https://doi.org/10.4018/978-1-7998-3292-8.ch018 lai, a., & cooper y. c. (in press). we are vr girls: becoming engaged citizens through steam-integrated projects. in m. bae-dimitriadis & o. ivashkevich (eds.), engaging youth civic participation: critical approaches to teaching digital media in art classrooms and communities. national art education association/taylor & francis. lambert, v. a., & lambert, c. e. (2012). qualitative descriptive research: an acceptable design. pacific rim international journal of nursing research, 16(4), 255‒256. https://he02.tcithaijo.org/index.php/prijnr/article/view/5805 larmer, j., mergendoller, j. r., & boss, s. (2015). setting the standard for project based learning: a proven approach to rigorous classroom instruction. acsd. laur, d. (2013). authentic learning experiences: a real-world approach to projectbased learning. routledge. creating project-based learning for online art classrooms 108 journal of effective teaching in higher education, vol. 4, no. 1 leavy, p. (2020). method meets art: arts-based research practice (3rd ed.). guilford press. nassaji, h. (2015). qualitative and descriptive research: data type versus data analysis. language teaching research, 19(2), 129‒132. https://doi.org/10.1177/1362168815572747 parker, w. c., lo, j., yeo, a. j., valencia, s. w., nguyen, d., abbott, r. d., nolen, s. b., bransford, j. d., & vye, n. j. (2013). beyond breadth-speed-test: toward deeper knowing and engagement in an advanced placement course. american educational research journal, 50 (6), 1424‒1459. https://doi.org/10.3102%2f0002831213504237 piaget, j. (1968). le point de vue de piaget [piaget’s point of view]. international journal of psychology, 3(4), 281‒99. https://doi.org/10.1080/00207596808246651 prensky, m. (2001). digital natives, digital immigrants part 1. on the horizon, 9(5), 1‒6. https://doi.org/10.1108/10748120110424816 prensky, m. (2010). teaching digital natives: partnering for real learning. corwin. rogers, r. r. h., & sun, y. (2018). engaging stem students from rural areas: emerging research and opportunities. igi global. https://doi.org/10.4018/978-1-5225-6341-9 seaman, j. e., allen, i. e., & seaman, j. (2018) grade increase: tracking distance education in the united states. the babson survey research group. https://eric.ed.gov/?id=ed580852 shiraz, m. p., & larsari, e. e. (2014). the effect of project-based activities on intermediate efl students’ reading comprehension ability. the journal of effective teaching, 14(3), 38–54. https://uncw.edu/jet/articles/vol14_3/poorverdi.html small, g. w., & vorgan, g. (2008). ibrain: surviving the technological alteration of the modern mind. harper collins. sweeny, r. w. (ed.). (2010). inter/actions/inter/sections: art education in a digital visual culture. national art education association. vaz, r. f., quinn, p., heinricher, a. c., & rissmiller, k. j. (2013). gender differences in the long-term impacts of project-based learning. in proceedings of the american society for engineering education annual conference, atlanta, ga. https://peer.asee.org/19648 vygotsky, l. s. (1978). mind in society. harvard university press. microsoft word phan_final3.docx journal of effective teaching in higher education, vol. 3, no. 1 the application of metacognitive note-taking skills in reading lessons to efl college students mi pham kieu phan, binh dinh college (vietnam), miphan86@gmail.com abstract. metacognition has been considered as a key element for successful selfregulated learning. however, it seemed foreign to efl college students. the present study examined students’ changes in metacognitive strategy use in the two conditions: one condition with the intervention of metacognitive note-taking skills (mnt) and one with the conventional teaching approach. fifty-four students were randomly assigned to an experimental group (n=27) and a control group (n=27). each group separately took part in a seven-session instruction outside school time, and then took a pretest and a posttest measuring their awareness level of reading strategy use. nineteen out of the experimental participants individually attended semi-structured interviews, exploring their perceptions of the effect of mnt use while reading. the tests’ results demonstrated that there were significant differences in their recognition of reading strategy use in both conditions. the qualitative findings reported on positive perceptions of the mnt application during reading sessions. also, the participants showed some difficulties in the application of mnt. this study potentially provides an effective and innovative educational tool to enhance students’ academic learning and their lifelong learning as well. keywords: metacognitive strategies, metacognitive note-taking, self-questioning, self-regulation metacognition is one of the most important objectives of teacher training, student learning and curricula development in higher education (alena et al., 2017). metacognition is regarded as a key factor to foster critical thinking, problem-solving and decision-making (zhao et al., 2014). additionally, it is an interconnection with self-regulated learning, the most efficient predictor of academic performance of modern student generations (pintrich & de groot, 1990). a conceptual framework for self-regulated learning demonstrates metacognitive processes (planning, monitoring, controlling and reflecting) in four core areas of self-regulation: cognition, motivation, behavior, and context (pintrich, 2004). although the implementation of metacognitive strategies in curricula is challenging, ample studies have shown that the frequent use of these strategies enables students to actively set out their learning plans, monitor their behaviors, affections, skills and knowledge, and reflect their thinking (boyle et al., 2014; zhao et al., 2014). selfreflection contributes to a meaning-making process which relates learners’ own experiences to the next deeper understanding and experiences (bruning et al., 2011). this study is developed from the work of phan (2019) on the mnt application in reading lessons to second-year english majors at a college. however, this term is still new to non-english-major students at binh dinh college. in the present efl context, students are heavily influenced by the grammar-translation method and the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 31 teacher-centered approach. efl university students have the lowest use of reading meta-strategies such as planning, organizing, orchestrating strategies, monitoring, and evaluating (thuy, 2018). the study investigated changes in the students’ awareness of metacognitive strategy use before and after two differently assigned instructions. it also discovered students’ perceptions of the effects of mnt in reading practice. hopefully, this study could improve students’ metacognitive reading behaviors with great effort to plan, monitor their comprehension skills, seek help and reflect reading outcomes. they were expected to metacognitively see their development in meaning-making process through three reading stages. furthermore, they could broaden extensive reading outside the class to support in-class reading lessons and apply mnt across different lessons and courses. literature review note-taking strategies rahmani and sadeghi (2011) present a difference between traditional notes and unconventional ones, graphic organizers or outlines. conventional notes are the results of students’ from-left-to-right full note-taking styles for a lecture or a speech, whereas the latter depict the organization of concepts in an abbreviated visual and verbal format. unconventional note-taking may foster students to realize important ideas and then write them down into notes, essential for interactive reading process (haghverdi et al., 2010). in addition, they assist students in relating their previous input to new concepts and presenting complex ideas in a creative format, using pictures, icons, numbers or key words (buzan & buzan, 2010). unconventional notes are products of students’ re-expressions, supporting concentration and retention (haghverdi et al., 2010). some beneficial unconventional notes include cornell method, outlining method, mind-mapping, and charting method (massey university, 2019). strategic reading reading is a complex receptive process involving a reader’s linguistic knowledge, world knowledge and knowledge of a reading topic (nunan, 1999). burchiellaro (2013) notes that a reader forms various representations for comprehension including the meaning of exact word, main ideas of the text and the thinking in the mind. reading is an interactive process between the text and a reader’s prior schemata (nunan, 1999). to attain reading final outcomes, a reader uses numerous strategies to activate their low-level reading processes (phonological recognition, word awareness and vocabulary knowledge) and higher-level processes (inferencemaking, working memory, questioning, synthesizing, and background knowledge) (chen et al., 2015). strategic reading aims to acquire a three-level series of developments of vocabulary, comprehension, and critical thinking (richards & eckstut-didier, 2011). mokhtari and reichard (2004) note good readers have high level of awareness of metacognitive strategies and strategy frequent use while reading. the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 32 questioning strategies questioning is an essential strategy in teaching and learning contexts, boosting learners’ own meaning-making construction (harvey & goudvis, 2007). selfquestions can be generated by instructors or learners (joseph et al., 2016). teacher’s questions used in the stages of reading lessons aim to increase students’ interests, assess comprehension and develop higher-thinking skills. meanwhile, learner-generated questions during reading process are likely to activate their background, self-monitor comprehension, and interact with the text. harvey and goudvis (2007) note that a proficient reader inclines towards employing strategic questions about authors, contents, events and ideas throughout reading process. questioning a text promotes readers’ expressions of uncertainties which drive them to explore plenty of possibilities (janssen, 2002). in addition, it can be more efficient to use higher-level questioning to boost learners’ guessing, inferential and evaluative cognition (ziyaeemehr, 2016). highquality questions by teachers can stimulate learners to strategically self-question while they read (williamson, 1996). joseph et al. (2016) revealed a multitude of effective instructions to promote effective questioning strategies for a range of diverse learners’ reading comprehension performances across various educational settings. for instance, strategies comprise art (ask, read with alertness, tell), kwl (what i know, what i want to know, and what i learn), sq3r (survey, question, read, recite, review), srq2r (survey, read, question, recite, and review), or twa (think before reading, think while reading, and think after reading). empirical studies there has been little research on mnt in reading lessons. however, the previous findings on the benefits of explicit instructions of metacognitive reading strategies, self-questioning and note-taking can spur the pursuit of the present study. mokhtari and reichard (2004) stated that the first-language (us) and secondlanguage (moroccan) readers were both aware of the importance of frequent use of metacognitive strategies such as planning, adjusting reading speed, paraphrasing, note-taking, self-questioning, help-seeking and evaluating when encountering problems. the instructions of metacognitive reading strategies were proved to boost students’ achievements in vocabulary and reading comprehension (djudin, 2017). the primary students in muara brunei district being exposed to the sixlesson treatment of metacognitive strategies to read expository lessons attained the higher mean of comprehension performance than the control group using conventional reading approach (othman et al., 2014). similarly, caliskan and sunbul (2011) found that the intervention of metacognitive reading strategies enabled experimental students to control their reading process and gain higher consequences in academic achievement. in particular, they reported that they used metacognitive strategies for further practice outside the classroom. the participants became more motivated and confident in utilizing metacognitive strategies after some interventions (zhao et al., 2014). the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 33 previous studies cited in the article review by joseph et al. (2016) reported mixed results on reading comprehension after the instructions of strategies to develop self-questioning abilities of k-12 students. however, positive findings might suggest that the self-questioning application could enhance reading comprehension performance across diverse students and across diverse academic settings. ninety students in adult education programs were randomly assigned to three experimental groups and two control ones (rich & shepherd, 1993). experimental participants were instructed self-questioning and summarizing strategies (total condition), self-questioning strategies only, or summarizing ones only. meanwhile, in control conditions, participants also took experimental materials without instruction and tests, or tests only. during six sessions, each lasting 45 minutes, experimental students were instructed assigned instructional methods. on the first measure, the participants in total condition and the one with self-questioning instruction outperformed the two control ones. in the next free recall task, experiment participants with selfquestioning and summarizing instruction were significantly better than the control groups. however, the subjects in total condition did not considerably outperform the other experimental conditions. furthermore, 72 nine-grade students studying science were randomly arranged to three conditions: questioning-training with the provisions of prompts (g1), question-generation without prompts (g2), and no question employment (garcia et al., 2014). the study found that g1 gained the highest means, and enhanced students’ metacognitive awareness, self-regulation, and learning control in science. ample studies investigated the effects of note-taking strategies across various learning contexts. despite inconsistent findings of the effects of graphic organizers (gos) on comprehension, the majority of researchers acknowledged that gos could benefit classroom learning in one way or the other (manoli & papadopoulou, 2012). note-taking strategies played an effective tool for students to cognitively remember concepts and recall knowledge (kiewra, 1985; piolate et al., 2005). significant changes were seen in the use of strategy categories such as analysis, inference, evaluation, inductive reasoning and deductive reasoning before and after the implementation of concept mapping on participants (cyr & all, 2009). gos’ effects at primary school level and graduate education were higher than figures for other educational levels such as high school education, undergraduate education and secondary school education accordingly (kansızoglu, 2017). thirty-five englishmajor subjects at a college were instructed metacognitive note-taking strategies in reading lessons (phan, 2019). after 8-week intervention, the majority of subjects showed their awareness of the use metacognitive reading strategies in reading practice. nearly half of participants acknowledged they became more motivated, thoughtful and communicative. the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 34 methods participants and setting fifty-four students, with 47 females, were invited from the tourism management faculty at binh dinh college at the mean age of 18.42. according to their official course, they take four english units: two units of general english in the first two semesters, and english for tourism in the following two semesters. these students were heavily affected by traditional approaches of learning reading skills, that is, learning vocabulary by rote, answering textbook-designed comprehension questions or translating an english text into vietnamese. moreover, the forms of reading tests they received were mainly based on content-based questions and vocabulary. it seemed to be difficult to take advantage of reading passages to promote other high-thinking levels. with the approval of college institutional board and the students’ consent, the students were randomly assigned to two groups: the experimental condition (n=27), and the control one (n= 27). each group separately took part in the seven 75-minute sessions after their school time. they were offered small gifts to attend the study. the experimental participants were guided to apply mnt while the control ones were taken the conventional teaching approach without mnt. the researcher designed the lesson plans for sessions in both conditions and then directly instructed the students in the seven-lesson instructions. procedures the experimental classroom included seven 75-minute sessions. in session one, the instructor introduced the effective forms of note-taking and guided the students how to take-note ideas. they were encouraged to employ graphic organizers with key words, symbols, pictures or icons to emphasize comprehension. in session two and three, the instructor presented some useful reading strategies (e.g., scanning, skimming, using dictionary, guessing, and inferring). the teacher explicitly directed the students to apply metacognitive reading strategies in each stage of reading practice such as setting up objectives based on some important cues, for example, headings, key words, charts or photos. they were instructed methods of learning control, for example, improving their attention, self-questioning for clarification, keeping on their objectives, paraphrasing, visualizing information, and evaluating their comprehension or strategies. in the next two sessions, they were provided a guide for applying mnt (appendix). this mnt model was adapted from the version designed by faculty innovate (2012). it supported a learner to take note their ideas through pre-reading, whilereading and post-reading stage. it contained heuristic questions which guided learners to formulate, regulate and present their cognition on paper in each reading stage. in session 6 and 7, the teacher engaged the participants in freely applying the mnt model individually or in groups. reading passages were taken from the book of falla and davies (2012). the students were encouraged to keep their own note-taking and then share them in groups for learning one another’s thinking, the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 35 evaluating comprehension performance and advancing their knowledge (chiu & kuo, 2009). the control group took the same reading materials as the experimental without the instruction of mnt in the independent condition. reading activities were already designed in the book of falla and davies (2012). a consecutive mixed methods approach was applied in this study. both groups took a pretest and post-test examining the levels of metacognitive strategy awareness before and after the assigned instructions. after that, nineteen experimental participants voluntarily took turn participating in face-to-face interviews which help respondents to provide elaborate answers or ask for clarification for interview questions (lavrakas, 2008). instrument all of the control and experimental participants filled out an anonymous questionnaire package given before and after the separately assigned instructions. this questionnaire measuring the recognition of metacognitive reading strategies was adapted from the survey of awareness of reading strategies (sars) (mokhtari & reichard, 2004). it included 30 items concerning three main domains: global strategies, problem-solving strategies and support strategies. the sars’s items involved numerous essential reading strategies, and global strategies are specifically in strong relation to metacognitive strategies (mokhtari & sheorey, 2002). the sars used a 5-point likert scale of 1-5: 1 (never), 2 (sometimes), 3 (often), 4 (usually) to 5 (always) (mokhtari & reichard, 2004). the package was stated as a reliable instrument with the high internal consistency reliability coefficient: 0.83 for global strategies; 0.81 for problem-solving strategies; 0.80 for support strategies (mokhtari & sheorey, 2002). the sars package was claimed to be clear and understandable for elf students (vo et al., 2014). qualitative collection procedure was conducted after the posttest to ensure that quantitative data were not impacted by interviewees. the researcher separately conducted 20-minute semi-structured interviews (figure 1) and recorded each respondent’s responses. interviews aimed to elaborate quantitative findings and explore students’ perceptions of mnt effects on different learning aspects. figure 1 interview questions 1 what learning benefits did you get from the application of mnt? 2 what difficulties did you encounter when applying mnt? 3 to what extent did mnt application involve you in collaborative tasks? 4 are mnt models are useful for other classes? the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 36 data analysis loaded into spss statistics package, all tests’ quantitative data from two conditions were kept secret to all participants. the descriptive statistics were implemented to obtain means and standard deviations, and then the paired-samples t-test was to determine whether there were any significant divergences in the metacognitive strategy awareness between the two tests’ scores in both conditions. qualitative analysis was based on “coding procedures” described by creswell (2014, pp. 267–269). firstly, the researcher examined the data to divide the text into information segments. secondly, the segments were labeled with initial codes, which then continued to be coded to a small number of themes. results table 1 differences in participants’ use of reading strategies by experimental group and control group strategy experimental group (n=27) control group (n=27) m sd -m sig m sd -m sig glob1 pretest 1.55 0.58 -1.3 0.000 pretest 1.63 0.49 -0.96 0.000 posttest 2.85 0.72 posttest 2.59 0.50 glob2 pretest 1.92 0.38 -1.45 0.000 pretest 1.81 0.55 -1.04 0.000 posttest 3.37 0.49 posttest 2.85 0.60 glob3 pretest 2.41 0.5 -0.89 0.000 pretest 2.41 0.50 -0.51 0.000 posttest 3.30 0.46 posttest 2.92 0.47 glob4 pretest 1.63 0.49 -1.22 0.000 pretest 1.85 0.60 -0.33 0.001 posttest 2.85 0.71 posttest 2.18 0.39 glob5 pretest 2.22 0.42 -0.7 0.000 pretest 2.22 0.42 -0.15 0.043 posttest 2.92 0.47 posttest 2.37 0.49 the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 37 strategy experimental group (n=27) control group (n=27) m sd -m sig m sd -m sig glob6 pretest 2.48 0.51 -0.63 0.000 pretest 2.44 0.50 -0.6 0.000 posttest 3.11 0.32 posttest 3.04 0.44 glob7 pretest 2.44 0.51 -0.63 0.000 pretest 2.33 0.48 -0.34 0.001 posttest 3.07 0.47 posttest 2.67 0.48 glob8 pretest 1.92 0.38 -1.19 0.000 pretest 2.03 0.52 -0.6 0.000 posttest 3.11 0.32 posttest 2.63 0.49 glob9 pretest 2.37 0.67 -0.74 0.000 pretest 2.33 0.55 -0.15 0.161 posttest 3.11 0.32 posttest 2.48 0.51 glob10 pretest 1.78 0.42 -1.48 0.000 pretest 1.81 0.48 -0.26 0.006 posttest 3.26 0.45 posttest 2.07 0.38 glob11 pretest 1.78 0.42 -1.15 0.000 pretest 1.74 0.45 -0.26 0.006 posttest 2.93 0.67 posttest 2.00 0.48 glob12 pretest 2.15 0.46 -1.15 0.000 pretest 2.18 0.56 -0.67 0.000 posttest 3.30 0.46 posttest 2.85 0.76 glob13 pretest 2.15 0.46 -0.48 0.001 pretest 2.15 0.36 -0.18 0.022 posttest 2.63 0.63 posttest 2.33 0.48 prob1 pretest 2.93 0.38 -0.33 0.000 pretest 3.04 0.59 -0.22 0.011 posttest 3.26 0.44 posttest 3.26 0.52 the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 38 strategy experimental group (n=27) control group (n=27) m sd -m sig m sd -m sig prob2 pretest 2.63 0.56 -0.59 0.000 pretest 2.74 0.65 -0.3 0.003 posttest 3.22 0.5 posttest 3.04 0.52 prob3 pretest 2.93 0.47 -0.37 0.001 pretest 2.81 0.68 -0.15 0.103 posttest 3.30 0.67 posttest 2.96 0.65 prob4 pretest 2.67 0.62 -0.59 0.001 pretest 2.70 0.61 -0.15 0.043 posttest 3.26 0.52 posttest 2.85 0.6 prob5 pretest 2.41 0.57 -0.74 0.000 pretest 2.52 0.51 -0.26 0.006 posttest 3.15 0.6 posttest 2.78 0.58 prob6 pretest 2.07 0.47 -1.26 0.000 pretest 2.30 0.46 -0.33 0.004 posttest 3.33 0.48 posttest 2.63 0.74 prob7 pretest 2.74 0.71 -0.63 0.000 pretest 2.70 0.61 -0.11 0.265 posttest 3.37 0.56 posttest 2.81 0.62 prob8 pretest 3.00 0.55 -0.52 0.001 pretest 2.85 0.53 -0.11 0.185 posttest 3.52 0.58 posttest 2.96 0.44 sup1 pretest 2.15 0.36 -1.44 0.000 pretest 2.04 0.34 -0.03 0.713 posttest 3.59 0.57 posttest 2.07 0.38 sup2 pretest 2.96 0.64 -0.11 0.083 pretest 2.48 0.64 -0.19 0.022 posttest 3.07 0.73 posttest 2.67 0.62 the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 39 strategy experimental group (n=27) control group (n=27) m sd -m sig m sd -m sig sup3 pretest 2.59 0.5 -0.37 0.002 pretest 2.56 0.64 -0.11 0.083 posttest 2.96 0.44 posttest 2.67 0.62 sup4 pretest 2.26 0.45 -1.07 0.000 pretest 2.33 0.62 -0.52 0.000 posttest 3.33 0.68 posttest 2.85 0.66 sup5 pretest 2.44 0.64 -0.81 0.000 pretest 2.56 0.64 -0.48 0.001 posttest 3.26 0.59 posttest 3.04 0.58 sup6 pretest 1.78 0.42 -1.26 0.000 pretest 1.93 0.47 -0.37 0.002 posttest 3.04 0.52 posttest 2.30 0.54 sup7 pretest 1.74 0.52 -1.00 0.000 pretest 1.81 0.68 -0.22 0.011 posttest 2.74 0.52 posttest 2.04 0.59 sup8 pretest 3.22 0.64 -0.41 0.001 pretest 3.07 0.67 -0.19 0.022 posttest 3.63 0.63 posttest 3.26 0.65 sup9 pretest 2.07 0.38 -1.23 0.000 pretest 2.30 0.54 -0.18 0.022 posttest 3.30 0.46 posttest 2.48 0.57 overall pretest 2.31 0.45 -0.86 0.000 pretest 2.32 0.39 -0.34 0.000 posttest 3.17 0.24 posttest 2.66 0.36 note: pretest to posttest change is significant at p≤0.05. table 1 indicates that there were, overall, prominent divergences in the use of three strategy domains before and after the interventions in both conditions. for the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 40 the experimental condition, there were significant divergences in 29 items (p≤0.05) apart from sup2 (reading aloud when meeting difficulties) between two tests. in the posttest, the mean result of reading strategy awareness ranged from a low of 2.63 to a high of 3.63. fifty percent of items were in a high usage with a mean of 3.25 or higher, whereas the rest fell in the medium range between 2.5 and 3.24. the posttest saw a dramatic rise in critically evaluating what you read (glob10, m=-1.48), using prior knowledge (glob2, -m=-1.45) and taking notes while reading (sup1, -m=-1.44). the other categories including setting purposes for reading (glob1), checking how text content fits purposes (glob4), visualizing information (prob6), using reference materials (sup6) and asking questions (sup9) also had a considerable mean gap about 1.22 to 1.30. six categories with small increases in mean values involved sup2, sup3 (summarizing text information), (sup8) going back and forth in text, (prob1) reading slowly and carefully, (prob3) adjusting reading rate and (glob13) confirming predictions. between two tests in the control condition, there were significant differences in all but six categories, that is, glob9 (using typographic aids), prob7 (re-reading for better understanding), sup3, sup1, prob3 and prob8 guessing vocabulary. considerable mean rises were in the use of glob2 (-m=-1.04) and glob1 (-m=0.96). prob1 and sup8 fell in high scores with a mean of just over 3.26. there were 18 items in medium usage compared to ten others in the low usage, namely, glob4, glob5 (skimming), glob9, glob10, glob11 (resolving conflicting information), glob13, sup1, sup6, sup7 (paraphrasing) and sup9. figure 2 means of three strategy domains by experimental group and control group as can be seen from the figure 2, the mean value of problem-solving subscale in the pretest was likely to be at the highest in both conditions (around 2.70), followed by the figure for support domain (approximately 2.30). despite rises in the posttest mean values regardless of different conditions, the experimental participants showed a significantly greater growth of awareness of three strategy domains than the counterpart. the problem-solving strategies still remained highest in the posttest with 3.30 in the experimental group and 2.80 in the control one. the mean results of two subscales of global and support strategies of the 0 0.5 1 1.5 2 2.5 3 3.5 pre-experimental post-experimental pre-control post-control global strategies problem-solving strategies support strategies the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 41 experimental group were in the range of 3.10 to 3.25 compared to those of the control group around 2.55. the participants’ positive responses in the interviews were coded into three broad themes: motivation, skill developments and interaction desire. the majority of respondents showed a high sense of self-efficacy, intrinsic motivation, extrinsic motivation and task value, essential elements of motivation scales (pintrich et al., 1991). one student stated, “i’m confident to show my opinions in this class.’’ another said, “i feel pleasant to present ideas on paper in a creative way”. they acknowledged they became motivated to freely broaden their knowledge and try to thoroughly understand concepts’ relationships. extrinsic motivation features were stated, “with mnt i can improve my reading results and other skills like speaking and writing” or “deal with long discussions”. five participants valued they would apply mnt across different english lessons and subjects. one reported, “mnt is a useful learning tool”, “applicable” or “is used in other classes”. regarding skill developments, sixteen respondents presented their significant increases in strategic reading. particularly, one stated, “firing-up questions stimulate me to set purposes before reading, summarize and evaluate their outcomes.” also, they showed they were more willing to make self-questions. a student said, “self-questioning reminds me of vocabulary and interesting ideas”, and “helps me actively find and check more information”. twelve students expressed that the choice of effective note-taking forms such as mindmapping or cornell methods could encourage them to keep or recall important concepts in a logically organized way. such notes allowed them to add more ideas after discussion or further reading. mnt aided them in the meaning-making progress (n=7). one told, “i can explain a concept with my prior knowledge.” another showed, “i can orally present a given topic with the aid of notes.”, or “compare given information and my prior knowledge”. for the third theme, ten students reported mnt engaged them to share their ideas and discuss for more clarification. one revealed, “a well-prepared note-taking helps me show my knowledge in a smart way.” to have ‘interesting notes’ they are involved in asking the teacher or friends for clarification and using dictionaries or the internet for help. a participant expressed, “my questions can be clarified by my peers or the teacher.”, or “i can understand my peers’ insights or their problems.” besides, respondents expressed some difficulties in using mnt. their creativity in note-taking was limited by the small note materials. they reported note-taking was time-consuming in designing a note-taking when discussing in groups. they lack vocabulary to re-express opinions while note-taking. additionally, some noted, “it was difficult to follow ideas in a complicated note.” discussion the pretest data from this study, all just under 2.70, were inconsistent with the results by mokhtari and reichard (2004) in which mean scores of the three main the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 42 subscales used by moroccan students were between 3.40 and 3.60 in sars packet. it is cited that the non-native participants in morocco were rated as skilled readers. it also suggests that reading strategies may be unpopular with vietnamese efl students. the pretest data also implies that problem-solving strategies and support strategies were more likely to be common than the global strategies (metacognitive strategies) to efl students. the results were in line with those by thuy (2018) in which metacognitive strategies were lower than other reading strategies relating to cognition, affection and sociocultural-interaction. ten global items witnessed a rise of over 0.70 in the experiment condition while only two global categories (glob1 and glob2) increased by over 0.70 in another group. besides, the overall post-experimental means of three strategy domains were in the frequency range ‘often’ – ‘nearly usually’. however, the counterpart’s figures were under ‘often’ level. this might be explained that mnt explicitly drew students’ attention to metacognitive skills, which in turn activate problem-solving and support strategies to gain more insights. these findings supported those in previous studies that explicit instruction of strategic reading, note-taking and selfquestioning could improve students’ metacognitive awareness (caliskan & sunbul, 2011; harvey & goudvis, 2007; murakami, 2014; williamson, 1996). however, the existence of fifteen items in the medium mean scores after mnt instruction imply that time allotment for the intervention seemed insufficient for participants’ awareness and skill developments. manoli and papadopoulou (2012) also pointed that longer interventions would be an important variable to master expected strategies. despite the conventional instruction, smaller increases in the post-control means of three domains might be explained that students were introduced interesting reading texts providing cultural information (falla & davies, 2012). attractive designs with pictures and lead-in questions could stimulate students to guess or relate their knowledge to texts. vocabulary tasks and comprehension questions elicited students’ concentration. they also had extra listening and speaking tasks based on a text’s topic. the qualitative results showed positive perceptions of the use of mnt in reading practice. these findings were expected to strengthen quantitative results when students acknowledged the influence of mnt on different skill developments. particularly, students reported mnt improved their use of global strategies like planning, monitoring, evaluating during reading lessons. students stated how they solved reading problems with self-questioning, note-taking, support-seeking. they explained mnt brought about various benefits beyond reading skills, namely, the improvements in vocabulary, social knowledge, writing and presentation. it is worth noting that qualitative reports may widen mnt’s effectiveness on other essential learning elements: motivation and collaborative learning. students with high sense of motivation show more engagement, commitment, curiosity and confidence while peer-learning helps widen their perspectives (pintrich et al., 1991). in essence, prior studies showed there were correlations between metacognitive skills and motivation (oguz & ataseven, 2016; phan, 2019). perceptions of the effect of mnt on interaction desire, moreover, contribute a promising solution to the improvement of social skills in educational contexts. the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 43 limitations there were some limitations in the study. first, the small number sampled from a single faculty of a college can limit the study’s generalizability. the majority of female participants also affect the applicability of the study to all population. next, the researcher directly instructed participants in two conditions may subjectively impact the participants’ perceptions of the use of mnt. additionally, the short-term interventions and practice were likely to reach surfaced conclusions. recommendations based on the findings, the researcher suggests some recommendations. the instructions of mnt should be regularly incorporated in the syllabus of reading lessons to enhance students’ awareness of metacognitive reading strategy use properly and efficiently. teachers can provide authentic materials to maximize effects of mnt in reading practice. mnt should be combined with textbook tasks to foster learning achievement to the fullest. in particular, students can have chance to use low-thinking strategies (e.g., scanning, guessing words, remembering, summarizing or note-taking) to high-thinking strategies (e.g., setting purposes, analyzing, relating prior knowledge, asking questions or evaluating). teachers should stimulate students’ creative note-taking and question-making to facilitate their understanding. students are encouraged to work in groups and find help for their clarifications. teachers give immediate facilitation in order to help students avoid wasting their time in designing notes. the time length of intervention should be longer for students’ practice and skill acquisition. further research is needed to measure mnt effects on different aspects of self-regulated learning. comprehension tests and class observation should also be applied in the future study for deeper analysis on the effect of mnt use. there should be a pilot study to measure the feasibility of sars in vietnamese language version. there need further studies with a large number of participants across diverse disciplines and english proficiency. conclusion after the seven-session instruction of mnt, the experimental participants showed significant changes in their awareness of metacognitive reading strategies. through the interviews, respondents claimed their growth of motivation, learning skills and interaction with the aid of mnt. the findings from the current study stated that the raise of self-regulation and metacognition awareness can be able to be teachable and transferable. in other words, this study was a promising predictor for the popularity of mnt application in educational contexts. however, to determine the success of mnt use, instructors should have deliberate plans of instruction and provide students with enough practice time. the large number of respondents and various research instruments should be considered to improve the research scale. it is proposed that metacognitive strategy instruction should be integrated into the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 44 syllabus design across different courses and english proficiency. the findings from the study can offer instructors at higher education systems an effective and innovative teaching tool to raise students’ awareness of strategy use in the reading lessons in particular and diverse skills or courses in general. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references alena, r., elci, a., & elci, a. (2017). metacognition and successful learning strategies in higher education. igi global. boyle, j. r., rosen, s. m., & forchelli, g. (2014). exploring metacognitive strategy use during note-taking for students with learning disability. international journal of primary, elementary and early years education, 44(2), 3–13. bruning, r. h., schraw, g. j., & norby, m. n. (2011). cognitive psychology and instruction (5th ed.). pearson. burchiellaro, g. (2013). learning how to read in english as a foreign language: issues in italian secondary school teaching and the role of strategy instruction. università degli studi di padova. buzan, t., & buzan, b. (2010). the mind map book : unlock your creativity, boost your memory, change your life. pearson. caliskan, m., & sunbul, a. (2011). the effects of learning strategies instruction on metacognitive knowledge, using metacognitive skills and academic achievement (primary education sixth grade turkish course sample). educational sciences: theory and practice, 11(1), 148–153. chen, x., dronjic, v., & helms-park, r. (2015). reading in a second language: cognitive and psycholinguistic issues. new york: routledge. chiu, m. m., & kuo, s. w. (2009). from metacognition to social metacognition: similarities, differences, and learning. journal of education research, 3(4), 1–19. creswell, j. w. (2014). educational research: planning, conducting, and evaluating quantitative and qualitative research (4th ed.). pearson. cyr, s. k. s., & all, a. c. (2009). concept mapping: a road to critical thinking. journal for nurses in professional development, 25(2), 70–74. the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 45 djudin, t. (2017). using metacognitive strategies to improve reading comprehension and solve a word problem. journal of education, teaching and learning, 2(1), 124–129. faculty innovate. (2012). developing your metacognitive skills – note taking guide. https://facultyinnovate.utexas.edu/sites/default/files/notetaking_metacog_in terventions_.pdf falla, t., & davies, d. a. (2012). solutions: elementary student's book. oxford university press. garcia, f. c., garcia, a., g., b. a. b., m.c, p., & justicia, f. (2014). the effects of question-generation training on metacognitive knowledge, self-regulation and learning approaches in science. psicothema, 26(3), 385–390. haghverdi, h. r., biria, r., & karimi, l. (2010). note-taking strategies and academic achievement. journal of language and linguistic studies, 6(1), 75– 109. harvey, s., & goudvis, a. (2007). strategies that work: teaching comprehension for understanding and engagement (2nd ed.). stenhouse publisher. janssen, t. (2002). instruction in self-questioning as a literary reading strategy: an exploration of empirical research. l1-educational studies in language and literature, 2(2), 95–120. joseph, l. m., cullen, j. m., & alber-morgan, s. r. (2016). the effects of selfquestioning on reading comprehension: a literature review. reading and writing quarterly, 32(2), 152–173. kansızoglu, h. b. (2017). the effect of graphic organizers on language teaching and learning areas: a meta-analysis study. education and science, 42(191), 139–164. kiewra, k. a. (1985). learning from a lecture: an investigation of note-taking, review and attendace at a lecture. human learning, 4(1), 73–77. lavrakas, p. j. (2008). encyclopedia of survey research methods. sage. manoli, p., & papadopoulou, m. (2012). graphic organizers as a reading strategy: research findings and issues creative education, 3(3), 348–356. massey university. (2019). note-taking method. http://owll.massey.ac.nz/studyskills/note-taking-methods.php mokhtari, k., & reichard, c. (2004). investigating the strategic reading processes of first and second language readers in two different cultural contexts. system, 32(3), 379–394. the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 46 mokhtari, k., & sheorey, r. (2002). measuring esl students' awareness of reading strategies. journal of developmental education, 25(3), 2–10. murakami, a. (2014). the impact of instructions: perceptions of notetaking and awareness of metacognitive listening for esl students (masters thesis). https://cornerstone.lib.mnsu.edu/cgi/viewcontent.cgi?article=1307&context= etds nunan, d. (1999). second language teaching & learning. heinle and heinle. oguz, a., & ataseven, n. (2016). the relationship between metacognitive skills and motivation of university students. educational process: international journal, 5(1), 54–64. othman, y., mahamud, z., & jaidi, n. (2014). the effects of metacognitive strategy in reading expository text. international education studies, 7(13), 102–111. phan, p. k. m. (2019). applying metacognitive note-taking skills in reading lessons to second-year english majors at binh dinh college paper presented at the proceedings of the 7th opentesol international conference 2019, ho chi minh city. pintrich, p. r. (2004). a conceptual framework for assessing motivation and selfregulated learning in college students. educational psychology review, 16(4), 385–407. pintrich, p. r., & de groot, e. v. (1990). motivational and self-regulated learning components of classroom academic performance. journal of educational psychology, 82(1), 33–40. pintrich, p. r., smith, d., grarcia, t., & mckeachie, w. j. (1991). a manual for the use of the motivated strategies for learning questionnaire (mslq). national center for research to improve postsecondary teaching and learning. https://eric.ed.gov/?id=ed338122 piolate, a., oliver, t., & kellog, r. t. (2005). cognitive effort during note-taking. applied cognitive psychology, 19(3), 291–312. rahmani, m., & sadeghi, k. (2011). effects of note-taking training on reading comprehension and recall. the reading matrix, 11(2), 116–128. rich, r., & shepherd, m. j. (1993). teaching text comprehension strategies to adult poor readers. reading and writing, 5(4), 387–420. richards, j. c., & eckstut-didier, s. (2011). strategic reading: level 3 student's book. cambridge university press. steiner, h. h., trivedi, n. h., & brown, j. (2019). bringing a learning strategies project to scale in a first-year seminar. journal of effective teaching in the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 47 higher education, 2(1), 27–44. https://jethe.org/index.php/jethe/article/view/36 thuy, b. t. n. (2018). reading strategies used by vienamese efl and esl university students. vnu journal of foreign studies, 34(2), 111–124. vo, t. t., luu, h. m., & luu, t. b. n. (2014). an inquiry into students' application of metacognitive strategies in reading technical materials. journal of language teaching and research, 5(6), 1283–1291. williamson. (1996). self-questioning: an aid to metacognition. reading horizons, 37(1), 1–19. zhao, n., wardeska, j. g., mcguire, s. y., & cook, e. (2014). metacognition: an effective tool to promote success in college science learning. journal of college science teaching, 43(4), 48–54. ziyaeemehr, a. (2016). use of questioning techniques and the cognitive thinking processes involved in student-lecturer interactions. international journal of humanities and cultural studies, 3(1), 1427–1442. the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 48 appendix a guide of metacognitive note-taking skills beginning of reading class planning date: course: what do i already know about this topic? how does it relate to something i already know about? what questions do i already have about this topic? middle of reading class – monitor learning monitoring learning notes create a left-hand column for noting key items of the text’s content or thoughts that arise. present ideas as headings. insights and “ah-huh!” moments. self-questions. responses strategies summary of main ideas follow-up items use on-the-right space for taking notes concerning the headings which are being presented in the left-hand column. possibly use a variety of note-taking if you suddenly realize that you understand something, make sure to write those thoughts down. you may also include feelings, other comments and make a connection between the current materials and your prior knowledge. write questions. questions are the best evidence showing that you are actually thinking about the material. note your responses to the questions you have. -what strategies you have used. note main ideas pay attention to what is happening inside your head (metacognition). using notetaking mind mapping method the cornell method outlining method the charting method the application of metacognitive note-taking skills journal of effective teaching in higher education, vol. 3, no. 1 49 end of class – reflect and evaluate learning at the bottom of your notes for each class, draw a line below your notes to write a summary. below are some guiding questions to assist with writing. the response can be presented in a variety of note-taking 1. what were the most important ideas of today’s class session? 2. what did i read today that is in conflict with my prior understanding? 3. how did the ideas of today’s class session relate to previous class sessions? 4. what do i need to actively go and do now to clarify my confusing questions? 5. what did i find most interesting about class today? microsoft word cox_raditch_final.docx journal of effective teaching in higher education, vol. 5, no. 1 https://doi.org/10.36021/jethe.v5i1.152 teaching online and cyberbullying: exploring cyberbullying policies thomas d. cox, university of central florida, thomas.cox@ucf.edu joseph raditch, university of central florida abstract. cyberbullying is a phenomenon increasingly affecting students and faculty in k-12 through higher education systems. cyberbullying in higher education negatively affects institutions and their respective stakeholders. the consequences range from faculty turnover to student suicide. research related cyberbullying in higher education in online learning is emerging. common understandings of cyberbullying vary by state, institution, and classroom level. furthermore, many states, including florida, defer conduct policies and their enforcement to the individual institution. in this article, a review of publicly available florida university policies regarding cyberbullying in higher education were explored by a florida professor and a higher education administrator. utilizing document analysis, this study analyzed policies from the 12 state universities to document the institutional definitions of cyberbullying and the recommended reporting practices for faculty. further, following the community of inquiry (coi) framework, this study examined policies and procedures to determine if they supported the construct of teacher presence. it was determined that there are multiple definitions of cyberbullying and that policies were publicly available but most lacked support for faculty to report related incidences. most policies were implied and not explicitly related to cyberbullying. finally, there was little to no evidence in the analyzed documents that would support teaching presence. keywords: cyberbullying; higher education policy; online teaching the repercussions of cyberbullying in higher education shocked the united states population in 2010 when tyler clementi committed suicide after being secretly filmed by a roommate during a sexual encounter with another man at rutgers university (parker, 2012; pilkington, 2010). dharun ravi, clementi’s roommate, had not only filmed clementi’s encounters but streamed the live video feed to other students at rutgers university. after clementi’s death, ravi was charged with and pled guilty to 15 counts of invasion of privacy (cherelus, 2016). however, the convictions were overturned in 2016 by an appeals court (mcgeehan, 2016). clementi's suicide began a public discussion about both cyberbullying and lesbian, gay, bisexual, and transgender (lgbt) issues in higher education, specifically questioning the university's support for cyberbullied and lgbt students (cherelus, 2016; hubbard, 2013). in the spring of 2017, nick lutz, a student at the university of central florida (ucf), was suspended after grading and subsequently tweeting his ex-girlfriend's apology letter following the dissolution of the relationship (langly, 2017; roll, 2017). the tweet of the graded message reportedly received over 121,000 retweets (coleman, 2017). ucf sanctioned the student for disruptive and harmful teaching online and cyberbullying 72 journal of effective teaching in higher education, vol. 5, no. 1 behavior. the charges were later revoked but the national attention that this incident brought about disrupted the learning environment and the lives of the people involved. these cases illustrate only a small portion of contemporary reports regarding cyberbullying within higher education. consistent across such cases are the immeasurable, detrimental, and even tragic consequences that may occur as a result of cyberbullying. moreover, the policy and legal intricacies underlying cyberbullying consequence reports portray the challenges that higher education has encountered in responding to and resolving cyberbullying incidents. as an example, perceptions and definitions of cyberbullying vary, leading to confusion, disruptions, and even the minimization of the actions that can harm social and emotional wellbeing. thus, higher education institutions need to implement a more comprehensive and standardized definition of and reporting strategies related to cyberbullying. as current professors and administrators, we have sought to investigate our own state and institutional policies and the ease of access to them. cyberbullying has a strong presence in the classroom. in 2017, marshall polston, a student at rollins college in florida, was accused of sending threatening emails to an adjunct world religions/humanities professor after receiving a failing grade on an essay (russon, 2017a). polston and the world religions professor, areej zufari, both reported acts of threats and retribution. according to zufari, polston disrupted face to face class sessions by contradicting and monopolizing class time. outside of class, polston reportedly sent emails to professor zufari accusing her of being "antichristian" and threatening to expose her bias to the student’s "friends in the national media." zufari submitted a report of the harassment incident to school administrators and filed for protection against stalking with orange county. these charges were later dropped based on video evidence. even though these charges were debunked and unfounded, there was evidenced of incivility that impacted the learning environment. this case serves as just one illustration of the possible ways in which bullying can extend beyond student-to-student or campus relationships and transcend into teaching environments and onto faculty members. the polston/zufari case is congruent with increasing research evidence that indicates that cyberbullying has influenced not only students and higher education faculty (baldasare et al., 2012; vance, 2010; walker et al., 2011). as the prevalence of mobile devices and the frequency of online interactions evolve, detrimental online interactions continue to grow (weber shandwick collective, 2018). because the worldwide pandemic has prompted more higher education courses to move online, cyberbullying may become more of a problem. therefore, the following investigation provides information about how institutions of higher education in florida define cyberbullying and documents the policies related to cyberbullying found in higher education environments. statement of the problem and purpose cyberbullying impacts students and faculty participating in online courses (vance, 2010). in online courses, cyberbullying includes harassment and bullying through teaching online and cyberbullying 73 journal of effective teaching in higher education, vol. 5, no. 1 online discussions. when these discussions are negative and personal, they may obstruct student participation within an online classroom (clark et al., 2012; stover, 2006). additionally, disruptive dialogue among the students affects students’ ability to interact with course content and other students effectively within an online learning community. according to garrison et al. (1999), disruptions within online courses may interrupt students during the higher-order thinking processes to address aggressive behavior. further, garrison asserted that instructors of online courses should facilitate and guide any dialogue within an online course to promote higher-order thinking, partially by using the tools made available to them by their institution. these tools would include the policies that govern student behavior. however, there are few state, federal, or institutional policies that address cyberbullying in higher education (washington, 2015). in place of state or federal laws, washington further recommends that institutions develop policies and procedures as well as professional development training for faculty to address cyberbullying on campuses. even though cyberbullying policies may exist at institutions of higher education, they may be difficult for faculty and students to find. barr and lugus (2011) concluded that when they found cyberbullying policies, many were improperly housed within campus technology-oriented policies where many faculty and students might not look. the confusion surrounding the existence of an institution’s policies and procedures regarding cyberbullying has been identified as a barrier for part-time faculty in reporting and addressing the phenomenon (minor et al., 2013). in response, we, as professors and administrators in florida, engaged in this qualitative study to examine how leaders of public institutions of higher education in the state of florida define cyberbullying and encourage reporting of cyberbullying incidents. in florida, leaders of each public university have maintained independent policies to regulate student behavior. there has been no comprehensive study reviewing or cataloging these policies related to cyberbullying across institutions within the state of florida. researchers have expressed the need for future studies to examine if institutions of higher education have crafted policies addressing cyberbullying (washington, 2015; watts et al., 2017). the purpose of this study was to use document analysis to analyze the policies at 12 florida public universities containing the definition of cyberbullying and the recommended reporting practices for faculty. by framing the results of the analysis through the community of inquiry (coi) theoretical framework (garrison et al., 1999), this study seeks to provide value to faculty seeking to strengthen their online teaching presence through providing clear guidelines established by each florida institution. it also provides value to administrators at institutions within the united states who are reviewing their policies addressing online abuse and cyberbullying by identifying common definitions currently used within public institutions. teaching online and cyberbullying 74 journal of effective teaching in higher education, vol. 5, no. 1 rationale for the study while there is a wealth of cross-discipline research on cyberbullying explicitly about the k-12 education system, the studies related to higher education are few. furthermore, as of 2018, comparisons of policies regarding cyberbullying in online courses at public institutions of higher learning in the state of florida do not exist. the present study will catalog and analyze the policies from institutions of higher learning in florida that pertain to cyberbullying, harassment, and disruptions within an online course, providing higher education with an accurate cyberbullying policy resource derived from one state’s public institutions. identifying common definitions and student conduct reporting strategies among public florida universities on the topic of cyberbullying can provide instructors, administrators, and instructional designers with cohesive resources to mitigate aggressive behavior in online courses. the current analysis may serve as a resource to improve faculty development in online teaching, the quality of online courses, and the learning experiences for students in public institutions of higher learning not only in florida but in other states as well. faculty members are uniquely positioned to detect cyberbullying through direct class observations or through student disclosure. therefore, faculty members are an essential part of helping to determine what instances of cyberbullying are occurring and providing information to administrators so that policies can be put in place to deter the offenses. our analysis will provide knowledge to faculty members as they seek to obtain current policies and modify or redefine existing policies. contemporary research on cyberbullying the three cases (e.g., clementi, lutz, and poston) previously described herein are reports that both researchers and the media have examined regarding cyberbullying within higher education (coleman, 2017; mccormick, 2010; quintana, 2017; roll, 2017; russon, 2017b). however, everyone has not always agreed about the frequency of cyberbullying in higher education. olweus (2012) labelled the phenomenon as overrated, citing low incident rates (4.5%) in his 5-year metaanalysis. however, hinduja and patchin (2012) argued the topic remained relevant as their 10 years of research on adolescents and k-12 students demonstrated that 1 in 4 youth experienced cyberbullying. while olweus’s (1995, 2012) and hinduja and patchin’s (2015) cyberbullying studies focused on adolescent and k-12 groups, other authors examining cyberbullying have revealed that the phenomena influence adult learners and faculty within higher education (baldasare et al., 2012; campbell et al., 2020; vance, 2010; walker et al., 2011). for example, according to vance (2010), students (12%) and faculty (35%) have reported being bullied within an online course. berne et al. (2013) reported that 11% of students at a large university indicated they personally experienced cyberbullying. these data demonstrate that cyberbullying impacts learners and instructors within higher education. teaching online and cyberbullying 75 journal of effective teaching in higher education, vol. 5, no. 1 though increasing evidence has indicated that cyberbullying has affected students and faculty within higher education, evidence reflects some administrators do not perceive cyberbullying to be an issue in their institutions. luker (2015) reported that 44.5% of administrators surveyed believed that cyberbullying was a rare occurrence at their home institution compared to their peer institutions but that only 13% of the institutions sampled reported not having a cyberbullying incident in the past 12 months. there may be a disconnect between administrative perceptions about cyberbullying and the reality of cyberbully occurrences within institutions. in addition to a perceptual disconnect about the occurrence of cyberbullying, faculty and administrators may be unprepared to manage cyberbullying incidents that may arise from coursework, as may have been the case in the 2017 rollins college incident (russon, 2017a, 2017b). palloff and pratt (2011) proposed focusing on maintaining authority through the syllabus by indicating specific expectations for classroom conduct and referring the student to any existing online harassment policies maintained by the institution. likewise, researchers have studied community building within online courses and proposed discussion management techniques to keep students on task rather than managing harassment (palloff & pratt, 2003, 2011). however, these discussion management techniques do not address cyberbullying occurrences in students’ online courses. while information regarding best practices is available to faculty teaching in the online environment, evidence addressing how to approach instances of cyberbullying within online courses is negligible. in the attempt to overcome this limitation, we curated and analyzed the policies of 12 public universities in florida pertaining cyberbullying, harassment, and disruptions within online courses. the present study sought to provide instructors with an accurate cyberbullying policy resource. issues defining cyberbullying adding to the difficulty of identifying and managing cyberbullying in online courses, researchers have not standardized the definition of cyberbullying. many researchers have defined cyberbullying as an individual using information and communications technology to promote deliberate and hurtful behavior with the intent to do harm (berne et al., 2013; haber & haber, 2007; walker et al., 2011). furthermore, the examination of cyberbullying policies reveals a similar finding: comparisons of policies regarding cyberbullying in online courses at public institutions of higher learning in the state of florida do not exist; moreover, there is no comprehensive review of higher education cyberbullying policies across the united states. however, there has been a review of bullying and cyberbullying policies in k-12 school in the state of louisiana (stickl haugen et al., 2019). the cyberbullying definition given by the legislature in florida illustrates the breadth and vagueness of the term’s interpretation: teaching online and cyberbullying 76 journal of effective teaching in higher education, vol. 5, no. 1 “cyberbullying” means bullying through the use of technology or any electronic communication, which includes, but is not limited to, any transfer of signs, signals, writing, images, sounds, data, or intelligence of any nature transmitted in whole or in part by a wire, radio, electromagnetic system, photoelectronic system, or photo optical system, including, but not limited to, electronic mail, internet communications, instant messages, or facsimile communications. (jeffrey johnson stand up for all students act, fla. stat. § 1006.147, 2018) in 2010, vance proposed that age influenced the definition of cyberbullying. he argued that adults who experienced aggressive behavior online were cyberharassed rather than cyberbullied. in addition, some researchers have included spam email and broad cyberattacks (e.g., scamming or phishing), within their definitions of cyberbullying (zorkadis et al., 2005). however, other researchers consider phishing and spam as separate types of cyberattacks from cyberbullying (hamby et al., 2018; wright, 2018). the discord of definitions and their diverse nuances reiterate the need for clearly stipulating and delineating the term cyberbullying so that its instances and effects can be better understood and prevention strategies more effectively developed. legal issues like the definition of cyberbullying, legislation and policies on cyberbullying and cyberharassment vary across the united states. each state maintains its laws about bullying and online bullying behavior for individuals under the age of 18. however, many states do not regulate the harassment of adults, including college-aged students (18 years and older). public institutions are not legally bound to protect adult-aged students from certain types of online harassment from individuals not associated with the school. yet, some state legislatures have delegated the responsibility of regulating student misconduct to the state college and university systems (horowitz & bollinger, 2014). florida regulates cyberbullying in the k-12 system but not in the state college or university system (fla. stat. § 1006.147, 2018). instead, florida’s legislature delegated the creation of policy to regulate student conduct to the state colleges and universities (fla. stat. § 1006.50, 2018; fla. stat. § 1006.62, 2018). while this delegation of power allows each state institution to address conduct as necessary, there are inconsistencies across institutions. additionally, federal requirements for title vii and title ix have mandated that institutions of higher learning regulate certain types of student behavior; access to federal funding can be denied or stopped for non-compliance. as such, school leaders have adopted technology-use policies, which restrict offensive, annoying, or harassing communications originating from campus-based resources, such as wireless networks or university managed computers (barr & lugus, 2011). teaching online and cyberbullying 77 journal of effective teaching in higher education, vol. 5, no. 1 community of inquiry (coi) the community of inquiry (coi) theoretical framework has been selected as a conceptual framework for this study. according to garrison et al. (1999), the coi refers to the educational experience within an online course as the culmination of the interaction between the social, cognitive, and teaching presences. garrison et al. concluded that the social and cognitive aspects of the learning process could not be separated from one another, and the researchers established the social and cognitive presences. the third element called the teaching presence includes the design, organization, discourse, and instruction of cognitive and social elements to foster positive and meaningful learning outcomes. teaching presence is highly predictive of student success. the primary mode of inquiry for this investigation was framed through the teaching presence. as described through the framework, the instructor builds the course, creates guidelines for the course using all tools available (including the institutional, state, and federal policies), and facilitates interactions between participants. the interdependent social, teaching, and cognitive presences are connected to generate the educational experience through the coi. educators consider all aspects of the online learning environment when establishing a positive teaching presence. cyberbullying acts as a disruptor, deriving from the social presence and working to separate each element simultaneously, and creates chaos within the learning environment. therefore, consideration should be given to creating a positive environment by establishing policies, procedures, and processes. methods the following qualitative study sought to analyze and compare available policies and codes of conduct from each of the 12 florida public universities regarding cyberbullying. according to bowen (2009), documents as a data source represent text and images that have been produced without intervention of the researcher. documents are typically produced and exist as “social facts” in that they are created and shared for social consumption (atkinson & coffey, 2011). these text artifacts may include public records, personal documents, or other types of physical evidence (atkinson & coffey, 2011; creswell, 2014; merriam & tisdell, 2015). documents were systematically collected, and definitions of cyberbullying were curated from publicly available documents on the websites of public universities within the state if florida. in doing so, we approached each document as a “social fact” that is created for social consumption and, in line with the purpose of this study, intended for social good. social facts can include the values and norms that govern social control and are commonly considered rules. research questions the study was guided by the following research questions: teaching online and cyberbullying 78 journal of effective teaching in higher education, vol. 5, no. 1 rq1. how do florida state public universities address cyberbullying in policies and codes of conduct? rq2. in what ways do policies or codes of conduct at florida state public universities provide instructors guidelines for responding to cyberbullying incidents? rq3. do policies and codes of conduct evidence the community of inquiry’s concept of teaching presence? setting and population the study was conducted in the context of florida’s public university system. the state university system of florida is under the jurisdiction of the florida board of governors (2016a, 2016b), with each university governed by a local board of trustees. there are 12 public universities in the state university system. in 2016, 352,116 students enrolled in state universities. eleven percent of students enrolled in the state university system exclusively took distance courses. the 2015–16 system accountability report (flbog, 2017h) reported that there were 13,634 fulltime and 3,185 part-time faculty employed by the state university system. furthermore, flbog (2017h) reported that 14% of all course sections were offered through distance learning. data collection and analysis policy documents that could reference cyberbullying were gathered from each of the florida public universities. documents gathered included, but were not limited to, student codes of conduct, faculty handbooks, and university policy and regulation documents. these documents provided the researchers with the official language and stance that each public university in the state of florida uses when providing guidance on understanding, recognizing, and handling cyberbullying. first the definitions of cyberbullying were classified by evidence: (a) explicit definitions, (b) implied definitions, (c) redirection, and (d) broad harassment no definition. explicit definitions used the term cyberbullying, and a definition was provided. implied definitions did not include the term cyberbullying but indicated actions associated with cyberbullying (e.g., sending email, posting or sending messages through the learning management system, and posts in virtual discussions). redirection references policies that were an antecedent to the current policy. finally, broad harassment represents guidelines without reference to technology means. teaching online and cyberbullying 79 journal of effective teaching in higher education, vol. 5, no. 1 results official university documents describing aspects of cyberbullying were collected and analyzed for evidence of cyberbullying polices, definitions, and approaches. the results are presented below. policy definition types each institution defined cyberbullying within its policies. four themes were identified in describing how institutions define cyberbullying:(a) explicit definitions, (b) implied definitions, (c) redirection, and (d) broad harassment – no definition. explicit definitions of cyberbullying emerged as a classification category from the document analysis. for example, the faculty handbook and sexual harassment policy from the university of north florida contained explicit cyberbullying definitions. though explicit definitions only appeared in two documents, the term was included within policies; however, the precise definition included the term cyberbullying as part of the definition of harassment. as such, cyberbullying is not viewed as being different from other types of harassing or aggressive behavior. instead, it is a child or subdimension of the harassment category. the implied definition was the most prominent classification. instead of explicitly using the term cyberbullying, the authors of the policies relied on contextual modifiers such as “email” or “electronic communication” to add a digital scope to the definition of harassment. included within the implied definition are references to cyberstalking, which many of the institutions defined as the repeated harassment of an individual through a digital medium (e.g., social media or email). redirection is the second most prominent classification to emerge in the analysis of the documents. the redirection category describes policies that rely on other policies to define the scope and merit or harassing behavior. in these policies, rather than redefining the terms, the authors refer to existing policies that have behaviors defined. as such, the redirected policy acts as a modifier to the original behavioral definition. for example, the acceptable use of technology resources policy from florida atlantic university (fau) states: laws and regulations: all users are responsible for adhering to all applicable federal, state, and local laws and regulations and all university regulations and policies, specifically including without limitation the university’s sexual harassment regulations and policies, those pertaining to the privacy of student records (ferpa), and the digital millennium copyright act (dmca). in this case, the acceptable use of technology resources policy is expected to augment all fau regulations and policies by adding technology resources to their existing definitions. redirection also supported the concept of teaching presence by connecting seemingly unrelated policies. where policies may have been weak on their own, the connection forged between each policy by redirection has the teaching online and cyberbullying 80 journal of effective teaching in higher education, vol. 5, no. 1 opportunity to strengthen the instructor's use of these resources in setting expectations for behavior in their online course. broad harassment – no definition is a category to describe policies that define harassment and aggressive behaviors, but make no reference to an electronic medium, use no contextual modifiers, and do not implement redirections to other policies. in using broad harassment – no definition, the policy makers use encompassing phrases such as "of any kind." while policies that do fall into this category do not redirect, other policies may redirect to the broad harassment policy. through this process, the broad harassment policy's applicability is augmented to include the redirected policy's scope. all 12 public universities in the state of florida maintained policies that contained a definition of harassment that encompassed cyberbullying behavior. only one institution, university of north florida, explicitly named cyberbullying as harassing behavior. the other 11 institutions used a combination of implicit, redirection, or broad harassment language to capture cyberbullying as prohibited behavior (see table 1). the number of mentions by type ranged from 2-7 times. table 1 cyberbullying definition frequency and type university explicit implied redirection broad harassment: no definition florida atlantic university 1 3 1 florida gulf coast university 5 1 florida international university 1 1 4 florida a&m 3 1 florida polytechnic university 2 3 4 florida state university 5 1 new college 2 2 1 university of central florida 4 1 2 university of florida 2 7 3 university of north florida 2 2 2 university of south florida 2 2 3 university of west florida 2 3 2 grand total 2 33 23 22 teaching online and cyberbullying 81 journal of effective teaching in higher education, vol. 5, no. 1 faculty guidelines for reporting cyberbullying next, 33 documents from the 12 public universities were examined to determine the available university-approved faculty guidance for reporting harassing behavior. the definition of harassing behavior included cyberbullying in varying degrees. overall, only one institution, the university of south florida, provided a detailed policy on the steps available to faculty for responding to harassing behavior inclass. the disruption of academic process policy advises the faculty member to ask the student about their behavior. if the student does not respond, the faculty member may remove the student from the class. the faculty member has 48 hours to complete an incident report. the faculty member is empowered to exclude the student from class until the conflict is resolved. oddly, the policy applies to online settings, but it does not advise faculty on the finer points of how to remove a student from an online course. florida international university faculty handbook advised faculty how to handle misconduct and did specify appropriate responses to student misconduct, but it was not a policy per se. like university of south florida, a formal command is required before removing a student from the class. the office of student conduct would be the agency to remove a student permanently from the class. community of inquiry (coi) and cyberbullying: teaching presence finally, the study analyzed policies, codes of conduct, and suggested instructor response approaches to determine the intersection of these policies with the coi’s concept of teaching presence. there was limited evidence that the policies supported teaching presence. the policies were generally created to support a safe and nurturing environment, but the connection to faculty practice was inadequate. policies and syllabi universities typically have standard statements for faculty to include in their syllabus that help to establish the culture, climate, and setting of the classroom, an integral aspect of the coi teaching presence. these statements set the expectations for communication and conduct. in the review of the documents from each university, no policies or faculty guidance required the inclusion of a cyberstalking statement. however, some universities did maintain policy statements for the syllabi that referenced conduct. additionally, the faculty handbook at florida international university linked the course syllabi to the conduct policies. seven of the universities had standing policies for faculty to include on their syllabi. however, only florida atlantic university, florida international university, and florida polytechnic provided suggestions to faculty on placing language about or directing attention towards anti-harassment policies. each institution addressed the inclusion of such policies differently. for instance, florida polytechnic university's administration required that university policies be placed within the syllabus. teaching online and cyberbullying 82 journal of effective teaching in higher education, vol. 5, no. 1 within the florida atlantic university’s guidelines for course syllabi, the authors suggested instructors include a statement on classroom etiquette policy supplemented by the phrase “if applicable.” “if applicable” suggests that the inclusion of an etiquette policy is entirely optional for the instructor. furthermore, the language found within the guidelines for course syllabi suggested that the inclusion of classroom etiquette policies should be owned by the instructor: if you have a particular policy relating to student behavior in the class, such as relating to tardiness or on the use of electronic devices in the classroom state so here. recognizing that the unique relationship between faculty and student and adhering to the principles of academic responsibility, any such policy must be reasonable, non-discriminatory, and not impede the educational mission. teaching presence is inclusive of design and organization, discourse, and direct instruction. policies and procedures upholding the tenets of responsible use of technology can contribute to the design and organization learning environments, a construct linked to teacher presence. technology responsibility fourteen policies regarding the use of technology with a relationship to a definition of cyberbullying were found to include statements about using technologies efficiently. the language used within each policy established two parameters for the use of technology resources: (a) using the media effectively and (b) improper use. for example, florida a&m university’s electronic connectivity policy contained the following statement on effective use: in order to enhance the efficiency and effectiveness of business and academic processes, it is the responsibility of fammail users to timely read notifications sent to them through fammail. other language addressed the improper use of the technology, such as using the network to break the law, harassing others through email and electronic messaging, and impairing others’ ability to effectively use the resources. the combination of effective use of the media and examples of improper user behavior builds the construct of technology responsibility and contributes to teacher presence when faculty address these policies with their class. conduct expectations expectations of student and faculty conduct were found within policies and documents from all universities. in total, 47 documents with definitions of cyberbullying included the expectation of conduct. the documents ranged from student codes of conduct to faculty handbooks to anti-harassment policies. conduct expectations aligned with policies, and documents that exhibited conduct expectations commonly presented language with guidance on proper interpersonal etiquette. teaching online and cyberbullying 83 journal of effective teaching in higher education, vol. 5, no. 1 the student codes of conduct described conduct violations and positive characteristics expected from all students, such as having integrity and thinking critically. for example, the student code of conduct from the university of north florida states the following: we value: the pursuit of truth and knowledge carried out in the spirit of intellectual and artistic freedom; ethical conduct; community engagement; diversity; responsibility to the natural environment; and mutual respect and civility. in many cases, the values to which the university community adhere included contributing to community understanding. establishing conduct expectations involves not simply stating the rules but explaining the philosophy behind the rules that drive the intellectual community. implications for teaching practice and recommendations for future research the following recommendations for institutions of higher education may support faculty in decreasing cyberbullying in online classes. 1. define or include examples of cyber-harassment or cyberbullying within conduct policies. 2. post the policies in prominent places that are easily accessible. 3. establish procedures to support classroom management. 4. link policies and regulations to reinforce teaching presence. define and include examples of cyberbullying cyberbullying behavior was identified in conduct policies at all 12 institutions. however, the behavior was included under the parent category of harassment. terms such as cyber-harassment, cyberstalking, cyberbullying should be classified and defined as harassment. definitions and scenarios describing the terms would promote shared meaning. in this study, similarities between the definition of harassment and cyberstalking were noted. similarly, other research has noted a spectrum of behaviors that constitute bullying and incivility in online learning spaces (campbell et al., 2020). similarities in defining distinct behaviors may cause confusion for instructors and students when reporting harassing behaviors. vance (2010) called for using the term cyber-harassment. while vance utilized the term cyber-harassment over cyberbullying based on the age of the participants, a better reason to use the term cyber-harassment is that it is a child concept to the parent category of harassment. furthermore, to improve the culture of the classroom, a component of teacher presence, posting policies regarding conduct in shared online spaces can serve as a reminder of expectations. teaching online and cyberbullying 84 journal of effective teaching in higher education, vol. 5, no. 1 establish procedures to support classroom management the current study found that only one out of the 12 institutions of higher education published a policy and procedure to manage students’ disruptive conduct. adopting a process known by students and faculty would support classroom management. minor et al. (2013) reported that barriers to reporting cyberbullying behavior for faculty included the lack of known authoritative resources and perceived lack of administration support. finally, documenting policies, processes, and procedures would afford faculty the authority to discipline misbehavior and affirm the support of the administration. link policies and regulations to reinforce teaching presence the documents analyzed demonstrated limited support of teacher presence in relationship to cyberbullying. perhaps this inadequacy related to the purpose of the policies to address cyberbullying at the institution level and not at the class level. as such, the policies were mainly related to the potential culture of the online teaching environment, but there was no written encouragement for faculty to adopt policies and procedures that would promote teaching presence. the current research study recognized the potential of linking policies together. therefore, institutions can effectively address cyberbullying and reinforce teaching presence. faculty could be provided boilerplate templates related to cyberbullying, classroom management, and other related topics to post in the course management system. faculty could be encouraged to create robust and collaborative learning experiences to discuss community culture and behavioral expectations thereby increasing teacher presence. furthermore, instructors can include conduct and technology policies within their syllabi that they will discuss with their learners as needed. institutions that have adopted policies should require instructors to include a statement on classroom conduct expectations including links to the institutional technology and conduct policies. future research and implications for practitioners additional research is recommended to determine the prevalence of cyberbullying in the online classroom. while this study examined the policies available at the institutional level, it did not consider at what level cyberbullying was occurring at the 12 institutions considered in this study. further consideration should be given as to how these policies are communicated with learners and the level of cyberbullying people have experienced at an institution. as higher education continues to grow in online spaces, policies should be frequently reviewed to maintain safety and structure within the online classroom. as these policies are adopted, a longitudinal study of incidents of bullying and students’ knowledge of policies and procedures could be conducted. teaching online and cyberbullying 85 journal of effective teaching in higher education, vol. 5, no. 1 conclusion cyberbullying in higher education continues to grow. the present study included a document analysis of policies, regulations, codes of conduct, and faculty handbooks at 12 state universities in florida to examine cyberbullying definitions and policies in higher education. as universities continue to educate through online modalities, other states and private institutions should review their policies to promote greater understanding of cyberbullying. common understanding and knowledge of procedures can contribute to a reduction in disruptive behavior online. furthermore, it is equally as important for instructors using online modalities to know the content of these policies so they can build better experiences within their online classrooms to increase their teaching presence to ensure student success. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references atkinson, p. a., & coffey, a. j. (2011). analyzing documentary realities. in d. silverman (ed.), qualitative research: theory, method and practice (3rd ed., pp. 77–92). sage. berne, s., frisén, a., schultze-krumbholz, a., scheithauer, h., naruskov, k., luik, p., & zukauskiene, r. (2013). cyberbullying assessment instruments: a systematic review. aggression and violent behavior, 18(2), 320–334. doi:10.1016/j.avb.2012.11.022 baldasare, a., bauman, s., goldman, l., & robie, a. (2012). cyberbullying? voices of college students. cutting-edge technologies in higher education, 5, 127– 155. doi:10.1108/s2044-9968(2012)0000005010 barr, j., & lugus, e. (2011). digital threats on campus: examining the duty of colleges to protect their social networking students. western new england law review, 33, 757. https://digitalcommons.law.wne.edu/lawreview/vol33/iss3/3/ bowen, g. a. (2009). document analysis as a qualitative research method. qualitative research journal, 9(2), 27–40. doi:10.3316/qrj0902027 campbell, l. o., tinstman jones, j., & lambie, g. w. (2020). online academic incivility among adult learners. adult learning, 31(3), 109–119. https://doi.org/10.1177/1045159520916489 cherelus, g. (2016, october 27). dharun ravi, roommate in rutgers webcam case, pleads guilty. the huffington post. https://www.huffingtonpost.com/entry/dharun-ravi-rutgersguilty_us_581272c2e4b064e1b4b17301?section=us_queer-voices teaching online and cyberbullying 86 journal of effective teaching in higher education, vol. 5, no. 1 clark, c. m., werth, l., & ahten, s. (2012). cyber-bullying and incivility in the online learning environment, part 1: addressing faculty and student perceptions. nurse educator, 37(4), 150–156. doi:10.1097/nne.0b013e31825a87e5 coleman, n. (2017). school reverses course after controversial cyberbullying suspension. cnn. https://www.cnn.com/2017/07/20/us/student-apologytweet-suspension-overturned-trnd/index.html creswell, j. (2014). research design: qualitative, quantitative, and mixed methods approaches (4th ed.). sage. florida board of governors. (2016a). fall student enrollment in state university system institutions [data set]. https://www.flbog.edu/universities/ florida board of governors. (2016b). universities. https://www.flbog.edu/universities/ fla. stat. § 1006.1 (47) (2018). http://www.leg.state.fl.us/statutes/index.cfm?mode=view%20statutes&sub menu=1&app_mode=display_statute&search_string=1006.147&url=10001099/1006/sections/1006.147.html fla. stat. § 1006.5 (2018). http://www.leg.state.fl.us/statutes/index.cfm?app_mode=display_statute&s earch_string=&url=1000-1099/1006/sections/1006.50.html fla. stat. § 1006.6 (2) (2018). http://www.leg.state.fl.us/statutes/index.cfm?mode=view%20statutes&sub menu=1&app_mode=display_statute&search_string=1006.62&url=10001099/1006/sections/1006.62.html garrison, d. r. (2011). e-learning in the 21st century: a community of inquiry framework for research and practice (2nd ed.). routledge. garrison, d. r., anderson, t., & archer, w. (1999). critical inquiry in a text-based environment: computer conferencing in higher education. the internet and higher education, 2(2–3), 87–105. doi:10.1016/s1096-7516(00)00016-6 haber, j. d., & haber, s. b. (2007). cyberbullying: a “virtual” camp nightmare. camping magazine, 80, 52–57. https://www.acacamps.org/staffprofessionals/news-publications/publications/camping-magazine hamby, s., blount, z., smith, a., jones, l., mitchell, k., & taylor, e. (2018). digital poly-victimization: the increasing importance of online crime and harassment to the burden of victimization. journal of trauma & dissociation, 19(3), 382– 398. teaching online and cyberbullying 87 journal of effective teaching in higher education, vol. 5, no. 1 hinduja, s., & patchin, j. w. (2015). cyberbullying legislation and case law: implications for school policy and practice. cyberbullying research center. https://cyberbullying.org/cyberbullying-legal-issues.pdf horowitz, m., & bollinger, d. m. (2014). cyberbullying in social media within educational institutions: featuring student, employee, and parent information. rowman & littlefield. hubbard, j. (2013, october 3). fifth gay teen suicide in three weeks sparks debate. abc news. http://abcnews.go.com/us/gay-teen-suicide-sparksdebate/story?id=11788128 jeffrey johnston stand up for all students act, fla. stat. § 1006.147(3) (2018). https://www.flsenate.gov/laws/statutes/2018/1006.147 langly, c. (2017, july 17). ucf student behind viral tweet grading breakup letter suspended from school. wftv. http://www.wftv.com/news/local/ucf-studentbehind-viral-tweet-grading-breakup-letter-suspended-fromschool/563094570 luker, j. m. (2015). perceptions of cyberbullying within higher education (113) [doctoral dissertation, school of education, hamline university]. https://digitalcommons.hamline.edu/hse_all/113 mccormick, r. (2010, october 10). rutgers president richard l. mccormick: reaffirming our values. web archive. https://web.archive.org/web/20101004013148/http://news.rutgers.edu/med rel/statements/2010/statement-of-preside-20101001 mcgeehan, p. (2016, september 9). conviction thrown out for ex-rutgers student in tyler clementi case. the new york times. https://www.nytimes.com/2016/09/10/nyregion/conviction-thrown-out-forrutgers-student-in-tyler-clementi-case.html?_r=0 merriam, s. b., & tisdell, e. j. (2015). qualitative research: a guide to design and implementation. john wiley & sons. minor, m. a., smith, g. s., & brashen, h. (2013). cyberbullying in higher education. journal of educational research and practice, 3(1), 15–29. https://scholarworks.waldenu.edu/jerap/ olweus, d. (1995). bullying or peer abuse at school: facts and intervention. current directions in psychological science, 4(6), 196–200. doi:10.1111/1467-8721.ep10772640 olweus, d. (2012). cyberbullying: an overrated phenomenon? european journal of developmental psychology, 9(5), 520–538. doi:10.1080/17405629.2012.682358 teaching online and cyberbullying 88 journal of effective teaching in higher education, vol. 5, no. 1 palloff, r. m., & pratt, k. (2003). the virtual student: a profile and guide to working with online learners. jossey-bass. palloff, r. m., & pratt, k. (2011). the excellent online instructor: strategies for professional development. john wiley & sons. parker, i. (2012, february 6). the story of a suicide. the new yorker. https://www.newyorker.com/magazine/2012/02/06/the-story-of-a-suicide pilkington, e. (2010, september 30). tyler clementi, student outed as gay on internet, jumps to his death. the guardian. https://www.theguardian.com/world/2010/sep/30/tyler-clementi-gaystudent-suicide quintana, c. (2017, april 10). rollins college says student’s suspension was not related to clash with professor. the chronicle of higher education. https://www.chronicle.com/blogs/ticker/rollins-college-says-studentssuspension-was-not-related-to-clash-with-professor/117733 roll, n. (2017, july 20). on second thought… inside higher education. https://www.insidehighered.com/news/2017/07/20/u-central-floridareinstates-student-suspended-over-tweet russon, g. (2017a, march 28). rollins student, professor clashed for weeks in religion class. orlando sentinel. https://www.orlandosentinel.com/news/education/os-rollins-investigate-freespeech-20170328-story.html#nt=interstitial-manual russon, g. (2017b, april 10). student suspended over facebook post, not fight with professor, rollins says. orlando sentinel. https://www.orlandosentinel.com/news/education/os-zufari-resign20170410-story.html rutgers university. (2010). events – project civility. https://web.archive.org/web/20101001104417/http://projectcivility.rutgers. edu:80/events seaman, j., allen, e., & seaman, j. (2018). grade increase: tracking distance education in the united states. survey research group. https://onlinelearningsurvey.com/reports/gradeincrease.pdf weber shandwick collective. (2018). civility in america 2018: civility at work and in our public squares. weber shandwick. https://www.webershandwick.com/news/civility-in-america-2018-civility-atwork-and-in-our-public-squares/ teaching online and cyberbullying 89 journal of effective teaching in higher education, vol. 5, no. 1 stickl haugen, j. l., sutter, c. c., tinstman jones, j., & campbell, l. o. (2019). school district anti-bullying policies: a state-wide content analysis. international journal of bullying prevention, 2, 309–323. https://doi.org/10.1007/s42380-019-00055-1 stover, d. (2006). treating cyberbullying as a school violence issue. the education digest, 72(4), 40–42. vance, j. w. (2010). cyber-harassment in higher education: online learning environments (order no. 3403660) [doctoral dissertation, university of southern california]. proquest dissertations publishing. walker, c. m., sockman, b. r., & koehn, s. (2011). an exploratory study of cyberbullying with undergraduate university students. techtrends, 55(2), 31–38. doi:10.1007/s11528-011-0481-0 washington, e. t. (2015). an overview of cyberbullying in higher education. adult learning, 26(1), 21–27. doi:10.1177/1045159514558412 watts, l. k., wagner, j., velasquez, b., & behrens, p. i. (2017). cyberbullying in higher education: a literature review. computers in human behavior, 69, 268–274. https://doi.org/10.1016/j.chb.2016.12.038 wright, m. (2018). cyberbullying victimization through social networking sites and adjustment difficulties: the role of parental mediation. journal of the association for information systems, 19(2), 113–123. zorkadis, v., karras, d., & panayotou, m. (2005). efficient information theoretic strategies for classifier combination, feature extraction and performance evaluation in improving false positives and false negatives for spam e-mail filtering. neural networks, 18(5/6), 799–807. doi:10.1016/j.neunet.2005.06.045 microsoft word bohan_perrotta_final4.docx a reflective study of online faculty teaching experiences in higher education katherine perrotta, mercer university, perrotta_ka@mercer.edu chara haeussler bohan, georgia state university abstract: despite the popularity of online course and degree offerings in higher education, a lack of data persists on the unique challenges and opportunities online faculty face. gaining insights about these experiences is important to ensure the quality of online teaching as colleges and universities continue expanding e-learning programs. therefore, the purpose of this study is to examine the online teaching experiences of two faculty members through the implementation of reflective study methods. major findings show that faculty access to professional development and mentoring, isolation and connectedness to the campus community, and academic freedom and curriculum control have significant implications for online teaching and student learning. in the wake of covid-19 as colleges across the nation suddenly are faced with moving to exclusively online learning, this study is needed more than ever. keywords: asynchronous learning, reflective study, faculty experiences, higher education, online teaching as the demand for online courses and degree offerings in higher education continues to increase, especially during the covid-19 outbreak, the need to examine the experiences of faculty persists. despite the increase in the popularity of e-learning, the faculty survey of student engagement (fsse) report, which measures faculty beliefs about engaged learning, revealed that none of its respondents taught online (2018, p. 5–8). furthermore, kimmel and fairchild (2017) contend that “little is known about [online faculty] experiences” (p. 53). this lack of data is problematic as colleges and universities need to gain insights about the unique challenges and opportunities online faculty face as e-learning programs continue to expand (allen & seamen, 2010; barr & miller, 2013, p. 10; bolliger & wasilik, 2009; suleman & gul, 2015). purpose of study and research question kimmel and fairchild (2017) argue that there is an urgency for greater examination of the challenges online faculty encounter that include, but not are limited to, curriculum design, pedagogy, teaching evaluations, connectedness to the campus community, and professional development. the main question that frames this study is as follows: how do the researchers’ experiences teaching online highlight professional challenges and opportunities for faculty considering transitioning to elearning settings in higher education? the first author taught over 30 sections of asynchronous online undergraduate survey history courses at three different institutions as a contingent faculty online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 51 member. the second author is tenured at a large university where she teaches face-to-face, hybrid, and asynchronous online graduate courses in education and political science. combined, the authors have over 10 years of experience teaching asynchronous online courses in higher education institutions. the purpose of this study is to gain deeper understandings about the conditions the authors faced teaching online. the authors engaged in reflective study in order to determine how these conditions impacted their teaching effectiveness. ultimately, teaching impacts student learning. literature on advantages and challenges of online teaching the following literature review highlights existing scholarship about the advantages and challenges of online instruction and the areas in which gaps exist with regard to how instructors’ experiences impact the quality of e-learning and teaching. there are several challenges faculty may face when teaching online. one challenge involves resistance to teaching online courses. one major finding of the 2017 educause survey of faculty information and technology found that “faculty have a love–hate relationship with online teaching and learning: they don’t want to do it but think they would be better instructors if they did (pomerantz & brooks, 2017, p. 7). additionally, only nine percent of 13,541 respondents to the educause survey indicated that they prefer to teach online (pomerantz & brooks, 2017, p. 4, 25). the pomerantz and brooks study was conducted in 37 states and 7 countries. there are several factors that contribute to faculty hesitance about teaching online. according to barr and miller (2013), some instructors “find it challenging to adjust [to a] new pedagogical form” (p. 12). schmidt et al. (2016) highlight that since college instructors often teach as they were taught, they may lack an example of what effective online teaching entails, especially if they never took an online course themselves as students. faculty may be required to switch from teacher-centered lectures that tend to dominate face-to-face teaching. examples of the shift from lectures to student-centered instructional methods include interactive and engaged lesson implementation via discussion board assignments, digital simulations and other instructional tools, and synchronous or asynchronous webinars. as a result, instructors who are more familiar with teacher-centered methods may experience challenges transitioning to online instruction that emphasize implementation of student-centered pedagogies (alexiou-ray & bentley, 2015; gregory & salmon, 2013). assumptions about e-learning may also contribute to challenges faculty face when teaching online. bart (2010) and mcquiggan (2012) note that prevailing suppositions regarding online instruction include the perceived easiness of teaching in a digital setting. moreover, mccormack (2015) and the 2017 educause faculty survey found that a majority of faculty who did not teach online “strongly disagreed that online learning helps students learn more effectively” (pomerantz & brooks, 2017, p. 25). given that the majority of faculty surveyed had unfavorable opinions of online teaching, these results highlight the need for wider distribution of scholarly data on online student learning, as well as examples of well-designed courses, in order to improve the quality of online pedagogy and instruction online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 52 (blumenstyk, 2016; kebritchi et al., 2017; pomerantz & brooks, 2017; sibley & whitaker, 2015). despite the challenges of teaching online, college and university instructors who engage in e-learning instruction may experience significant benefits. first, shea et al. (2001) and skibba (2011) found that online faculty tend to engage in greater self-reflection than face-to-face instructors. by providing frequent, timely, and constructive feedback on students’ assignments and interactive communicative features such as discussion boards, online faculty can simultaneously reflect upon their instructional techniques and curricular design (neuman et al., 2017; alexiouray & bentley, 2015). as a result, increased reflective thinking may lead to greater instructor engagement in teaching and learning, professional success, and academic curiosity. second, online teaching affords faculty several incentives concerning time and money. for instance, shea (2007) found that many faculty, particularly contingent faculty and women instructors, teach online due to the convenience this mode of instruction can provide when taking care of personal and professional commitments, especially if these courses involve asynchronous methods. moreover, sibley and whitaker (2015) contend that some higher education institutions combat resistance to teaching online by offering financial incentives, such as paid professional development and extra funds to teach online courses, to attract more online instructors. as a result, faculty may receive intrinsic rewards, such as greater schedule flexibility, and extrinsic rewards, such as financial compensation, when they transition to teaching in the online classroom. ecological theoretical framework the authors used an ecological theoretical framework to shape this reflective study. tinkler and tinkler (2019) state that an ecological framework in educational research can “forge systems to promote justice, a justice sustained through care, curiosity, and humility” (p. 62). dinkelman (2003) contends that the roots of an ecological perspective in reflective education research can be traced to john dewey’s scholarship “on the nature of thinking, problem solving, democracy, and educative growth” (p. 8). dewey’s works, specifically how we think (1933), experience and education (1938), and democracy in education (1944) lay the groundwork for reflection as a “rigorous and systematic” research method because the practitioner must 1) describe observations from an experience in detail, 2) analyze the experiences to generate theories, and 3) test theories about experiences through interactions between one’s self, others, and environment (rodgers, 2002, p. 863). the scholarship of elliot eisner and bruce uhrmacher also provide strong theoretical underpinnings for the significance of an ecological framework in reflective educational studies. according to uhrmacher (1997), the implementation of an ecological framework allows researchers and practitioners to explore how “new ideas and practices affects extant ones” (p. 74). furthermore, eisner (1998) contends that examination of teachers’ experiences within an ecological framework online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 53 “can expand our conception of human cognition and help us develop new forms of pedagogical practice” (p. 245). drawing upon these works, harvey et al. (2016) developed a theoretical framework for the ecology of reflection constituted from the following assumptions: 1. reflection is a process that supports learning 2. reflection may be engaged with at different levels, for different purposes, and from different perspectives 3. connections exist between critical reflection and the higher order of cognitive processes of self-regulation and metacognition 4. reflection includes many contexts and applications in teaching and learning 5. reflective skills can be developed through strategic interventions and scaffolding 6. reflection on experience provides a link to praxis (pp. 5–9). with these assumptions in mind, harvey et al. (2016) provide a model of three different ecologies that align with the theory of the ecology of reflection. these ecologies are 1) the learner, 2) the learning ecology, and 3) the experientiallearning ecology (p. 14). the authors used the experiential-learning ecology model for this study because they examined the online environments at four higher educational institutions to evaluate the challenges and opportunities they faced in the context of the policies, histories, populations, and culture of these unique colleges and universities. methodology, data collection, and analysis reflective study methodology was conducted for this research. according to goodrick (2014), reflective methodology “involves the analysis and synthesis of the similarities, differences, and patterns across two or more cases that share a common focus or goal” (p. 1). andrew et al. (2016) state that reflective methods are “the process of reflection and inquiry [that] can prompt faculty to question and change their pedagogies in ways that enhance student learning” (p. 291). the authors implemented this methodology in order to examine how the challenges and opportunities they faced teaching online could contribute to the growing body of scholarship on e-learning instruction in higher education. several data collection techniques were employed for this study. according to silverman and marvasti (2008), examples of data that can be analyzed in reflective studies include artifacts that record what people do, say, produce, or write. the authors collected 1) university guidelines for online instruction, 2) personal reflective observations, 3) supervisory evaluations, and 4) student evaluations and feedback. the authors manually coded the documents and determined emergent themes in order to identify areas in which experiences aligned with the theoretical framework. next, the authors implemented samaras’ (2011) five key characteristics for effective reflective educational research to analyze the data. these characteristics include 1) a personal situation inquiry; 2) critical collaborative online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 54 inquiry; 3) improved learning; 4) a transparent and systematic research process; and 5) knowledge generation and presentation. the authors applied samaras’ criteria by examining the context of the institutions where they taught and collaborating on data analysis to determine significant trends or themes from their reflective and other qualitative data. in addition, they also identified areas in which they experienced challenges and advantages teaching online and examined the type of professional development that was offered by their institutions to support online instruction. study participants the first author taught over 30 asynchronous online undergraduate history survey courses as a contingent faculty member at three different higher education institutions over the course of ten years. she began teaching online courses due to her interest in e-learning instructional technology and pedagogy and to meet her teaching load. additionally, she taught online courses because of the flexibility this kind of instruction provided while she pursued her doctoral degree and cared for a young child. munro (2011) attributes people in the millennial generation to be born after 1980. although the first author is considered to be a part of the “millennial generation,” she never took an online course during her undergraduate and graduate studies. the second author is a full professor with twenty-five years of experience teaching multiple social studies education, educational policy, and educational history courses at universities in the southwest and southeast. recently, she facilitated asynchronous online courses for political science graduate students to meet her teaching load. she did not take online courses during her undergraduate or graduate studies. unlike the first author, the second author was reluctant to teach online. however, as enrollments declined, the university pressured colleges and departments to expand online course offerings and programs. study settings college a was a two-year college located in a metropolitan region of the southeast that at the time of this study enrolled over 20,000 students. since conducting this study, college a merged with a larger state university where the enrollment is over 50,000 students. students at college a included traditional and non-traditional students, international students, and veterans. the first author was responsible for using the learning management system (lms) to develop and implement course curriculum that included the syllabi, course content, assignments, and exams. college a used blackboard and desire2learn (d2l) to teach online courses. college b is a non-profit private university located in a suburban area in the northeast. the student body was comprised of traditional and non-traditional students and active military personnel. terrestrial campus enrollment is over 3,000 students. the online campus enrollment is over 100,000 students. the online courses were content enhanced, meaning an instructional design team created the course curricula that included the syllabus, discussion board questions, grading online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 55 rubrics, and research projects posted on the blackboard lms. the first author was expected to facilitate the provided curriculum and follow the college’s policies with regard to accepting late or missing work and grading assignments. college c was a public four-year institution located in a metropolitan region of the southeast that enrolls over 30,000 students. the first author participated in a semester-long professional development course on the d2l lms, which focused on online course design, curriculum development, assessment, and aligning assignments to learning outcomes. furthermore, the author’s newly created course was peer reviewed before the first author was certified by the college to teach online. similar to college a, instructors designed all assignments, assessments, and course curricula while adhering to the criterion of the institution’s rubric for an online course. college d is a large public university located in a metropolitan region of the southeast with over 50,000 students enrolled. the second author has been a tenured education professor at college d for over a decade. formal professional development completion was not required for faculty to teach online. however, the author did obtain assistance from the center for excellence in teaching and learning (cetl). college d used the d2l lms for online and hybrid course instruction. findings several themes emerged from the authors’ reflections about the advantages and challenges of teaching online courses. all four institutions placed a similar emphasis on 1) maintaining student retention, 2) meeting learning outcomes, and 3) providing rigorous instruction. these schools, however, possessed different policies concerning curricular goals and pedagogy, academic freedom, and faculty evaluation. analysis of these experiences highlight important perspectives with regard to the challenges and opportunities faculty may face when teaching online. academic freedom and curriculum control the authors enjoyed considerable latitude designing the curriculum, assessments, and instructional materials while teaching online at colleges a, c, and d. both designed the course syllabus and schedule with respect to when readings and assignments were due. they used a variety of resources to differentiate instruction in the e-learning environment. some of these materials included supplementary documentaries and informational videos on educational sites such as www.ted.org and www.learner.org to supplement course readings, recorded lectures on course readings, and power point presentations with narrations and images that outlined major points from the course textbooks. institutional expectations of academic freedom were drastically different at college b. faculty were required to administer the assignments and policies that were provided by the university. several assignments were mandated for students to complete at college b, including weekly responses to discussion board questions with textbook citations and replies to classmates’ posts. additionally, weekly multiple-choice quizzes were embedded online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 56 in the electronic textbook. college b’s policy mandated that students be allowed to re-take the quizzes in order to allow them to review incorrect answers to improve comprehension of content (johnson & kiviniemi, 2009). students were also required to write a research paper from a list of pre-approved topics and preapproved sources to reference. obviously, the first author saved substantial time not having to prepare the assignments, rubrics, and instructional resources at college b. although maintaining standardization for assessments was crucial for college b to monitor student progress, the first author found that not being able to develop course content or establish course policies was extremely challenging to negotiate. as a result, the lack of flexibility to create curriculum, grade assignments at instructor discretion, and instill individual policies with regard to completing late work was difficult for the first author to manage and address. faculty mentoring and evaluation colleges a, b, c, and d differed greatly with regard to faculty mentorship and departmental evaluation. according to vaill and testori (2012), “mentoring is a vital part of online faculty development process” (p. 116). the first author was never formally observed or evaluated by a departmental supervisor as an online teacher at college a and c. however, she received high scores by meeting all essential criterion for evaluation on her peer-reviewed course at college c, which certified her to teach online. the second author was not formally evaluated by peers for her online courses at college d. both authors received formal student evaluations in all courses. two themes emerged from these student evaluations during the authors’ reflections about teaching online. these themes included promoting stronger student engagement and producing better videos. the first author received several evaluations from students who recommended “tweaks” to the videos, including the use of special effects and a “pseudo script” to make the content clearer and more “enjoyable” (field notes, december 12, 2018). in the student evaluations of the second author, 12.5% strongly disagreed, whereas 33.3% agreed and 50.0% strongly agreed that the lecture videos were helpful (student evaluations, september 21, 2018). furthermore, students recommended that the second author make the discussion board assignments more interactive, noting that at times these assignments “felt contrived and inauthentic” (field notes, august 1, 2018). another student noted in the comments section of the evaluation that the discussion threads were not as engaging as hoped (student evaluations, september 21, 2018). lloyd et al. (2012) state that mentoring is a key factor in effective online teaching, particularly with regard to fostering student engagement in the e-learning environment. because departmental feedback was not available, both authors sought other resources such as attending conferences and collaborating with each other in order to improve their instructional skills in their online classrooms. when meeting together, both authors would ask questions about each other’s experiences, which prompted further reflection about online teaching. in addition, the second author came later to online teaching as compared to the first author, online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 57 and thus found that the mentor-protégé relationship became reciprocal as both discussed common advantages and challenges related to online teaching. college b provided considerable administrative oversight of faculty teaching and evaluation. online instructors were part of a team with other faculty who worked under the directorship of one mentor. the mentors posted templates for email messages to send to struggling students and to use for feedback on assignments. they also facilitated discussion boards on pedagogy and shared digital resources for instructors to access on their lms site. overall, the first author found working with the mentors at college b to be a beneficial experience. for example, one mentor noted on the first author’s midpoint evaluation that she observed her “attending to student needs, providing excellent resources and extra content, and giving students constructive and helpful feedback” (field notes, september 14, 2016). the mentors usually responded to questions about the course, grading, and other concerns in a very timely manner. hence, they helped instructors remain engaged with colleagues and improve instructional and student outreach techniques. although mentors used a standard rubric to evaluate online instructors, faculty evaluation at college b was highly subjective. the first author was assigned to over a dozen different teams, hence working with different faculty mentors each term she taught. as a result, the feedback she received was inconsistent. for instance, one mentor commented that the frequency and content of her announcements were “exemplary” (field notes, january 2, 2015). the following term, a different faculty mentor noted that she posted too many announcements each week (field notes, march 3, 2015). consequently, the lack of consistency with regard to how the first author was evaluated at college b became a significant challenge. of course, faculty evaluations can be inconsistent in a face-to-face setting. however, because faculty at college b lacked autonomy over the curriculum, the first author was not always able to correct issues noted on mentor evaluations. professional development both authors found that attending professional development opportunities about 1) online pedagogy, 2) instructional design, and 3) the use of web-conferencing tools like skype to increase faculty social presence in the online classroom to be extremely beneficial. aust et al. (2015) note that “the success of any faculty training program hinges on creating a program that effectively delivers appropriate content in a supportive environment” (p. 13). the first author participated in professional development at colleges a, b, and c. college a’s cetl offered periodic “best practices” workshops that faculty could attend. the first author presented a workshop about promoting engaged learning in online survey history courses. she also presented data on her implementation of discussion boards as a means to promote high impact practices at a faculty meeting at college c. the second author attended professional development opportunities at college d’s cetl where she received more than 10 hours of guidance. furthermore, additional support from a graduate research assistant was provided for her online political science course. college b offered online webinars as professional development opportunities with regard to using rubrics to assess assignments for online courses. the workshops online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 58 that the first author attended included topics such as assessment calibration and the effective use of rubrics for grading assignments. the training for online teaching that the first author received at college c was the most rigorous as compared to the other colleges. the training was a semester-long course facilitated by faculty at the college who provided in-person workshops and detailed handbooks to implement instructional tools such as panopto and kaltura recording software and softchalk for designing interactive notes for online students. she was evaluated on the materials she created in a “sandbox” course shell using the college’s lms where the instructors provided her comprehensive feedback on how to improve the design and implementation of her assignments for online students. college c’s cetl frequently conducted refresher workshops and various professional development sessions for faculty that the first author attended. the availability of these professional development opportunities encouraged both authors to pursue further professional and scholarly research pertaining to engaged learning and pedagogical effectiveness in the online academy. connectedness to the campus community and isolation faculty establishment of a strong social presence in the e-learning setting is extremely important in order to promote engaged teaching and learning. among the conditions that promote a strong social presence in online classes include, but are not limited to, posting announcements and discussion board threads, hosting webinar sessions, and corresponding with students via email (aragon, 2003; plante & asselin, 2014; shea et al., 2001; taverna et al., 2015). both authors implemented these internet-based communications strategies in conjunction with opportunities to meet with online students in face-to-face settings in order to foster greater connectedness to the campus community and rapport with online students. for example, some online students attended a panel discussion the first author gave at a conference hosted at one of college a’s campuses. furthermore, the first author organized field trips to historical sites for online students at college c to attend (field notes, april 2019; field notes, april 2016). the second author met with students during her office hours as well as off-campus at a coffee shop in order to discuss assignments and other course materials (field notes, june 2019). neither of the authors visited the terrestrial campus of college b. although maintaining a vibrant social presence in e-learning environments is vital for effective teaching and learning, both authors found that teaching online can be isolating (field notes, july 2018; field notes, april 2016). online instructors who do not have access to an on-campus office space may feel like an “outsider” from the academic community (kezar & sam, 2013). feelings of isolation are important considerations that relate back to professional development and to dispelling assumptions about online teaching. kebritchi et al. (2017) note that faculty may find that the online environment can be a “cold and distant” place for students and instructors (p. 18). the lack of regular in-person interactions with colleagues and students caused both authors to feel disconnected at times from their academic communities, especially since they both taught asynchronous online courses. although meyer (2004) notes that the disconnect between online faculty and the online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 59 terrestrial campus environment is not a new concern, the authors believe that the issue of isolation is a major tension that online faculty may face, which could impact instructional effectiveness and student learning. discussion the authors confronted different policies and expectations for teaching asynchronous online courses at colleges a, b, c, and d. however, they found several themes across institutions that indicate analogous conditions that support best practices for teaching in e-learning environments. having access to on-going professional development, maintaining academic control over curriculum, and receiving consistent teaching evaluations were major factors that created a positive experience for the researchers when teaching online. isolation from campus meetings and activities, the inability to design curricula at one institution, and the confusion resulting from contradictory supervisory and student feedback posed significant challenges for the researchers when teaching asynchronously online. overall, their findings suggest that higher educational institutions should provide faculty with opportunities to design, implement, and revise curricula; to participate in campus gatherings to foster connectedness; and to engage in professional development to support the complexities of online teaching. creating an interactive and engaging online environment is not necessarily intuitive for faculty, especially those who never engaged in learning in online environments. institutions that provide such support to faculty ultimately will attain the goals of promoting teaching effectiveness and student learning in asynchronous environments. limitations and recommendations for future research there are limitations to this study. since reflective methods were implemented for this research, both authors’ subjectivities impact how they interpreted the challenges and opportunities faced as online faculty. in order to make this study more credible, the first author initially sought guidance and consultation from the second author to assist with member checking the reflective data and with analyzing the benefits and drawbacks of teaching online. both authors find that reflection coupled with mentorship and collaboration with colleagues is critical for improving online teaching and minimizing feelings of isolation. however, additional member checks with other online faculty or instructional designers, who were not acquainted with the authors, could strengthen the findings in this study. second, there are limitations with the use of student evaluations for supporting data on how the authors’ experiences teaching online impacted student as well as faculty learning. while the intention of student evaluations is to improve teaching through reflection, mcdonald (2019) notes that the validity of these evaluations is questionable due to the fact “students often respond subjectively based on how they perceive the instructor makes them feel as an individual and/or learner” (p. 7). empirical studies by macnell et al. (2015), mitchell and martin (2018), and rosen (2017) found that students gave unfavorable evaluations to women faculty who taught identical online courses as male colleagues. additionally, the american online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 60 association of university professors (aaup) stresses that student evaluations of teachers should not be used for tenure and promotion purposes due to issues of bias concerning gender, perceptions of the ease of passing a class, and low submission rates (lawrence, 2018). since both authors are women, it is unclear as to whether the comments they received on their student evaluations were valid. however, these evaluations helped the authors ponder areas for improvement of their online course offerings. with these limitations in mind, the authors recommend three items for future research on how reflective studies on faculty experiences can further contribute to improved pedagogy and learning outcomes with online courses at higher educational institutions. first, further scholarship about wise practices in professional development for online faculty should be pursued. despite the prevalence of literature that highlights the importance of professional development for online faculty, the american association of state colleges and universities (aascu) online learning at public universities report found that only 37% of participating aascu schools mandated pedagogical training for faculty in online settings (magda, 2019, p. 7). the authors concur with mandernach and holbeck (2016) that further research is needed about whether professional development that focuses on course design and the facilitation of teaching quality can enhance the achievement of student learning outcomes. second, the authors suggest further studies on the impact of regular faculty mentorship and its effect on online instructional effectiveness. the aascu report highlights that faculty evaluations by supervisors and peers “rarely occur” (magda, 2019, p. 7). vaill and testori (2012) recommend that an experienced online instructor serve as guest observer to give feedback on course design for novice online instructors, especially given the controversies with using student evaluations of teaching. frequent mentorship may provide new and experienced online faculty the support and guidance they need to engage in effective pedagogy and to receive reliable and constructive evaluative feedback for improvement. such mentorship can promote excellence in teaching and enhance professional satisfaction (magda, 2019). third, the matter of academic freedom while teaching online is a significant aspect of the faculty experience that the authors suggest requires future research. as more colleges expand their online degree programs, there are concerns as to whether faculty are becoming “content providers” instead of scholars engaged in teaching and learning (kessler & wall, 2016, p. 3). consequently, continued scholarship on issues of autonomy with regard to designing and teaching curricula in online courses is needed to further explore the impact of academic freedom on teaching effectiveness, professional development, and faculty satisfaction in elearning environments. online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 61 conclusion overall, both authors’ online teaching experiences were positive, particularly with regard to professional development and faculty support for designing and teaching online content. however, significant challenges both authors faced included feelings of isolation from students and colleagues, inconsistent teaching evaluations, and varying degrees of academic freedom and curriculum control. although these situations may be experienced by faculty in face-to-face environments, harvey et al. (2016) “invite colleagues…in validating and further developing the theory [of the ecology of reflection]” in the online classroom (p. 14). the authors suggest that further reflective studies of online faculty experiences are needed, especially since the abrupt and massive shift to online teaching as a result of the covid-19 outbreak. faculty members’ dramatic conversion to digital instruction may provide opportunities for large-scale studies that evaluate the unique challenges and benefits of online teaching. given the sudden urgency for faculty to engage in online teaching in the wake of the coronavirus, reflection on the process is critically important. when the pandemic has receded, faculty need to be able to consider opportunities and challenges that resulted from this great online teaching experiment. indeed, understanding online instructional effectiveness ultimately should advance the goal of enhancing student learning. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references alexiou-ray, j., & bentley, c.c. (2015). faculty professional development for quality online teaching. online journal of distance learning administration xviii(4) https://www.westga.edu/~distance/ojdla/winter184/ray_bentley184.html allen, i. e., & seaman, j. (2010). learning on demand: online education in the united states, 2009. babson survey research group & the sloan consortium. http://files.eric.ed.gov/fulltext/ed529931.pdf andrew, k., richards, k.a.r., & ressler, j.d. (2016). a collaborative approach to self-study research in physical education teacher education. journal of teaching in physical education 35(3), 290–295. https://doi.org/10.1123/jtpe.2015-0075 aragon, s. (2003). creating social presence in online environments. new directions for adult and continuing education 2003(100), 57–68. https://doi.org/10.1002/ace.119 online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 62 aust, p., thomas g., powell, t., randall, c. k., slinger-freidman, v., terantino, j., reardon, t. (2015). advanced faculty professional development for online course building: an action research project. internet learning 4(2), 103–118. doi:10.18278/il.4.2.8 barr, b. a., & miller, s. f. (2013). higher education: the online teaching and learning experience. https://files.eric.ed.gov/fulltext/ed543912.pdf bart, m. (2010). a checklist for facilitating online courses. faculty focus. http://www.facultyfocus.com/articles/distance-learning/a-checklist-forfacilitating-online-courses/ bolliger, d., & wasilik, o. (2009). factors influencing faculty satisfaction with online teaching and learning in higher education. distance education 30(1), 103– 116. https://doi.org/10.1080/01587910902845949 blumenstyk, g. (2016, july 21). which ed-tech tools truly work? new project aims to tell why no one seems eager to find out. chronicle of higher education. http://www.chronicle.com/article/which-ed-tech-tools-truly/237196 dewey, j. (1933). how we think. prometheus books. (original published in 1910). dewey, j. (1938). experience and education. collier books, macmillan. dewey, j. (1944). democracy and education. free press. (original published 1916). dinkelman, t. (2003). self-study in teacher education: a means and ends tool for promoting reflective teaching. journal of teacher education 54(1), 6–18. https://doi.org/10.1177/0022487102238654 eisner, e. (1998). the enlightened eye: qualitative inquiry and the enhancement of educational practice. prentice hall. faculty survey of student engagement (fsse). (2018). disciplinary area report. university of indiana. http://nsse.indiana.edu/pdf/nsse_annual_results_2018.pdf goodrick, d. (2014). methodological briefs impact evaluation no.9: comparative case studies. united nations children’s fund. https://www.unicefirc.org/publications/pdf/brief_9_comparativecasestudies_eng.pdf gregory, j., & salmon, g. (2013). professional development for online university teaching. distance education 34(3), 256–270. https://doi.org/10.1080/01587919.2013.835771 harvey, m., coulson, d., & mcmaugh, a. (2016). towards a theory on the ecology of reflection: reflective practice for experiential learning in higher education. online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 63 journal of university teaching & learning practice 13(2), 1–20. https://ro.uow.edu.au/jutlp/vol13/iss2/2 johnson, b.c., & kiviniemi, m. t. (2009). the effect of online chapter quizzes on exam performance in an undergraduate social psychology course. teaching of psychology 36(1), 33–37. https://doi.org/10.1080/00986280802528972 kebritchi, m., lipshuetz, a., & santiague, l. (2017). issues and challenges for teaching successful online courses in higher education: a literature review. journal of educational technology systems 46(1), 4–29. https://doi.org/10.1177/0047239516661713 kessler, s., & wall, a. f. (2016). do the best parts equal the best whole? a critique of online teaching and learning. the journal of e-learning and higher education. 2016(2016), p. 1–9. http://10.0.20.51/2016.827620 kimmel, k. m., & fairchild, j. l. (2017). a full-time dilemma: examining the experiences of part-time faculty. the journal of effective teaching 17(1), 52– 65. https://uncw.edu/jet/articles/vol17_1/kimmel.html lawrence, j. w. (2018). student evaluations of teaching are not valid. academe. https://www.aaup.org/article/student-evaluations-teaching-are-notvalid#.xpwrqc3mzea. lloyd, s.a., byrne, m.m., & mccoy, t.s. (2012). faculty-perceived barriers to online teaching. merlot journal of online learning and teaching 8(1), 1–12. https://jolt.merlot.org/vol8no1/lloyd_0312.pdf macnell, l., driscoll, a., & hunt, a. n. (2015). what’s in a name: gender bias in student ratings of teaching. innovative higher education 40(4), 291–303. doi: 10.1007/s10755-014-9313-4 magda, a. j. (2019). online learning at public universities: recruiting, orienting, and supporting online faculty. the learning house, inc. https://49hk843qjpwu3gfmw73ngy1k-wpengine.netdna-ssl.com/wpcontent/uploads/2019/02/201902-onlinelearningpublicuniversities-aascutlh.pdf mandernach, j. b., & holbeck, r. (2016). teaching online: where do faculty spend their time? online journal of distance learning administration xviii(4). https://www.westga.edu/~distance/ojdla/winter194/mandernach_holbeck19 4.html mccormack, s. k. (2015). teaching history online to today’s community college students. the journal of american history 101(4), 1215–1221. https://histsotleurope.wordpress.com/journal-of-american-history/ online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 64 mcdonald, d. (2019). investigating intentionality and mindfulness of storytelling as pedagogy: what student evaluations reveal about my practice. curriculum and teaching dialogue 21(1&2), 97–110. mcquiggan, c. a. (2012). faculty development for online teaching as a catalyst for change. journal of asynchronous learning networks 16(2), 27–61. https://files.eric.ed.gov/fulltext/ej971044.pdf meyer, k. r. (2004). disembodiment for the sake of convenience: an exploratory case study of student and faculty experiences and a nursing program's technology initiative. [unpublished doctoral dissertation]. university of st. thomas, minnesota. https://www.learntechlib.org/p/119492 mitchell, k. m, & martin, j. (2018). gender bias in student evaluations. political science and politics 51(3), 648–652. https://doi.org/10.1017/s104909651800001x munro, l. (2011). go boldly and dream large!: the challenges facing non-traditional students at university. australian journal of education 55(2), 115–131. https://doi.org/10.1177/000494411105500203 neuman, y., neuman, e., & lewis, s. (2017). determents of self-reflective learning and its consequences in online master degree programs. online journal of distance learning administration xviii(1). http://www.westga.edu/~distance/ojdla/spring201/neumann_neumann_le wis201.html plante, k., & asselin, m. e. (2014). best practices for creating social presence and caring behaviors online. nursing education perspectives 35(4), 219–223. doi:10.5480/13-1094.1 pomerantz, j., & brooks, c. d. (2017). ecar study of faculty and information technology, research report. ecar. https://library.educause.edu//media/files/library/2017/10/facultyitstudy2017.pdf rodgers, c. (2002). defining reflection: another look at john dewey and reflective thinking. teachers college record 104(4), 842–866. rosen, a. s. (2017). correlations, trends, and potential biases among publicly accessible web-based student evaluations of teaching: a large scale study of ratemyprofessors.com data. assessment & evaluation in higher education 43(1), 31–44. https://doi.org/10.1080/02602938.2016.1276155 samaras, a. p. (2011). self-study teacher research: improving your practice through collaborative inquiry. sage publications. schmidt, s., tschida, c., & hodge, e. m. (2016). how faculty learn to teach online: what administrators need to know. online journal of distance learning online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 65 administration xix(1), http://www.westga.edu/~distance/ojdla/spring191/schmidt_tschida_hodge1 91.html shea, p. (2007). bridges and barriers to teaching online college courses: a study of experienced online faculty in thirty-six colleges. journal of asynchronous learning networks. 11(2), 73–128. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.453.4788&rep=re p1&type=pdf shea, p. j., pelz, w., fredericksen, e. e., & pickett, a. (2001). online teaching as a catalyst for classroom-based instructional transformation. in j. bourne, & j. c. moore, eds. elements of quality online education, volume 3 in the sloan-c series. the sloan consortium. https://urresearch.rochester.edu/institutionalpublicationpublicview.action?ins titutionalitemid=2499&versionnumber=1 sibley, k., & whitaker, r. (2015). engaging faculty in online education. educause review. https://er.educause.edu/articles/2015/3/engaging-faculty-in-onlineeducation silverman, d., & marvasti, a. (2008). doing qualitative research: a comprehensive guide. sage publications. skibba, k. (2011). an investigation of how faculty members learn to teach adult learners in blended programs (doctoral dissertation). https://search.proquest.com/openview/390fb81b746f83773daca73a714c6ce d/1?pqorigsite=gscholar&cbl=18750&diss=y&casa_token=abqtfmpgr5oaaaaa:bt3n lblzkkzwrawyfyk9vzhej1jht8wfxe2j1gqirmlcpynuopecaeh3vvninbrvrj7n 4lnxdq suleman, q., & gul, r. (2015). a comparative study of directly selected, in-service promoted and online selected subject specialists regarding teaching effectiveness in kohat division, pakistan. journal of education and practice 6(10), 25–36. https://files.eric.ed.gov/fulltext/ej1081655.pdf uhrmacher, p. b. (1997). evaluation change: strategies for borrowing from alternative education theory into practice 36(2), 71–78. https://doi.org/10.1080/00405849709543749 taverna, f., kushnir, l. p, berry, k., & harrison, l. (2015). keeping it real: factors that impact social presence, feelings of isolation, and interactivity in online learning. in s. carliner, c. fulford & n. ostashewski (eds.), proceedings of edmedia 2015—world conference on educational media and technology, 1343–1353. association for the advancement of computing in education (aace). https://www.learntechlib.org/p/151523/ online faculty teaching experiences journal of effective teaching in higher education, vol. 3, no. 1 66 tinkler, b., & tinkler, a. (2019). forging pathways to understanding: using an ecological lens to frame a school reform initiative. curriculum and teaching dialogue, 21(1&2), 51–64. vaill, a.l., & testori, p. a. (2012). orientation, mentoring, and ongoing support: a three-tiered approach to online faculty development. journal of asynchronous learning networks 16(2), 111–119. https://files.eric.ed.gov/fulltext/ej971048.pdf microsoft word cornell et al. final version.docx journal of effective teaching in higher education, vol. 2, no. 2 sense of community in an online graduate program heidi r. cornell, wichita state university, heidi.cornell@wichita.edu donna sayman, wichita state university jason herron, wichita state university abstract. a sense of community among learners is recognized as contributing to positive student outcomes. synchronous learning opportunities may serve as one way to create a sense of community among online learners. findings are shared from a mixed-method study that examined the impact of synchronous discussion sessions on students’ sense of community, as well as additional factors to consider when creating a sense of community. cultivating trusting relationships by providing students with multiple opportunities for authentic, spontaneous, supportive discussion with people they know well emerged as central to fostering a sense of community. implications for online learning in higher education and future research needs are discussed. keywords: online learning, sense of community, synchronous learning, higher education online learning opportunities have become increasingly prevalent over the past few decades as universities seek unique avenues to expand their student population (allen & seaman, 2013; shackelford & maxwell, 2012). distance learning offers a low cost, flexible platform for working adults to earn their degree (boling et al., 2012). as leader-janssen, nordness, swain, and hagaman (2016) affirmed, online courses must contain the same rigor and quality as traditional on-campus courses. in fact, cummings and townly (2016) recognized that as online course offerings are growing, students question whether the quality of the course is equivalent to the face-to-face environment. researchers have discovered that creating and maintaining a sense of community among online learners is a crucial factor for the students’ success (hart, stewart, & jimerson, 2011; shackelford & maxwell, 2012). sense of community mcmillan and chavis (1986) define sense of community as “a feeling that members have of belonging, a feeling that members matter to one another and to the group, and a shared faith that members’ needs will be met through their commitment to be together” (p. 9). the concept of sense of community is recognized as an essential component to supporting social and intellectual learning goals (rovai, 2002). additionally, a sense of community is recognized as an important factor to foster social and teaching presence within the community of inquiry (coi) framework that is commonly used to explain effective online learning (kim, kwon & cho, 2011; shea et al., 2005). as such, the importance of creating a sense of community is widely recognized as essential to ensure effective online learning programs. sense of community in a special education online graduate program 118 journal of effective teaching in higher education, vol. 2, no. 2 creating a sense of community is critical because it is associated with a variety of positive student outcomes. for example, a sense of community among online learners has shown to help with student retention (kuo et al., 2013; lorenzo, 2015). broadly, research also continues to show a connection between students’ sense of community, learning outcome, and academic achievement (e.g., akyol & garrison, 2011; arbaugh et al., 2008; wighting & derrick, 2015). in addition, a sense of community has been found to lead to deeper thinking among students (hulon, 2013; larson & kieper, 2002), as well as increased satisfaction and motivation (boling et al., 2012; hart et al., 2011; lorenzo, 2012). in fact, many students have identified a sense of community as a helpful aspect of their learning experience (hart, stewart & jimerson, 2011; shackelford & maxwell, 2012) as it has the potential to minimize feelings of isolation and help students feel better connected to other learners. given the important role sense of community has on student outcomes in the online learning environment, some research has attempted to identify general recommendations to promote a sense of community. for example, shackelford and maxwell (2012) found that the following have an impact on developing students’ sense of community in online learning: introductions, collaborative group projects, contributing personal experiences, entire class online discussions, and exchanging resources. haythornwaite, kazmer, robins, and shoemaker (2000) suggested the following instructional strategies to promote sense of community: promoting initial bonding, monitoring and supporting interaction and participation, and providing multiple ways of communication. in addition, palloff and pratt (2007) suggested that active interaction, collaborative learning, socially constructed meaning, resource sharing, and expressions of support and encouragement would all promote a sense of community in the online environment. although these general recommendations are useful, very few studies have yet to explore the specific relationship between varied amounts and types of synchronous learning opportunities and students’ sense of community. synchronous learning opportunities synchronous learning opportunities facilitate teacher-student social interaction in a virtual classroom space (szeto & cheng, 2016). a number of studies have demonstrated the benefits of online synchronous teaching and learning, such as enhanced learning, improved communication and strong group cohesion (boling et al., 2012; chatterjee & juvale, 2015; croxton, 2014). for example, wdowik (2014) provided students with an online learning community through the use of blackboard collaborate and found that it enhanced student engagement levels and quality of student learning by allowing the instructor to be more accessible and supportive. additionally, synchronous online learning is recognized as one way to enhance the element of social presence, which is defined as “the ability of learners to project their personal characteristics into the community of inquiry, thereby presenting themselves as ‘real people’ (rourke et al., 2001). in other words, it is the feeling of being present with another person (i.e. instructor and/or classmates) in an online learning space. szeto and cheng (2016) argue that the flow of interactions that are thought to contribute to a sense of community in a face-to-face classroom can be sense of community in a special education online graduate program 119 journal of effective teaching in higher education, vol. 2, no. 2 similarly created online using videoconferencing technologies. however, very few studies have examined specific strategies and/or methods for how to best use these technologies to create synchronous learning opportunities that promote a sense of community. moreover, despite the benefits that are starting to emerge, synchronous online learning is an area within the distance education field that is still largely understudied, and it is recognized that additional research is needed (martin, ahlgrim-delzell, & budhrani, 2017). as such, the current study sought to examine the impact of synchronous learning opportunities on student’s sense of community. in addition, this study sought to identify additional factors to consider that might increase sense of community among students enrolled in a special education online graduate program. specifically, the research questions were: research question 1: do synchronous discussion sessions increase student’s sense of community in an online course as measured by the sci-2? research question 2: in what ways do students perceive a sense of community was and can be promoted in their online special education graduate degree program? research question 3: how do the qualitative findings contribute to a more comprehensive and nuanced understanding of how sense of community is promoted through the use of synchronous discussion sessions? methods a convergent parallel mixed-methods design was employed to allow qualitative measures to elaborate, enhance, and/or illustrate the results from a quantitative measure (fetters, curry, & creswell, 2013). consistent with this design approach, both qualitative and quantitative data were collected and analyzed during relatively similar timeframes, with integration occurring after the data collection and analysis process had been completed. furthermore, the purpose of this design approach was to yield a more nuanced examination of the impact of synchronous discussion sessions, as well as to identify additional strategies or things to consider that might contribute to creating a sense of community within the online learning environment. participants & setting in the fall of 2017, the special education programs at one midwestern university began delivery of their programs completely online. in the past, only a few courses were online and most were taught using a hybrid format. at the start of the fall 2017 semester, a total of 22 graduate students in a special education program were recruited to participate in this study during the first week of their advanced level methods course. these students were asked to participate because they would have already completed at least one semester of their online graduate program. during the first learning module, students were asked to watch a short video about the study and if willing to participate, were asked to give consent. a total of 18 students agreed to participate in the study. all students who participated in the study were taking online courses and also working full-time as special education teachers. all but one student was female and self-identified as caucasian. one student was male and self-identified as hispanic. it is possible that the participants may have taken previous courses or may work in the same school district. sense of community in a special education online graduate program 120 journal of effective teaching in higher education, vol. 2, no. 2 delivery of synchronous discussion sessions in two different graduate advanced methods courses, synchronous discussion sessions were created in order to examine the impact these opportunities had on students’ sense of community. each course was taught by a different instructor. the course instructors were both tenure track faculty members who work closely together to deliver the graduate program online. all students were expected to participate in the online synchronous discussion sessions, regardless of their willingness to participate in the study. students earned participation points for attending each zoom discussion session. the zoom platform (https://zoom.us) was used to facilitate all synchronous discussion sessions. during the first learning module in each course, students were asked to sign up for four 30-40 minute mandatory small group sessions (approximately once a month). each small group session included no more than four students. as such, there were a total of two small groups in one of the graduate courses and four small groups in the other, for a total of 24 small group discussion sessions across both courses. the course instructor participated in each discussion session and told students at the start of the course that the purpose of the synchronous sessions was to reflect on, discuss, and clarify course related content. moreover, students were asked to complete a pre-meeting worksheet and submit it to their instructor no later than one day prior to their scheduled synchronous discussion session. the pre-meeting worksheet asked students to respond to three questions and included an agenda to be followed in order to best facilitate meaningful discussions in a short amount of time. discussion sessions were well-attended by all students enrolled in both courses. quantitative strand: survey data instrument. the sense of community index-2 (sci-2; chavis, lee, & acosta, 2008) questionnaire was used to gain information related to the extent to which synchronous teacher-student interaction experiences increased students’ sense of community. the sci-2 questionnaire is a survey that measures sense of community and is frequently used in the social sciences (chavis et al., 2008). the instrument allows researchers to define the “community” as being “the individuals that are a part of their online learning course experience”. the sci-2 consists of 24 items that are presented on a likert-type scale: 0 (not at all), 1 (somewhat), 2 (mostly), and 3 (completely). the sci-2 yields four subscale scores (e.g., reinforcement of needs, membership, influence, shared emotional connection) and a total sense of community index. the subscale ‘reinforcement of needs’ measures the extent to which “needs are met by the resources received through membership in the group” (mcmillan & chavis’, 1986, pg. 9), and is scored by summing questions 1 through 6. the subscale ‘membership’ measures the feeling of belonging or of sharing a sense of personal relatedness” (mcmillan & chavis’, 1986, pg. 9), and is scored by summing questions 7 through 12. the subscale ‘influence’ measures the “sense of mattering, of making a difference to a group and of the group mattering to its members” (mcmillan & chavis’, 1986, pg. 9), and is scored by summing question 13 through 18. the subscale ‘shared emotional connection’ measures “commitment and belief that members have shared and will share history, common places, time together, and similar experiences” (mcmillan & sense of community in a special education online graduate program 121 journal of effective teaching in higher education, vol. 2, no. 2 chavis’, 1986, p. 9), and is scored by summing questions 19 through 24. lastly, the ‘total sense of community index’ is a combination of these subscales and is intended to measure a sense of community as defined by mcmillan & chavis (1986), which states that sense of community is “a feeling that members have of belonging, a feeling that members matter to one another and to the group, and a shared faith that members’ needs will be met through their commitment to be together (mcmillian & chavis, 1986, p. 9).the total sense of community index is scored by summing all items (i.e. questions 1 through 24) together. among a survey of 1,800 people, the sci-2 showed to be a very reliable measure (coefficient alpha =.94). the specific subscales also indicated reliability with coefficient alpha scores of .79 to .86 (chavis et al., 2008). data collection procedure. the sci-2 questionnaire was entered into qualtrics for dissemination. participants were asked to complete a pre-course and post-course sci-2 survey. the pre-course survey was administered at the start of the first learning module. participants were explicitly asked to answer questions as they related to previous courses they had taken in their online special education graduate program. prior to this study, synchronous discussion activities had not been implemented in courses that were part of this online program. similarly, a link to the post-course survey was accessible to participants in their last learning module. for the post-survey, participants were explicitly asked to answer questions as they related to their experience in the advanced level methods course they had just completed, which included a series of required synchronous group discussion meetings. all 18 participants completed both the pre and post survey questionnaire. data analysis. survey data was entered into spss statistical software for analysis. a series of repeated measures anova (rmanova) were conducted in order to assess the extent to which there were significant mean differences between participant’s scores on the dependent variable (sense of community) subscores between administrations. this approach was appropriate because data at different administration points are correlated due to within subject factors (girden, 1992; lovie, 1981). qualitative strand: focus groups focus groups were conducted to gain a comprehensive understanding of how synchronous discussion sessions contributed to students’ feelings of sense of community and to understand other ways a sense of community could be created in the online special education graduate program. moreover, the intent of the focus groups was to collect data to complement the quantitative survey data by allowing students to provide contextual elaborations that could enhance, illustrate, and/or confirm survey results. data collection procedure. participants were asked to sign up (via link to google doc) for a focus group session during the first learning module in their advanced level methods course. a total of four, 30-45-minute focus group sessions were held the week after the course ended and after the post-survey to ensure that survey results were not influenced by participation in the focus group or expected sense of community in a special education online graduate program 122 journal of effective teaching in higher education, vol. 2, no. 2 course grade. initially, focus groups were scheduled to have 3-4 participants in each group. however, only seven students participated in the focus group sessions, which resulted in two small focus groups of two and three participants, and two individual interviews. we believe that the lack of participation in this phase of the study was a result of the focus groups being scheduled the week after the course had ended. despite the lack of participation from all those who had completed the survey, results still yielded important themes to help explain quantitative data and answer the study research questions. focus groups (and/or individual interviews) were conducted at the scheduled times virtually through zoom (https://zoom.us ). a semi-structured interview protocol (see table 1) was followed to guide the focus group session. the questions used also served as prompts in order to obtain more in-depth information from participants as it related to the study research question. the audio and video from each focus group was recorded and saved to a secure study database. each focus group session was then transcribed by the authors’ graduate research assistant and also uploaded to the study database for analysis. table 1 semi-structured focus group/interview questions 1. how important is it to you to establish a connection with your peers in class? why or why not? 2. did you feel that you could share problems and concerns with other people in this class? 3. to what extent did the collaboration help you to establish a sense of community with the other people in the class? 4. what aspects of the class did you like the best? 5. what aspects did you like the least? 6. would you recommend this course/program to others? 7. do you have anything else you would like to share regarding the course, course design, or tools and activities in this course? data analysis. the analysis of focus group data began with the reading and rereading of all transcriptions of recorded focus groups (stake, 2006). the researchers then separately coded focus group data using a content analysis approach to identify themes from the data relating to study research questions. qualitative content analysis is defined as method for “the subjective interpretation of the content of text data through the systematic classification process of coding and identifying themes or patterns” (hsieh & shannon, 2005, p. 1278). transcripts were analyzed using line by line analysis, allowing for categories, subcategories and themes to emerge. both researchers met weekly to discuss discrepancies that emerged from their independent efforts and until they came to full agreement about the coding. the analysis process was iterative and continued until no new information emerged from the data. as a result, several assertions were generated that captured the codes and themes and were supported by the data. sense of community in a special education online graduate program 123 journal of effective teaching in higher education, vol. 2, no. 2 results combining quantitative and qualitative data in a mixed methods study yields a deeper understanding of the issues under analysis (creswell, 2014). in the following sections, results from the analysis of quantitative and qualitative strands of this study are presented. consistent with approaches used in publications of other mixed methods studies (i.e., garwood, werts, varghese, & gosey, 2018; mcauley et al., 2006), quantitative and qualitative strands are integrated and presented through two central themes in the discussion of these results that follow. quantitative strand a total of five rmanova were conducted (e.g., reinforcement needs, membership, influence, shared emotional connection, total sci index). participant (n = 18) subscale scores were grouped between administrations to facilitate comparisons between pre and post-test. the mean sum scores for the reinforcement needs, membership, and influence subscales, as well as the total sci index increased from pre to post survey (see table 2). however, results indicated that there was only a statistically significant [wilks lambda = .784, f (1,17) p = .045, partial eta2 = .216, power = .532] mean difference between the pre (m = 9.444, sd = 4.259) and post survey (m = 11.556, sd = 3.416) sum scores on the reinforcement needs subscale. as such, analysis indicated that there was a non-significant [wilks lambda = .877, f (1,17) 2.377, p = .142, partial eta2 = .123, power = .307] mean difference between sum scores of the total sci index and between administrations of the remaining subscales: membership [wilks lambda = .853, f (1,17) p = .105, partial eta2 = .147, power = .366], influence, [wilks lambda = .916, f (1,17) p = .228, partial eta2 = .084, power = .219], shared emotional connection [wilks lambda = .999, f (1,17) p = .889, partial eta2 = .001, power = .052]. table 2 anova tables – sense of community subscales subscale pretest m (sd) posttest m (sd) fa p partial η2 reinforcement needs 9.444 (4.259) 11.555 (3.416) 4.67 4 .045* .216 membership 7.833 (3.959) 9.444 (4.003) 2.93 9 .105 .147 influence 9.833 (3.601) 11.000 (4.256) 1.56 3 .228 .219 shared emotional connection 8.6111 (4.717) 8.500 (3.944) .999 .889 .001 total sci index 35.722 (14.96) 40.500 (12.39) .877 .142 .123 note: af wilks lambda multivariate tests used for all subscales * the mean difference is significant at the .05 level. sense of community in a special education online graduate program 124 journal of effective teaching in higher education, vol. 2, no. 2 qualitative strand a total of five themes were identified that relate to ways students perceive a sense of community was and can be promoted in their online classes: (a) makes people “real”, (b) authentic, spontaneous discussion, (c) shared experiences, (d) shared learning, and (e) trusting relationships. make people “real”. all students reported that a sense of community can be created by making people real. they found this was done in the current course with the use of synchronous discussion sessions. for example, one student stated that she “enjoyed being able to see [her] classmates”. another student stated, “the meetings helped to make my classmates more "real" to me by being able to interact and hear their voices and their comments.” similarly, one student stated, “the best ways i’ve felt a sense of community is when i can simply see another person’s face.” the synchronous discussion sessions made people real because their classmates were no longer just a name. for example, one student stated, “i was just going to say i really like putting a face to a name because after fourteen, fifteen weeks you really do get to notice those names. but when it’s all online you don’t put a face to it.” another student made a similar point by stating, “i like putting a face with a name. i guess when we use all of our senses, it means more” authentic, spontaneous discussion. the majority of students also felt that creating opportunities for authentic, spontaneous discussion contributed to improved sense of community. students felt that the synchronous discussion sessions were more authentic than other asynchronous discussion forum activities because they weren’t forced to reply or discuss specific topics. for example, one student stated, [the synchronous discussion sessions] helped with the contribution of a sense of community way more than our message board posts. to be honest, i hate responding to 3 of my peers every week. i don’t particularly always have something to say about what others have written, so i find myself making up something i don’t really feel deeply about. on the other hand, when we do the meetings, i am constantly engaged with my peers. i’d even go so far to say that i look forward to them. students also reported that these more authentic, less forced discussions allowed for the opportunity to have more spontaneous discussions that were less structured. students found these authentic, spontaneous discussions to be beneficial. for example, one student stated, [i] enjoyed the impromptu, less structured conversations that could happen in this format. by meeting in this way, i was better able to understand what types of settings others were teaching in and know who might be a good resource for certain questions that i had. with this said, one student diverged slightly from the views of others by saying that she valued authentic, spontaneous discussion, but didn’t feel it helped her improve her sense of community. she felt she benefited just as much, if not more, from sense of community in a special education online graduate program 125 journal of effective teaching in higher education, vol. 2, no. 2 asynchronous interactions (i.e. blogs, journals, and discussion forums) with other students in the online classroom environment and in some instances preferred them more than the synchronous discussions. for example, she stated, i felt there was a sense of community and i think that helps. i really enjoyed the blogs and the journals. i enjoy that aspect of online class….. i liked the zoom meetings tooi just had a little trouble getting on and then i always feel like i want to talk over people and so i feel like i have a little of trouble with the zoom meetings because i keep wanting to jump in. i have a little trouble waiting my turn. shared experiences. the majority of students also shared that having opportunities to share similar experiences contribute to improved sense of community as it “brings people together”. they expressed comfort in hearing that there “were other special education teachers that were struggling too.” students shared that they have opportunities to share experiences through asynchronous discussion forums, blogs, and journal assignments, but the authentic, spontaneous discussion that occurs during the synchronous discussion session provides the best opportunity for sharing similar experiences. for example, one student stated, i think so, i think the zoom meetings especially like when we would be talking about stuff even with the people that i didn’t work with in my usd, district or anything. it was nice to see that a lot of us had the same questions or issues. even in some of the larger groups and we would be like well we have a behavior team, well we don’t have that but you guys are going through the same stuff but we just don’t have a team. that was really nice. it was like even though we are all very different in areas we all kind of dealing with the same issues. another student stated, i like meeting new people, and talking about our crazy jobs together. it is hard to find many people to vent to about hard days because one simply cannot convey what this job looks like until you are actually living it! shared learning. several students indicated that a sense of community could be created by having opportunities to engage in shared learning activities where students could work together to create a common goal or task. for example, students expressed that a sense of community could be promoted through collaborative group activities and their desire to engage in shared learning, such that one student stated, well i guess i’m kind of the older generation but i thought that was part of learningyou learn with othersthat’s what makes the classroom different than… personally i think i learn from other people a lot more and i know this kind of setting is quite different – we get to see each other like we are doing right now for a short period of time. we don’t have the timeit’s a high priority for me because i think learn a lot from others. trusting relationships. many students shared the need to build trusting relationships with their classmates to facilitate deeper discussion that would lead to sense of community in a special education online graduate program 126 journal of effective teaching in higher education, vol. 2, no. 2 a stronger sense of community. they stated that it is important to get to know their classmates and that this can be done by interacting with the same students across multiple activities and classes. although the synchronous discussion sessions afforded some students an opportunity to get to know their classmate better, some still felt they didn’t get to know their classmates well enough because each discussion session had different students, which resulted in their reluctance to fully participate in the discussion and feel connected to others. for example, one student stated, [you] have to continue with that’s who you are meeting withzoom meetings….i don’t think i really saw the same people every zoom meeting i had…. you know that way there is a connection. you know that is one way that i thought you were putting us in a group so that we could see each other at our zoom meetings. that way we could see each otherwho we are and you know it was never that way. another student stated, not really because i didn’t really feel secure or that i knew the people [in my group] enough to say we are going to keep it together or that we are going to keep it within our family. you know i see kids all the time and i deal with kids all the time and they aren’t going to tell you what’s on their minds until they feel comfortable and they trust you. discussion the importance of creating and sustaining a sense of community in online learning programs is widely recognized (hart et al., 2011; kuo et al., 2013; lorenzo, 2015; shackelford et al., 2012). as online course offerings continue to grow (allen & seaman, 2013; cummings & townly, 2016), identifying specific methods and strategies online instructors can use to target improvements in students’ sense of community is critical. as such, the purpose of this study was twofold: to examine the impact of synchronous discussion sessions on students’ sense of community, as well as identify additional methods or strategies that might increase sense of community among students enrolled in an online graduate program. qualitative findings contributed to a more comprehensive and nuanced understanding of how sense of community was and can be promoted through the use of synchronous discussion session. synchronous online discussion sessions as a way to increase students’ sense of community integrated analysis suggest that synchronous online discussion sessions are a promising way to increase students’ sense of community. for example, there was an increase in the total sci-2 score and three of the subscale scores (e.g., reinforcement needs, membership, and influence) from pre to post administration with one of the subscale score increases being statistically significant (e.g., reinforcement needs). participants confirmed these improvements by indicating they indeed felt the synchronous discussion sessions helped create a sense of community. more specifically, participants attributed their improved sense of community to the opportunity they had to share similar experiences in a more sense of community in a special education online graduate program 127 journal of effective teaching in higher education, vol. 2, no. 2 authentic, spontaneous way. it may be that by sharing similar experiences, challenges, priorities, and personal stories, students were able to identify certain values that they shared with their classmates. mcmillian & chavez (1986) share, as part of their sense of community theory, that having shared values is something individuals have who are part of strong communities. similarly, in a study conducted by keehn (2015), students felt that sharing personal experiences during synchronous learning encounters was “engaging, enjoyable and integral to their learning”. as such, it is likely that the interactions afforded to participants in the current study allowed participants to obtain the reinforcement (i.e., reinforcement needs) they needed to grow closer together as part of a learning community (mcmillian & chavis, 1986). given these promising findings, online instructors should consider synchronous learning opportunities where students are given opportunities to receive and offer support through the sharing of similar experiences and challenges. additional strategies for creating a sense of community although there was an increase in the mean subscale scores for membership and influence, integrated analysis revealed additional strategies that are likely needed to reach levels of significance. first, it appears to be important for instructors to create ways for students to build trusting relationships with their classmates in order improve the element of membership. mcmillian and chavez (1986) suggest that students who feel as if they are members have been provided with the “emotional safety necessary for needs and feelings to be exposed and for intimacy to develop” (p. 9). although the majority of participants found the synchronous discussion sessions beneficial, a few expressed that they did not participate much because they did not know their classmates well enough or did not trust them to “keep things in the family”. this may indicate that not all students felt emotionally safe enough for the level of intimacy to develop that is needed to have complete membership as part of the community. students offered suggestions on how trusting relationships might be created, such as enrolling students as a cohort, creating partner and/or group assignments, and engaging in discussions with some of their same classmates multiple times, across more than one course. moreover, synchronous learning opportunities may have the most influence on student’s sense of community when they are created across multiple courses within a program with similar classmates. in terms of influence, qualitative analysis revealed that instructors should consider adding opportunities for students to take on leadership roles and engage in collaborative shared learning activities. in addition, instructors may want provide online students with more choice in how their classes are structured to embrace more of a constructive approach to teaching and learning. it appears that doing these things might afford students a way to feel more influential (i.e., influence subscale), which will likely lead to feeling more connected to their learning community (i.e. a better sense of community). sense of community in a special education online graduate program 128 journal of effective teaching in higher education, vol. 2, no. 2 limitations results from this study must be interpreted with caution. first, using a pre/post experimental design, some generalizations can be made to online graduate programs, but generalizability is limited due to a smaller sample size. another limitation is the small sample size of the focus group. it is not possible to determine why those 7 chose to participate in the focus group following the end of the semester. they could have felt strongly about the course delivery or may have participated to appear to be compliant to the instructor. despite the minor limits to generalization, findings do have a higher degree of transferability due to the mixed method nature of the study. regardless, it should be recognized that the experiences of the students in this study might not be reflective of the experiences of students in other programs. second, the administration of the sci-2 survey prior to focus group interviews may have influenced group discussion due to an ordering effect (israel & taylor, 1990). lastly, participants may not have been completely honest or forthcoming because they are not used to being asked what they think about course delivery (messick & jackson, 1961; watson, 1992). although students participated voluntarily and were told that their participation and answers would not affect their grade, there is a chance that they may have felt pressured to give positive responses as not to negatively impact their course grade. conclusions with the increase in online learning, the importance of maintaining a sense of community will continue to become more urgent. although future research is needed to compare asynchronous and synchronous online instructional strategies, the experiences shared by participants in this study provide insight into the positive impact of synchronous discussion sessions, as well as additional factors instructors may want to consider when creating a sense of community among students in an online program. as such, cultivating trusting relationships by providing students with multiple opportunities for authentic, spontaneous, supportive discussion with people they know well emerged as central to fostering a sense of community among online learners. as online learning continues to grow, we can move forward by continuing to use research based strategies to make informed decisions when designing quality learning experiences. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references akyol, z., & garrison, d.r. (2011). assessing metacognition in an online community of inquiry. the internet and higher education, 14(3), 183-190. allen, i. e., & seaman, j. (2013). changing course: ten years of tracking online education in the united states. newburyport, ma: sloan consortium. sense of community in a special education online graduate program 129 journal of effective teaching in higher education, vol. 2, no. 2 arbaugh, j. b., cleveland-innes, m., diaz, s. r., garrison, d. r., ice, p., richardson, j. c., & swan, k. p. (2008). developing a community of inquiry instrument: testing a measure of the community of inquiry framework using a multi-institutional sample. the internet and higher education, 11(3-4), 133-136. boling, e. c., hough, m., krinsky, h., saleem, h., & stevens, m. (2012). cutting the distance in distance education: perspectives on what promotes positive, online learning experiences. the internet and higher education, 15(2), 118126. retrieved from: https://teachingcommons.yorku.ca/wpcontent/uploads/2017/11/cutting-the-distance-in-distance-education.pdf chatterjee, r., & juvale, d. (2015, march). a mixed method study of the relationship between online collaborative learning activities and students’ sense of community in an online environment in higher education. in society for information technology & teacher education international conference (pp. 223-228). association for the advancement of computing in education (aace). chavis, d. m., lee, k. s., & acosta, j. d. (2008, june). the sense of community (sci) revised: the reliability and validity of the sci-2. in 2nd international community psychology conference, lisboa, portugal. creswell, j. w. (2014). a concise introduction to mixed methods research. thousand oaks, ca: sage publications. croxton, r. a. (2014). the role of interactivity in student satisfaction and persistence in online learning. journal of online learning and teaching, 10(2), 314. retrieved from: https://pdfs.semanticscholar.org/2a3c/ab58d3d0637d20d907d67fecf3c34685 1393.pdf cummings, c. & townly, g. (2016). navigating academia: a mixed methods analysis of sense of community, student-faculty relationships, and student success. psu mcnair scholars online journal, 10(1), 4. retrieved from: https://pdxscholar.library.pdx.edu/cgi/viewcontent.cgi?article=1166&context =mcnair fetters, m. d., curry, l. a., & creswell, j. w. (2013). achieving integration in mixed methods designs—principles and practices. health services research, 48(6pt2), 2134-2156. retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/pmc4097839/ garwood, j.d., werts, m.g., varghese, c., & gosey, l. (2018). mixed-methods analysis of rural special educators’ role stressors, behavior management, and burnout. rural special education quarterly, 37(1), 30-43. https://doi.org/10.1177/8756870517745270. girden, e. r. (1992). anova: repeated measures. newbury park, ca: sage. https://doi.org/10.4135/9781412983419. hart, s. r., stewart, k., & jimerson, s. r. (2011). the student engagement in schools questionnaire (sesq) and the teacher engagement report formsense of community in a special education online graduate program 130 journal of effective teaching in higher education, vol. 2, no. 2 new (terf-n): examining the preliminary evidence. contemporary school psychology, 15, 67-79. retrieved from: https://files.eric.ed.gov/fulltext/ej934707.pdf haythornthwaite, c., kazmer, m., robins, j., & shoemaker, s. (2000). making connections: community among computer-supported distance learners. in association for library and information science education 2000 conference. san antonio, texas. retrieved july (vol. 3, p. 2003). hsieh, h. f., & shannon, s. e. (2005). three approaches to qualitative content analysis. qualitative health research, 15(9), 1277-1288. doi: 10.1177/1049732305276687 hulon, s. (2013, march). face-to-face discussions versus online threaded discussions: can we have the best of both worlds? in society for information technology & teacher education international conference (pp. 577-582). association for the advancement of computing in education (aace). israel, g. d., & taylor, c. l. (1990). can response order bias evaluations? evaluation and program planning, 13(4), 365–371. doi: doi.org/10.1016/0149-7189(90)90021-n kahn, p., everington, l., kelm, k., reid, i., & watkins, f. (2017). understanding student engagement in online learning environments: the role of reflexivity. educational technology research and development, 65:203–218. doi 10.1007/s11423-016-9484-z keehn, m. g. (2015). “when you tell a personal story, i kind of perk up a little bit more”: an examination of student learning from listening to personal stories in two social diversity courses. equity & excellence in education, 48(3), 373-391. https://doi.org/10.1080/10665684.2015.1056712 kim, j., kwon, y., & cho, d. (2011). investigating factors that influence social presence and learning outcomes in distance higher education. computers & education, 57(2), 1512-1520. doi:10.1016/j.compedu.2011.02.005 kuo, y. c., walker, a. e., belland, b. r., & schroder, k. e. (2013). a predictive study of student satisfaction in online education programs. the international review of research in open and distributed learning, 14(1), 16-39. larson, b. e., & keiper, t. a. (2002). classroom discussion and threaded electronic discussion: learning in two arenas. contemporary issues in technology and teacher education, 2 (1), 45-62. retrieved from: https://www.learntechlib.org/primary/p/14452/. leader-janssen, e. m., nordness, p. d., swain, k. d., & hagaman, j. l. (2016). students’ perceptions of an online graduate program in special education for emotional and behavioral disorders. teacher education and special education, 39(4), 246-258. https://doi.org/10.1177/0888406416637411 lorenzo, g. (2015). a research review about online learning: are students satisfied? why do some succeed and others fail? what contributes to higher retention rates and positive learning outcomes?. internet learning, 1(1), 5. sense of community in a special education online graduate program 131 journal of effective teaching in higher education, vol. 2, no. 2 lovie, a.d. (1981). on the early history of anova in the analysis of repeated measure designs in psychology. british journal of mathematical and statistical psychology, 34(1), 1-15. martin, f., ahlgrim-delzell, l., & budhrani, k. (2017). systematic review of two decades (1995 to 2014) of research on synchronous online learning. american journal of distance education, 31(1), 3-19. https://doi.org/10.1080/08923647.2017.1264807 mcmillan, d. w., & chavis, d. m. (1986). sense of community: a definition and theory. journal of community psychology, 14 (1), 6-23. retrieved from http://aisel.aisnet.org/amcis2005/306 mcauley, c., mccurry, n., knapp, m., beecham, j., & sleed, m. (2006). young families under stress: assessing maternal and child well-being using a mixedmethods approach. child & family social work, 11(1), 43-54. messick, s., & jackson, d.n. (1961). acquiescence and the factorial interpretation of the mmpi. psychological bulletin, 58(4), 299. paley, j. (2016). phenomenology as qualitative research: a critical analysis of meaning attribution. new york, ny: routledge press. palloff, r. m., & pratt, k. (2007). online learning coƒmmunities in perspective. in r. luppicini (ed.) online learning, (pp.3-16). charlottesville, nc: information age publishing. rourke, l., anderson, t., garrison, d. r., & archer, w. (2001). methodological issues in the content analysis of computer conference transcripts. international journal of artificial intelligence in education (ijaied), 12, 8-22. retrieved from: https://telearn.archives-ouvertes.fr/hal-00197319/document rovai, a. p. (2002). sense of community, perceived cognitive learning, and persistence in asynchronous learning networks. the internet and higher education, 5(4), 319-332. doi: 10.1016/s1096-7516(02)00130-6 shackelford, j. l., & maxwell, m. (2012). sense of community in graduate online education: contribution of learner to learner interaction. the international review of research in open and distributed learning, 13(4), 228-249. shea, p., li, c. s., swan, k., & pickett, a. (2005). developing learning community in online asynchronous college courses: the role of teaching presence. journal of asynchronous learning networks, 9(4), 59-82. stake, r.e. (2006). multiple case study analysis. new york: the guilford press. szeto, e., & cheng, a. y. (2016). towards a framework of interactions in a blended synchronous learning environment: what effects are there on students' social presence experience?. interactive learning environments, 24(3), 487-503. doi:10.1080/10494820.2014.881391 watson, d. (1992). correcting for acquiescent response bias in the absence of a balanced scale: application to class consciousness. sociological methods & research, 21(1), 52–88. retrieved from: https://www.researchgate.net/profile/dorothy_watson2/publication/2496939 sense of community in a special education online graduate program 132 journal of effective teaching in higher education, vol. 2, no. 2 16_correcting_for_acquiescent_response_bias_in_the_absence_of_a_balan ced_scale_an_application_to_class_consciousness/links/588602b992851c21 ff4d3044/correcting-for-acquiescent-response-bias-in-the-absence-of-abalanced-scale-an-application-to-class-consciousness.pdf wdowik, s. (2014). using a synchronous online learning environment to promote and enhance transactional engagement beyond the classroom. campus-wide information systems, 31(4), 264-275. https://doi.org/10.1108/cwis-102013-0057 wighting, m., & derrick, g. (2015, march). relationships between sense of community and academic achievement: a comparison among teacher preparation candidates in an online program. in society for information technology & teacher education international conference (pp. 3507-3510). association for the advancement of computing in education (aace). microsoft word steiner final version.docx journal of effective teaching in higher education, vol. 1, no. 2 bringing a learning strategies project to scale in a first-year seminar hillary h. steiner, ph.d., kennesaw state university, hsteiner@kennesaw.edu nirmal h. trivedi, kennesaw state university joshua a. brown, kennesaw state university abstract: this paper describes the impact of a learning strategies intervention conducted in first-year seminar courses that, 1) disaggregated components of academic skills into meaningful components for first-year students, 2) taught students academic skills within an authentic context, and 3) scaled-up the intervention for implementation at a programmatic level. this work is grounded in research on metacognition, self-regulation, and motivation, as well as literature on the academic transition to college. results reinforced earlier findings indicating significant improved use of metacognitive learning strategies, even when the intervention was expanded to include multiple instructors in multiple course sections. further research is needed to determine the precise factors precipitating improvement when the project was brought to scale. keywords: first-year seminar, metacognition, self-regulated learning, learning strategies teaching academic skills in a first-year seminar has been an established practice for several years. in the national resource center’s 2012-2013 national survey of first-year seminars (nsfys), developing academic skills was reported as the third most common course objective after developing a connection to an institution and orienting students to campus resources (young & hopp, 2014). a precise description of what is meant by “academic skills,” however, varies tremendously, depending on the course characteristics and institution type. four-year colleges often include critical thinking skills, motivation, time management, oral communication, and faculty-student communication as academic skills. two-year colleges might include the development of study skills, academic planning, grammar, and note-taking (young & hopp, 2014). acquiring these skills is important for college success; however, designing adequate curricula is challenging for first-year seminar programs given the vast range of learning skills and study habits of beginning college students (balduf, 2009). the present paper presents the results of an implementation of a learning strategies project in a three-credit hour, letter-graded first-year seminar conducted first as a pilot study and then scaled-up for implementation in several course sections, with an eventual goal of implementation in an entire first-year seminar program. the strategy project assignment (see appendix) is unique in that it disaggregates “academic skills” into component parts that enable the first-year student to understand the differences and relationships between time management, motivation, study skills, and metacognition. furthermore, the strategy project is context-specific: to complete the project, students plan, study, and reflect on their bringing a learning strategies project to scale in a first-year seminar 28 journal of effective teaching in higher education, vol. 1, no. 2 planning for and execution of an exam taken in another course, thus providing learners with an authentic context for developing specific academic skills. the research study described herein was guided by the question: does the strategy project impact first-year students’ self-regulation and metacognition strategies as measured by the motivated strategies for learning questionnaire (mslq) and selfreported strategy use when brought to scale for a large first-year seminar program? literature review research on self-regulated learning has a long history in the educational psychology literature, originating from work by zimmerman, schunk, pressley, and others on goal setting, motivation, and strategy use (zimmerman, 2008). as zimmerman (2008) notes, a self-regulated learner is one who is motivated, reflective, and proactive in his use of learning strategies. for college students, self-regulation of learning strategies involves taking the initiative to learn and implement appropriate strategies for the task, set goals for learning, and reflect on one’s approach, modifying it for the next task if necessary. metacognition, most commonly defined as the knowledge and regulation of one’s own cognition, is closely related to selfregulation in that a metacognitive student is aware of what she does and does not know, and plans for, monitors, and reflects on the acquisition of that knowledge. students exhibit good metacognitive behaviors when they choose and use appropriate learning strategies for the task at hand and reflect on the results of their efforts. self-regulating and metacognitive behaviors are proven predictors of academic success (pintrich & degroot, 1990). however, many first-year students have not yet developed metacognitive and self-regulated learning strategies that are sufficient for success in their new academic environment. in college, learning strategies that go beyond memorization and passive knowledge acquisition are essential, as students who use higher-level metacognitive strategies tend to be more successful in their courses (dunlosky, rawson, marsh, nathan, & willingham, 2013; kitsantas, 2002). for example, students who are taught to monitor their comprehension and evaluate their learning as they read show deeper processing of the material and better retention than students who passively read the text (nashditzel, 2010). in addition, first-year students who successfully manage their time and seek out existing resources show greater achievement in the first year (thibodeaux, deutsch, kitsantis, & winsler, 2017; tuckman, 2003). many students enter college believing that they are prepared for this challenge due to their success in high school (balduf, 2009). however, because learning tasks at the college level require different skills and greater personal responsibility for managing those skills, many high school students begin to show deficits in these abilities once they reach the college level (cohen, 2012). this is not limited to poor performing students. while some students may have better self-regulation strategies than their peers in middle and high school (e.g., zimmerman & martinezpons, 1990) those same students may reach the ceiling of their strategic ability in college where deeper processing is required (balduf, 2009). this suggests the teaching of self-regulating and metacognitive learning strategies is important for bringing a learning strategies project to scale in a first-year seminar 29 journal of effective teaching in higher education, vol. 1, no. 2 college students of all achievement levels. furthermore, when strategies are embedded in or taught alongside a content area, students are better able to transfer the strategies to future tasks and reflect on the outcome of that transfer (tuckman, 2003). ample evidence shows that self-regulated learning strategies, as primary contributors to academic success (pintrich & degroot, 1990), can and should be taught (cohen, 2012; paris & paris, 2001; weinstein, tomberlin, julie, & kim, 2004; tuckman, 2003), yet faculty attempting to teach them face many challenges. for example, many students are reluctant to change the learning strategies with which they were successful in high school (dembo & seli, 2004; dunlosky et al., 2013). many students also are unable to judge whether or not they are actually learning—a key aspect of metacognition—which can lead to overconfidence (koriat & bjork, 2005). the intervention presented in this paper encourages students to practice confronting their existing habits, monitor their knowledge acquisition, and reflect on outcomes of learning. motivation theory suggests that students are more likely to engage in activities for which there is a high degree of personal value and little cost (barron & hulleman, 2015). the strategy project assignment attempts to maximize value and minimize cost by engaging students in activities that will help them achieve a higher grade in a class in which they are currently enrolled. method and findings both studies described in this paper seek to answer the question: are first-year seminar students more likely to show evidence of metacognition and self-regulation after completing the strategy project assignment? in this section, we will describe the methods and findings for a pilot study, in which one member of the research team measured the effects of the strategy project on her own students, and for the current study, in which this intervention was brought to scale for a group of trained instructors who used the strategy project in their first-year seminar courses and were compared to a matched group of instructors who did not use the strategy project. both institutional review board-approved studies were conducted with students at a large, public university in suburban atlanta, georgia, where a firstyear seminar is required for all first time, full-time students in their first semester of college. strategy project assignment the strategy project assignment (see appendix) is a multistep project requiring students to plan, monitor, and evaluate their newly learned strategies as they prepare for a test in a course in which they are currently enrolled. each step in the process is meant to disaggregate “learning skills” into meaningful components that students are empowered to integrate for their individual purpose. in this assignment, students are directed to complete the following activities related to an exam in another course that they have chosen for the project. students must: 1) create a plan for their study time; 2) interact one-on-one with the instructor of their chosen course; 3) use active reading and/or active note-taking strategies as they learn new material; 4) implement additional active learning strategies that are bringing a learning strategies project to scale in a first-year seminar 30 journal of effective teaching in higher education, vol. 1, no. 2 appropriate for the content area, such as self-quizzing or concept mapping; and 5) reflect on their exam performance in a final paper. students receive instruction on the use of self-regulated learning strategies as well as research-based evidence on why the strategies work. this lesson is extended to a discussion on metacognition and the benefits of deep cognitive processing. after instruction in the various strategies, students identify a test in a course for which they have ample time to prepare and reflect. typically, this is the second or third test in a semester-long course. when given the option to choose their own course for the project, choices can vary widely, from psychology to math to chemistry; however, most students choose a course in which they feel less confident and prepared. students meet briefly with the first-year seminar instructor to review the requirements and determine if any substitutions should be made. next, the student meets with the instructor of the course they have chosen for the project. this is a significant piece of the strategy project, as it allows students not only to gain knowledge about how to succeed in the course but also to build a relationship with a faculty member, an important part of the transition to college (goodman, baxter magolda, seifert, & king, 2011). requiring this interaction as part of the strategy project may help ease the initial discomfort that many new college students feel in speaking with their professors. students then plan their approach to the strategy project by mapping out a time and task-specific “plan of study.” as an essential component of self-regulation for college students (kitsantas, winsler, & huie, 2008; tuckman, 2003), time management is another important piece of the assignment. with increased task demands and more responsibility for allocating their time, college students must be taught strategies for managing their study time. specific learning strategies are the next components of the strategy project. the active reading portion allows for practice of textbook annotation, an active strategy that promotes deep processing and engagement with the material (simpson & nist, 1990). in the notetaking portion, students are instructed to take notes for the duration of the exam-preparation time using the cornell method of note-taking (pauk, 1962) or a similar active note-taking method. finally, students are directed to choose two additional strategies from among those discussed that are appropriate for test preparation in their course. evidence of good quality textbook annotations, class notes, and study strategies are presented as a portion of the final strategy project. finally, students reflect on the results of their strategy project through a written assignment and oral presentation. both of these final products empower students to analyze critically the efficacy of each portion of the project. by sharing these reflections with classmates, students share peer-driven models of self-regulation and co-develop discipline-specific strategies that may help their fellow students succeed in other courses. bringing a learning strategies project to scale in a first-year seminar 31 journal of effective teaching in higher education, vol. 1, no. 2 pilot study to determine whether the mslq (pintrich, smith, garcia, & mckeachie, 1991), described below, revealed any changes in metacognition and self-regulation after completion of the strategy project assignment, a pilot study was conducted with fifty-two students recruited from two sections of the first author’s first-year seminar. just under 54% percent of the students identified as female and about 42% identified as male. when asked to choose the best description of their ethnicity, 65% of students indicated white (caucasian), 12% african-american, 12% hispanic, 4% asian, and 2% other. the students’ median age was 18 years. the study did not include data from students younger than 18. pilot study procedures. an online preand post-test version of the mslq (pintrich et al., 1991) was distributed via the learning management system during weeks 2 (prior to the introduction of the assignment) and 14 (well after the assignment had been completed) of a 16-week semester. completion of each survey counted for 1% of the course grade, respectively. the study was explained to students before obtaining their consent, and during each survey administration an alternative assignment was offered to students who were under 18 or who did not wish to participate in the study. measurement instrument. the mslq has been widely used for decades to investigate cognitive and metacognitive strategies, motivation, and selfregulation. in both studies described in this paper, the full 81-item version of the mslq, containing fifteen subscales, was used. the subscales included: intrinsic goal orientation, extrinsic goal orientation, task value, control of learning beliefs, self-efficacy for learning performance, test anxiety, rehearsal, elaboration, organization, critical thinking, metacognitive self-regulation, time and study environment management, effort regulation, peer learning, and help seeking. participants were instructed to answer the questions based on the course they chose for their strategy project. the survey also included a section where participants were asked to indicate which strategies they currently used to study for the chosen course. participants indicated that they completed one or more of the following strategies during their study time for the chosen course: read the textbook; took notes on the textbook; took notes in class; read over class notes; made flashcards; created a chart, diagram, or concept map; studied with a group; completed practice problems; or engaged in self-quizzing. in addition, the survey included questions to gather demographic data, including age, gender, and ethnicity. pilot study results. following recommendations from jamieson (2004) and others who suggest likert scale data be treated as ordinal-level data, nonparametric statistics were used in the analysis. a wilcoxon signed rank test revealed a statistically significant preand post-strategy project difference on the total scores of the mslq subscale pertaining to metacognitive self-regulated learning, z = -2.757, p < .01, with a fairly large effect size (r = .40). the median bringing a learning strategies project to scale in a first-year seminar 32 journal of effective teaching in higher education, vol. 1, no. 2 score on the subscale increased from pre-project (md = 17.5) to post-project (md = 18.0). none of the other subscales showed significant differences. a series of related-samples mcnemar’s tests revealed statistically significant increases for the use of several self-reported strategies (see table 1), including active reading, using flashcards, creating charts/diagrams/concept maps, group studying, and self-quizzing. no significant differences were found for textbook reading (passive reading), re-reading class notes (also a passive strategy), completing practice problems, or taking notes in class. table 1 results of mcnemar’s tests on study 1 participants’ (n=52) use of study strategies over time pre-strategy project post-strategy project strategies used n % of group n % of group read textbook 23 44% 26 50% took notes on textbook* 10 19% 17 33% read over class notes 39 75% 36 69% made flashcards*** 2 15% 29 56% created chart, diagram or concept map** 2 4% 11 21% group study 12 23% 20 39% completed practice problems 23 44% 28 60% took notes in class 42 81% 45 87% engaged in self-quizzing** 16 31% 30 64% * preto post-project change is significant at p<.05 ** preto post-project change is significant at p<.01 *** preto post-project change is significant at p<.001 note: students in this group significantly increased their use of textbook annotations, flashcards, charts/diagrams/concept maps, and self-quizzing. there were no significant decreases in strategy use. current study the current study sought to investigate whether the strategy project could be scaled up by providing targeted faculty development to a select group of instructors who used the strategy project in their first-year seminar courses. these instructors were matched with a group of instructors who had not previously used the strategy project in their courses. participants. five hundred eight students in thirty-three sections of a firstyear seminar participated in the study. close to fifty-seven percent of the students identified as female and 43% as male. when asked to choose the best description of their ethnicity, 57% of students indicated white (caucasian), 23% africanbringing a learning strategies project to scale in a first-year seminar 33 journal of effective teaching in higher education, vol. 1, no. 2 american, 8% hispanic, 6% asian, and 6% other. the students’ median age was 18 years. the study did not include data from students younger than 18. procedures. instructors teaching first-year seminar were recruited to participate in the study via emails describing the strategy project and associated faculty-training workshop. the ten instructors who responded and agreed to attend the workshop were assigned to the treatment group and were matched on three variables—years of experience, number of course sections taught, and educational background—to a comparison group of instructors who also participated in the study. both groups included fulland part-time instructors from a variety of disciplinary backgrounds. instructor training. instructors in the treatment group attended a fourhour workshop conducted by the researchers on the use of the strategy project. during this workshop, instructors received an overview and background information about the project, discussed instructional content related to the project including a timeline and assessment practices, and engaged in individual course planning. after the workshop, the research team continued to reach out to treatment group instructors for individual consultations and held multiple drop-in sessions for instructors to ask questions about the project. instructors in the comparison group were told that their students would be taking surveys about their use of learning strategies but were not given any special direction in curriculum planning. strategy project intervention. the strategy project, described above, was assigned in each of the sections taught by instructors in the treatment group. twenty-five percent of the semester grade was allotted to the project in each section. instructors used four class periods early in the semester for strategy project-related instruction, following the protocol given during the training workshop. survey implementation. an online version of the mslq was distributed via the learning management system as a pre-test to all students in both groups during weeks 2-3 of the semester. a member of the research team made a brief visit to each course section in both groups to encourage students to complete the survey. as in the pilot study, completion of the survey counted for 1% of the course grade. for grading purposes, a member of the research team sent frequent updates to instructors noting which of their students had completed the survey. in week 14 of the semester, a post-test version of the mslq was given to all students in both groups. this survey also contributed to 1% of the course grade. during each survey administration, an alternative assignment was offered to students who were under 18 or who did not wish to participate in the survey. instrument. as in the pilot study, the full 81-item version of the mslq, containing fifteen subscales (see description above), was used. participants were advised to answer the questions based on the course they chose for their strategy project (treatment group) or to answer the questions based on a course they were currently taking (comparison group). as in the pilot study, treatment and comparison group surveys also included a section where participants were asked to bringing a learning strategies project to scale in a first-year seminar 34 journal of effective teaching in higher education, vol. 1, no. 2 indicate which strategies they currently used to study for the chosen course (see table 3 for a list of strategies). in addition, the survey included questions to gather demographic data, including age, gender, and ethnicity. current study results. to compare changes on the mslq preto post-project between the two groups, change variables were computed for each subscale by subtracting pre-test subscale scores from post-test subscale scores. as with the pilot study, a conservative approach was taken with the likert scale data in the use of non-parametric statistics, and estimated effect size is labeled as “r” and interpreted using cohen’s (1988) criteria. a series of mann-whitney u tests revealed significantly greater change for the treatment group as compared to the comparison group on the total scores of four mslq subscales: task value, metacognition and self-regulated learning, time and study environment management, and peer learning (see table 2). effect sizes were small, ranging from r = .10 to r = .16. table 2 results of mann-whitney u tests comparing treatment and comparison groups on each mslq subscale mslq subscales n (t) n (c) md (t) md (c) u z p r internal goal orientation 252 251 .333 .333 31052.5 -3.53 .724 ns external goal orientation 252 252 -.250 -.250 30480 -.781 .435 ns task value 252 252 -.250 -.167 28178.5 -2.189 *.029 .10 control of learning beliefs 252 252 .000 .000 30671.5 -.663 .507 ns self-efficacy 252 252 -.125 -.125 28556.5 -1.956 .050 ns test anxiety 252 252 -.200 -.200 31246 -.310 .757 ns rehearsal 251 248 .250 .000 28186 -1.830 .067 ns elaboration 251 248 .167 .000 29635 -9.26 .354 ns organization 251 249 .000 .000 30292.5 -.594 .553 ns critical thinking 251 248 .000 .200 29266.5 -1.155 .248 ns metacognition and self-regulated learning 251 249 .083 -.083 27311 -2.44 *.015 .11 time and study environment management 251 249 .000 -.250 25496.5 -3.565 **.000 .16 effort regulation 251 248 -.250 -.250 29160.5 -1.223 .221 ns peer learning 251 248 .333 .000 27696.5 -2.136 *.033 .10 help seeking 255 251 .000 .000 31693.5 -.189 .850 ns bringing a learning strategies project to scale in a first-year seminar 35 journal of effective teaching in higher education, vol. 1, no. 2 * significant at p<.05 ** significant at p<.001 note: (t) indicates treatment group; (c) indicates comparison group. estimated effect sizes were listed for those subscales with significant results; when nonsignificant, “ns” was noted. finally, a series of mcnemar’s tests examining the preto post-change of selfreported strategies between the treatment and comparison groups revealed several significant differences between the two groups. the comparison group significantly increased their use of flashcards and group studying, while decreasing textbook reading and note-taking. in contrast, the treatment group significantly increased their use of most strategies listed, including note-taking, flashcards, creating charts/diagrams/concept maps, group studying, completing practice problems, and self-quizzing. tables 3 and 4 provide details about these strategy changes. table 3 results of mcnemar’s tests on the treatment group’s (n=255) use of study strategies over time pre-strategy project post-strategy project strategies used n % of group n % of group read textbook 121 47% 131 51% took notes on textbook 91 36% 86 34% took notes in class** 220 86% 241 95% read over class notes 180 71% 189 74% made flashcards*** 79 31% 160 63% created chart, diagram or concept map*** 9 4% 35 14% group study*** 60 24% 121 47% completed practice problems*** 114 45% 154 60% engaged in self-quizzing*** 106 42% 166 65% ** preto post-project change is significant at p<.01 *** preto post-project changes is significant at p<.001 note: students in the treatment group significantly increased their use of class notes, flashcards, charts/diagrams/concept maps, group study, practice problems, and self-quizzing. there were no significant decreases in strategy use. bringing a learning strategies project to scale in a first-year seminar 36 journal of effective teaching in higher education, vol. 1, no. 2 table 4 results of mcnemar’s tests on the comparison group’s (n=253) use of study strategies over time pre-strategy project post-strategy project strategies used n % of group n % of group read textbook* 159 63% 136 53% took notes on textbook 101 40% 95 38% took notes in class* 230 91% 213 84% read over class notes 197 78% 185 73% made flashcards* 39 15% 56 22% created chart, diagram or concept map 19 8% 30 12% group study** 49 19% 71 28% completed practice problems 136 54% 117 46% engaged in self-quizzing 98 39% 106 42% *preto post-project change is significant at p<.05 **preto post-project change is significant at p<.01 note: students in the comparison group significantly increased their use of flashcards and group study. they significantly decreased their use of the textbook and class notes. discussion both studies discussed in this paper were guided by the question: does the strategy project impact first-year students’ self-regulation and metacognition strategies as measured by the mslq and self-reported strategy use when brought to scale for a large first-year seminar program? findings from the pilot study and current study indicate that the strategy project does positively affect students’ metacognition and self-regulation, but the effect as measured by the mslq varies. in the current study, students who experienced the strategy project assignment had a statistically significant increase in their scores on four subscales, all of which reflect various aspects of metacognition and selfregulation, when compared with their peers: task value, which reflects “beliefs about the importance, utility, and relevance of the task” (pintrich, 2004, p. 365); metacognitive and self-regulated learning; management of time and study environment; and peer learning. these subscales were closely aligned with our goals for the project. of particular interest was the “management of time and study environment” subscale, which “is a measure of how well you manage your time and schedule, and your use of a place to study” (pintrich et al., 1991), and reflects a major challenge for first-year students (tuckman, 2003). notably, median scores on this subscale actually decreased for the comparison group during the semester but increased significantly for students using the strategy project. the change in peer learning subscale scores was also significantly different between groups. this subscale measures the degree to which students report collaborating with peers to increase learning. instructors using the strategy project encouraged this bringing a learning strategies project to scale in a first-year seminar 37 journal of effective teaching in higher education, vol. 1, no. 2 collaboration by making explicit the advantages of group studying, and by including group study as a choice in the metacognitive strategies portion of the assignment. more impressive, perhaps, is that students in the treatment group significantly increased their use of metacognitive strategies like creating concept maps, completing practice problems, and self-quizzing, while the comparison group did not. recent research indicates that self-quizzing, in particular, is a highly effective metacognitive strategy (dunlosky et al., 2013; rodriguez et al., 2018). one explanation for this success lies in the fact that the strategy project explicitly requires students to practice, then observe the effect of, metacognitive and selfregulation strategies. what is notable is that the post-test measure was taken during week 14 (out of 16) of the semester, several weeks after the strategy project itself had taken place, suggesting that students continued to use metacognitive strategies in the course chosen for the strategy project when they were no longer formally required and encouraged by their instructors. this finding confirmed the researchers’ intent to empower individual students to become strategic and self-directing in their use of specific “learning skills.” in other words, it is likely that students began to themselves disaggregate the utility of various strategies depending on the task at hand. limitations like most studies involving the scholarship of teaching and learning, this study was quasi-experimental due to the impossibility of randomly placing students into the various sections of first-year seminar. as in most universities, students selfselected their own course section. however, prior to registration, most sections were not yet staffed with instructors, and students were not aware which instructors were participating in the study, nor were they aware to which group participating instructors belonged. in order to reduce instructor differences as a confounding variable, the research team was very intentional about matching instructors for comparison. however, it is possible that the instructors in the two groups differed in some way that acted as a confounding variable. it is also possible that the two groups of students differed at the outset of the study. to investigate this possibility, demographic data for the treatment and comparison group were compared. a t-test revealed no significant difference in age, and chi-square analyses revealed no significant difference in gender or ethnicity between the two groups. additional comparisons between the comparison and treatment groups were made on the pre-mslq subscale scores and self-reported strategies to establish that the two groups were equivalent at the beginning of the study in their approach to metacognition and self-regulated learning. a series of ttests did reveal significant differences on some of the pre-test subscales, including task value, metacognition and self-regulated learning, and management of time and study environment. a series of chi-square analyses revealed significant differences on the use of textbook reading, flashcards, and practice problems. in all of these cases except for the use of flashcards, the comparison group scored higher than the treatment group on their pre-tests, which meant that the treatment group often had more room for improvement. the research team could find no plausible bringing a learning strategies project to scale in a first-year seminar 38 journal of effective teaching in higher education, vol. 1, no. 2 explanation for this initial difference between the groups. pre-test group differences such as these are not uncommon in educational research (mcgrath, 2016), but it is a potential moderator for how the key findings of the study are interpreted. conclusions and future directions essential skills like metacognition, time management, and self-regulation are critical for first-year student success and are often within the purview of a first-year seminar (young & hopp, 2014). first-year students, in particular, need appropriate guidance from faculty about the component parts that constitute complex concepts like motivation and “academic skills” so as not to reinforce faulty prior knowledge about how such academic skills are developed. in a large first-year seminar program, it can be a struggle to maintain consistency and quality of instruction, especially for programs that rely on instructors with varying disciplinary backgrounds and few years of teaching experience. therefore, there is a need for high quality, evidence-based practices that can be brought to scale by providing appropriate faculty development and support. based on the evidence described in this paper, the strategy project assignment may be a viable option for first-year programs seeking to scale up students’ use of time management, learning strategies, “study skills,” or “academic skills” in the authentic environment of another course in which they are enrolled. future studies should establish the replicability of the strategy project assignment in other contexts; one such study is currently being conducted at another institution (steiner, sturges, & langdon, 2018). in addition, longitudinal studies of students’ strategy use would reveal whether the changes brought about by the strategy project have lasting impact beyond the first semester. dunlosky and colleagues (2013) have suggested that students continue to use ineffective study techniques because they don’t know—and haven’t been taught— which techniques are most effective, perhaps in part because instructors themselves don’t know. the project discussed in this paper could serve as a springboard for both students’ and instructors’ learning about learning. in a postproject feedback survey, instructors in the treatment group touted the benefits of the project, indicating that the project made “more of a difference to my students than any other assignment i've ever taught.” another appreciated that it was a “great tool for teaching these necessary study habits and strategies, all rolled into one project.” this anonymous instructor also said that it sparked his/her own interest in metacognition and self-regulation: “study strategies are not my strong suit so i didn't feel highly equipped, but did the best i could and learned a lot. [i] may want to become an expert in the future!” when complex assignments like the strategy project are paired with faculty development, both students and faculty may benefit. the studies described in this paper suggest that these important benefits can be brought to scale for even the largest first-year seminar programs in order to maximize student success. bringing a learning strategies project to scale in a first-year seminar 39 journal of effective teaching in higher education, vol. 1, no. 2 references balduf, m. (2009). underachievement among college students. journal of advanced academics, 20(2), 274-294. barron, k. e., & hulleman, c. s. (2015). expectancy-value-cost model of motivation. in j.d. wright (ed.), international encyclopedia of the social & behavioral sciences, 2nd edition (vol. 8, pp. 503-509). oxford: elsevier. cohen, j.w. (1988). statistical power analyses for the behavioral sciences (2nd ed). hillsdale, nj: lawrence erlbaum. cohen, m. t. (2012). the importance of self-regulation for college student learning. college student journal, 46(4), 892-903. dembo, m. h., & seli, h. p. (2004). students' resistance to change in learning strategies courses. journal of developmental education, 27(3), 2-11. dunlosky, j., rawson, k. a., marsh, e. j., nathan, m. j., & willingham, d. t. (2013). improving students’ learning with effective learning techniques: promising directions from cognitive and educational psychology. psychological science in the public interest, 14, 4-58. everett, m. c. (2013). reflective journal writing and the first-year experience. international journal of teaching and learning in higher education, 25(2), 213-222. goodman, k. m., baxter magolda, m., seifert, t. a., & king, p. m. (2011) good practices for student learning: mixed-method evidence from the wabash national study. about campus, 16(1), 2-9. jamieson, s. (2004). likert scales: how to (ab)use them. medical education, 38, 1212-1218. kitsantas, a. (2002). test preparation and test performance: a self-regulatory analysis. journal of experimental education, 41, 231–240. kitsantas, a., winsler, a., & huie, f. (2008). self-regulation and ability predictors of academic performance during college: a predictive validity study. journal of advanced academics, 20(1), 42–68. koriat, a., & bjork, r. a. (2005). illusions of competence in monitoring one's knowledge during study. journal of experimental psychology: learning, memory, and cognition, 31(2), 187-194. mcgrath, a. (2016). searching for significance in the scholarship of teaching and learning and finding none: understanding non-significant results. teaching & learning inquiry, 4(2). nash-ditzel, s. (2010). metacognitive reading strategies can improve self regulation. journal of college reading and learning, 40(2), 45-63. paris, s. g., & paris, a. h. (2001). classroom applications of research on self regulated learning. educational psychologist, 36, 89–101. pauk, w. (1962). how to study in college. boston, ma: houghton mifflin. penuel, w. r., & fishman, b. j. (2012). large-scale science education intervention research we can use. journal of research in science teaching, 49(3), 281– 304. pintrich, p. r., & degroot, e. v., (1990). motivational and self-regulated learning components of classroom academic performance. journal of educational psychology, 82(1), 33-40. pintrich, p. r., smith, d., garcia, t., & mckeachie, w. j. (1991). a manual for the bringing a learning strategies project to scale in a first-year seminar 40 journal of effective teaching in higher education, vol. 1, no. 2 use of the motivated strategies for learning questionnaire (technical report 91-b-004). the regents of the university of michigan. pintrich, p. r. (2004). a conceptual framework for assessing motivation and self regulated learning in college students. educational psychology review, 16(4), 385–407. rodriguez, f., kataoka, s., janet rivas, m., kadandale, p., nili, a., & warschauer, m. (2018). do spacing and self-testing predict learning outcomes? active learning in higher education, 385-407. simpson, m. l., & nist, s. l. (1990). textbook annotation: an effective and efficient study strategy for college students. journal of reading 34, 122-129. steiner, h.h. (2016). the strategy project: promoting self-regulated learning through an authentic assignment. international journal of teaching and learning in higher education, 28 (2), 271-282. steiner, h.h., sturges, d., & langdon, j. (2018, october). promoting self-regulated learning in the first-year seminar: evidence and future directions. workshop presented at the annual meeting of the international society for the scholarship of teaching and learning. bergen, norway. thibodeaux, j., deutsch, a., kitsantas, a., & winsler, a. (2017). first-year college students' time use: relations with self-regulation and gpa. journal of advanced academics, 28(1), 5-27. tuckman, b. w. (2003). the effect of learning and motivation strategies training on college students' achievement. journal of college student development, 44(3), 430-437. weinstein, c. e., tomberlin, t. l., julie, a. l., & kim, j. (2004). helping students to become strategic learners: the roles of assessment, teachers, instruction, and students. in j. ee, a. chang, & o. tan (eds.), thinking about thinking: what educators need to know (pp. 282–310). singapore, china: mcgraw-hill. young, d. g., & hopp, j. m. (2014). 2012-2013 national survey of first-year seminars: exploring high-impact practices in the first college year. research reports on college transitions no. 4. national resource center for the first-year experience and students in transition. columbia, sc: national resource center zimmerman, b. j. (2008). investigating self-regulation and motivation: historical background, methodological developments, and future prospects. american educational research journal, 45, 166–183. zimmerman, b. j., & martinez-pons, m. (1990). student differences in self regulated learning: relating grade, sex, and giftedness to self-efficacy and strategy use. journal of educational psychology, 82, 51-59. bringing a learning strategies project to scale in a first-year seminar 41 journal of effective teaching in higher education, vol. 1, no. 2 appendix strategy project this assignment will help you see the connections between what you are learning in your first-year seminar and your other courses. in this project, you will apply the strategies and hints that we have learned in seminar to preparing for a test in another class. the test you select should be between sept. 30 and oct. 30, and should be a test that requires you to study. to complete the strategy project, you will complete and submit a variety of test preparation strategies along with written reflections about your experience. you will submit these materials as noted below. all papers should be double-spaced, with 12-point font. strategy project timeline and activities: contract (5 points). list the details of your chosen test, choose the strategies you plan to use, and mark the necessary due dates. we will meet briefly in class to ensure your chosen test and strategies are appropriate. please bring the printed and signed contract to class on [date]. if you need to make changes to your contract after this meeting, please notify me. no changes may be made after week 5 of the semester. _________________________________________________________________________________________ pre-project survey (5 points). this portion of the project will require you to take the mslq survey (found at this link) or complete an alternative assignment in accordance with irb policy (see your instructor for details). if you are younger than 18, please ask your instructor for the alternative assignment. your instructor will only grade the survey for completion, not for how you answered the questions. pre-project reflection (25 points). please reflect on your current habits in a 2-3 page essay. when writing your essay, please consider the following questions: ● what do you feel are your strengths when it comes to college success? ● what do you feel are your weaknesses when it comes to college success? ● what are you most worried about when it comes to college studying and learning? ● what are your favorite study strategies, and why? where did you learn these strategies? ● is there anything you would like to improve this semester that could lead you to be a successful college student? submit the essay via the assignments tool on d2l. the survey and reflection must be completed by [date]. professor interaction (15 points). attend office hours or a drop-in session with the instructor of the course you have chosen for the project and write a oneor two-paragraph description of the questions you asked, what you learned, and why attending office hours might be helpful. a list of suggested questions is below, but please feel free to add your own: ● what is the format of the upcoming test? what do you think are the best ways to study for this test? ● what do you think differentiates the most successful students in your class from the least successful students? ● what do you like most about being a college instructor? bringing a learning strategies project to scale in a first-year seminar 42 journal of effective teaching in higher education, vol. 1, no. 2 ● do you conduct research? if so, what is it about? submit this via the d2l assignments tool by the due date indicated on your contract. plan of study (15 points). using the suggestions from your professor interaction as well as what you’ve learned in ksu 1101, please outline your plan for studying for this test. you should create a detailed study schedule that describes what you will do and when you will do it, demonstrating your plan to spread out your studying over at least two weeks, using a variety of active learning strategies. this outline should be created and submitted via the d2l assignments tool by the due date indicated on your contract. _________________________________________________________________________________________ metacognitive study strategies (60 points). these study strategies should be submitted in one folder or binder in class on the date indicated on your contract. please make sure everything is clearly labeled, and that your signed contract is in the front of the folder. you must choose at least four of these strategies, one of which must be active notetaking or active reading. please choose your strategies carefully, taking into consideration the course, the professor, and the type of test. choose at least one: ● active notetaking. use the cornell or a modified cornell method to take notes in your chosen class for at least two weeks during the period leading up to your test, making sure to set aside space for self-quizzing and daily summaries. submit these notes (or a copy of these notes) in your folder. ● active reading. preview and annotate all textbook chapters associated with this test, paying special attention to the visual aids, vocabulary, and concepts. summarize these annotations by distilling all important information into a one-page study sheet. copy a 4-page sample of your annotations and include it and your onepage study sheet in your folder. choose three (or two, if you chose both strategies above): • complete practice problems from an outside source. this is a good strategy for math, physics, chemistry, and computer science courses. • quiz yourself by creating your own test. • create flashcards. this flashcards must include explanations of the topic in your own words as well as examples. you must create at least fifteen flashcards. • create a concept map. • meet with a study group and describe your experience in 1-2 paragraphs. • attend a tutoring session and describe your experience in 1-2 paragraphs. • other strategies? propose a strategy that is appropriate to your course in consultation with your instructor. _________________________________________________________________________________________ post-project survey (5 points). complete the mslq survey (found at this link) or complete an alternative assignment in accordance with irb policy (see your instructor for details). again, your instructor will only grade the survey for completion, not for how you answered the questions. bringing a learning strategies project to scale in a first-year seminar 43 journal of effective teaching in higher education, vol. 1, no. 2 post-project reflection (20 points). after you take your test but before you receive it back, please estimate the grade you think you received. once you receive your feedback, please write a reflection paper 2-3 pages in length about your experiences with the strategy project. in particular, please reflect on which elements of the project you think helped and which did not. please also reflect on the grade you received, whether it was an improvement over your last grade (list both grades), and whether it matched what you thought you’d receive. also address whether or not your study strategies have changed since you’ve been in college. submit your reflection paper via the d2l assignments tool. the survey and reflection must be completed by the date indicated on your contract. _____________________________________________________________________________________ strategy project presentation--if required by your instructor (15 points). please plan to share your strategy project with fellow students in an informal presentation of about 10 minutes in length on [date]. you may use the doc cam to show elements of your project. this is a great chance for other students to learn strategies that may be useful to them in the future. in particular, your fellow students would like to know: what is your course like? was the professor receptive to your questions? what elements of the strategy project worked in your course? bringing a learning strategies project to scale in a first-year seminar 44 journal of effective teaching in higher education, vol. 1, no. 2 strategy project contract name__________________________________________ course you will use for this project__________________ date of test (between 9/30 and 10/30): __________________ i would like to choose the following study strategies for my project. please choose at least four. ☐ active notetaking and/or ☐ active reading ☐ complete practice problems ☐ quiz yourself ☐ create flashcards ☐ create a concept map ☐ meet with a study group ☐ attend a tutoring session ☐ other (please describe): i understand that my due dates for this project are the following: _______________ contract _______________ pre-reflection paper and survey _______________ professor interaction (three weeks prior to test) _______________ plan of study (two weeks prior to test) _______________ active notetaking (next ksu 1101 class meeting after your test day) _______________ active reading (next ksu 1101 class meeting after your test day) _______________ additional test preparation strategies (next ksu 1101 class meeting after your test day) _______________ post-reflection paper and survey (two weeks after your test day) _______________ presentation (if required by instructor) __________________________student signature _______________________instructor signature microsoft word 136-article text-878-1-18-20210421.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.136 college ready or not? engaging and supporting english language learners in higher education katie d. lewis, york college of pennsylvania, klewis15@ycp.edu stephanie l. brown, york college of pennsylvania abstract. in recent years, there has been a greater emphasis on ensuring that high schools across the united states focus on college readiness standards and skills, with the goal that more students will persist and find greater success in college. despite this focus, there are still significant gaps among the students who are deemed “ready” and are actually persisting and finding that success, specifically for english language learners (ells). in this article, we illustrate the landscape of ells enrolled in institutions of higher education. we then explore how ell students are classified and how these different profiles intersect with the limited range and types of english as a second language (esl) support available at the institutional level. finally, we contribute to the very limited practical knowledge base on ells in the college setting, with a summary of promising best practices for college faculty across disciplines to consider in their instruction. keywords: english language learners, higher education, college readiness, instructional strategies recently, the united states experienced a decline in their role as a global leader in college completion rates. one of the factors contributing to the lower completion rates is the increased need for remedial coursework (scott-clayton et al., 2014). the u.s. department of education (n.d.) reports that “about a third of american students require remedial education when they enter college” (para. 3). recognizing the need for reform in college preparedness, federal funding was allocated through the every student succeeds act (essa), which provides federal funding to prek-12 schools for educational reform to directly address college readiness through the following measures: (1) sets high standards for aligning with the state’s higher education institutions (ihe) entrance requirements; (2) develops accountability measures and standards for college and career readiness; (3) provides federal funding to support student preparedness and supports transition from high school to ihe; (4) calls for a development of a partnership between prek12 and ihe (malin et al., 2017; minnich et al., 2016). states like california are among few in the nation that have a college readiness framework in place and have instituted other policy initiatives to provide more opportunities to ells (e.g., the multilingual education act of 2016; the el roadmap). despite this focus, significant gaps exist among the students who are persisting and finding success, particularly for the racially/ethnically and linguistically diverse students who are historically underrepresented in higher education (kanno & cromley, 2012; kanno & kangas, 2014; martin et al., 2017). conley (2012) summarizes college readiness as, “a student who is ready for college and career can qualify for and succeed in entry-level, credit-bearing college courses leading to college ready or not? 110 journal of effective teaching in higher education, vol. 4, no. 1 a baccalaureate or certificate, or career pathway-oriented training programs, without the need for remedial or developmental coursework” (p. 1). college readiness focuses on four distinct areas: cognitive strategies, content knowledge, self-management skills, and knowledge about postsecondary education (conley, 2012). higher education expects first-year students to be able to think critically, investigate, evaluate, and problem solve in various contexts. along with the cognitive strategies, first-year students are expected to have a strong level of foundational content knowledge. understanding key concepts and systems of structure within a content area is essential to the development of deeper understanding. the third key area of college readiness focuses on self-management skills; while these skills develop during college, first-year students are given greater responsibility for managing all aspects of their academic and social life. finally, college readiness means that students understand how postsecondary education functions from the financial aid, the application process, required admissions exams, as well learning about the culture of higher education opposed to high school. while some conceptual research exists (see perez & morrison, 2016), few studies have focused on the college planning process for english language learners, let alone their advancement process once they do make their way onto this path (kanno & cromley, 2015). college readiness is an admirable goal of secondary education, yet it is not a perfect system. professors, instructors, lecturers, and adjuncts often still encounter underprepared college students who struggle within all four of the college readiness areas. the national center for education statistics reports that 67.7% of the degree-granting postsecondary institutions offer remedial services to first-year students (2018–19). those college students who struggle the most are often those who face the greatest challenges due to cultural, linguistic, or economic (cled) diversity. the varied educational experiences and linguistic and cultural backgrounds of ells enrolled in universities and colleges make it challenging to find one uniform solution. in this article, researchers explored the current literature related to the following questions: 1. who are the ells in higher education? 2. what support systems are available to ells in higher education? 3. what are best practices for instruction for working with ells in higher education? theoretical framework the following theories guided the authors’ approach to understanding the role of ells in ihe and practical applications for increasing the effectiveness of instruction for ells across the disciplines. bourdieu’s theory of cultural capital of the three forms of cultural capital bourdieu (1986) proposes (institutionalized, college ready or not? 111 journal of effective teaching in higher education, vol. 4, no. 1 embodied, objectified), the institutionalized form is relevant to this review because of the way higher education can position an ell for upward mobility. the cultural capital they gain from a college degree can be a game changer for ells. first, however, they have to acquire knowledge of the “rules of the game,” which is often up to prek-12 school personnel, such as guidance counselors and teachers, or in ihe, the professors and advisors. this leads to the second relevant form of embodied cultural capital: linguistic capital, which, for ells, is an often volatile currency yet vital for educational attainment. actor-network theory there is an added level of complexity that emerges at the institutional level for ells; a barrier that comes as a result of persistent power dynamics that are deeply embedded in the fabric of education in america. to account for this, actor-network theory (ant) provides a framework that captures these “non-human” elements and linkages that often reify the challenges ells face when it comes to access to, let alone participation and persistence in, higher education (fenwick & edwards, 2010; latour, 2007). ant provides a lens to understand the interplay between human and non-human (environmental) networks that can either interfere with or support their goals for higher education. zone of proximal development social constructivist, lev vygotsky (1978) developed the concept of the zone of proximal development (zpd). zpd is the area where student learning occurs best with scaffolded support from the teacher. limited academic growth occurs outside of this zone, where the student experiences boredom and limited academic growth due to unchallenging work or learning does not occur due to high levels of frustration because the work is too difficult. scaffolding is a support system that is removed over time as the student becomes more independent. scaffolding instructional practices are essential to engaging ell students. gibbons (2009) conceptualized intellectual challenging instruction with appropriate scaffolding for ells. ells who experience high levels of challenge and scaffolding from the teacher experience the greatest academic growth. it is within this theoretical viewpoint that authors explored practical applications for the college professor. understanding the access barriers ells must overcome to persist in higher education as well as the means through which learners construct new understanding is imperative to providing meaningful instruction across disciplines. methods this review of the literature included peer-reviewed journal articles and grey literature (e.g., technical reports, government documents) from the past ten years, with a specific focus on the academic discipline of education. book reviews, dissertations, and editorials were excluded. table 2 illustrates the search strategies and search terms/keywords that were used in the review, all conducted through the college ready or not? 112 journal of effective teaching in higher education, vol. 4, no. 1 online database onesearch, through the authors’ institutional library. we focused on the academic disciplines of education, social sciences, and sociology. table 2 search strategies and key terms search strategy examples of search terms/keywords terminology variations english language learners (ells); english learners (els); english as a second language (esl); longterm english language learners (ltells); limited english proficiency (lep); synonyms for “instructional strategies” instructional practices; best practices; pedagogy; curriculum associated terms paired with key terminology college readiness; career readiness; college access, higher education; instructional strategies; developmental education; community colleges; every student succeeds act (essa) characteristics of ells in higher education the learner profile of ells in higher education varies greatly from the academic to cled background of english language learners. it is paramount that professors recognize these differences to provide appropriate support in the classroom. international students in 2019, 1,095,299 (5.5%) of the total united states student body in higher education were international students (bustamante, 2020). of these international students, 431,930 were undergraduate students. about 70% of the international students enroll in 200 different institutions across california, new york, and texas. the most common countries of origin are canada, china, india, saudi arabia, and south korea (moody, 2019). the top fields of study for these students include business management, engineering, math/computer science, and social, physical, and life sciences. as part of the admissions process, international students complete a standardized english language proficiency exam. their scores guide the institution in admission decision-making. many institutions have a cut off score and/or offer an esl program for students who are below the cut off score (kice, 2014). college ready or not? 113 journal of effective teaching in higher education, vol. 4, no. 1 immigrant students the united states has a long history of being a nation of immigrants. first generation immigrants are those who were born outside of the united states and later moved to the united states. in 2016, 14% of the us population were immigrants, a percentage that is steadily rising. first generation immigrants and their children made up about 20% of all students enrolled in college in the united states in 2015 (postsecondary national policy institute, 2019). second generation students are those students who were born in the united states, but whose parents were born in another country. it is important to note that while first generation immigrants are readily enrolling in higher education, there has also been an increase in the second-generation students enrolling (postsecondary national policy institute, 2019). age upon arrival to the united states plays a major role in an immigrant’s success in higher education, “47% of all full-time undergraduate students who immigrated to the u.s. prior to age 12 went on to earn a degree” (postsecondary national policy institute, 2019). undocumented immigrants may face financial hardships, as they are not eligible for federal student aid. language may also be a barrier for an immigrant student in higher education. the us census bureau reports that approximately 47% immigrants identified as limited english proficient (batalova et al., 2020). limited english proficiency (lep) refers to a person whose first language is not english and has a limited ability to speak, read, or write in english. migrant students little data is collected on the number of migrant students who attend institutions of higher education. migrant students are “children and youth ages 0-21 whose families work in the agricultural and/or fisheries industries and who will often move across districts and state lines several times within a 12-36 month period of time” (lundy-ponce, 2010). migrant children often experience gaps in their k-12 education as a result of frequent moves. the migrant student leadership institute (msli) at the university of california, los angeles, is one of the few programs that specifically address the college readiness of migrant students (nuñez, 2009). this program focuses on providing college readiness preparation as well as social and cultural support for migrant students seeking a four-year degree. migrant students who completed the msli program were more likely to attend an institution of higher learning as well as more likely to be successful in higher education (nuñez, 2009). college assistance migrant program (c.a.m.p.) is federal funding that is available to assist migrant students during their first year of college. the c.a.m.p. program serves 2,000 participants annually (office of elementary and secondary education, 2020). generation 1.5 students generation 1.5 students are students who straddle first generation and secondgeneration worlds but belong to neither one. sharing characteristics of both generations and cultural groups makes it difficult for the student to identify with college ready or not? 114 journal of effective teaching in higher education, vol. 4, no. 1 one culture. generation 1.5 students may have been born in the united states but grew up speaking a language other than english at home. some may have been born in another country but came to the united states for education or moved during their high school years (deanza college, n.d.). generation 1.5 students have learned english for the most part by listening and speaking, rather than reading and writing. they may sound like a native english speaker yet have weaker academic literacy skills. these disparities in their language abilities and inconsistent attendance often results in misplacement in k-12 schools into esl or low ability classrooms where they struggle to master academic subject areas. generation 1.5 students struggle in higher education; even though they have graduated from us high schools, they often are lacking the strong foundational knowledge and skills needed to be successful (deanza college, n.d.). just as understanding the diversity of the ell students in the classroom, it is equally important to understand the terminology used to label ell students. the term ell “refers to students who are not currently proficient as english speakers and are in the process of developing their english language skills” (eslteacheredu.org, 2020). esl is the term used to identify either teachers of or programs for ell students. esl programs work with ell students to develop fluency in academic and social language as well as serving as cultural bridge (eslteacheredu.org, 2020). supports for ells in higher education ihe vary greatly in their levels of support for students who are ells. most institutions require an english language proficiency exam prior to admission (bergey et al., 2018). those students who score below the cut off score may be offered conditional admission requiring enrollment in an esl program run either through the university or private sector. beyond this initial esl program, few supports are consistently found in higher education settings. even in the research, the majority of the focus is on effective instructional practices and pedagogies for ells in the k-12 setting (gallagher & haan, 2018). additionally, there is even more of a dearth of research when it comes to the types of support, if any, for faculty responsible for helping esl students find success in their college coursework. considerations for instructional practices ells are successful in higher education when fostered in a rich learning environment that recognizes and supports their unique academic backgrounds. ideally, these students are entering the university setting with the college readiness and skills to be successful, yet this is often not the reality. course professors can help ensure the ell student is successful by considering the instructional practices and characteristics of ell students discussed below. culturally responsive pedagogy culturally relevant pedagogy (crp) was first developed by gloria ladson-billings to understand the african american student population and has evolved to include college ready or not? 115 journal of effective teaching in higher education, vol. 4, no. 1 additional socially constructed "cultures," such as ells (gay, 2010). ladson-billings (1995) views crp as something "that empowers students to maintain cultural integrity, while succeeding academically." she popularized the teaching approach in the early 1990's with her work with teachers of african american students. in her discussion about ells, stacey lee (2012) builds on ladson-billings’ work, calling this culturally "responsive" pedagogy a more action-oriented approach. one of the culturally responsive strategies professors can employ is perspective taking, a way to operationalize what it means to “show empathy” (warren, 2018). in his seminal study, ezra stotland (1969) found two dimensions to perspective taking—“imagine self” (imagining how you would feel in the other person’s situation) and “imagine other” (imagining how the other person feels in their situation). surprisingly, few researchers have endeavored to build on this often ignored framing of empathy. batson et al. (1997) note that the “imagine self” approach to perspective taking is less congruent with crp principles, in that it tends to promote a more egocentric response often laden with implicit bias and privilege. on the other hand, with an “imagine other” approach to perspective taking, a person, in this case a course professor, “acknowledges the range of external social and cultural variables that may be determining the student’s academic performance” (batson et al., 1997, p. 174). professors can begin by learning to pronounce their students’ names correctly and take the time to briefly check in with them before or after class or during office hours. building these respectful and empathic relationships can make a significant difference in the college experiences of ell students. professors across disciplines can be culturally responsive in their teaching by providing explicit opportunities for students to integrate their cultural backgrounds and lived experiences into coursework. for example, teacher educators can provide opportunities for their preservice teachers to reflect on their own lived experiences with schooling and encourage them to use these reflections to scaffold their thinking about their own cultural identities. in turn, learning opportunities like this can also help them understand how their cultural identity intersects with their identity as a future educator. for ell students in particular, this type of activity could surface important connections between their personal and professional experiences that could propel them into the teaching profession with a new sense of purpose. lastly, it is important for professors to still maintain and explicitly communicate high (not hurried) expectations of ell students. this may include maintaining eye contact during interactions, providing meaningful and in-depth feedback on assignments, and calling on ell students to answer questions in class (echevarria, 2018). academic background considerations understanding and recognizing that ell students come from a variety of cultural and educational backgrounds with different lived experiences is paramount to facilitating their success. international students may lack cultural knowledge of u.s. specific events as the focus in history was on their own country’s history. it is beneficial for these students to provide supplemental resources with additional content background knowledge. immigrants and generation 1.5 students may be proficient in oral english speaking and listening skills but lacking formal academic college ready or not? 116 journal of effective teaching in higher education, vol. 4, no. 1 english knowledge and skills. providing these students with the information for on campus writing and tutoring centers is essential in helping them improve their academic language. these few examples highlight the complexity of teaching ell students. kaur and singh (2019) identified several challenges in academic writing practices for esl learners. the first barrier recognizes that a lack in english language proficiency overridingly affects the students’ ability to succeed in higher education. esl students who come from weaker academic backgrounds struggle with the rigorous expectations of higher education. a second barrier is intentional plagiarism; those esl students who were weaker in their english language proficiency were more likely to cut and paste text together to create an assignment. additionally, some esl students were unaware of plagiarism and the problems that it presents. for instance, merkel (2020) describes the variation that exists between l1 (first language) writing practices in students’ home cultures compared to the u.s. plagiarism in relation to ells remains a complex and multilayered issue across the literature. the third barrier relates to difficulties in expressing ideas. esl students struggled to elaborate and explain in detail their thoughts in written assignments (kaur & singh, 2019). a significant barrier for learning is the integrated learning of content and english language. often in higher education, ell students are struggling to learn new subject matter content and, at the same time, they are still mastering the english language. in essence, this duality of learning complex academic content and the english language concurrently is the crux of the problem for course instructors. second language acquisition the process through which one acquires a second language is often under much scholarly debate. originally, it was believed that second languages were best learned through a repetitious process focused on grammar and mechanics (diazrico, 2018). more recently, second language acquisition theories recognize the importance of ownership over one’s learning and the intersectionality of academic and social learning. rather than drill and kill, second languages are learned more effectively through an interactive process of learning. cognitive psychology offers several theories (baker, 2001; cohen, 1996; cummins, 1980) to language learning from which one may derive key understandings of ells in higher education. first, the level of anxiety of the learner directly impacts their learning. if students have high anxiety related to language learning, then they will learn the language a slower rate (diaz-rico, 2018). anxiety levels may increase as individuals reach adulthood due to a variety of factors including lower levels of self-efficacy or a fear of failure. adults are more aware of their errors and take fewer risks in front of peers when learning a second language. college professors can reduce student anxiety by providing a safe learning environment. second, students who are fluent in their l1 are more likely to grasp a second language (l2) quicker because they are able to transfer foundational language understandings (cummins, 1980; diazrico, 2018). one such concept might be the idea that letters correlate to spoken or unspoken sounds that combine create words that are then organized into sentences college ready or not? 117 journal of effective teaching in higher education, vol. 4, no. 1 to create meaning. students who have a weak l1 foundation tend to struggle more with academic literacy in l2. this is especially something to consider among generation 1.5 students who may have graduated high school without l2 proficiency. third, language strategies, both direct and indirect, are essential to ensuring the development of academic language. indirect language strategies effective for developing l2 are acquired naturally in the learning process (cohen, 1996). direct language strategies may be taught and focus on the higher levels of learning. fourth, intentional applications of instructional strategies are essential to ensuring the development of academic language. in the following section, the authors will explore selected strategies, which are beneficial across disciplines in higher education. opportunities for critical thinking and academic literacy second language acquisition often begins informally through the acquisition of social language. individuals learn to navigate social situations through listening and conversing with peers. formal academic language acquisition occurs in the classroom setting and may develop slower than social language. academic literacy encompasses critical thinking, reading, writing, and speaking skills as well as students’ disposition for advanced academic work (intersegmental committee of academic senates, 2002). entering first-year students should have a “fundamental understanding of audience, tone, language usage, and rhetorical strategies to navigate appropriately in various disciplines” (intersegmental committee of academic senates, 2002, p. 13). academic literacy is complex and filled with nuances, which are often discipline specific (fenton-o’creevy & van mourik, 2016). growth of academic literacy skills requires development of critical thinking skills. critical thinking skills encompass the ability to synthesize and analyze a text, engage in discourse, and question the text as well one’s own claims (intersegmental committee of academic senates, 2002). academic disciplines, similar to genres, vary in tacit rules and practices (fenton-o’creevy & van mourik, 2016), which may create challenges for the english language learner. therefore, it is necessary to foster opportunities for engaging in academic literacy and critical thinking in the classroom setting. table 3 provides selected instructional strategies with an explanation along with an alignment to research. college ready or not? 118 journal of effective teaching in higher education, vol. 4, no. 1 table 3 selected instructional strategies for fostering critical thinking and academic literacy strategy description selected examples research alignment structured assignments assignments where the format remains the same but the prompt or topic changes chapter summaries, lab reports, mathematical proofs helmy, 2016 revise and resubmit allows students to produce drafts at various points in time research paper draft process grabe & kaplan, 2014; kaur & singh, 2019 modeling provides examples of what is expected of students demonstration of a task or thought process, completion of the first problem in a set, show of previous work of students kim & bowles, 2019; witt & soet, 2020 collaborative structures in class or out of class activities where students are required to work together to accomplish the task think-pair-share activities; group work with team roles; discussion board contribution nokes-malach et al., 2015; scager, et al., 2016; watanabe & swain, 2007 visual representati on of content graphic displays of content timelines, graphs, concept maps baxendell, 2003; halwani, 2017; petrie, 2003; uba et al., 2017 structured assignments structured assignments are assignments where the format remains the same but the prompt or topic changes. the benefits of these assignments allow the students to focus on their reading and writing skills rather than worrying about the format of the assignment. familiarity with the format enables the students to apply the previous assignments’ feedback in an authentic manner. one of the greatest college ready or not? 119 journal of effective teaching in higher education, vol. 4, no. 1 downfalls of instructor feedback on student work is that the assignment is completed. therefore, the student may read the comments but never have the opportunity to apply their new understanding. in economics, the structured assignment may involve reviewing current events and synthesizing and analyzing the economic impact on the local economy. this recurring assignment provides opportunities for the student to internalize and implement the instructor’s recommendations from the previous assignment. additional benefits are the use of familiar vocabulary and terms for ell students. applying the same processes and vocabulary used during lecture to a structured assignment provides a knowledge base from which the ell student can springboard from (helmy, 2016). in an economics class, reviewing terminology and examples of the poverty gap index related to regions prior to assigning a similar assignment focused on individual countries is another example of a structure assignment. a key to successful structured assignments is the repetition, which provides an opportunity for developing critical thinking skills based on instructor feedback as well as developing academic literacy skills within the discipline. embedded within structured assignments are scaffolded assignments, which benefit all undergraduate students, but the benefits for ell students are exponential. instead of assigning a term paper with one due date, scaffolded assignments break down the major assignment into smaller more manageable segments. scaffolded assignments are essential in developing academic literacy within disciplines. for example, in a writing-intensive education course, undergraduate students are required to complete an action research project focused on a current educational issue. in this introductory course, the assignment is divided into twelve smaller assignments, each of which build on the previous one. the benefits for ell students are opportunities for feedback, clarification of directions throughout the assignment process, and demonstrating growth throughout the semester. revise and resubmit english language learners struggle in academic writing more so than other subjects because of the cultural differences in rhetorical styles. these differences in rhetorical styles prove difficult because the ell students struggle with phrasing even when their grammar and vocabulary may be correct (grabe & kaplan, 2014; kaur & singh, 2019). english language learners benefit from multiple opportunities to revise and resubmit written assignments. the revision process enables the students to correct grammatical errors as well as to elaborate on ideas based on the professor’s feedback. it is through the revision process where writers become aware of mechanical and stylistic errors. not only should the revision process be allowed, but it should also be a requirement for english language learners. model in class while all students can reap the benefits of effective modeling in class, it is especially important for ell students and one of the most flexible strategies professors across disciplines can implement. one way professors can do this is by providing a partial or complete demonstration of what is expected of students on a college ready or not? 120 journal of effective teaching in higher education, vol. 4, no. 1 particular task. this can be done by the professor or by other students in the class if they have already shown mastery of the task or skill. for example, in a philosophy class, where the content and vocabulary can be very abstract, the professor could share an example of a gettier case then ask students to devise their own according to the same basic structure. relatedly, a professor can model not only a task but also a thought process. this may be especially useful in math courses, where the process of getting to the solution is just as important as the solution itself. a professor can “think out loud” with the students in a collaborative fashion and invite them to follow the steps needed to arrive at a solution (kim & bowles, 2019). finally, providing students with visual examples of high quality student work completed in previous semesters can serve as another form of modeling that can greatly benefit ell students. some may argue that modeling is a disservice to students and could stifle original thinking. however, ells may have had significantly less exposure to the types of independent tasks expected of them in higher education. they also may have had many more scaffolds from an esl teacher to support task completion. modeling can be a starting point from which students can build. collaborative structures collaborative structures are beneficial to ells for many reasons. first, the nature of collaborative structures means that the student will have opportunities to listen and participate in academic conversations with a peer. these structured opportunities for discourse are important in the learning process. they provide a setting for the ell student to gain cultural understandings that are often taken for granted during a lecture setting. second, understanding of content knowledge and skills may increase when the ell student collaborates with a high performing peer (watanabe & swain, 2007). collaborative learning promotes critical thinking as students engage in conversations about the topic while interacting, questioning, and explaining their ideas to each other (scager et al., 2016). finally, collaborative learning helps students to develop social skills necessary for success in the workplace. without social interactions, collaboration falls short as it is through the discussion of ideas, crafting of rebuttals, and combining of perspectives/ideas that one develops critical thinking and academic literacy skills (scager et al., 2016). in higher education, collaborative structures are most frequently implemented as group projects to be completed outside of the class meeting time (scager et al., 2016). to ensure successful collaboration, it is important that the instructor create an assignment that fosters interdependence and autonomy. interdependence links all members of the group together by requiring equal amounts of participation as well as the ultimate success of the group hinging on each person completing a task (scager et al., 2016). autonomy to make decisions as a group is equally important for successful group work in higher education. learners crave the independence to make choices on their own; however, a balance must be struck because too vague of an assignment may result in failed outcomes. in science laboratories, there are many benefits of collaborative structures stemming from conversations during the lab where knowledge is pooled together, peers correct students’ errors, and students engage in observational learning. nokes-malach et al. (2015) found that college ready or not? 121 journal of effective teaching in higher education, vol. 4, no. 1 individuals achieved more when working in a collaborative structure when multiple perspectives are engaged and the workload is shared. an untapped instructional strategy in higher education is in-class collaborative structures. think-pair-share is an example of a collaborative strategy that could be interwoven during a larger lecture class. this strategy begins by the instructor posing a question and each student thinking about the problem/question. next, the students turn to their neighbor and pair up and share their thoughts or solutions. this provides an opportunity for the affirmation of ideas or a chance to revise proposed solutions. in the final step, the instructor selects students to share out to the larger group. within smaller class sizes, jigsaw activities are an effective strategy for reading a text that is either lengthy or complex through a collaborative structure. the instructor assigns groups, and each group is responsible for one section of the text. rather than having to read the entire text, individuals are only responsible for their group’s section. this benefits ells who may struggle to read a complex or lengthy passage in a short amount of time. each group reads and extracts key information from their section. finally, all groups share out their findings and everyone is responsible for the information learned in the text. visual representation of content a learner gathers information through linguistic and non-linguistic components. for the ell learner, nonlinguistic components often clarify understanding and subsequently result in greater comprehension of the content (halwani, 2017; petrie, 2003). the use of color, images, and graphics enhances comprehension and provides memory hooks. features such as images, tables, charts, and figures all enhance a nonfiction text, particularly college textbooks. the strategic use of these elements both outside of class as well as during the lecture is beneficial to the ell’s understanding of the material. for example, a history professor may share an image of “washington crossing the delaware” and ask students to describe the people and action in the image and predict the implications of the event. benefits for the ell students include interacting within group conversations, engaging in critical thinking, and making connections between visual and verbal (halwani, 2017). visual aids, such as graphic organizers, venn diagrams, timelines, and concept maps are particularly beneficial to ell students in higher education. these nonlinguistic tools organize information, highlighting the essential concepts that provide clarification and make a visual-verbal connection (halwani, 2017). the use of graphic organizers in higher education leads to higher engagement for ells as well as opens the door for their greater comprehension of course content (baxendell, 2003; uba et al., 2017). modifying instruction for ells may seem daunting at first; however, these scaffolded strategies provide the infrastructure to support learning at the highest levels. these strategies also provide appropriate scaffolding and rigor to ensure that ells are working in the zpd (gibbons, 2009). modifying course assignments does not equate to lowering expectations or rigor, rather it includes the creation of college ready or not? 122 journal of effective teaching in higher education, vol. 4, no. 1 meaningful assignments after careful consideration of the student’s learner profile, language proficiency, and appropriate levels of scaffolding. these instructional strategies may be adapted across all disciplines and executed in a variety of ways to develop critical thinking and academic literacy for ells. final thoughts as we consider what it means to be “college ready,” it is important not to lose sight of the academic and social emotional needs of ells across all levels, p-16. this is especially paramount as we navigate two pandemics, both of which have only exacerbated the barriers for ells in terms of equity and access to educational opportunities. ihes may see increased enrollments of ells if more financial resources are provided to prek-12 settings for dedicated positions that can better serve the variety of needs of ells. this also may mean reimagining the role of prek-12 guidance counselors and esl personnel when it comes to educating ells and their families about the college preparation and application process. this is particularly true of those in rural areas who are often ignored for bilingual education programs and key esl personnel supports due to low enrollments (coady, 2020). in turn, more research is needed to help ihes understand what works for different types of ells in college settings and what role faculty have to play in supporting their persistence. this knowledge can then inform the development of the types of support faculty may need to help students find success in their courses. with community colleges serving as a common entry point for ells, future research could also continue exploring the unique policies, programs, and practices that may be happening in this space and could potentially be scaled up to larger universities as well (bergey et al., 2018). the practices identified here are by no means meant to be exhaustive but provide a meaningful snapshot of strategies professors across different fields of study might consider implementing in their classes to help these “college ready” ell students achieve the success their high school guidance counselor assured them they could find. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references baker, c. (2001). foundations of bilingual education and bilingualism (3rd ed.). multilingual matters. bustamante, j. (2020, april 12). international student enrollment statistics. educationdata.org. https://educationdata.org/international-studentenrollment-statistics/ college ready or not? 123 journal of effective teaching in higher education, vol. 4, no. 1 batalova, j., blizzard, b., & bolter, j. (2020, february 14). frequently requested statistics on immigrants and immigration in the united states. migration policy institute. https://www.migrationpolicy.org/article/frequentlyrequested-statistics-immigrants-and-immigration-unitedstates#demographic batson, c. d., early, s., & salvarani, g. (1997). perspective taking: imagining how another feels versus imagining how you would feel. personality and social psychology bulletin, 23(7), 751–758. baxendell, b. w. (2003). consistent, coherent, creative: the 3 c’s of graphic organizers. teaching exceptional children, 35, 46–53. https://doi.org/10.1177/004005990303500307 bergey, r., movit, m., baird, a. s., & faria, a. m. (2018). serving english language learners in higher education: unlocking the potential. american institutes for research. bourdieu, p. (1986). the forms of capital. in j. richardson (ed.), handbook of theory and research for the sociology of education (pp. 241–258). greenwood. coady, m. r. (2020). rural english learner education: a review of research and call for a national agenda. educational researcher, 49(7), 524–532. cohen, a. (1996). second language learning and use strategies: clarifying issues. working paper series #3. university of minnesota center for advanced research on language acquisition. conley, d. t. (2012). a complete definition of college and career readiness. educational policy improvement center. cummins, j. (1980). the cross-lingual dimensions of language proficiency, linguistic interdependence: implications for bilingual education and the optimal age issue. tesol quarterly, 14(21), 175–187. deanza college. (n.d.). who are generation 1.5 students? http://deanza.edu/english/resources/generation.html diaz-rico, l.t. (2018). the crosscultural, language, and academic development handbook: a complete k-12 reference guide (6th ed.). pearson. echevarria, j. (2018, november 14). what are your expectations for english learners? https://www.janaechevarria.com/?p=1167 eslteacheredu.org. (2020). what is an esl teacher? https://www.eslteacheredu.org/what-is-an-eslteacher/#:~:text=teachers%20of%20english%20to%20speakers,to%20des college ready or not? 124 journal of effective teaching in higher education, vol. 4, no. 1 cribe%20educating%20english%20learners.&text=ell%20students%20are %20referred%20to,that%20they%20are%20integrated%20into fenton-o’creevy, m., & van mourik, c. (2016). ‘i understood the words but i didn’t know what they meant’: japanese online mba students’ experiences of british assessment practices. open learning: the journal of open, distance and e-learning, 31(2), 130–140. https://doi.org/10.1080/02680513.2016.1177503 fenwick, t., & r. edwards. 2010. actor-network theory in education. routledge. flores, s. m. (2014). does english language learner (ell) identification predict college remediation designation?: a comparison by race and ethnicity, and ell waiver status. review of higher education, 38(1), 1–36. https://doi.org/10.1353/rhe.2014.0041 gallagher, c. e., & haan, j.e. (2018). university faculty beliefs about emergent multilinguals and linguistically responsive instruction. tesol quarterly, 52, 304–330. https://doi.org/10.1002/tesq.399 gay, g. (2010). culturally responsive teaching (2nd ed.). teachers college press. gibbons, p. (2009). english learners, academic literacy, and thinking: learning in the challenge zone. heinemann. grabe, w., & kaplan, r. b. (2014). theory and practice of writing: an applied linguistics perspective. routledge. halwani, n. (2017). visual aids and multimedia in second language acquisition. english language teaching, 10(6), 53–59. https://files.eric.ed.gov/fulltext/ej1143525.pdf helmy, h.e. (2016). a lottery on the first day of classes! an innovative structured steps assignment on a partially randomly selected topic. international review of economics education, 21, 41–47. https://doi.org/10.1016/j.iree.2015.12.002 intersegmental committee of the academic senates. (2002). academic literacy: a statement of competencies expected of students entering california’s public colleges and universities. https://senate.universityofcalifornia.edu/_files/reports/acadlit.pdf kanno, y., & cromley, j. g. (2012). english language learners’ access to and attainment in postsecondary education. tesol quarterly, 47(1), 89–121. college ready or not? 125 journal of effective teaching in higher education, vol. 4, no. 1 kanno, y., & cromley, j. (2015). english language learners’ pathways to four-year colleges. teachers college record, 117(12), 1–44. kanno, y., & kangas, s. e. (2014). "i'm not going to be, like, for the ap": english language learners' limited access to advanced college-preparatory courses in high school. american educational research journal, 51(5), 848–878. https://doi.org/10.3102/0002831214544716 kaur, m., & singh, m. (2019). academic reading and writing challenges among international efl master’s students in a malaysian university: the voice of lecturers. journal of international students, 9(4), 972–992. https://doi.org/10.3102/0002831214544716 kice, a. r. (2014, september). how u.s. colleges gauge international students’ english skills. u.s. news & world report. https://www.usnews.com/education/blogs/international-studentcounsel/2014/09/11/how-us-colleges-gauge-international-students-englishskills kim, h. r., & bowles, m. (2019). how deeply do second language learners process written corrective feedback? insights gained from think-alouds. tesol quarterly, 53(4), 913–938. ladson-billings, g. (1995). toward a theory of culturally relevant pedagogy. american educational research journal, 32(3), 465–491. https://doi.org/10.2307/1163320 latour, b. 2007. reassembling the social: an introduction to actor-network-theory. oxford university press. lee, s. j. (2012). new talk about ell students. phi delta kappan, 93(8), 66–69. lundy-ponce, g. (2010). migrant students: what we need to know to help them succeed. colorincolorado! https://www.colorincolorado.org/article/migrantstudents-what-we-need-know-help-them-succeed malin, j. r., bragg, d. d., & hackmann, d. g. (2017). college and career readiness and the every student succeeds act. educational administration quarterly, 53(5), 809–838. https://doi.org/10.1177/0013161x17714845 martin, n. d., spenner, k. i., & mustillo, s. a. (2017). a test of leading explanations for the college racial-ethnic achievement gap: evidence from longitudinal case study. research in higher education, 58(6), 617–645. doi:10.1007/s11162-016-9439-6 college ready or not? 126 journal of effective teaching in higher education, vol. 4, no. 1 merkel, w. (2020). a case study of undergraduate l2 writers’ concerns with source based writing and plagiarism. tesol journal, 11(3), e00503. minnich, c., martin, r., pernsteiner, g., & gandal, m. (2016). leveraging essa to increase college readiness and completion. https://edstrategy.org/wpcontent/uploads/2017/10/hehs-leveraging-essa-092816.pdf moody, j. (2019, november). where, how international students are studying in the u.s. u.s. news & world report. https://www.usnews.com/education/bestcolleges/articles/where-howinternational-students-are-studying-in-the-us national center for educational statistics (2018–19). percentage of degree-granting postsecondary institutions with first-year undergraduates offering remedial services, by control and level of institution: 1989–90 through 2018–19. https://nces.ed.gov/ipeds/search?query=remedial&query2=remedial&resultt ype=all&page=1&sortby=relevance&overlaydigesttableid=202005 nokes-malach, t. j., richey, j. e., & gadgil, s. (2015). when is it better to learn together? insights from research on collaborative learning. educational psychology review, 27, 645–656. doi:10.1007/s10648-015-9312-8 nuñez, a. (2009). migrant students’ college access: emerging evidence from the migrant student leadership institute. journal of latinos and education, 8(3), 181–198. https://doi.org/10.1080/15348430902888781 office of elementary and secondary education. (2020). college assistance migrant program. https://oese.ed.gov/offices/office-of-migrant-education/collegeassistance-migrant-program/ perez, c. p., & morrison, s. s. (2016). understanding the challenges of english language learners and increasing college-going culture: suggestions for school counselors. ideas and research you can use: vistas, 18, 1–12. petrie, g.m. (2003). esl teachers’ views on visual language: a grounded theory. the reading matrix, 3(3), 137–168. https://pdfs.semanticscholar.org/0800/0b72c5acaad03c3c8cb383bf287259cb d035.pdf postsecondary national policy institute. (2019, december 11). immigrant students in higher education. https://pnpi.org/immigrant-students-in-higher education/ scager, k., boonstra, j., peeters, t., vulpershorst, j., & wiegant, f. (2016). collaborative learning in higher education: evoking positive interdependence. cbe life sciences education, 15(4), 1–9. https://doi.org/10.1187/cbe.16-070219 college ready or not? 127 journal of effective teaching in higher education, vol. 4, no. 1 scott-clayton, j., crosta, p.m., & belfield, c.r. (2014). improving the targeting of treatment: evidence from college remediation. educational evaluation and policy analysis, 36(3), 371–393. https://www.jstor.org/stable/43773472 stotland, e. (1969). exploratory investigations of empathy. in berkowitz, l. (ed.), advances in experimental social psychology (vol. 4). academic press. uba, e., oteikwu, e.a., onwuka, e., & abiodun-eniayeken. (2017). a research based evidence of the effect of graphic organizers on the understanding of prose fiction in esl classroom. sage open, 7(2), 1–9. https://doi.org/10.1177/2158244017709506 u.s. department of education. collegeand career-ready standards. https://www.ed.gov/k-12reforms/standards vygotsky, l. s. (1978). mind in society: the development of higher psychological processes. harvard university press. warren, c. a. (2018). empathy, teacher dispositions, and preparation for culturally responsive pedagogy. journal of teacher education, 69(2), 169 –183. watanabe, y., & swain, m. (2007). effects of proficiency differences and patterns of pair interaction on second language learning: collaborative dialogue between adult esl learners. language teaching research, 11(2), 121–142. https://doi.org/10.1177/136216880607074599 witt, d., & soet, m. (2020, july 13). 5 effective modeling strategies for english learners. https://www.edutopia.org/article/5-effective-modeling-strategiesenglish-learners microsoft word worsley&roby_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi 10.36021/jethe.v4i2.166 fostering spaces for black joy in stem-rich making and beyond ti’era worsley, university of north carolina at greensboro, tdworsle@uncg.edu reanna s. roby, vanderbilt university abstract. what does it mean to express black joy and loving blackness through stem-rich making? what does it mean for black youth in community-based, youthfocused makerspaces to express black joy and loving blackness? we look at how black youth alongside their facilitators co-create spaces of black joy through making. these makerspaces are located at two local boys and girls clubs in the us midwest and the southeast. makerspaces are informal sites where youth are encouraged to work collaboratively while building digital and physical artifacts. as two black female stem educators working with black youth we frame our work in critical race theory. specifically we draw on the tenets of whiteness as property and counter-narratives. using critical ethnographic methods, we explore the ways in which black youth produce counter-narratives that disrupt whiteness as property through stem-rich making. data sources include fieldnotes; artifacts, such as youth work; interviews; and video recordings. the first vignette highlights how two black girls navigate choosing and creating characters using scratch. the second vignette focuses on a brother and sister duo who center their making on family and their shared maker identity. we then discuss the freedoms afforded to youth with flexible co-designed curriculum with facilitators and how we foster open spaces. we address this special issue’s driving question by asking, how do we, as stem facilitators, counter anti-blackness in/through stem by fostering space for black joy with youth in making? keywords: black joy, informal, makerspace, stem historically, stem (science, technology, engineering, and mathematics) has been depicted as a bland, white space with little to no spice, pizzaz, or energy. similarly, innovations within these domains have been enriched and made more appealing with the centering of black thought and joy, for example, black twitter and tik tok (dates & ramirez, 2018). the same can be said about maker education that centers black joy and love within such spaces. this article highlights expressions of black joy and loving blackness through stem-rich making: “stem-rich making refers to making projects and experiences that support makers in deepening and applying science and engineering knowledge and practice, in conjunction with other powerful forms of knowledge and practice” (calabrese barton & tan, 2018, p. 3). specifically, this piece is designed for science education researchers who speak to the development of curriculum and policy that directly and indirectly impacts science learning. as such, we see the work of calling attention to anti-blackness in stem and ideally creating environments for black joy as a circular conversation. as two black women with co-conspirators (love, 2019), we have co-created a community that celebrates and enacts black joy through relationships with students, families, and staff within two boys and girls clubs (bgc). in these making-based clubs, youth have fostered a community where they have strong bonds and sense of ownership (calabrese barton & tan, 2018; roby & calabrese fostering spaces for black joy in stem-rich making and beyond 119 journal of effective teaching in higher education, vol. 4, no. 2 barton, 2019). we support youth in having significant agency over what they make and how and why they want to make, especially in ways that challenge the settled boundaries of making. we employ the following critical race theory (crt) tenets, whiteness as property (harris, 1993) and counter-narrative (stefancic & delgado, 2000), as means of highlighting the liberatory praxes made possible through making while also reimagining participation and epistemologies by/with minoritized youth. our work in community-making counters black suffering while keenly focusing on black joy and brilliance (bullock et al., 2012; dumas, 2014). black suffering within the context of stem is a silenced discussion, though central to many of the practices implemented in pk-20 in formal and informal learning contexts, especially in stem (bastile & lopez, 2015; mcgee, 2020). for example, stem curriculum diminishes how race was employed as a variable to define humanness, not only with regard to education but also history (nelson, 2016). additionally, leaning into color-evasive ideologies (annamma et al., 2017) to advance science teaching and program development not only works to erase the experiences of black learners but to rationalize whiteness as a form of objectivity that is above critique (habig et al., 2021). together these factors insinuate stem is not “a place” for black youth (roby & calabrese, 2019; vakil & ayers, 2019). we focus on the joy fostered throughout the making process from learning new skills to completing community-needed projects. the makers from and with whom we have the privilege of learning exude black joy and brilliance in a variety of ways. as such, answering the research question, how do we, as stem facilitators, counter anti-blackness in/through stem, by fostering space for black joy with youth in making?, is in service of radically imagining making that celebrates and honors black youth. literature review making, as an extension for stem, generally takes place in informal science settings, such as community centers, museums, and science centers. within the context of makerspaces and other informal science spaces, black youth are presented with the opportunity to leverage their community cultural wealth (yosso, 2005) and engage in stem through its multiple entry points (calabrese barton & tan, 2010; dawson, 2017). yosso (2005) defines community cultural wealth as an array of knowledge, skills, abilities, and contacts possessed and utilized by communities of color to survive and resist macroand micro-forms of oppression (p. 77). while making serves as a site of possibilities, it still remains dominated by white, heteronormative, male, and middle-class adults (halverson & sheridan, 2014) and remains largely inaccessible by black and brown communities. as such, the dominant maker culture exudes whiteness as property by legitimizing their discourses, practices, and tools in making (keenan-lechel, 2019). critical maker educators aim to broaden what counts as making and who is or can be a maker (peppler et al., 2016) by challenging dominant discourses and practices (barajaslópez & bang, 2018; gollihue, 2019). within the last decade, minoritized youth, who have been the focus of a wide range of maker intervention programs, and yet have been positioned as novice/outsider in need of being mentored into making through deficit lens and lack of acknowledgment of their cultural wealth. it is fostering spaces for black joy in stem-rich making and beyond 120 journal of effective teaching in higher education, vol. 4, no. 2 important to acknowledge how minoritized communities have always been engaged in making in ways that matter to them, their families, and communities (vossoughi et al., 2016). however, not all opportunities to make in maker programs acknowledge these histories or support such meaningful making. to the contrary, making has been presented as elitist and perpetuates anti-blackness in subtle, yet still harmful ways. the structural inequities and forms of oppression that limit opportunities to make in ways that matter include the powered relationalities that inform life in makerspaces (gollihue, 2019). there are current models of informal science that look to embrace the totality of blackness specifically while countering its elitism and anti-black values. king and pringle (2018) present a summer program called i am stem. i am stem is a community-based, non-residential summer program for black girls in grades 4–8 that focuses on how informal experiences transfer into the classroom. one of the main principles of i am stem is to tend to the whole child (mind, body, and spirit) and maintain “culturally healthy” students. the participants were co-constructors of knowledge and worked alongside researchers. the focus of this study was for black girls to share their counter-narratives about their experiences in stem. three themes that emerged were field trips and authentic stem experiences, continuation of stem activities, and the role of race in their formal stem experiences. the participants shared that field trips were a key factor for them to engage in stem learning and expanded their knowledge. participants also began to look for stem opportunities outside of i am stem that included formal and informal settings. when participants reflected on their formal experiences, they felt racialized in their science classroom as they noticed their teacher treated white girls better (king & pringle, 2018). critical and political studies of maker learning over the past several years have offered a more robust, heterogeneous view of what maker culture can be. for example, barajas-lópez and bang (2018) describe how maker programs centered on clay making within indigenous communities cultivate a culture built around socially and ecologically just nature-culture relations, where “knowing, being, and doing are simultaneously unfolding and are essential/defining elements in the practice of material storytelling” (p. 17). studies that have followed youth through makerspaces to inform iterative makerspace design have also contributed to expanded understandings of what maker practices are possible and could be more explicitly acknowledged, supported, and centralized (martin et al., 2018). for these reasons, we employ critical race theory (crt) as a tool to combat the anti-black nature of making, while providing a counter-narrative of black joy. conceptual framework given the centrality of black joy within the context of this study, we employ crt as a conceptual framework. crt within the context of education and research has a history reaching back to its roots in critical legal studies and critical theory (bell, 1987). applying crt in education makes it possible to analyze practices and ideologies through a race-conscious lens, which can help to frame critical questions fostering spaces for black joy in stem-rich making and beyond 121 journal of effective teaching in higher education, vol. 4, no. 2 addressing the traumas that directly affect communities of color (mcgee & stovall, 2015). for the purpose of this study, we focus primarily on the tenets of whiteness as property and counter-narrative. whiteness as property (harris, 1993) can be understood as a means to call attention to the danger and pervasive nature of whiteness within the context of science, making, and knowing. within the context of science and science education and by extension making, the dominant discourses of whiteness serve to disenfranchise while also withholding or limiting participation and access for minoritized youth (mensah & jackson, 2018; miles, 2019; wright & riley, 2021). we trouble this practice by centering narratives of black youth coopting science and making it their own. crt operationalizes counter-narrative in three different ways: 1) lifts the voices of people of color so that race and racism are seen from an alternative lens; 2) pushes against ethnocentrism and one-world views to construct a new reality; and 3) works against silencing oppressed individuals and provides legitimated space for the “counter” perspective/narrative to be heard (solórzano & yosso, 2002; stefancic & delgado, 2000; tate, 1994). the valuing of stories and experiences acknowledges that “…those who lack material wealth or political power still have access to thought and language, and their development of those tools…differs from that of the most privileged” (matsuda, 1987, p. 65). the practice of centering counter-narrative serves as a way to locate black joy within the context of making while also pushing back on anti-blackness within the dominant epistemological understandings and practices of science, which suggest black youth are in some ways sub-human and, as such, unable to feel or express emotion (coles, 2019; haynes et al., 2016). positionality statement as scholars deeply invested in equity and justice in science and science education by way of informal science, this study is a testament to the power of love and joy we experience with youth as we engage in this work. as such, we are invested in not only calling out anti-blackness within the context of science education, extension, and maker education but, more importantly, highlighting the active resistance to such in the form of black joy as a means of disruption, resistance, and liberation. the authors of this paper identify as first-generation, cisgendered, black women from the rural south who have not only witnessed black joy within the context of making with youth but have lived and experienced such as a result of our ontological experiences with science and making in unorthodox ways from girlhood. author 1 — ti’era worsley i identify as a black woman and a tinkerer. as a child, my greatest joy was to take things apart to understand how they worked and then put them back together. my family always encouraged and supported my interests and skills. i enrolled in an engineering program at a pwi located in the southeast, and this experience showed me the harm that can be done when others feel that you do not “fit” in. as a black woman who works with black youth, i make it my personal goal to encourage them fostering spaces for black joy in stem-rich making and beyond 122 journal of effective teaching in higher education, vol. 4, no. 2 in all ways possible. it can be easy for someone to point out all of the ways that you do not match rather than to focus on the brilliance (leonard & martin, 2013) of what you can contribute. author 2 — reanna s. roby i identify as a cisgender black woman from the deep rural south. my love for science stemmed from the intimate familial relationship with my grandparents and their engagement with informal science (farming and gardening) in addition to numerous hours spent in the beauty salon on saturday mornings. this love for informal science was enriched with a passion for mathematics that prompted a love for chemistry. as an undergraduate at my hbcu, i was nurtured and supported in my chemistry engagement. however, upon graduation and matriculation to graduate school, i quickly noticed the overrepresentation of whiteness in the field. this motivated me to start a new line of inquiry that critically assessed participation in science fields in addition to the history of science. my ontological experiences coupled with my training as a scientist and social scientist equip me with a unique lens to ideally reimagine science differently. our cross-generation, multi-racial/ethnic research team across sites consists of scholars ranging from undergrads to graduate students to full professors. we are situated in the us midwest and southeast. within the context of this work, we see ourselves as co-learners and co-authors within the makerspaces with youth makers and youth authored-knowledge. methods this study is grounded in two stem-rich makerspaces housed at boys and girls clubs within the midwest and southeast regions over the course of a year. because our study aims to highlight black joy by critiquing what it means to make, we employ critical ethnographic approaches. critical ethnography starts with an ethical responsibility to call out injustices within the communities in which we work (madison, 2011). as black women and critical ethnographers in this space, we are hyper aware of how power, as it relates to race and gender, are operationalized in/within stem, making, and larger society (evans-winters, 2019). as authors deeply entrenched in the makerspaces, spending an average of 15-20 hours per week (2017–present) during the academic year and summer with the youth, we acknowledge our roles as integral members of the community. data sources included fieldnotes; artifacts, such as student work; interviews; and video recordings and vlogs. fieldnotes were taken after programming sessions. artifacts included projects made by youth as well as scaffolding resources to aid in the development of projects. artifact interviews were conducted over the span of project development while vlogs (video logs) were recorded at the end of meeting sessions. data were analyzed using a grounded theory approach (glaser, 1992) that sought to identify critical episodes where youth were supported and reveled in black joy. the criteria for these episodes were youth’s challenge of dominant views to prompt counter-narratives of making and joy for black youth. in concert with grounded theory, the moments described within this text were not sought after but, fostering spaces for black joy in stem-rich making and beyond 123 journal of effective teaching in higher education, vol. 4, no. 2 instead, surfaced as we considered the functioning of joy in direct opposition to anti-blackness. mid-west green club the mid-west green club has functioned for over more than a decade within the community providing youth with stem-rich making experiences and more. the midwest green club offered programming and events for youth grades three and up focused on stem-rich making, coding, robotics, and forensics. during the signature program of the mid-west green club, which centered making for middle school youth, participants would identify issues pertinent to them and develop ways to address them. the socially just community that makes up the mid-west green club is a collective of youth, research practitioners, and their families. the mid-west green club is centrally located to the community and within walking distance for some of the participants. each session was hosted twice and week and started with refreshments and orienting discussions before diving-into making. southeast green club within the southeast green club, worsley had dual roles as a researcher and facilitator. southeast green club meets once weekly, and the theme of programming at the time of the vignette was coding. southeast green club consisted of about 12 youth participants (in sixth and seventh grade), who were predominately black. in southeast green club, youth have fostered a community where they have a strong bond and sense of ownership. youth have significant agency over their work and space (such as moving freely in the room, visiting other friends to look at their work) and a lot of input with how weekly programming goes. youth are also provided a place where they can share and vent about their daily lives, which heavily revolve around school. there is no set curriculum for the program that directs youth in a linear, step by step process. for example, when youth are learning a new coding concept, facilitators provide a general overview of how the different codes work. youth are then able to take that direction and explore what they can produce with it. the facilitators usually go in depth about how to do something when it is requested by the youth or they are presenting a new concept/feature. this is referred to as just-in-time teaching (calabrese barton & tan, 2018). vignettes can we become the characters in our scratch? — felicia and kia the focus of the southeast green club at the time (2018–2019) was coding with the use of scratch and ozobots. ozobots are tiny robots that use a color-coding system on paper by using a combination of colors (red, blue, green, and black) and can also be used with a block-based coding program online (ozoblockly.com). scratch is an online block-based coding program created by massachusetts institute of technology (mit) where users can create games and share them with an online community. the project that youth were working on was to create a storyboard fostering spaces for black joy in stem-rich making and beyond 124 journal of effective teaching in higher education, vol. 4, no. 2 with 12 scenes. the themes of these stories came from the responses from their community ethnography. youth were tasked with interviewing bgc staff and other youth about what they appreciated about their community and something that they wished for their community. youth paired up, were given an ipad to record, and practiced their interviewee questions. youth practiced their interviews in pairs to develop their interview etiquette, such as starting with, hello, how are you? and do you have a moment to do a quick interview with us? they then practiced probing questions, such as, can you tell me more? or can you give me an example? once youth felt ready, they left the room to find interviewees. when youth returned they used the responses they collected to fill out their storyboards. some of the findings included that staff liked working there because of the kids, liked that the programs taught kids new stuff, and, based on one staff member’s response, felt that youth were her kids. some youth responses included activities, such as sports, games, friends, help with homework, and helping people. once their storyboards were completed, youth began coding. two makers, felicia and kia, had interviewed a few staff members about what they liked most about working at bgc. as mentioned above, a common response among the staff was that they enjoyed working at bgc because of the kids. felicia and kia incorporated this data into their scratch. as they were thinking about how they wanted to design their scratch, they knew they wanted to include kids. they started to look through the pre-loaded scratch characters to find ones that looked like kids. as they searched through the characters, they found one that they wanted to add. they raised their hand, and worsley walked over to assist. they explained their dilemma saying that they had located a character that they felt was representative of a kid, but they wanted to alter the character to be more reflective of a black kid. they asked if there was a way that they could paint the characters. to clarify, i asked felicia and kia what they meant by paint the character. they responded by saying they wanted to paint the skin on the character to make it look darker. scratch has an editing feature (costumes) where you can edit characters. users have many options to alter characters such as add shapes, flip the image, fill the background, and many others. first, together we looked at the colors, which were presented on a color wheel. the colors immediately presented are red, yellow, green blue, purple, pink, and back to red. to create different shades of these colors, the user must adjust the color by saturation and brightness. felicia and kia were looking for shades of brown, and brown is not a pre-selected color in the color wheel. to create brown, felicia and kia had to select the red color and then adjust the saturation and brightness until they created the desired shade of brown. after selecting the right shade of brown, felicia and kia used the paintbrush tool to find the right size. the size of the paintbrush can be changed on a scale of 1–100 (1 being extremely thin strokes and 100 being extremely thick strokes). felicia and kia decided to use a thinner size so they could carefully apply the brown color on top of the character. this required felicia and kia to zoom in on the character and paint small sections at a time. this process was tedious to the girls as they had to constantly zoom in and out on the character and while staying in the lines. fostering spaces for black joy in stem-rich making and beyond 125 journal of effective teaching in higher education, vol. 4, no. 2 after, worsley showed them what they would need to do, felicia and kia continued to work. because of the long and tedious process of painting the kid character, felicia and kia stopped painting the character. they then asked one of the other facilitators if they could take pictures of themselves and make the images a character instead. the facilitator said that they could do that, and together they left the room to find a location to take their photo. felicia and kia decided to take their photo in a doorway that led to one of the work rooms, with an apartment building in the background. when felicia and kia returned back to the room, they were all smiles as they waited to see how their photo would transfer from the cell phone to scratch. the facilitator emailed the picture and then downloaded the attachment to the computer. once on the computer, the facilitator selected the upload scratch option for adding characters and then the images appeared in their scratch. felicia and kia had to adjust the size of their characters so that they could fit within the frame boundary of scratch. felicia and kia decided to make their pictures the same size and placed kia to the left (figure 1) and felicia to the right. felicia and kia instantly began to smile when they saw themselves as characters in their scratch. figure 1 kia as a character in her scratch fostering spaces for black joy in stem-rich making and beyond 126 journal of effective teaching in higher education, vol. 4, no. 2 analysis by engaging in the process of stem-rich making, we see felicia and kia provide a counternarrative of who can participate in stem. for two black girls, the lack of representation within stem by not seeing themselves as pre-loaded characters could have led them to settle for a character that they did not want to include. however, we see them offer a counter-narrative by disrupting whiteness within the context of scratch and reauthoring who is making for and for what purpose. together they leverage the support of their facilitators and utilize them as resources to learn the skills needed to create black girl characters. felicia and kia had the option to upload any character that they wanted into their scratch. they could have searched the internet for other characters that were black girls, but with much enthusiasm, they chose to use images of themselves, thereby decentering whiteness. usually, when the facilitators take pictures of the girls specifically they tend to cover their faces, shy away from the camera, or immediately demand to see the picture. choosing to be seen on the scratch they were creating was a powerful statement that challenged notions of whiteness as property that pushed back on who has the right to be represented as characters in games while also countering dominant discourses of how black youth experience and embody joy. to support felicia and kia’s vision of their scratch, we see the facilitators being allies and playing an integral role to support their vision. when felicia and kia first asked if they could “paint” the character’s skin, worsley did not deter them from their idea because of the additional work involved. she took the time to show felicia and kia the skill of how to alter characters using the costumes tool. when felicia and kia decided that painting the skin of the character would be a tedious task, they decided to use images of themselves. again we see the facilitator support their vision by showing felicia and kia how to download and upload images as characters into scratch. the facilitators provided skills that felicia and kia would continue to draw on for the remainder of the year. making with family in mind at mid-west green club, roby had the honor of working as a research practitioner and also as a liaison between the mid-west green club and the university. our facility, which had its own space dedicated to the mid-west green club as well as teenage members, was a thriving environment full of energy. tasked with bridging the mid-west green club for middle schoolers and teens prompted a peculiar tension, but one roby leaned into with grace, that is, she was ready for the challenge and the odds it may have presented. the joy in making and beyond described below is reflective of my relationship with a group of siblings who were between the makerspace in the teen zone. “why your knees look like raspusha[1]?” suzanne asked, as i held the door to open to the maker space. taken a back, but not knocked down completely, i responded “because that’s how god made me.” fostering spaces for black joy in stem-rich making and beyond 127 journal of effective teaching in higher education, vol. 4, no. 2 suzanne was a popular black girl at the club, but not for all of the “right” reasons. as a result, she would sometimes be reprimanded and have her “mid-west green club'' privileges revoked by club staff above my pay grade. her big, bold spirit was endearing, and during one-on-ones, her tenderness was made plain or evident, as exhibited during the process of creating a board that featured her name. as a black girl from the deep south, suzanne’s name plate connected her to her former home and the family there as well as her immediate family. during the development of her name board, suzanne was intentional about not only naming herself but also creating a board reflective of the things that brought her joy and brought her family into a space that some may or may not occupy. as such, suzanne selected colors that made her board pop and that also represented her brothers, her mom, and other relatives. each letter on the name board used specific rubber band colors to designate certain family members—her mom, her oldest brother, her other brother, and more. this was a work suzanne was proud of and used to not only express her love of her family but also the joy of making which invited them into the space—directly and indirectly. while suzanne was intentional about making for herself, the selflessness reflected in her board also included others and brought them into the space. within the context of each letter and each color featured on suzanne’s board, there was a narrative that connected to others and served as an example of how her making was not just for her but was a message conveyed to her by her brother mac. mac, suzanne’s brother, was a member of mid-west green club , long before suzanne. during his time as a middle school maker, he led making and design efforts to support his peers with the community in mind. when he and another maker designed the timmy, a timberland boot with a heating core to support wearers in the brutal mid-west winter, it was with the message that making is not a self-serving endeavor and, as such, helping others is one way to bring about community and joy. figure 2 suzanne and an iteration of her name board fostering spaces for black joy in stem-rich making and beyond 128 journal of effective teaching in higher education, vol. 4, no. 2 figure 3 mac starting his own name board after an impromptu visit to mid-west green club analysis while suzanne and mac’s experience in the mid-west green club is one example of family sharing in making, it is also one that seeks to disrupt antiblack narratives of representation and feelings/shared emotions. historically, black people, to include black youth, have been framed as a group of people with limited ability feel or to actually have family (coates, 2015; hoffman et al., 2016). this discourse was used not only to rationalize the inhumane treatment of enslaved people in the us but also to further subjugate black people (haynes et al., 2016). suzanne and mac’s engagement within the makerspace exhibits how their making together serves as a site to consider the generational knowledge shared among siblings and well as a countering of dominant discourses with the joy and love shared through the process. while suzanne was proud of her work and would oftentimes share it throughout the club, it also brought her joy to be supported in tinkering and navigating the makerspace as an extension of her older brother. by getting mac back into the makerspace, both siblings were able to engage in the joy of making and bonding simultaneously. in a world where black youth are often demonized for their creativity, and not provided safe spaces for such, it was important for the midwest green club to be something they could claim as their own and as a space for them to explore with support and care. additionally, leaning into their identities as scientists and makers, suzanne and mac counter who can be a scientist and who science and making are for. their makerspace, which was located in a community center, already troubled the class and racialized privilege of making. their participation, though, went another level by troubling who making is for and to what ends making could be used as a space to tell and amplify stories, not centered in struggle, but in speaking truth to power—in this case, the power of black joy. fostering spaces for black joy in stem-rich making and beyond 129 journal of effective teaching in higher education, vol. 4, no. 2 discussion much of informal science learning tends to further minoritize black thoughts, and access is limited by location and cost. in the vignette of felicia and kia, we see them enact their freedom practice through their determination to authentically represent the black kids at their bgc by bringing their vision to reality. the preloaded characters were not sufficient for the imagined way that they saw their game being represented. however, without the role of the facilitators, this could have become a missed opportunity. the facilitators supported felicia and kia’s choice of adding images of themselves into their game. in addition, this provided an opportunity for felicia and kia to develop their scratch expertise by learning new skills. without the allyship of their facilitators, felicia and kia could have chosen to not continue further with their scratch. this shows the importance of facilitators working alongside youth to engage in possibilities that challenge historically who gets to make and what making has been. in the vignette centering the brother-sister duo, mac and suzanne, we see the siblings engage black joy in part by being together and focusing on the development of their maker projects. creating maker projects that centered their names and how they wanted to show up in the world is deeply connected to black familial practices that extend throughout the diaspora. this can be reflected in how suzanne developed a naming board that brought family in the space, both physically and abstractly. additionally, it builds on the historical practice black people within the united states have leveraged to combat white supremacist practice that disregard them and their names (martin, 1991; ortiz et al. 2019). as research practitioners and facilitators, it is not enough to provide material resources, but also to call upon and include family in the making process. this practice is one way to expand who making is for and what participation in making could look like. the ability to engage making as a freedom practice with family provides unique opportunities for creativity and a love ethic. this works seeks to address a void in maker education literature by specifically considering how black joy can foster spaces of pride, creativity, and ownership for and by black youth. countering anti-blackness by defining, creating, designing and naming one's own worlds, characters, and realities is in essence a form of resistance as well as black joy. implications for stem educators seek to explore how access and the choice to innovate on one’s own terms is a freedom practice (love, 2019; wright & riley, 2021). to consider stem learning as a freedom space that actively counters anti-blackness requires moving from trauma-centered narratives to imagining from perspectives that are not limited to the confines of stem and making as defined by the dominant culture. conclusion we have argued for the freedoms afforded to youth when facilitators are flexible and open in their curriculum and co-design alongside youth (as detailed in the vignettes). by co-designing, the learning environment opens for youth and makes room for their voices to be acknowledged and heard. freedom as a construct has fostering spaces for black joy in stem-rich making and beyond 130 journal of effective teaching in higher education, vol. 4, no. 2 been limited because of whiteness as property. our work acknowledges that our own fullness and capacity to embrace freedom but also to embrace the liberatory praxis that comes with making is not only an act but, in part, a celebration of joy through making. to be able to do this within an informal makerspace is reminiscent of the making that we as practitioners have engaged in through our own experiences with making. however, we acknowledge this work is not easy and having the support of our ontological experiences as well as that of other research practitioners makes it doable for the youth we have the honor to collaborate with. it is for those reasons we explore and uplift the unique and telling ways black youth resist deficit narratives which would suggest that making, and by extension stem, is not for them and neither is joy. seeing how the youth take on projects that decenter whiteness while centering blackness in all of its complexities is one way in which we as practitioners can better engage in developing curriculum that actively combats anti-blackness. looking forward, it is important to consider how we as facilitators can create spaces where black youth feel that they can bring their brilliance. as facilitators we must engage in critical reflection of our praxis and ask where we are supporting the youth’s freedom to explore that brilliance. additionally acknowledging the power that black youth possess and seeing it as an opportunity and not a nuisance requires a level of vulnerability necessary to further intergenerational models and practices of radically imagining something different. this is necessary, as the historical and current conditions of stem in formal contexts and informal contexts, were not necessarily created with black youth in mind as creators, authors, and knowledge producers. thus, the counter-narratives presented within the context of this work are necessary in fully embracing the abundance of blackness and the joy that we witness as black women research practitioners. acknowledgments we thank and acknowledge drs. angela calabrese barton and edna tan for their support and guidance on this paper as well as their mentorship from this research project. we would also like to thank the many makers who we have the privilege of learning from and working with. we also thank the national science foundation for their support (award# 1712834). conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references annamma, s., jackson, d., & morrison, d. (2017) conceptualizing colorevasiveness: using dis/ability critical race theory to expand a color-blind fostering spaces for black joy in stem-rich making and beyond 131 journal of effective teaching in higher education, vol. 4, no. 2 racial ideology in education and society. race ethnicity and education, 20(2), 147–162. barajas-lópez, f., & bang, m. (2018). indigenous making and sharing: claywork in an indigenous steam program. equity and excellence in education, 51(1), 7–20. basile, v., & lopez, e. (2015). and still i see no changes: enduring views of students of color in science and mathematics education policy reports. science education, 99(3), 519–548. bell, d. (1987). law, litigation, and the search for the promised land. georgetown law journal. bullock, e., gholson, m., & alexander, n. (2012). on the brilliance of black children: a response to a clarion call. journal of mathematics education, 5(1) 1–7. calabrese barton, a., & tan, e. (2010). we be burnin'! agency, identity, and science learning. the journal of the learning sciences, 19(2), 187–229. calabrese barton, a., & tan, e. (2018). a longitudinal study of equity-oriented stem-rich making among youth from historically marginalized communities. american educational research journal, 55(4), 761–800. coates, t. n. (2015). the black family in the age of mass incarceration. the atlantic, 316(3), 82. coles, j. a. (2019). the black literacies of urban high school youth countering antiblackness in the context of neoliberal multiculturalism. journal of language and literacy education, 15(2), n2. dawson, e. (2017). social justice and out-of-school science learning: exploring equity in science television, science clubs and maker spaces. science education, 101(4), 539–547. dates, j. l., & ramirez, m. m. (2018). from blackface to black twitter: reflections on black humor, race, politics, & gender. peter lang incorporated, international academic publishers. dumas, m. j. (2014). “losing an arm”: schooling as a site of black suffering. race ethnicity and education, 17(1), 1–29. evans-winters, v. e. (2019). black feminism in qualitative inquiry: a mosaic for writing our daughter's body. routledge. glaser, b. (1992). emergence vs forcing: basics of grounded theory analysis. sociology press. fostering spaces for black joy in stem-rich making and beyond 132 journal of effective teaching in higher education, vol. 4, no. 2 gollihue, k. n. (2019). re-making the makerspace: body, power, and identity in critical making practices. computers and composition, 53, 21–33. habig, b., gupta, p., & adams, j. (2021). disrupting deficit narratives in informal science education: applying community cultural wealth to youth learning an engagement. cultural studies of science education, 16, 509–548. halverson, e. r., & sheridan, k. m. (2014). the maker movement in education. harvard educational review, 84(4), 495–505. harris, c. i. (1993). whiteness as property. harvard law review, 106(8), 1707– 1791. haynes, c., stewart, s., & allen, e. (2016). three paths, one struggle: black women and girls battling invisibility in us classrooms. the journal of negro education, 85(3), 380–391. hoffman, k. m., trawalter, s., axt, j. r., & oliver, m. n. (2016). racial bias in pain assessment and treatment recommendations, and false beliefs about biological differences between blacks and whites. proceedings of the national academy of sciences, 113(16), 4296–4301. keenan-lechel, s. f. (2019). a co-exploration and centering of youths’ funds of knowledge in a stem-rich makerspace. [doctoral dissertation, michigan state university]. msu libraries | digital repository. https://d.lib.msu.edu/etd/48029 king, n., & pringle, r. (2018). black girls speak stem: counterstories of informal and formal learning experiences. journal of research in science teaching, 56(5), 539–569. leonard, j., & martin, d. b. (eds.). (2013). the brilliance of black children in mathematics. iap. love, b. l. (2019). we want to do more than survive: abolitionist teaching and the pursuit of educational freedom. beacon press. martin, b. l. (1991). from negro to black to african american: the power of names and naming. political science quarterly, 106(1), 83–107. martin, l., dixon, c., & betser, s. (2018). iterative design toward equity: youth repertoires of practice in a high school maker space. equity & excellence in education, 51(1), 36–47. matsuda, m. j. (1987). looking to the bottom: critical legal studies and reparations. harvard civil rights-civil liberties law review, 22, 323. mcgee, e. o. (2020). black, brown, bruised: how racialized stem education stifles innovation. harvard education press. fostering spaces for black joy in stem-rich making and beyond 133 journal of effective teaching in higher education, vol. 4, no. 2 mcgee, e., & stovall, d. (2015). reimagining critical race theory in education: mental health, healing, and the pathway to liberatory praxis. educational theory, 65(5), 491–511. madison, d. s. (2011). critical ethnography: method, ethics, and performance. sage publications. mensah, f. m., & jackson, i. (2018). whiteness as property in science teacher education. teachers college record, 120(1), 1–38. miles, m. r. (2019). against the grain: science education researchers and social justice agendas. cultural studies of science education, 14(2), 283–292. nelson, a. (2016). the social life of dna: race, reparations, and reconciliation after the genome. beacon press. ortiz, n., morton, t., roby, r.s., and ridgeway, m. (2019). what about us: exploring the challenges and sources of support influencing black students’ stem identity development in postsecondary education. journal of negro education, 88(3), 311–326. peppler, k. a., halverson, e., & kafai, y. b. (2016). makeology: makerspaces as learning environments (volume 1). routledge. roby, r. s., & calabrese barton, a. (2019). historical exclusion as an onus for inclusion and home in stem. in p. boda (eds.), essays on exclusion: our critical, collective journey toward equity in education. diopress. solórzano, d. g., & yosso, t. j. (2002). critical race methodology: counterstorytelling as an analytical framework for education research. qualitative inquiry, 8(1), 23–44. stefancic, j., & delgado, r. (eds.). (2000). critical race theory: the cutting edge. temple university press. tate, w. f. (1994). from inner city to ivory tower: does my voice matter in the academy? urban education, 29(3), 245–269. vakil, s. & ayers, r. (2019) the racial politics of stem education in the usa: interrogations and explorations. race ethnicity and education, 22(4), 449– 458. vossoughi, s., hooper, p., & escudé. (2016). making through the lens of culture and power: toward transformative visions for educational equity. harvard educational review, 86(2), 206–232. https://doi.org/10.17763/00178055.86.2.206 fostering spaces for black joy in stem-rich making and beyond 134 journal of effective teaching in higher education, vol. 4, no. 2 wright, c., & riley, a. (2021). mitigating the need for resiliency for black girls: reimagining the cultural brokering through a lens of science as white property. cultural studies of science education, 16, 495–500. yosso, t. j. (2005). whose culture has capital? a critical race theory discussion of community cultural wealth. race ethnicity and education, 8(1), 69–91. microsoft word gallagher.docx journal of effective teaching in higher education, vol. 3, no. 2 “i teach, therefore i am?”: exploring the teaching personas of college professors kristel m. gallagher, thiel college, kgallagher@thiel.edu abstract. this exploratory study sought to identify the existence of a “teaching persona” in college professors. specifically, an examination of self-reported differences in traits displayed as teachers versus in everyday life was conducted. also investigated were feelings of job satisfaction and burnout in relation to these differences. findings suggest that professors see themselves as more extraverted, emotionally stable, caring/supportive, professionally competent, and better communicators as teachers than in their everyday lives. however, adjunct professors lacked distinctiveness in all areas. social science professors were less open to new experiences as teachers than in their everyday lives, while natural science professors were more open. in a few instances, larger differences between one’s teaching persona and everyday life were significantly related to more burnout and less job satisfaction. being more conscientious, feeling more capable, and perceiving oneself as a better communicator in the teacher-role was associated with more job burnout. keywords: personality traits; persona; college professor; emotional labor; teacher burnout in his 2001 hit song ‘i don’t have to be me,’ musician steve azar describes the freedom he experiences leaving work on a friday, not having to worry about “being me ‘til monday.” he goes on to say “i can do what i wanna do; be who i wanna be” as he basks in his escape. while most college professors are not as likely to “escape” on a friday and leave all their responsibilities behind until monday, azar’s sentiments do offer an intriguing perspective on how professors might separate themselves from their roles as academics. do college professors modify the presentation of their personality and character traits on the job? do they perform under the guise of a persona that is unlike that of their everyday life? and, if they do, does this help or hinder their work experience? the present study seeks to explore this idea of teaching personas in the college setting as well as the potential relationship between the use of teaching personas and job satisfaction and feelings of burnout. although teacher personality, teacher burnout, and the association between the two has been investigated previously, and most recently in meta-analytic form (kim et al., 2019), the concept of the teaching persona is less present in the literature. mask wearing and the teaching persona psychologist carl jung is well known for his ideas on the concept of “mask wearing” or the persona (hall & nordby, 1973). in roman times, the term persona was often used to represent the mask that actors wore in theatrical presentations. jung “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 29 believed that all individuals wear masks and that these masks are social in nature— we wear them during interactions with others to portray our desired personality characteristics. mask wearing is thought to be an energy drainer as it can create an inner conflict and requires constant management to present, maintain, and adjust (jung, 1989). the concept of the persona, jung’s term for the masks we wear, has been applied to teaching. craig (1994) noted how teachers of all varieties use personas as well how it might be nearly impossible for those in this helping profession not to do so. for many, the act of “being the teacher” becomes automatic to the extent that it is easier to identify with the teaching persona than to recognize and develop the genuine self. in fact, craig (1994) argued that many teachers are not even aware that they are operating through a persona. like other personas, the teaching persona develops as a result of outside pressures, such as the expectations of administrators, colleagues, students, and even society regarding what it means to “be a teacher.” a small body of literature supports the idea that teachers do “wear masks,” though the term persona seems restricted to the jungian psychology literature. for example, sociologist mckinney (1988) proposed that entertainment should be considered an integral component of quality teaching. she asserted that professors are to some degree performers, acting on a stage, and must put on a show in order to effectively convey their subject matter. in fact, mckinney (1988) suggested that professors “entertain (their) students through impression management techniques” (p. 300). griggs (2001) supported this idea, drawing parallels between the training techniques used by actors to convey a genuine and believable performance of a character and the way teachers tend to become their own version of the teacher character they wish to portray to their students. the portrayal of the teacher character, or persona, is likely to be manifested in different forms for different individuals. for some, it may come in the form of trying to look like a teacher—from clothing choice to displays of nonverbal communication. the teaching persona may also manifest itself in behaviors. this might be in the form of exuding professional competency in one’s discipline, being caring and supportive of students, or managing and communicating with confidence and authority. an interesting proposition is to consider whether the teaching persona can also be revealed in distinctive personality traits. for example, one’s teaching persona might be extraverted and conscientious, yet their genuine self might be more reserved and disordered. the current study was inspired by the latter two possibilities—behaviors and personality traits that are distinctive only to one’s teaching persona and not necessarily reflective of one’s everyday portrayal to the outside world. consequences of mask wearing the literature on emotional labor argues that acting on the job, such as one may do when putting on a “teacher mask,” can be harmful. emotional labor is a form of self-regulation in which individuals manage their displays of emotion in the “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 30 workplace setting (hochschild, 1983). for example, it is generally not acceptable for one to display negative emotions, such as irritation and boredom, on the job. however, enthusiasm and cheerfulness, even if not natural emotional responses, are often expected displays. in terms of how these emotions are managed, there are two main strategies that tend to be employed—surface acting and deep acting (grandey, 2000). surface acting is described as faking in “bad faith” as this type of acting is essentially the ultimate faking of emotional displays that govern a work role or job expectation (grandey, 2000). in contrast, deep acting is described as faking in “good faith” because in this case the individual is trying to be genuine in their expressions (grandey, 2000). in other words, when engaging in deep acting, individuals will modify their outward emotional display to conform to the expectations of the workplace role (much like surface acting); however, they will also attempt to feel those emotions on the inside. in surface acting, there is no underlying attempt to match the outer display with inner emotions (grandey, 2003). both strategies of the management of emotional displays have been examined as they relate to employee and workplace outcomes. surface acting, in particular, is related to a number of negative outcomes—emotional exhaustion at work, depression, and job burnout (e.g., abraham, 1998; brotheridge & grandey, 2002; liu et al., 2008; morris & feldman, 1996). deep acting, on the other hand, is related to more positive outcomes—higher customer service ratings, positivity at work, personal accomplishment, and job efficacy (e.g., brotheridge & grandey, 2002; grandey, 2003; liu et al., 2010). teaching requires emotional labor as teachers often must make a conscious effort to inhibit, generate, and manage their expressions of emotion to align with expectations and beliefs of what it means to “be a teacher” (yin et al., 2013). research examining the emotional labor of teaching reports mixed outcomes. for example, naring et al. (2006) found that surface acting in secondary school teachers was significantly associated with depersonalization at work, a component of job burnout. they also found that the absence of emotional labor led to feelings of personal accomplishment at work. tsang (2011) reviewed nearly 20 studies on the emotional labor of teaching at the elementary and secondary level and reported that while 4 showed that emotional labor strategies lead to negative outcomes, the rest argued that emotional labor does not always lead to negative outcomes and, in some cases, may be positive for teachers. the present study there are some important gaps in the literature, however. first, most of the research on emotional labor in teaching was conducted using teachers who work in the k-12 setting (for the five exceptions to this, see bellas, 1999; constanti & gibbs, 2004; mahoney et al., 2011; tunguz, 2016; and zhang & zhu, 2008). second, of those studies that were conducted at the college level, only three (mahoney, et al., 2011; tunguz, 2016; zhang & zhu, 2008) included quantitative “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 31 components, versus relying only on qualitative methods and/or theoretical reviews. as noted by kim and colleagues (2019), there is a need for research on teacher personality to employ “recognized framework” (p. 163) and “common descriptors” (p. 187), such as those found within empirical scales. the use of these established scales aids in the dissemination, application, and replication of personality research findings, and also can inform educational researchers who may wish to construct comprehensive theories of teacher personality from existing research (kim et al., 2019). finally, while the current literature broadly suggests that college professors do “act” in their professional roles, the specific characteristics that define these teaching personas is yet to be examined. gaining knowledge in these areas necessarily can contribute to a better understanding of the college professorial domain, specifically as it relates to the teaching role of professors. if, as griggs (2001) suggests, teaching should be considered a form of acting and most good teachers are, in fact, actors playing a role, then effective teaching at the college-level should be viewed as a role to master and a persona to develop. further, if the development of a teaching persona was to be encouraged as component of effective teaching practices, it would become imperative to understand the characteristics and traits that should comprise that persona. on the other hand, if this teacher-acting or role playing is harmful, as suggested by the literature on surface acting and emotional labor (grandey, 2003), effective teaching practices would seek to encourage professors to find ways to balance their self-presentation with authentic and genuine expressions of emotion (possibly through deep acting). at the very least, an awareness of just the existence of the teaching persona (and the potential connection of such a persona to job satisfaction and burnout) may help college professors to become more effective teachers as they learn to mindfully examine their self-presentation in various contexts. kim et al. (2019) remark that “the field of teacher personality [research] is expanding, and the potential implications of such research are exciting” (p. 189). in line with this observation, the present study is exploratory in nature with the novel goal of examining the existence of the teaching persona in college professors. additionally, this study seeks to explore how differences in self-reported “everyday” personas and teaching personas might relate to the overall work experience of college professors. following below are the specific research questions that guided the present study: o rq1: do college professors have teaching personas that are distinct from their self-reported “everyday” personas, and if so, what traits are most distinctive in teaching personas versus everyday personas? § rq1a: do distinctive traits in teaching personas vary based on the academic rank, academic discipline, and/or gender of the professor? o rq2: are feelings of job satisfaction and burnout related to differences in the teaching personas versus everyday personas of college professors? “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 32 method participants participants (n=71) were faculty at three teaching-focused post-secondary institutions in the continental united states (two private liberal arts, n=59; one public state, n=12). the majority of participants indicated their rank as either an assistant (n=18; 25.4%), associate (n=23; 32.4%), or full (n=14; 19.7%) professor, with the remaining 22.5% (n=16) reporting rank as either instructors or adjunct faculty. most (80.3%) participants reported holding a doctorate (e.g., ph.d. or ed.d.) as their highest academic credential. participants reported a mean of 10.5 years of employment at their current institution (sd = 11.8 years; range 1-35 years) and 13.9 years of teaching experience at the post-secondary level (sd = 10.5; range 1-40 years). a wide range of academic disciplines were represented in the sample (indicated by participants as the discipline in which they felt they “best fit”): humanities (n=28; 39.4%), social sciences (n=18; 25.4%), natural sciences (n=12; 16.9%), professional studies (n=7; 11.3%), and formal sciences (n=5; 7.0%) (n=1 participant reported “other”). there were more female (56.3%) than male (42.3%) participants (1.4% reported gender non-conforming), with nonhispanic white (81.7%) being the highest represented race/ethnicity. participants’ ages ranged from 28 to 75 years, with mean age of 47 years (sd = 12 years) procedure participants were recruited using both convenience and snowball sampling. contacts at each of the participating institutions distributed an invitation email from the principal investigator to all faculty at their institution (using institutional email addresses only). the invitation email contained a short description of the purpose/goals of the study (worded as a study to “gather data on the behaviors and personalities of instructors of higher education…both in the classroom and in your everyday life outside of the classroom”), a statement assuring participant anonymity, the irb approval number from the principal investigator’s institution, contact information for the principal investigator, and a qualtrics web link to access the study. the study questionnaire was made available to all faculty who first completed the informed consent procedure. the study was completed entirely online in a single session and most participants took between 10 and 20 minutes to complete it. data collection took place for a full academic semester (approximately 4 months). no compensation was provided to study participants. participants completed measures concerning basic personality dimensions, specific behavioral traits, and feelings of job satisfaction and burnout. the satisfaction and burnout measure was completed once, while the measures of personality dimensions and specific behavior traits were completed twice by each participant. the first time participants were asked to consider their “teacher-self” – that is, how much they felt each dimension/trait described them in the classroom/lab, in office hours, when conducting research with students, in advising/mentoring roles with students, and any other situations that involved them interacting with students. the second time participants were asked to consider their “everyday-self”—that is, “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 33 how much they felt each dimension/trait described them outside of the classroom and outside of working with students (including at home, with friends, in public settings, interacting with strangers, and/or when alone, among other things). measures basic personality dimensions participants’ basic personality dimensions were measured using the ten item personality inventory (tipi), a short-form measure of the five-factor model of personality (gosling et al., 2003). participants indicated their strength of agreement/disagreement on a 1 (strongly disagree) to 7 (strongly agree) likert scale. responses were scored to create five personality factor subscales. each subscale contained two items; the two items were summed to create each personality factor subscale with a minimum possible score of 2 and maximum possible score of 14 for each factor. the five personality factor subscales included extraversion (being social and outgoing; enjoyment of being with people; energized by presence of others), agreeableness (cooperative, kind, and polite; concern for the welfare of others; empathetic), conscientiousness (organized and mindful of details; goal-directed, thoughtful, and prepared), emotional stability (calm and even-tempered; less reactive to stress; able to handle adversity), and openness to new experiences (enjoyment of new things; being imaginative and curious; having an open mind). each factor was calculated twice, once for the “teacher-self” and once for the “everyday-self.” specific behavioral traits participants self-reported how often they engaged in a number of specific behavioral traits using the 28-item teacher behavior checklist (buskist et al., 2002; keeley et al., 2006). participants indicated how often they engaged in a variety of behaviors on a 1 (never) to 5 (always) likert scale. when necessary, items in the “everyday-self” version of the measure were modified from their original form to include language generalizable to situations outside of the classroom (e.g., “students” changed to “others”; “teacher” changed to “person”; “in class” removed). effort was made to ensure that the underlying trait/quality each item was designed to measure was not significantly impacted when modifications were made. examples of items that were modified (and the modification) include “i have realistic expectations of students/of others,” “i effectively manage class time/my own time,” and “i strive to be a better teacher/better person.” responses were scored to create two subscales: a caring and supportive factor and a professional competency and communication factor. the caring and supportive subscale contained 15 items, while the professional competency and communication subscale contained 13 items. items in each subscale were summed to create the behavioral trait factor score. the minimum possible score for the caring and supportive subscale was 15 and for the professional competency and “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 34 communication subscale was 13, while the maximum possible score for the caring and supportive subscale was 75 and for the professional competency and communication subscale was 65. each subscale was calculated twice, once for the “teacher-self” and once for the “everyday-self.” job satisfaction and burnout participants’ feelings of job satisfaction and burnout were measured using the 19item maslach burnout inventory educators survey (mbi – es; boles et al., 2000). participants indicated their strength of agreement/disagreement on a 1 (strongly disagree) to 7 (strongly agree) likert scale. responses were scored to create three subscales: emotional exhaustion, depersonalization, and personal accomplishment. the emotional exhaustion and personal accomplishment subscales each contained 7 items, while the depersonalization subscale contained 5 items. items in each subscale were summed to create the specific subscale score. the minimum possible score for the emotional exhaustion and personal accomplishment subscales was 7 and the maximum possible score was 49. the minimum possible score for the depersonalization subscale was 5 and the maximum possible score was 35. results research question 1: do college professors have teaching personas that are distinct from their self-reported “everyday” personas, and if so, what traits are most distinctive in teaching personas versus everyday personas? paired samples t-tests were conducted to compare participants’ self-reported teaching personas to their everyday personas as operationalized by the five basic personality factors (extraversion, agreeableness, conscientiousness, emotional stability, openness to new experiences) and the two behavioral trait factors (caring and supportive; professional competency and communication). means for each paired personality factor are displayed figure 1 and for each paired behavioral trait factor in figure 2. in terms of distinctiveness in personality factors, participants rated their teacherself as significantly more extraverted (t (70) = 10.58, p = < .01, cohen’s d = 1.25) and emotionally stable (t (70) = 4.55, p = < .01, cohen’s d = .54) than their everyday-self, while rating their teacher-self as significantly less conscientious than their everyday-self (t (70) = -4.01, p = < .01, cohen’s d = .47). there were no significant differences in ratings of agreeableness and openness to new experiences (p’s > .05). regarding behavioral trait factors, participants rated their teacher-self as significantly more caring and supportive than their everyday-self (t (70) = 8.55, p = < .01, cohen’s d = 1.02), and also indicated that their teacher-self was significantly more professionally competent and a better communicator than their everyday-self (t (70) = 8.71, p = < .01, cohen’s d = 1.03). figure 1 “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 35 mean score of personality factors for teacher-self vs. everyday-self note. significant differences denoted with the * symbol (all p’s < .01). the minimum possible score for each factor is 2 and the maximum possible score for each factor is 14. “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 36 figure 2 mean score of behavioral trait factors for teacher-self vs. everyday-self note. significant differences denoted with the * symbol (all p’s < .01). the minimum possible score for the caring and supportive factor is 15 and the maximum possible score is 75, while the minimum possible score for the competency and communication factor is 13 and the maximum possible score is 65. research question 1a: do distinctive traits in teaching personas vary based on the academic rank, academic discipline, and/or gender of the professor? paired samples t-tests were again conducted to compare participants’ self-reported teaching personas to their everyday personas as operationalized by the five basic personality factors and the two behavioral trait factors. however, these tests were split according to the academic rank, academic discipline, and gender of the professor (independently) such that deviations from the overall pattern of findings could be examined. only significant deviations from the overall pattern of findings are reported. “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 37 some deviations from the overall pattern of findings were found when comparing the teaching personas to the everyday personas of professors of different academic ranks. most notably, there were no significant differences (all p’s > .05) in the teaching versus everyday personas of adjunct faculty (across all personality and behavioral trait factors), suggesting that adjunct professors do not portray a distinctive persona in the classroom as teachers. further, full professors were similarly emotionally stable as teachers and in their everyday lives (p > .05) and assistant professors were similarly conscientious as teachers and in their everyday lives (p > .05). there were also deviations from the overall pattern of findings when comparing the teaching persona to the everyday persona of professors from different academic disciplines. professors from the social sciences were significantly less open to new experiences as teachers than in their everyday lives (t (17) = -3.21, p = .01, cohen’s d = .76), while natural science professors were significantly more open to new experiences as teachers than in their everyday lives (t (11) = 2.58, p = .03, cohen’s d = .75). additionally, natural science professors and professional studies professors were equally conscientious and emotionally stable as teachers and in their everyday lives (p’s > .05). no significant deviations in the overall pattern of findings (across all personality and behavioral trait factors) occurred as a function of the gender of the professor. research question 2: are feelings of job satisfaction and burnout related to differences in the teaching personas versus everyday personas of college professors? difference scores were created to quantify the distinctiveness in participants’ selfreported teaching personas as compared to their everyday personas for the five basic personality factors (extraversion, agreeableness, conscientiousness, emotional stability, openness to new experiences) and the two behavioral trait factors (caring and supportive; professional competency and communication). difference scores were created by subtracting the everyday-self factor from the teacher-self factor such that positive scores represent greater distinction of each factor as the teacher-self, negative scores represent greater distinction of each factor as the everyday-self, and scores close to zero represent a lack of distinction (or similarity) of each factor between the teacher-self and the everyday-self (see table 1 and table 2 for difference scores). a correlational analysis was then conducted to assess whether feelings of job satisfaction and burnout were related to how much participants’ self-reported teaching and everyday personas differed (i.e., the amount of distinctiveness). distinctiveness in conscientiousness was the only personality trait factor to be significantly related to feelings of burnout (r (71) = .31, p = .01; table 3). as represented by the positive correlation coefficient, increases in distinctiveness as one’s teacher-self compared to one’s everyday-self (i.e., higher difference score for conscientiousness) was related to increases in feelings of burnout in the form of depersonalization. in other words, when participants reported higher levels of conscientiousness in their teacher role than in their everyday life, they also “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 38 reported more feelings of burnout. likewise, when participants reported lower levels of conscientiousness in their teacher role than in their everyday life, they reported less feelings of burnout. even though the mean difference score for conscientiousness characterized the sample as being more conscientious as their everyday-self than as their teacher-self (i.e., the difference score was a negative value; see table 1), this analysis provides a more complete view of the manner in which burnout is associated with dissimilarity between the teacher-self and everyday-self in either direction. table 1 personality factor difference scores in teaching vs. everyday personas trait m (sd) min. max. extraversion 3.61 (2.87) -1.00 10.00 agreeableness 0.30 (2.11) -5.00 6.00 emotional stability 1.39 (2.58) -7.00 10.00 conscientiousness -1.44 (3.02) -7.00 6.00 openness -0.44 (3.14) -7.00 7.00 note. positive scores represent greater distinction as the teacher-self, negative scores represent greater distinction as the everyday-self, and scores close to zero represent a lack of distinction (or similarity) between the teacher-self and everyday-self. table 2 behavioral trait factor difference scores in teaching vs. everyday personas trait m (sd) min. max. caring and supportive competency and communication 4.32 (4.26) 3.87 (3.75) -5.00 -7.00 17.00 13.00 note. positive scores represent greater distinction as the teacher-self, negative scores represent greater distinction as the everyday-self, and scores close to zero represent a lack of distinction (or similarity) between the teacher-self and everyday-self. to further examine this finding, exploratory analyses compared the burnoutdepersonalization scores of participants who reported being more conscientious as their teacher-self versus those who reported being more conscientious as their “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 39 everyday-self. the sample was split into two groups using the difference scores of each participant. participants with positive difference scores were placed in the group that represented more conscientiousness in the teacher role (n=22), participants with negative difference scores were placed in the group that represented more conscientiousness in everyday life (n=35), and participants with a difference score of 0 (n=14) were excluded from this exploratory analysis. there was a statistically significant difference in feelings of burnout between those participants who reported being more conscientious as their teacher-self compared to those who reported being more conscientious as their everyday-self, t (55) = -2.71, p = .01, cohen’s d = .73. feelings of burnout were significantly higher when participants portrayed themselves as more conscientious in their roles as teachers (m = 14.91; sd = 4.98) than in their everyday lives (m = 11.49; sd = 4.39). distinctiveness in the professional competency and communication behavioral trait factor was significantly related to feelings of burnout and job satisfaction. higher levels of professional competency and communication as one’s teacher-self compared to one’s everyday-self (i.e., more distinctiveness) was related to greater feelings of burnout in the form of emotional exhaustion (r (71) = .25, p = .03) and decreased feelings of job satisfaction in the form of personal accomplishment (r (71) = -.27, p = .02). there was no significant relation observed between any other personality or behavioral trait factor and feelings of job satisfaction and burnout (all p’s > .05; see table 3) table 3 correlations among trait difference score variables and burnout sub-scales difference score variables emotional exhaustion depersonalization personal accomplishment extraversion .09 -.08 -.21 agreeableness .13 .22 -.21 emotional stability .18 .01 -.12 conscientiousness .17 .31** -.07 openness .01 -.16 -.08 caring and supportive competency/communication .20 .25* -.14 .18 -.14 -.27* note. *correlation is significant at .05 level; **correlation is significant at .01 level. discussion the purpose of this exploratory study was to consider the possibility that college professors may perform their duties as teachers under the guise of personas that are unlike that of their everyday lives, while also examining the potential relationship between persona distinctiveness and the overall work experience of professors. while some literature exists to support the notion that teachers and professors “wear masks” and therefore necessarily have distinctive personas, this is “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 40 the first study to use a quantitative approach to examine the specific characteristics that may define these teaching personas. this exploratory study adds to the small body of literature on the ways in which college professors manage/display their emotions, act in their roles as teachers, and experience job satisfaction and burnout. interpretation of results the results of this study suggest that college professors, with the possible exception of adjunct faculty, do have distinctive teaching personas. overall, professors see themselves as more extraverted and emotionally stable as teachers than in their everyday lives as well as more caring, supportive, competent, and better communicators. however, professors see themselves as equally agreeable and open to new experiences in their everyday lives as in their teacher-selves. further, professors report being more conscientious in their everyday life than in their teacher role. adjunct professors appear to have little distinction between their teacher-selves and their everyday-selves in these particular areas. one way to interpret this difference between full-time faculty members and adjunct faculty is to consider the nature of many adjunct positions. often, institutions hire adjunct faculty in a temporary role that may include limited time in the classroom (e.g., teaching a single class or working part-time) and limited additional faculty responsibilities (e.g., student advising or college service). many adjunct faculty members are employed as full-time professionals outside of academia and hired specifically to teach courses in their specialty (e.g., a business lawyer teaching a business law course or a social worker teaching an advanced social work course). thus, adjunct faculty members may not take on a distinct teaching persona in the classroom because they may not view themselves as teachers and therefore do not see the need to adapt their persona. another possibility is that adjunct faculty already have a work-related persona (e.g., lawyer-self or social worker-self). the results examining the association between the teaching persona and feelings of burnout were revealing, yet also inconclusive. in most cases (18 of 21 analyses), there was no significant association found between the different dimensions of burnout examined and the personality and behavioral trait factors. there were three instances, however, in which the use of a teaching persona was related to job burnout. feeling more capable and perceiving oneself as a better communicator in the teacher-role compared to in everyday life was related to increased feelings of emotional exhaustion and decreased feelings of personal accomplishment. distinctiveness between one’s teacher-self and everyday-self in conscientiousness was related to increased feelings of burnout in the form of depersonalization of students, a finding supported by the work of naring et al. (2006) where surface acting on the job was found to be related to depersonalization. the present study did not explore a causal explanation of this particular finding, or why the inauthentic display of conscientiousness in particular is associated with depersonalization of students; however, the nature of this personality trait may “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 41 help to shed light on this finding. indeed, conscientiousness is a personality trait that entails being hardworking, detail-orientated, organized, prepared, and wanting to exceed expectations in performance—all behaviors that require a great deal of self-regulation. individuals who try to “turn on” this trait as teachers, while not necessarily living the rest of their lives in this manner, may feel disconnected or disengaged from their students as a consequence of having to put in the extra effort to “appear” conscientiousness. it is perhaps the case that some professors, in their teacher-role, feel compelled to exude conscientious behaviors. for those who are not conscientious in their everyday lives, this forced display of inauthenticity may lead to feelings of burnout characterized by the depersonalization of students. future research should seek to test this, and other possible explanations, experimentally so that a fuller understanding of this relationship is possible. future research should also continue to explore the correlational findings surrounding feelings of personal accomplishment and emotional exhaustion for possible causal explanations. the results of this study showed that when professors reported less distinctiveness between their teacher-self and everyday-self in behaviors representing competency and communication, they also reported more feelings of personal accomplishment, but when they reported more distinctiveness, they were more emotionally exhausted. naring et al. (2006) found that the lack of acting on the job (i.e., being genuine; acting similar in and outside of work) leads to increased feelings of personal accomplishment. other research mirrors these findings, showing that deep acting (i.e., the attempt to both display and actually feel emotions that are expected in a workplace context) is related to job satisfaction in areas such as personal accomplishment and job efficacy (brotheridge & grandey, 2002; grandey, 2003; liu et al., 2010), while surface acting is related to emotional exhaustion (brotheridge & grandey, 2002; morris & feldman, 1996). indeed, these findings tentatively suggest that, at least in certain domains, displaying authenticity when in one’s teaching role may be the better approach. this interpretation naturally lends itself to the issue of what to make of a situation in which professors might perceive their everyday-selves to have traits that are not conducive to effective teaching, such as disorganization or insensitivity. would trying to be more organized and caring as a teacher be determinantal to job satisfaction and lead to feelings of burnout? while the correlational nature of this study does not definitively allow this question to be answered, the literature on emotional labor does offer some perspective (grandey, 2000). this literature suggests that if a professor were to genuinely attempt an expression of a particular trait or characteristic, that is to engage in deep acting, then little negative consequences should result. in other words, if professors were to not only display signs of care toward students as their teacher-selves, but also genuinely try to feel a sense of care for the students, then likely job satisfaction would not be at risk and burnout not a concern. if, on the other hand, professors who were to only outwardly express care for their students by simply ‘going through the motions’, that is to engage in surface acting, might experience negative consequences concerning job satisfaction and burnout. “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 42 limitations as this study was exploratory in nature and based on correlational data, the findings should be interpreted with caution until additional studies replicate and/or extend the results. though an interpretation of the notable findings has been presented here, care should be taken in extending them too far beyond the original context of this study. the main goal of this study was to consider the possibility that college professors have teaching personas that are distinct from their everyday personalities; indeed, this study provides some evidence to support that possibility. however, several methodological limitations restrict the generalizability of these findings until future studies are conducted. one limitation concerns how the teaching and everyday personas were operationalized and measured. these personas were operationalized as basic personality dimensions (classified using the five-factor model of personality) and specific behavioral traits (classified using a behavior trait checklist), thus any reported difference between the teacher-self and everyday-self was necessarily limited by the specific dimensions of these scales. further, while both self-report scales used in this study have demonstrated strength in the general dimensions of reliability and validity (gosling et al., 2003; keeley et al., 2006; landrum & stowell, 2013), the framework in which participants were asked to consider their personalities and behaviors may have unintentionally biased their responses. for example, self-report measures are limited by the social desirability bias (krumpal, 2013). participants in this study specifically may have been unwilling to self-report teacher traits and/or behaviors perceived as “negative,” even if they were traits or behaviors that they were aware of engaging in. similarly, participants may have overexaggerated teacher traits they perceive as “positive” and overreported how often they engage in teacher behaviors perceived as “positive.” in addition, wording in several behavior trait checklist items was modified from its original, validated form so that participants could complete it while considering their teacher-self and everyday-self separately. these modified items have not been validated, thus are necessarily a limitation to consider. the actual act of having participants complete each scale twice as they considered two “versions” of themselves is a limitation. the results of this study hinge on the assumption that participants were able to both honestly and accurately examine their own personality traits and behaviors in two different contexts. however, this approach has not been used elsewhere in the literature, so future research is needed to determine whether this approach is reliable and valid. in terms of reliability, participants could be asked to complete the measures on several different occasions throughout a semester or academic year. regarding validity, the self-reported personality and behavioral traits could be compared to outside reports of participants’ personality and behavioral traits by, for example, colleagues or students (teacher-self) and a spouse/partner or close friends (everyday-self). an alternative approach to the measurement of persona distinctiveness for future research could be to consider the use of the “teacher emotional labor strategy scale” (telss; yin, 2012). this scale measures the amount of emotional labor “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 43 teachers feel they experience by examining the components of surface acting, deep acting, and the expression of naturally felt emotions. the measurement of surface acting, specifically, would allow for the quantification of the amount of “acting” one performs as a teacher. compared to deep acting, higher levels of surface acting would be similar regarding having a greater distinction between the teacher-self and everyday-self. while use of the telss has the advantage of being a validated scale, the disadvantage of such an approach would be the loss of the ability to identify which specific traits comprise one’s teacher-self versus their everyday-self as the current study did. future research should also expand the size and diversity of the study sample. regarding the size, this study reports the results of analyses conducted with under 100 participants (n = 71). the specific focus on examining the experiences of college professors from teaching-focused institutions in this study was intentional; however, recruitment of such participants proved to be challenging. future researchers might explore alternative avenues for recruitment (e.g., soliciting in teaching listservs, social media groups, and/or at conferences) or expanding recruitment to include college professors from research-focused or more equally balanced (teaching-research) institutions. regarding diversity, participants for this study were recruited from one of three teaching-focused post-secondary institutions in a similar geographic region of the united states. thus, diversity was lacking in several demographic and professional/academic domains. conclusion as the results of this study seem to suggest, many college professors do appear to live out steve azar’s (2001) sentiments regarding not “being [teacher] me ‘til monday” in a number of different ways when they leave campus on a friday. for some professors, it may be that “i teach, therefore i am”; these individuals wear the mask of the teacher, play the part as the teacher, and must go home in order to be their authentic self. for other professors, however, it may be that “i teach, therefore i am…still me”; these individuals are less likely to “act” in the classroom and more likely to express their authentic self to their students. understanding that each of these mindsets may lead to different outcomes concerning job satisfaction and burnout is an important catalyst for future research seeking to enhance the professional lives of college professors. these preliminary findings are intended to encourage future researchers to continue to explore the teaching personas of college professors, including the implications of such personas in various facets of the higher education domain. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references abraham, r. (1998). emotional dissonance in organizations: antecedents, “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 44 consequences, and moderators. genetic, social, and general psychology monographs, 124(2), 229–246. azar, s. (2001). i don't have to be me ('til monday). on waitin' on joe [cd]. mercury nashville records. bellas, m. l. (1999). emotional labor in academia: the case of professors. american academy of political and social science, 516, 96–110. boles, j. s., dean, d. h., ricks, j. m., short, j. c., & wang, g. (2000). the dimensionality of the maslach burnout inventory across small business owners and educators. journal of vocational behavior, 56, 12–34. brotheridge, c. m. & grandey, a. a. (2002). emotional labor and burnout: comparing two perspectives of “people work.” journal of vocational behavior, 60, 17–39. buskist, w., sikorski, j., buckley, t., & saville, b. k. (2002). elements of master teaching. in s. f. davis & w. buskist (eds.), the teaching of psychology: essays in honor of wilbert j. mckeachie and charles l. brewer (pp. 27–39). lawrence erlbaum associates, inc. craig, r. p. (1994). the face we put on: carl jung for teachers. clearing house, 67(4), 189–191. constanti, p. & gibbs, p. (2004). higher education teachers and emotional labour. international journal of educational management, 18(4), 243–249. gosling, s. d., rentfrow, p. j., & swann, w. b., jr. (2003). a very brief measure of the big five personality domains. journal of research in personality, 37, 504–528. grandey, a. a. (2000). emotional regulation in the workplace: a new way to conceptualize emotional labor. journal of occupational health psychology, 5(1), 95–110. grandey, a. a. (2003). when “the show must go on”: surface and deep acting as determinants of emotional exhaustion and peer-rated service delivery. academy of management journal, 46, 86–96. griggs, t. (2001) teaching as acting: considering acting as epistemology and its use in teaching and teacher preparation. teacher education quarterly, 28(2), 23–36. hall, c. s. & nordby, v. j. (1973). a primer of jungian psychology. oxford, taplinger. hochschild, a. r. (1983). the managed heart: commercialization of human feeling. “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 45 university of california press. jung, c. g. (1989). memories, dreams, reflections. trans. by r.c. winston. vintage books. keeley, j., smith, d., & buskist, w. (2006). the teacher behaviors checklist: factor analysis of its utility for evaluating teaching. teaching of psychology, 33(2), 84–91. kim, l. e., jorg, v., & klassen, r. m. (2019). a meta-analysis of the effects of teacher personality on teacher effectiveness and burnout. educational psychology review, 31, 163–195. krumpal, i. (2013). determinants of social desirability bias in sensitive surveys: a literature review. quality and quantity, 47, 2025–2047. landrum, r. e. & stowell, j. r. (2013). the reliability of student ratings of master teacher behaviors. teaching of psychology, 40, 300–303. liu, y., prati, l. m., perrewé, p. l., & brymer, r. a. (2010). individual differences in emotional regulation, emotional experiences at work, and work-related outcomes: a two-study investigation. journal of applied social psychology, 40, 1515–1538. liu, y., prati, l. m., perrewé, p. l., & ferris, g. r. (2008). the relationship between emotional resources and emotional labor: an exploratory study. journal of applied social psychology, 38, 2410–2439. mahoney, k. t., buboltz, w. c., jr., buckner v, j. e., & doverspike, d. (2011). emotional labor in american professors. journal of occupational health psychology, 16(4), 406-423. mckinney, k. (1988). faces: five components of quality teaching. teaching sociology, 16(3), 298–301. morris, j. a. & feldman, d. c. (1996). the dimensions, antecedents, and consequences of emotional labor. academy of management review, 21, 986– 1010. naring, g., briet, m., & brouwers, a. (2006). beyond demand–control: emotional labor and symptoms of burnout in teachers. work and stress, 20(4), 303 315. tunguz, s. (2016). in the eye of the beholder: emotional labor in academia varies with tenure and gender. studies in higher education, 41(1), 3–20. tsang, k. k. (2011). emotional labor of teaching. educational research, 2(8), 1312–1316. “i teach, therefore i am?” journal of effective teaching in higher education, vol. 3, no. 2 46 yin, h. b. (2012). adaptation and validation of the teacher emotional labour strategy scale in china. educational psychology, 32, 451–465. yin, h., lee, j. c. k., zhang, z., & jin, y. (2013). exploring the relationship among teachers’ emotional intelligence, emotional labor strategies, and teaching satisfaction. teaching and teacher education, 35(2013), 137–145. zhang, q. & zhu, w. (2008) exploring emotion in teaching: emotional labor, burnout, and satisfaction in chinese higher education. communication education, 57(1), 105–122. microsoft word vengadasalam.docx journal of effective teaching in higher education, vol 3, no.2 transformative pedagogy and student voice: using s.e.a. principles in teaching academic writing sarbani sen vengadasalam, rutgers university, sarbani@english.rutgers.edu abstract. this paper describes the principles of transformative pedagogy that lead to the development of distinct student voices in academic writing classes. whether the course is taught at the undergraduate level through research, expository, and argumentative writing assignments or at the graduate level through literature review essays, research articles, and dissertation writing tasks, students need to be able to develop their voices and make their contributions to knowledge. correspondingly, professional writing teachers need to teach students how to write voiced project documents such that they have the student’s unique signature even when situated within a paradigmatic boundary. the article expands on how facilitators of academic writing courses can incorporate s.e.a. principles of scaffolding, empowerment, and awareness as triple enablers into their teaching methodologies in order to develop student voices and usher in transformation successfully. as one of the few articles to examine how graduate and undergraduate academic writing instruction, including w.a.c. (writing across the curriculum) and w.i.d. (writing in the discipline) teaching, can be recast to develop student voices, the paper can be helpful to readers looking for resources and recommendations to incorporate transformative pedagogy into their teaching. keywords: academic writing, transformative pedagogy, scaffolding, empowerment, awareness writing is a process of being and becoming. the way one writes reflects the way one thinks, so any change in the writing process causes a change in the thinking process as well. as academic writing instructors and teachers who facilitate w.a.c. (writing across the curriculum) and w.i.d. (writing in the discipline) classes, we train students on how to write and, therefore, influence how they interpret what they read as well as how they organize their ideas on paper. teaching writing can be a way to develop student voices such that they can contribute to academic dialogs, knowledge creation, and even social change. if developing voices in students is the goal of writing teachers, what is the pedagogy that teachers of undergraduate or graduate academic writing classes can adopt to better achieve course goals? while there have been numerous publications on the topic over the last four decades as the theme is neither new nor ever irrelevant, this may be one of the few articles to examine the role of both undergraduate and graduate writing instruction in the development of student voices as well as investigate s.e.a. or the principles of scaffolding, empowerment, awareness as triple enablers in the teaching effort. what is student voice? voice, as used in this article, is how each student contributes to scholarship. transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 13 developing a voice in writing is not easy; it involves struggle, risk-taking, and reconstruction. as the process of acquiring a voice through education in general and writing, in particular, is a way of “becoming,” voice acquisition is an important marker of academic success. however “becoming” does not always happen—the student’s voice can get shut down because of various factors. as “becoming” does not occur in isolation, students need one another and a mentor in their instructor in order to develop their voices. since many students today are not producing “voiced” writing, reframing of composition instruction with a new pedagogical underpinning can not only preempt the blocking of student voices but also metamorphose the classroom into a creative space where each student asserts "i am” in his or her writing. can a voice echo or should it be unique? since voice is the active expression of an individual and distinguishes one writer from another, it should exist by itself and in contrast to other voices. also, voice signals participation and is an active part of the social production of meaning. if voice, as simon (1987) perceives it, is a "discursive means whereby students make themselves present and define themselves as active authors of their world" (p. 377), voice, as described by mayaba et al. (2018), is an instrument or “force for social change” (p. 1). in the context of higher education, voice, as a marker of presence or a tool for social innovation, emerges from what harrell-levy & kerpelman (2010) describe as the students’ critical analysis of a subject using their individual and collective experiences as and in a class community (p. 80). byker et al. (2017), on their part, define student voice as a term that “honors the participatory roles that students have when they enter learning spaces like classrooms” and point out that “student voice is the recognition of students' choice, creativity, and freedom” (p. 119). what is important to note here is that there is an emergent consensus that voice is a major indicator of successful participation and scholastic growth in academic writing classes that include w.a.c. and w.i.d. courses. the absence of voice, by implication, is the negation of a student's identity. when a student is taught to parrot or merely repeat another’s thoughts, even if it is in the name of research, it is a sign of a creativity vacuum. when it happens in the classroom, it may even indicate that oppressive conditions from the outer world have infiltrated within. if students feel marginalized, silenced, or afraid of expressing their individual interpretations in their academic work, it signifies their loss of voice either under the weight of academic norms or from societal inequities. in this context, a writing classroom, by focusing on voice development, can make a difference. when a teacher succeeds in promoting diverse voices in the classroom, it implies that every student has gained the power to be heard. since to drown voice is to deny students their basic humanity, teaching students—including those from marginalized communities—to acquire voices and usher in change through their writing restores equity into the classroom and academic writing that includes w.a.c. and w.i.d. writing. students develop voices and express themselves when the writing teacher succeeds in creating a safe, dynamic, learning space in the classroom despite what may be happening outside. developing a voice thus becomes, as lensmire (1998) puts it, “a way to distinguish yourself from others and a way of embedding yourself, your transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 14 writing, and your interpretations into public or semi-public spheres such as the classroom” (p. 273). since the classroom is a microcosm of the world outside and our writing reflects the way we think, fostering voice, whether the student is interpreting readings or writing professionally, is not only integral to the development of the students’ persona but can even lead to social transformation. given this background, it is obvious that the role of the teacher in the development of voice is both central and critical. teachers need to operate on the premise that teaching and learning is a two-way proposition and accept the role of mediators in identity formation. they need to recognize that student-writers faces difficulties in articulating complex thoughts in academic prose even if they are native speakers of the english language. while lensmire (1998) maintains that the writing teacher needs to help students tide over difficulties in finding “outer words to express inner meanings” (p. 273), harrell-levy & kerpelman (2010) contend that the teacher’s job is to develop a group of identity explorers and establish: a community that fosters the type of openness and daily feedback that facilitates the identity exploration of each student. when students feel respected and safe enough to air their honest thoughts, an atmosphere of trust and community develops. in such an atmosphere, a process of sharing and reflecting leads to a different, deeper type of learning about the material, how to think about the material, and how to think about oneself. (p.83) even if the teacher is an important aspect of the students' voice acquisition, it must be noted that the teacher’s function is to be supportive, not normative. voice development cannot happen when, as turner (2006) points out, “a teacher tells the students what to do, when to do, and how to do everything” (p. 28). student voices are never static or inert, but evolving and diverse. a cookie-cutter approach cannot lead to voice acquisition in academic and professional writing contexts both in the undergraduate and the graduate classroom. students acquire their voices in the writing class at a point in life when they do not imagine themselves as capable of doing so. however, if the opportunity is missed, students may end up not knowing how to express their voices all through their lives. a text’s meaning and value, as also that of the research, are dependent on how it is read and interpreted, by whom and from where, and through which ideology and framework. if the acquisition of a research persona or voice, as lillis et al. (2015) point out, “is to acquire the capacity for semiotic mobility” (p. 24), it needs a student-centered classroom environment to develop and manifest itself. if students are to mature as writers, adopt positive attitudes towards written work, and demonstrate growth in writing performances, they need an academic writing classroom where risk-taking is expected, trust is established, choice is available, authority is shared, and writing is viewed as a meaning-making event. the process by which writing teachers guide students through existing paradigms in the complex world of academia involves some handholding. since new writers and scholars not only learn the intricacies of writing discourses but also the way to grow their academic voices and academic persona from their instructor, they are like “apprentices” who have come to learn the necessary writing skills from the “guru” or the writing teacher. transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 15 what kind of teaching method adopts such a mentoring approach towards writing instruction? what type of pedagogy successfully rewards and recognizes students’ attempts at voice creation? what are the markers of such a pedagogic system? how does the practice affect the teacher and the student individually? how, if at all, does it transfigure the framework of undergraduate and graduate academic writing classes including w.a.c. and w.i.d. courses? to be able to answer these questions systematically, the paper is divided into the following sections. section two discusses transformative learning and how it nurtures student voices. section three elaborates on s.e.a or the scaffolding, empowerment, and awareness principles of transformative pedagogy for their ability to create an academic environment where student voices can be fostered. section four wraps up the discussion by pointing out that a transformative pedagogic approach not only grows students’ voices but also empowers writing instructors and alters their self-perceptions. what is transformative pedagogy? transformative theory, in the academic context, posits that students can achieve their potential through transformative experiences via participation in an academic community. in this sense, transformative pedagogy can be especially relevant to the teaching of academic and professional writing discourses. in fact, transformative pedagogy may well be a methodological breakthrough in writing pedagogy since it enables students to see the interconnectedness of texts or argue for a new line to problem-solving or approach to research through their writing. the theory of transformative learning, first propounded by mezirow (1978), proclaimed that learning can revise the way we think, feel, and act. believing in transformation as the primary goal of education, this theory impacted these fields in particular: social activism, higher education, adult literacy, and human resources development. mezirow (1991) eventually turned away from social activist implications of transformative learning in favor of a focus on individual growth and development. transformative learning, as he saw it, triggers a "critical assessment of assumptions,” "exploration of options for new roles," and "building of competence and self-confidence in new roles and relationships" (p. 109–110). transformative learning, hence, was described by bass (1998) as seeking ways to include other ways of knowing, of expanding our concepts of reasoning, and, finally, of transforming writers as well as institutions (p. 254). the symbiotic relationship of students with one another and with their surroundings, including the teacher, is central in this pedagogy. transformative pedagogy not only involves critical questioning that raises students’ awareness of their assumptions but also marries contemplation of the subject matter with self-scrutiny and scrutiny of the surrounding environment. in other words, transformative pedagogy reframes a student’s relationship with content, fellow students, teacher, disciplinary contexts, and professional writing paradigms. as the transformative approach aims to promote students’ awareness of other perspectives, donnell (2007) correctly indicates that transformative teachers urge students to "think creatively and critically” and foster “collaborative learning practices” (p. 225). meyers (2008) takes the thought forward when he points out that transformative instruction transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 16 discourages teaching styles that increase the power differential between teachers and their students (p. 220). transformative practice, in fact, disrupts the imperial setup of the traditional classroom where the teacher is at the center and the source of all information flow. instead, transformative writing teachers situate themselves at the hub of a more democratic, crisscross flow of ideas making it clear to all stakeholders that academic writing is dialogic and can even be oppositional within a local, social, or disciplinary context. if academic writing teachers, including w.a.c. and w.i.d. instructors, have to succeed in their mission of expanding students’ awareness of professional contexts and the big picture so students can shift paradigms or create new ones, they will need to embrace transformative pedagogy and its theoretical perspectives to guide and support their efforts. the following section briefly discusses how academic writing theorizing is reframed in transformative pedagogy before expanding on how the transformative approach can be implemented in both undergraduate and graduate classes in the effort to nurture and grow student voices. academic writing & the transformative approach academic writing, as wideman (2005) describes it, is a way to attain and demonstrate disciplinary knowledge as well as a means to acquire a presence in the academic world (p. i). academic writing is taught to undergraduates under various names: expository writing, professional writing, argumentative writing, and so on. graduate academic writing, too, is taught under different appellations and typically works around literature review papers, academic articles, or dissertation writing. all academic writing courses, including w.a.c and. w.i.d. courses, end in student writing outputs such as is common in the field of higher education. born out of a dialectical interplay between writer, reality, audience, and conventions, the writing output presents research and may or may not include an actionable component. as academic writing classes almost always teach research reporting and citation conventions, davis (2009) aptly declared that academic writing in the agonistic tradition a. is based on the discourse of the academic elite b. includes critique c. is derived from the works of others d. reflects the student (or the self) as constructed by the ‘insider’ discourse of the academy. (p. 11) agreeing and extending the notion further, ingle et al. (2015) termed academic writing as an “examination of the multiple identities that one has to negotiate in the process of producing a piece of academic prose, an awareness of how these identities interact with wider structures and relations existing in academia and beyond, and a consciousness of the processes and practices surrounding the production, transmission, and use of academic texts” (p. 154). what this implies is that if academic writing, including professional writing classes, have to develop student voices in both undergraduate and graduate students, it has to begin with the setting up of befitting course outcomes. put differently, course setup, classroom practice, and assignment construction have to focus on “voice” development if the course is to achieve that objective directly. transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 17 how does transformative pedagogy impact the setting up of an academic writing class? in an undergraduate writing class, a teacher is no longer content with focusing only on training students to write a five-paragraph or a five-page essay with opening and closing arguments, thesis statements, and topic sentences with research support at appropriate places. instead, they choose to have students connect complex seminal articles from various disciplines in unexpected ways in expository writing essays. such tasks lead to the acquisition of voice as it entails that students not only probe into layered arguments to arrive at the learning from one essay but that they also envisage how that idea connects with the learning and ideas of another layered text. similarly requiring students to do feasibility studies or draw up recommendation plans for change in their workplace, schools, or townships using fieldwork, published best practices, and case studies require student writers to take unique, innovative departures within a theoretical matrix leading to new patterns, pathways, and paradigms. such w.a.c. and w.i.d. assignments can go a long way in catalyzing change both in students and in their surroundings. with a transformative pedagogy, the undergraduate academic writing teacher offers students opportunities and strategies about how to make their papers voiced, not just well cited. such teachers create writing prompts that move students from conventional summarizing to connective thinking, problem-solving, and imaginative ideation. feedback in the transformative undergraduate writing classroom accordingly rewards and recognizes the students’ ability to grapple and wrestle with complex ideas in order to create new meanings or real world linkages. graduate writing classes face both similar and different challenges. as graduate students write out their literature reviews, academic articles, and master or doctoral dissertations, they often feel, as stevens (2015) highlights, a "loss of identity during the course of their studies and hold academic writing responsible for it. this loss is expressed as a stripping away of creativity and being made to write in a way that felt abstract and not representative of who they are or want to be. academic writing feels like something i've produced that is separate to me and is passed on to the audience" (p. 268). graduate academic writing teachers, using a transformative pedagogy, can fix the sense of loss by urging students not to practice research reporting, uncritical acceptance, and assimilation but highlight their understanding of what is satisfactory, generative, and meaningful in their research. in the transformative schemata, graduate academic writing teachers help students to create knowledge rather than train them just to follow disciplinary norms and academic conventions. they coach graduate academic writers to be nonderivative even when operating within discourses of learning and writing. in transformative pedagogy, graduate academic writing is neither an elitist exercise nor a ticket to academy insidership. hence the task, the instruction, and the feedback focus on how graduate students display their acquired mastery over current research while finding gaps within it wherein to situate their unique study. like the undergraduate teacher, the academic writing instructor in a graduate writing class nurtures and rewards the student writers’ attempts to find their academic niches. in this sense, transformative undergraduate and graduate writing teachers both work at developing voices in students’ writing even if the scope of graduate writing tasks and papers may be larger. transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 18 transformative writing adopts a new take not only on the relationship between students and the teacher but also on the relationship between the student and texts, between the student and peers, as well as the student and the audience. as the writing teachers embrace these changes, they may do well to be grounded in s.e.a. or the scaffolding, empowerment, and awareness principles of transformative pedagogy. the three subsections to follow explore how the three principles of transformative pedagogy can power teachers’ efforts and recast academic writing instruction for more positive outcomes. the empowerment of transformation writing in an academic setting should empower and enrich rather than diminish a writer's sense of self. empowerment becomes possible when students do not turn in lifeless, guarded responses, but develop individual takes on their readings and research. how can the teacher create or incorporate the empowerment principle in the undergraduate classroom? as indicated in the previous section, one way the academic writing teacher can create transformative pedagogic experiences for students is by assigning creative writing tasks and composing inspired writing prompts. what is signified by a creative or inspired prompt? tasking students to find relationships between disparate readings through the power of their ideas, such as in the sample prompt1 used in an expository writing class at rutgers university does, can work. when students feel empowered enough to interplay nonlinked disciplines and unrelated texts such as a war memoir, a psychology lesson, and a religious tenet through layered connective thinking in response to the transformative writing prompt, they become participants in a transformative pedagogic experience. since interdisciplinarity has valid and strong connections with the experience of transformation, asking students to take and defend innovative positions on interdisciplinary texts can be both handy and effective. again, when student-writers are encouraged to interpret or illuminate one single text, say rebecca skloot’s the immortal life of henrietta lacks (2017), from various disciplinary standpoints, be it scientific, business, racial, historical, ethical, social, literary, or a mix of them—as we see in skloot’s prompts2—students develop voices. put differently, distinctive academic experiences result, as virtue et al. (2018) testify, when academic writing teachers empower their undergraduate writing students to make connections and pick perspectives depending on their interests and backgrounds as with skloot’s text. correa (2010) credits such empowerment outcomes to be the resultant of the teacher’s efforts at “helping students recognize not only the various types of voices that can be brought into a text but also the sources of those voices, the cultural or disciplinary ways in which these voices can be brought in, and how the voices can be creatively recombined with other voices to achieve certain purposes (e.g. to argue or explain a point)”(p. 81). similarly, specialized business or technical writing classes taught in a w.a.c. or w.i.d. program can grow student voices when they encourage students to adopt and justify their unique approach to solving a problem in their project documents that may range from white papers to presentations, business proposals to technical reports. when students take innovative lines of argument within the professional or technological paradigmatic framework in their project writing, voice and originality are achieved and manifested. transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 19 when empowerment happens, students write about what they like, like what they write, and share it with others. to levin (2000), the manifested connective thinking and interdisciplinary takes expressed by students in their writing is a consequence of their empowerment, since students now “recognize unseen capacities and knowledge in themselves and others, writ[e] in new voices, have a stronger sense of voice, question conventional roles, ideas, and stereotypes, bring out feelings and thoughts they didn't know they had, and experience new ways of learning and being” (p. 45). through empowerment, the academic writing teacher emboldens the undergraduate pupil to exhibit connective, interdisciplinary, and changeoriented thinking or voiced writing. at the graduate level, empowerment, and the development of an authoritative voice, comes through the slant and the originality of the research. the teacher’s task here is to demonstrate how authority emanates from being able to speak from a unique vantage point within a discourse community, be it humanities, social sciences, or the scientific disciplines. even if writing for publications and writing of dissertations require learning of disciplinary and citation conventions, the academic writing teacher should be careful not to teach these to the exclusion of all else. empowering graduate academic writing instruction urges and audits how researchers are developing influential voices by what correa (2010) describes as “the opportunity to discuss the value of both the discourses they already possess and of the discourses they are being asked to produce” (p. 80). graduate students develop a sense of self and voice from the depth and breadth of their research or literature reviews, the innovativeness and exclusiveness of their approaches, and the positions they take as insiders or outsiders of the communities to which they wish to gain affiliation. just as with the undergraduate instructor, the graduate academic teacher focuses all instruction, feedback, and grading around voice manifestation. it must be pointed out that a writer's presence in any form of academic writing, whether undergraduate or graduate, is not connected to the use of the personal pronoun "i". voice, in the empowerment context of transformational academic writing, is not a resultant of the use of the personal pronoun but is, as comfort (1995) points out, "a convergence of elements: choice of subject matter for phenomena, methods of working through arguments, types of evidence,…and underpinning assumptions, even what is left unsaid” (p. 37). such a discourse is empowered and voiced and, even when “conditioned by an array of social, historical, and cultural influences within the hierarchical power relationships that constitute academic institutions” (ibid.), questions, critiques, and interrogates it successfully. the expanded awareness of transformation expanding awareness in students is another strategic and fundamental principle of transformative pedagogy. student voices acquire authenticity and expansiveness through critical questioning. critical questioning is the kind of questioning that marries contemplation of the subject matter with self-scrutiny and cognizance of various perspectives. as per lillis (2015), a transformative pedagogy foregrounds transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 20 such questions of the text: how have particular conventions become legitimized—and what might alternatives be? to what extent, do they serve knowledge making—and are other ways of making knowledge, and other kinds of knowledge/knowing possible? whose epistemological and ideological interests and desires do these reflect and enable—and whose interests and desires may be being excluded? (p. 9) when students ask questions when they write about texts or about their projects, they become more aware of how and where their writing is to be situated. what is transformed through the “transformative approach” is the way of seeing and being. for instance, awareness is expanded when students connect or relate various readings or identify problem areas and propose changes. from the new critical awareness acquired, the academic writing student—whether undergraduate or graduate, whether in a w.a.c. or a w.i.d. class—learns how to synthesize and interplay varied contexts and discourses. like with empowerment, expanded awareness leads to the opening up of dialogic and community spaces in undergraduate writing classrooms. as with imaginative prompts, critical questioning push students toward transformative goals or what mitchell (2013) describes as “intertextuality, alternative understandings, and discovery of new frontiers of knowledge” (p. 16). harrington (2015) endorses the idea when she points out that transformative writers need “to step back, imagine, and actually begin to do things differently—more creatively, more thoughtfully, and more radically” (p. 13). with expanded awareness, such possibilities can be realized in academic writing classes including w.a.c. and w.i.d. courses. as awareness is not a finished state, but something that is achieved through critical questioning, the writing teacher would want to set up communal spaces and co-operative relationships between all classroom stakeholders that include the student, peers, texts, instructor, and audience. when undergraduate students are self-aware, they move from an attitude of "submission" to authors whose works are being read and researched into to connecting and holding a conversation with them and with the audience who may be the students’ project patrons. expanded awareness in graduate academic writing class reveals itself in multivocality since graduate academic writers are now encouraged to question existing ideas and adopt distinctive research angles. with the help of the teacher, they learn how to present their research innovatively. in transformative graduate writing, expanded awareness generates interdisciplinary conversations and new relationships between the writer’s voice, the research hypothesis, and the disciplinary field. since today’s academic writing students are tomorrow’s professionals, dialoguing about research discoveries is necessary for progression and development. since conflict exists between self-expression and the conformity that disciplinary practices and conventional paradigms impose, writers need to be aware of the complex of activities, experiences, and purposes that are clubbed under the category of academic writing norms even if they intend to transcend them. as the transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 21 task is challenging, students need scaffolds to negotiate through expected norms and not be limited by them. hence, in addition to empowering students through transformative prompts and providing them with an expanded awareness of disciplinary practices, teachers may want to adopt the principle of scaffolding to show how students, as emergent researchers and novice writers, can resolve any conflict that exists and acquire voices. the next section discusses how incorporating the scaffolding principle in teaching academic writing, including w.a.c. and w.i.d. courses, can grow student voices and provide consistent results. the aim of the section is to explore how scaffolding works and can be used effectively to nurture students' voices. the scaffolding of transformation academic writing is challenging because it is located within a global complex of signs and an international knowledge economy. while empowerment has much to do with assignment design and expanded awareness impacts course objectives, scaffolding affects course delivery. scaffolding, as bliss and askew (1996) describe it, involves keeping the task constant but manageable (p. 39). scaffolding entails breaking up the writing work into chunks and building one part onto another so the parts contribute to the whole seamlessly. as they impact teaching methodology as well as course delivery, scaffolds have to be well defined and comprehensible. if it has to be successful, scaffolding requires organized procedures to be welded onto and developed around the writing process and its three key phases of outlining, drafting, and revising. while scaffolding is required for all writing instruction, it is even more fundamental in transformative pedagogy because the task of nurturing an original voice is an onerous one. scaffolding, in the context of voice, refers to a set of techniques that the instructor uses to move students towards understanding how and finally manifesting their voices independently. while a literature search shows an agreement with the viewpoint of read (2010) that instructors' inputs during the scaffolding process are all important when it comes to students’ voice development, some critics like bodrova and leong (1998) caution that the goal is to progressively decrease the level of assistance they initially provide so students can develop their voices on their own and become self-sufficient (p. 5). what this means is that even though the teachers provide the pupil with a range of resources through the outlining, drafting, and revising stages of the writing process, they should leave the choice to students to use the resources as per their needs. in the next three paragraphs, i share strategies from the scaffolding toolkit i have evolved and used successfully since 2008 while teaching transformatively at both levels in my academic, business, and technical writing classes. a key scaffolding strategy that works well and can be used across the board in undergraduate academic writing classes is modeling. in the outlining stage, graphic organizers can be used to model and help students with ideation. mini-lessons could be devised using newspaper editorials to model aristotle’s persuasive triad of ethos, pathos, and logos. as students go through various levels of outlining—so an idea grows into a leveled detailed plan—students can examine models provided by the teacher with the intent to learn how to attach microdetails to their macro outlines. transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 22 as students begin drafting, workshops for analyzing successful voiced writing models can be held where current students closely read, dissect, and analyze writing samples of former students (after obtaining their permission of course) individually and in small and big groups. this can go a long way in promoting student understanding of how transformation works and voice is manifested. students not only comprehend how to isolate writing stratagems that sample writers used to manifest their voices in these writing workshops, but they also figure out how to mix and match the strategies while writing their own essays. in the revision stage when the essays are being completed, it becomes necessary to hold stepped revision cycles and guided peer review sessions. so that the peer review workshops are useful, questions, criteria, and checklists that focus on voice manifestation are required. students can use them both for peer and self-review of their essays and projects while revising, proofing, and finalizing them. scaffolds and models can be used similarly in a graduate writing class to help students incorporate transformative processes into their writing. as in undergraduate classes, each stage has a distinct focus, with the teacher offering feedback particularly at the beginning and end of every stage. in the outlining stage, the teacher could proffer models and mind maps to help in the creation of a multi-leveled plan. components of an introduction such as an opening statement, need-establishment, literature review, hypothesis construction, and essay overview can be innovatively demonstrated with reading and writing exercises that incorporate an examination of model papers published in reputed journals. what the scaffolding and modeling does here is to offer graduate writing students practice in expanding their range and comfort in reading and writing for academia. this is important because transformative pedagogy at the graduate level prompts students to move beyond the normative “academic socialization approach,” as jacobs (2015) terms it, to the kind of academic writing where the literacy practices of disciplines are critiqued and contested (p. 152). covert tensions may exist between entrenched, legitimized, scientific writing conventions, such as the classical i.m.r.a.d. (or introduction, methods, results, analysis & discussion) template for scientific writing and their disciplinary variations, or even within the thematic arrangement that the arts and the humanities require students to adopt. as per stevens (2015), departmental academic writing cultures and university guidelines, where these exist, also inevitably, come into play (p. 276). graduate students could evolve a scholastic guide for themselves, with the teacher’s help wherever necessary, through searching, isolating, and evolving best practices from a close and thorough inspection of model articles in periodicals picked by the students themselves. techniques for discussing the literature or the methodology whether they are related to a laboratory setup, market research, or sample distributions could be scrutinized. guidelines have to be extracted and practice has to be offered in various ways of presenting hypotheses and interpreting artifacts, outlining procedures and analyzing results, highlighting inferences and forecasting impact through scholarly article analyses for modeling purposes. what transformative graduate writing pedagogy does here is not just tender models as scaffolds to facilitate the writing of each subsection of the article within the disciplinary macrostructure, but also proffer ways to turn granular presentational practices into co-acting synergistic ones. whether graduate students are writing transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 23 scientific studies or literary papers, sociological treatises or marketing investigations, a scaffolded transformative writing pedagogy offers models and exercises as well as self-study opportunities and group review platforms to ensure that students are moving away from homogenized to original, inventive, and voiced writing, irrespective of the disciplines they belong to. the goal here is to help graduate students place their research at boundaries or intersections of various theories and perspectives so they can manifest their voices and carve their own niches in the graduate academic writing world. scaffolding is central to student success whether in an undergraduate or a graduate academic writing classroom or a w.a.c. & w.i.d. course. a common element in teaching at both levels is not only the structuring of the process into three distinct phases of outlining, drafting, and revising, but also the marking of the milestones with peer review workshops. both undergraduate and graduate instructors are encouraged to put in much care into devising the peer review workshops as they are important scaffolds to support and nurture the growth of student voices. since the output of one process becomes the input for the other, students get to experience how each stage is linked and builds on the one that went before. if time permits, one-on-one student-teacher conferences can also be held at the end of each stage as it can prove a useful scaffold for the student writer. in fact, the scaffolding principle itself goes a long way in bringing in structuring and organization into academic writing instruction as well as in ensuring that positive results are achieved consistently in growing student's voices in both undergraduate and graduate academic writing courses. conclusion the value of transformative learning is often difficult to gauge because it is an outcome of invested deliberations, creative ruminations, and change-oriented thinking. it results in students developing close reading skills, layered writing skills, and independent thinking skills. these skills take time to manifest themselves, but what is important is that undergraduate students leave a transformative academic writing class with an awareness of their emergent voices and enhanced skills of interpretation and argumentation. the takeaways for graduate students from their academic writing class similarly are enhanced composition skills and the ability to balance and interplay individual expression and interdisciplinary components. when academic writing students—whether undergraduate or graduate, or from a w.a.c. or a w.i.d. class—understand the value of transformative writing practices in the development of their professional identities in academic writing and community spaces, they become aware of their ability to contribute to change and scholarship even as they develop individualized distinctive voices. from the teacher's standpoint, the rewards are high as well. incorporating s.e.a. principles not only make instructors acutely aware of their augmented role in developing student voices, but it also empowers them to reach out to students more effectively through creating scaffolds. as individuals who transformed students’ writing, academic writing teachers experience elevated self-esteem. while receiving positive student evaluations is a bonus, what is perhaps the most rewarding for transformative writing instructors is the immense satisfaction they transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 24 gain from the knowledge that they helped their students develop voices and contribute to the academy and the community. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. notes 1 the sample transformative writing prompt that follows is from my expository writing class. in “selections from reading lolita in tehran,” war memoir writer azar nafisi and her select group of students find that imaginatively engaging with fictional works help them contend with the “absurd fictionality [that] ruled our lives” under a totalitarian regime. how far, if at all, does that engagement reflect the smart shopping of the psychological immune system that daniel gilbert, social psychologist, discusses as happening “behind closed doors, in the back room, outside of our awareness" in “immune to reality”? or would you rather say that the secret deliberations lead to the group developing interconnectedness that buddhist thinker, robert thurman, terms in his essay, “wisdom,” as the manifestation of the deepest awareness that comes “when your consciousness begins to turn inward and gaze upon itself”? 2 the immortal life of henrietta lacks by rebecca skloot narrates the case of henrietta lacks, “a poor black tobacco farmer whose cancerous cells, taken without her knowledge in 1951, became one of the most important tools in medicine, vital for developing the polio vaccine, cloning, gene mapping, in vitro fertilization, and more. henrietta’s cells have been bought and sold by the billions, yet she remains virtually unknown, and her family can’t afford health insurance” (book cover/promotion). the incident sparked off a host of debates around questions of racial discrimination, medical ethics, scientific research, and medical waste ownership. a list of transformative argumentative essay prompts on this non-fiction book is available on pages 19 to 28 of the teacher's guide. here is the link to the resource: http://rebeccaskloot.com/wpcontent/uploads/2011/03/rhsklootteachersguidelores.pdf transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 25 references bass, e. (1998). a narrative analysis of a transformative pedagogies impact on academic literacy (publication no. 9838459). [doctoral dissertation, temple university]. proquest dissertations & theses global. bliss, j. & askew, m. (1996). effective teaching and learning: scaffolding revisited. oxford review of education, 22(1), 37–61. bodrova, e. & leong, d. (1998). scaffolding emergent writing in the zone of proximal development. literacy teaching and learning, 3(2), 1–18. byker, e. j., putman, s. m., handler, l., & polly, d. (2017). educational technology and student voice: examining teacher candidates' perceptions. world journal on educational technology: current issues, 9(3), 119–129. comfort, j. (1995). negotiating identity in academic writing: experiences of africanamerican women doctoral students (publication no. 9544542). [doctoral dissertation, the ohio state university]. proquest dissertations & theses global. correa, d. (2010). developing academic literacy and voice: challenges faced by a mature esl student and her instructors. profile: issues in teachers’ professional development, 12(1), 79–94. davis, k. m. (2009). personal writing, academic writing, and critical pedagogy in the community-based classroom. (publication no. 3350301). [doctoral dissertation, wayne state university]. proquest dissertations & theses global. donnell, k. (2007). getting to we: developing a transformative urban teaching practice. urban education, 42, 223–249. gilbert, d. (2006). immune to reality. in richard e. miller and kurt spellmeyer (eds.), the new humanities reader (pp. 141–50). houghton mifflin. harrell-levy, m.k., & kerpelman, j. l. (2010). identity process and transformative pedagogy: teachers as agents of identity formation. identity, 10(2), 76–91. harrington, k. (2015). border crossing. in t. lillis et al. (eds.), working with academic literacies: case studies towards transformative practice (pp. 11– 13). the wac clearinghouse/parlor press. harrington, k. (2015). border crossing. in working with academic literacies: case studies towards transformative practice. the wac clearinghouse/parlor press, 13–14. ingle, j. & yakovchuk, n. (2015). writing development, co-teaching and academic literacies: exploring the connections. in t. lillis et al. (eds.), working with transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 26 academic literacies: case studies towards transformative practice (pp. 143– 154). the wac clearinghouse/parlor press. jacobs, c. (2015). opening up the curriculum: moving from the normative to the transformative in teachers’ understandings of disciplinary literacy practices. in t. lillis et al. (eds.), working with academic literacies: case studies towards transformative practice (pp. 131–142). the wac clearinghouse/ parlor press. lensmire, t. j. (1998). rewriting student voice. journal of curriculum studies, 30(3), 261–291. levin, j. a. (2000). lifework: connecting academic writing, teaching, and the sense of self (publication no. 9967389). [doctoral dissertation, the university of nebraska]. proquest dissertations & theses global. lillis, t. (2015). towards transformative design. in t. lillis et al. (eds.), working with academic literacies: case studies towards transformative practice (pp. 8–12). the wac clearinghouse/parlor press. lillis, t., harrington, k., lea, m., & mitchell, s. (2015). working with academic literacies: case studies towards transformative practice. the wac clearinghouse/parlor press. mayaba, n.n., ralarala, m. & angu, p. (2018). student voice: perspectives on language and critical pedagogy in south african higher education. educational research for social change, 7(1), 1–12. meyers, s. a. (2008). using transformative pedagogy when teaching online. college teaching, 56(4), 219–224. mezirow, j. (1978). perspective transformation. adult education, 28(2), 100–110. mezirow, j. (1991). transformative dimensions of adult learning. jossey-bass. mitchell, s. (2013). open-ended transformation: ethnographic lens and a suspicious tendency. in t. lillis et al. (eds.), working with academic literacies: case studies towards transformative practice (pp. 15–25). the wac clearinghouse/parlor press. nafisi, a. (2006). selections from reading lolita in tehran. in richard e. miller and kurt spellmeyer (eds.), the new humanities reader (pp. 312–24). houghton mifflin. read, s. (2010). a model for scaffolding writing instruction: imsci. the reading teacher, 64(1), 47–52. simon, c. r. (1987). empowerment as a pedagogy of possibility. language arts, 64(4), 370–81. transformative pedagogy and student voice journal of effective teaching in higher education, vol 3, no.2 27 skloot, r. (2017). the immortal life of henrietta lacks. broadway paperbacks. skloot, r. (2020, june 12). teacher’s guide. retrieved from http://rebeccaskloot.com/wpcontent/uploads/2011/03/rhsklootteachersguidelores.pdf stevens, f. (2015). there is a cage inside my head and i cannot let things out: an epistemology of collaborative journal writing. in t. lillis et al. (eds.), working with academic literacies: case studies towards transformative practice (pp. 267–79). the wac clearinghouse/parlor press. thurman, r. (2004). wisdom. in richard e. miller and kurt spellmeyer (eds.), the new humanities reader. (pp. 434–450). houghton mifflin. turner, y. (2006). chinese students in a uk business school: hearing the student voice in reflective teaching and learning practice. higher education quarterly, 60(1), 27–51. virtue, e., wells, g., mackusick, c., murphy-nugen, a., rose, a., & snyder, m. m. (2018). the immortal life of henrietta lacks: using a common read to transform a learning community. learning communities research and practice, 6(1), 4. wideman, j. e. (2005). brothers and keepers: a memoir. houghton mifflin. microsoft word hammond final version.docx journal of effective teaching in higher education, vol. 2, no. 2 experiential learning and peer teaching to develop powerpoint slide formatting skills kay m. hammond, auckland university of technology, kay.hammond@aut.ac.nz abstract. construction of visual material to enhance audience understanding of an oral presentation is an important skill in educational and professional settings. many first-year undergraduates may not be familiar with the basic principles of effective slide design to increase audience understanding. however, faculty face time pressures to train students on effective use of information technology for educational activities. this paper reports on the development and use of a timeefficient, engaging, in-class activity involving specific learning criteria for slide design in an academic literacy course. the activity is based on experiential learning and peer teaching to increase student ability and confidence with creating powerpoint slides for first year undergraduates. students evaluated the activity positively and were able to achieve most of the learning criteria. recommendations for further development of the activity are provided. keywords: oral presentation, powerpoint, experiential learning, peer teaching, undergraduate oral presentations skills are a key aspect of achieving professional competency across a wide range of contexts. communication skills are a valued component of many graduate profiles in higher education (van ginkel et al., 2015) and a significant aspect of academic life (barrett & liu, 2016; de grez, valcke, & roozen, 2009; živković, 2014). oral presentations are included in assessments of many undergraduate courses as a desirable skill set in terms of thinking and communication. one of the key skills is to connect verbal information with visuals (aryadoust, 2015). in professional oral presentations, presenters frequently include communication through visual forms such as slides. furthermore, the increasingly multimedia rich world will require communication skills beyond the current dominance of the written word in essay form that characterizes current university practice (mitchell, 2012). despite the recognized importance of oral presentations, students are given limited training in how to do them (andrews & higson, 2008; jones, 2003). an oral presentation is a multi-skill activity that takes time and experience to develop. in addition, this can be frustrating and nerve-wracking for students who are not native speakers of english (aryadoust, 2015; barrett & liu, 2016), or those with little experience of, and who fear presenting and may disengage from presentation activities. therefore, a focused and manageable subset of skills for presentations can be useful within such contexts to quickly develop engagement, skill, and confidence. students benefit from focused learning with specific guidelines (barrett & liu, 2016) and such guidelines assist staff to focus their instruction and assessment activities within available time allowances. powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 24 there are several options for slide software such as powerpoint, keynote, prezi, slideshare and adobe spark. although the use of powerpoint has become extremely common, critics report the constraints of information into titles and bullet points as a restriction to creativity and other forms of knowledge presentation. furthermore, that such presentations perpetuate monotonous and ritualistic presentation formats (kernbach, bresciani, & eppler, 2015). despite criticisms of powerpoint, it offers an easy-to-use tool that introduces users to the basic principles of working with visual communication. thus, instead of attempting to propel early adopters into cutting-edge experience with newer and emerging slide tools, students can build skills and confidence in a time-efficient and measurable way that enables new undergraduates to master the basic principles of slide design with powerpoint software. furthermore, with powerpoint being a commonly used tool for current students, academics, and industry professionals, then being familiar with this format can be considered a starting point from which later development into more creative and non-text-based formats of presentation can launch. another advantage of powerpoint is the ease of uploading slides to common learning management systems, such as blackboard or moodle for assessment and to have them all located in one place on the classroom computer for each presentation session. in addition, with many people being familiar with powerpoint, there is a greater pool of potential help by friends and family outside of the class. in this article i will describe some challenges with teaching introductory slide design and combine two effective teaching strategies of experiential learning and peer teaching into an experiential slide design activity. following this is a description of the local context in which i developed and applied a slide design activity using experiential learning and peer teaching to fit within a single class period. student perceptions of their engagement, learning and perceived usefulness of this exercise are presented along with their suggestions for improvement. teacher evaluation of the students’ slides is also presented. this article will be useful for educators and/ or students who require an effective, time-efficient learning activity for designing slides with specific learning outcome criteria for assessment. challenges with teaching slides time limitation of time is a concern for both teachers and students. performance elements of presentations such as vocal variety and body language need practice physically. designing slides is also performative in that it includes the creation of an artefact mediated by technology and requires physical manipulation of digital objects and an awareness of how their placement affects audience learning. however, educators often do not have time to teach presentation skills (barrett & liu, 2016). van ginkel et al. (2015) summarized research pointing out the challenge of offering sufficient opportunity for skill development within increasingly tight time constraints within higher education. restricted time also reduces the amount of individualized instruction the teacher can provide. although there are many resources available on how to design slides (e.g., duarte, 2008), students may not have time to access or prioritize these skills. student lack of time has been noted as an issue for study (eriksson, adawi, & stöhr, 2016). thus, instead of powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 25 developing their design skills, students and educators may rely on what they know of writing for other formats and apply these to their slides (e.g., presenting large chunks of text). this may enable them to ‘get by’ without spending additional time. clarity & consistency educators need to make explicit what outcomes the students will be assessed on and how to achieve these (biggs & tang, 2011), so students know what is expected of them and how these skills will apply after graduation from undergraduate tertiary study (national institute for learning outcome assessment, 2016). in courses with large enrolments, several markers may assess students, some of which may not be teaching on the course. therefore, a clear set of requirements for consistency in grading is important for consistency. this is particularly important when marking rubrics, constrained by limited space, describe the requirements for slides vaguely as “appropriate” or “enhancing” visuals without specifying what these terms mean. furthermore, clarification of how to reference images is also important to respect copyright (jones, 2003) and academic referencing conventions. unfamiliarity with slide making software a barrier to using slide-making software may be the reluctance of those slower to adopt technologies (jones, 2003). this reluctance may be irrespective of age and therefore includes young as well as mature students. jones stated one of the benefits of using powerpoint was the ease of achieving basic-level proficiency. however, the downside of this is the tendency for educators to use slides as notes and to reproduce large chunks of text in lectures (jones). such examples may have influenced students to imitate a similar design. i have noticed the excessive use of text by my students when they appear to use their slides to reproduce their essay and serve as cue cards for themselves rather than as a method to enhance the audience’s understanding of their topic. i have heard students make similar comments about their lecturers. although people can get by with this practice, brock and joglekar (2011) found higher teaching effectiveness was rated by students as connected to less text on slides. spending focused and structured time on presentation skills is beneficial for students who are required to acquire a presentation skill set. incorporating slides into a presentation proceeds in three phases: planning, design, and delivery (mollerup, 2014). during the design phase, students need to consider how to select and arrange key content on a slide to enhance the communication of that content. the proposed activity in this paper focuses on the design phase. effective teaching practices experiential learning according to constructivist views of learning, students construct knowledge through engagement with activity (shuell, 1986) and activities need to align with the powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 26 required learning outcomes (biggs & tang, 2011). many skills required for oral presentations require active experience to support deep level learning. if students are expected to engage with an activity on developing their slide design skills, then it needs to be a performative activity, narrowed to an achievable set of skills that aligns with the learning outcomes and can be covered in the time available. proponents of experiential learning argue that people learn more effectively through experience and reflecting on that experience. in doing so, students engage more of their senses for processing information (jacobson & ruddy, 2015; kolb, 1984). a combination of experiential techniques and more direct instruction is recommended to assist students with little prior knowledge of a task (lalley & miller, 2007). beard and wilson (2006, 2018) viewed learning as holistic and so the space, emotions, senses, and levels of challenge are important when designing learning experiences. learning occurs through the interaction between the learner’s inner world and outer world using their senses. bodily movement, social interaction, language, and building things support learning. the opportunity to talk aloud while building something stimulates thinking and a sense of reality. therefore, beard and wilson (2006) recommended several elements for experiential learning such as: a clear target of learning, progressive learning journey, exercising intelligences, reality-based, social collaboration, restricted scope, challenge/risk, time limits, and an opportunity for reflection. peer and self-assessment for reflective learning in a review of studies, marton, mccullough, and ramnanan (2014) found peer teaching led to increased confidence in teaching, oral presentation skills and giving feedback. sociocultural views of learning highlight learning through social interaction and language within context over disconnected information transmitted from a teacher. examples of this view are shown in vygotsky’s (1978) conception of the zone of proximal development (zpd) and lave and wenger’s (1991) concept of the community of practice (campbell, macpherson, & sawkins, 2014). ramaswarmy, harris, and tschirner (2001) recommended that teachers play an active role during peer teaching session as they can identify aspects that have been missed and recognize student progress. local context and aims healthcare workers need to learn to teach others, including their colleagues and patients with their families. peer teaching is common in health education and assists in the development of this important skill (marton, et al., 2014). in a core academic literacy course for first year health science students in a large, publiclyfunded university in new zealand, one of the assessments is to write a 1250-word academic essay on a topic of their choice. a subsequent assessment is to make a five-minute verbal presentation of their essay, accompanied by six powerpoint slides. six slides encourage conciseness and consistency for the assessment artefact. thus, the oral presentation component is integrated within the academic literacy course rather than being a stand-alone presentation course. competence in powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 27 the oral presentation is required to pass a learning outcome of “producing written and verbal work at an appropriate academic standard”, of which “using visual aids to enhance the presentation” is a criterion. another assessed learning outcome requires academic integrity in which students must correctly reference any text or images used from other sources. students must meet the oral presentation standards to progress in their studies. the aforementioned course was run in a flipped classroom format in which lecture content was viewed online prior to attending a 90-minute weekly tutorial in a class sizes of approximately 20─40 students (overall enrolment ranges from 350─1500 students of considerable age and ethnic diversity). there were two class periods focused on the oral presentations: one for creating slides and the other for the presentation delivery skills. prior to the slide creation tutorial, students were required to watch a five-minute online video on slide creation that included information on: content conciseness, font type and size, background selection, contrast, animations/distractions, selection of informative images and proofreading. the follow-on in-class activity required students to view and critique three examples of previous students’ slides. however, there were several problems with our approach to teaching students how to design effective powerpoint slides in this way. despite the instructional video and in-class examples, teachers noticed that many students did not appear engaged as very few students would offer critical comments in class. furthermore, during the oral presentation assessment, we saw several slides containing cut and pasted blocks of text with little attempt at formatting or selection of informative images to increase audience understanding. although most students created slides that could pass as adequate, we believed they could achieve higher standards. we also realized that despite the young age of many students, several were not familiar with designing slides and this was also the case for mature students returning to education. our marking rubric was also vague and offered little to guide student learning as it stated, “visual aids well prepared, managed and appropriate for presentation”. considering these characteristics of our students and the non-specific wording in the marking rubric, the tutorial needed a more engaging activity that would involve all students rather than just those who volunteered answers. we also realized that much of our knowledge on good design was tacit and varied across instructors. therefore, we needed to narrow our assessment criteria to a manageable set of specific design principles and make our knowledge explicit to students and consistent across assessors. we also needed students to be more aware of how the design principles increase understanding. to achieve these goals, the author designed an in-class activity based on the principles of experiential learning and peer-teaching that students could engage in and help learn more about the design criteria. there would also be an opportunity for reflection on their learning. ethical approval was obtained through the university’s ethics committee. powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 28 method recruitment immediately after students completed the learning activity, the teacher stepped out of the classroom. a staff member external to the class introduced the study and offered students the opportunity to participate by completing an anonymous survey about their learning. activity design based on the experiential learning program elements outlined by beard and wilson (2006, 2018), and considering the recommendation to include some direct instruction, the activity includes the following elements: (1) target – create a clear slide to enhance communication of the provided information. (2) journey, construction, organizing and functional skills – progression from selecting, and physically formatting and positioning text and images to create the slide using powerpoint. (3) exercise intelligences and senses – achieving the goal through talk, touch, thinking and social interaction. (4) reality – create an actual slide following the guidelines that also form the basis of assessment in the coming assessment. (5) social collaboration – working in pairs or threes (or alone if preferred). (6) restrictions – set of guidelines provided to assist with slide construction. (7) challenge/risk – making the slides available for the class to view and compare. deliberate typos included in the text and inclusion of noninformative (cliché) images. (8) time constraint – 10 minutes to construct the slide. (9) reflection – comparing their slide with those of their peers. materials and steps questionnaire. the slide creation post activity survey included five sections: 1) demographic questions on sex and age, 2) questions on how students experienced the activity according to the experiential learning dimensions (rated on a 5-point scale), 3) questions on to what degree students believed they had learned the 11 slide formatting techniques in comparison to what they knew before the activity (rated on a 3-point scale of: “same as before”, “a little better than before”, and “a lot better than before”), and an open question box to write what they perceived they had learned most 4) an overall rating of engagement with the activity from 1 (not at all) to 10 (very), and 5) an open question on any suggestions for improvement of the activity. powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 29 teacher observation sheet. i created a teacher observation sheet to make three key observations: 1) whether students were looking at their laptop screens displaying the activity, and 2) whether students were communicating with their peers (or focused on task if doing the task solo). i used a white box in the center of the sheet to sketch the student groupings and locations in the classroom. i made five observations of the students in each class. the author noted on a printout of the format guidelines for slide layout given to students whether the slide created by the individual, pair or group had achieved the learning outcomes. laptops and computers. students require laptops or computers for this activity. if a student does not have a laptop in class, they can work in a pair with someone who does; otherwise this activity could be completed in a computer lab or at home. tablets or mobile phones are not effective for this activity as the screens are too small and it is difficult to manipulate text and images. the following procedure outlines the steps required to set up the activity in class. preparation step 1. consider the learning outcome(s) the slides are associated with and list the observable criteria required to be demonstrated. for example, the outcome of facilitating understanding may include easily readable font size (24 or greater) and type (arial, new times roman or calibri). step 2. display these assessment criteria in a format guide for students to work with (either as a handout or electronic version). this becomes the checklist for feedback to students. for a useful set of presentation design guidelines, see jones (2003) and duarte (2008). table 1 shows the guide used in the current study. although sans serif fonts are most advised, we allowed times new roman to be included in the suitable font to reduce the complexity for beginning level students. for flipped classroom formats these criteria are covered in an online video that students are expected to watch prior to attending class. table 1 formatting guides for slide layout format includes the use of: good background text that contrasts strongly with the background appropriate font size 24 or larger for body text appropriate font type easy to read types – e.g. arial, times new roman or calibri powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 30 appropriate title keywords appropriate to the content amount of text one point & not too cluttered relevant picture(s) images that increase the understanding of text text hierarchy titles and subtitles in larger font than the body text reference the picture(s) small font citation under the image resize objects in powerpoint resize option to make objects an appropriate size avoid distracting items & animations animation only if it enhances understanding & do not use items that do not enhance understanding step 3. collect or create text appropriate for the course level (reading level and content) and adapt it so students are required to edit it to achieve the learning goals. for example, if a goal is accuracy, then include errors. if a goal is conciseness, then include more text than necessary. the text may include titles, subtitles, or body text. step 4. collect or create images suitable for the learning goal. if visual images are required, then provide choices between informative images (pictures that add understanding to the text) and cliché images (pictures that do not add further understanding). for example, on the topic of physical rehabilitation using water therapy, an informative picture could show a therapist working with a client in a pool. a cliché picture would just show a pool. step 5. develop a practice content slide by placing content information and images. practice content slide. figure 1 shows the content elements which include: suggested keywords for a possible title, enough sample text so that some could be deleted, deliberate spelling mistakes for proofreading, images that either illustrate points made in the text (e.g. an instructor working with a client, or instructions on equipment) or are cliché (e.g. the general gym scene, dumb-bell or dumb-bell lifter in figure 1). pictures will need deleting or resizing, and the font type and size will need to be changed. powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 31 figure 1. content elements slide. step 5. following the content slide, include a blank slide for students to transfer content and images (optional use). step 6. develop a set of instructions for students to follow. in the current study the instructions were: in the next ten minutes, you can work alone or in pairs or threes on your laptops to create one body slide to illustrate some information contained in this content slide. you should include a title, some text and at least one image. you do not have to use all the material here. use the keywords to get ideas for the title. you may resize, move, or delete anything on the slide. at the end of the ten minutes, we will move the screens to face the center of the room to form a gallery and we will look around at what others have done. any questions? powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 32 step 7. embed the activity content slide and blank slides into student versions of the overall lesson slides, or save as a separate file for students to download and work on. activity in practice the steps here were completed in 25 minutes with a class attendance of approximately 15 students per class. time could be adjusted for varying class sizes. step 1. setup have students sit with their laptops or arrange themselves in pairs or threes with a laptop to share. they open the file with the practice content slide and blank practice slide (3 minutes). step 2. instructions give or read the instructions to students to begin the activity and answer any questions (2 minutes). step 3. feedback in process during the activity, move around the students to answer any questions, check progress, and offer guidance (10 minutes). step 4. feedback after activity while the last students are finishing their slide creation, the teacher starts giving feedback on each slide as to whether it met the requirements described in the formatting guide. continue until all students have received feedback (5 minutes). step 5. the gallery exhibition at the conclusion of slide creation and teacher feedback, the students position their laptops on the desks to form a circle of screens facing the centre of the room. this arrangement forms the gallery. all students then move around the gallery looking at the differing versions of the slides. as students move around the gallery they compare the slides and notice what they liked about the content choices and formatting of their peers. the teacher can also direct students’ attention to good examples of formatting and content choice (5 minutes). step 6. evaluate the activity and use the feedback to develop further contextspecific iterations. analysis analysis of the quantitative data from the questionnaires and teacher observations included calculations of descriptive statistics. a selection of qualitative responses offered further illustration of findings using the students’ voices. results during the activity, teacher observations recorded that students focused on the activity during the allocated time or until the task was completed. following the activity, 76 students completed an anonymous in-class survey. this represented powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 33 41% of my class enrolments and 100% of students completing the activity in my classes. the majority of the participants were female (59.2%, male, 34.2%, no response, 6.6%) and under 20 years old (42.1%, 20–24, 30.3%, 25+, 23.7%, no response, 3.9%). the average rating of engagements was 8.5 out of 10 (standard deviation = 1.3, mode = 10). table 2 shows the average and modal responses from 1 (negative) to 5 (positive) for each of the experiential learning characteristics. table 2 average and mode for each experiential learning characteristic experiential learning characteristic average mode clear goal 4.7 5 clear process 4.6 5 use of physical action 4.6 5 use of intelligences 4.3 5 task based on reality 4.8 5 social collaboration 4.5 5 set of rules 4.5 5 challenge 3.2 3 time constraint 3.6 4 opportunity to reflect 4.5 5 table 2 shows students perceived most of the experiential learning characteristics as positive. students rated the time limit and level of challenge less positively. teaceher observations noted approximately half the students completed their slide (59%), and the remaining students were near completion at the end of the allocated time. social collaboration was rated by 63 students as the others worked alone. table 3 presents the percentages of student evaluations of their learning in each criterion, falling across three levels of learning on the scale of 1 (same as before), 2 (a little better than before) and 3 (a lot better than before). there was only one missing value for the ratings of font size and relevant picture. table 3 percentage of students reporting each degree of learning per learning cutcome criterion learning outcome criteria degree of perceived learning same little better lot better chisq p reference picture 4 20 76 66.0 <.001* relevant picture 16 28 55 19.0 <.001* powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 34 avoid distracting items 25 28 47 6.82 .03* amount of text 16 33 51 14.4 <.001* make a slide to enhance understanding 3 54 43 33.5 <.001* font size 21 37 41 5.0 .08 title using keywords 22 41 37 4.3 .12 text hierarchy 34 30 36 .34 .84 font type 37 28 36 1.1 .57 contrast between text & background 39 30 30 1.3 .50 resize objects 47 22 30 7.5 .02* *p=<.05 table 3 shows nearly all students (97%) reported the activity helped them a little or a lot better than prior to, in making a slide to enhance their understanding. at least half the students reported learning a little better or a lot better than prior to, for all learning outcomes and in particular, for the learning outcomes of: referencing a picture (96%), the amount of text to use (84%) how to select a relevant picture (82%) and the avoidance of distracting items (73%). a chi-square goodness-of-fit test was performed with two degrees of freedom per test. the distribution of the responses in these five learning outcome criteria reached statistical significance. in addition, the proportions of responses for resizing objects also reached significance – in this case for nearly half of students reporting their knowledge of resizing objects was the same as prior to the activity. at the conclusion of the activity, the teacher assessed whether the students had achieved the specified learning outcome criteria. a total of 43 slides were assessed across the six classes. table 4 shows the percentages of slides achieving the formatting requirements at the conclusion of the task. table 4 percentage of slides showing degree of each learning outcome criterion format achieved partially achieved not achieved background contrast 95.3 0.0 4.7 text hierarchy 95.3 0.0 4.7 no distractions 95.3 0.0 4.7 appropriate title 90.7 7.0 2.3 font type 86.0 7.0 7.0 font size 79.1 14.0 7.0 relevant picture 74.4 9.3 16.3 powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 35 text amount 67.4 23.3 9.3 picture reference 65.1 9.3 25.6 resize objects 39.5 25.6 34.9 table 4 shows most slides included acceptable background contrast, picture referencing, absence of distractions, an appropriate title, and font type. achievement dipped below 80% for the other criteria. resizing objects was not frequently achieved as many students did not resize, or only marginally increased the size of the image. during the reflection gallery, i observed students commenting to each other on what they liked about the other students’ designs. from the open questionnaire item asking students what they had learned from the activity, most referred to an aspect of creating slides. the following response illustrates this: to review the slide drafts, i have made to make images more relevant and strongly connected to the words. to review font sizes so same over whole [slide] deck. (student 50) a few other responses mentioned the awareness of the audience when constructing slides. for example: i learnt how to summarize key points, and that pictures are important in getting a message across to the audience. (student 5) others mentioned how the guideline gave them an idea was what standard was expected. for example: [i learned] what was required of the upcoming assignment, which was useful as now i am much more confident. (student 1) few students made suggestions for improvement, but those who did offer ideas mentioned: doing the activity with a slide they would actually use for an assessment, using a computer lab, having more time, having a little more instruction at the beginning, design more slides in the activity, practice presenting the slide, and to have an example of a ‘good’ slide at the end. discussion engagement the average rating of engagement was high (8.5/10). this rating indicated that the students perceived the activity as matching the characteristics of experiential learning described by beard and wilson (2006) and students found it engaging. thus, the current form of the activity appears to have worked well. this was an improvement on the previous passive critique of example slides during class in which only a few students actively participated. furthermore, the class activity powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 36 aligned more closely with learning outcomes for the course. this offered more clarity of expectations for students (biggs & tang, 2011). the two aspects of the activity receiving the lowest ratings were the level of challenge (3.2/5) and the time limit (3.6/5). even though the challenge rating was moderate, the learning benefits reported by students for the other criteria suggest this level of challenge is appropriate. high engagement can be achieved when people use their skills to work on a manageable challenge (csikszentmihalyi, 1997). if the challenge rating is too high, then some people may give up. the challenge rating here may also be due to the basic nature of the slide design principles, which are easy to understand and consciously perform. educators will need to assess the competency level of their students when deciding what criteria to include in the activity and when to increase the level of difficulty and associated challenge. the time limit was slightly restricting in that some students had not quite completed their slide. an additional five minutes to complete the task may be helpful. educators also need to be mindful of time allowances that enable most of the task to be completed by all students – if too much time is given, students who complete the task quickly may become disengaged. alternatively, faster students could be given an additional task while waiting, such as working on a slide they will use for a future presentation. learning most students felt their understanding of how to make a slide to increase audience understanding had improved, particularly in the aspects of: referencing pictures, selecting a relevant picture, selecting an appropriate amount of text and avoiding distracting items. students achieved high levels of success (approximately 80% or higher) for six of the design principles, and moderate (above 65%) for the others. the criteria perceived improving the least were: resizing objects, contrast between text and background, and font type. although more than 50% of students reported learning about these criteria at least “a little better than before”, it is possible that these principles take a little longer to master for some students than one class. students may have to unlearn habits of using larger amounts of text and copying images without referencing them. although the chi square test showed statistical significance for six of the learning outcomes, it was based on the assumption of expected frequencies of one third in each response category. it is unlikely that many students would randomly say they had learned a lot in an activity when they had not – as they did not do so for resizing an object. thus in practical terms, even though some learning outcomes did not achieve statistical significance in this case, the relatively large percentages of students reporting some gain in knowledge about the learning outcome warrants inclusion of those outcomes. although students reported learning the least from resizing objects, the least achieved skill was resizing objects (only 39.5% of slides were deemed to achieve this). resizing objects was seen as something nearly half the students felt they did not learn more than what they knew before and yet this was the least achieved item. students may have considered the concept of resizing images to mean the powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 37 physical manipulation of the images to change their size, rather than understanding the concept of white space and a greater awareness of what is visible to people at the back of the presentation room (similar to understanding choice of font size). the current list of criteria appears suitable for this student group. there were a number of skills that were achieved well and other skills that could still be improved. the number of skills was enough to provide just-in-time learning and just enough for their assessment. this is useful when mastering technology skills (jay & conners, 2005). in addition, students receiving time to immediately apply the information after the brief instruction period facilitates their ability to remember and perform the skills, which in turn provides them with established associations they can connect future learning with (klemm, 2010). if student cohorts become more proficient in future, extra criteria could be included to advance the skills of the more proficient students, such as: adding hyperlinks to break up the liner flow of the presentation, strategic use of animation to highlight points or control reading (e.g., revealing one line of text at a time), or as jones (2003) suggests, choosing slightly less contrast between text and background to assist readers with dyslexia. however, this needs to be carefully considered so that it aligns with specific learning outcome criteria. furthermore, if the aim is to have the entire cohort achieve the learning outcome criteria, focusing on these criteria together will enable peer interactions and teaching center on those criteria rather than students splitting into individual levels. limitations research design and procedure the study design was mostly quantitative. a control group would enable a comparison between students doing the activity and those who watched a more passive explanation of slide formatting. in the absence of a control group, a couple of extremely easy learning outcomes could be added to test that students are responding accurately. for example, asking them to open powerpoint and also to type text on a slide. it would be expected that nearly all students would already know that and therefore answer “same” for the degree of learning compared with previous knowledge. engagement with the learning activity could also be studied through greater focus on how the students experienced the activity and how it impacted their learning. this may be achieved through a narrative inquiry of the students’ experiences of the activity. lack of laptops in some contexts, students may not have laptops. if there are not enough students with laptops for at least one between three, the task could be assigned for a computer lab or as homework when students may have access to a computer. in this study students were sometimes in groups of three and although they seemed powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 38 engaged, it limited the amount of practice individual students could have in using software on a laptop. some students insisted on working alone, but they still benefitted from seeing peers’ work during the gallery section of the activity. students not watching the flipped classroom video it is possible that some students had not watched the explanatory video prior to attending the class. a summary of research has reported the value of the flipped classroom format, but also cautions some students may not have good online access or feel that preparation time outside of the classroom is too much (betihavas et al., 2016). in the current study, students were not asked if they had previewed the video prior to attending class. further research could determine what value the video had as part of the activity. knowledge transfer student evaluation of the activity was based on anonymous questionnaires immediately after the activity. further ethical approval would be required for permission to include actual student examples of slides or to follow their progress to the slides they submitted for the summative assessment. the aim at this stage was to obtain honest student opinions without fear of association with their actual assessment. now that student engagement with the activity is established, future research could follow through to determine how knowledge and skills gained in the activity transfers to the slides submitted for the summative assessment. implications and recommendations the slide formatting learning outcomes in this study appeared to be engaging and useful learning for most students in the cohort. therefore, these are recommended for other contexts that include students with few or basic skills in slide design. considering the mismatch between student belief in their ability to resize and the lack of achievement of this learning outcome, more instruction is needed about resizing objects. overall, flexibility in the content or timing of this exercise can be adjusted for different contexts with more advanced learners. teachers can change the content and images to suit the topics of their students. for example, information may vary in terms of text, graphs, figures, equations, or images required for the students’ discipline. the level of skill required can also be adjusted to suit the competence level of the student cohort. educators can share their knowledge of their students in collaborating with learning advisors to develop effective activities for their students’ learning (national institute for learning outcomes assessment, 2016). as with the activity reported here, ten important principles of slide design that were most appropriate to display discipline-relevant content can be selected and also used to align with learning outcomes. in the open questions, students mentioned wanting the option to work on actual assessment slides. in classes with more generous time allowances, students could return to their slides to make modifications based on what they observed from others during the gallery section. if more time is available, greater authenticity powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 39 could be achieved by having students select content they will be using for an assessment and select a picture from the internet or their own images. students could also work on reformatting a complete set of slides for an entire presentation. in classes with large numbers and limited time, peers could check each other’s work for achievement of the formatting requirements. conclusion oral presentation skills often include the design of effective visuals such as slides. academic and professional competency require these skills; however, training in these skills in higher education can be limited due to time, clarity of expectations, and student unfamiliarity with design software. the slide activity reported here created a time-efficient, experiential activity that students found engaging, and felt they had learned skills that were clearly aligned to the learning outcomes of their future assessment. the results of this study show that the creation of digital artefacts, such as slides, benefits from experiential learning rather than traditional didactic lecturing. furthermore peer-teaching assisted the engagement with the concepts and design software. the ease in which this task can be modified to suit other contexts makes it a valuable activity to consider when engaging students in experiential learning activities on basic slides design principles within a limited amount of time. acknowledgements the author acknowledges the anonymous reviewers for their helpful advice on the preparation of this manuscript. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references andrews, j., & higson, h. (2008). graduate employability, ‘soft skills’ versus ‘hard’ business knowledge: a european study. higher education in europe, 33(4), 411─422. doi:10.1080/03797720802522627 aryadoust, v. (2015). selfand peer assessments of oral presentations by firstyear university students. educational assessment, 20, 199–225. doi:10.1080/10627197.2015.1061989 barrett, n. e., & liu, g-z. (2016). global trends and research aims for english academic oral presentations: changes, challenges, and opportunities for learning technology. review of educational research, 20(10), 1–45. doi:10.3102/0034654316628296 beard, c., & wilson, j. p. (2006). experiential learning: a best practice handbook for educators and trainers (2nd ed.). london: kogan page. powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 40 beard, c., & wilson, j. p. (2018). experiential learning: a practical guide for training, coaching and education (4th ed.). london: kogan page. betihavas, v. bridgeman, h., kornhaber, r., & cross, m. (2016). the evidence for ‘flipping out’: a systematic review of the flipped classroom in nursing education. nurse education today, 38, 15–21. doi:10.1016/j.nedt.2015.12.010 biggs, j., & tang, c. (eds.). (2011). teaching for quality learning at university (4th ed.). birkshire, england: mcgraw hill. brock, s., & joglekar, y. (2011). empowering powerpoint: slides and teaching effectiveness. interdisciplinary journal of information, knowledge, and management, 6. retrieved from http://www.ijikm.org/volume6/ijikmv6p085-094brock545.pdf campbell, c., macpherson, s., & sawkins, t. (2014). preparing students for education, work, and community: activity theory in task-based curriculum design. tesl canada journal, 31(8), 68–92. retrieved from https://files.eric.ed.gov/fulltext/ej1052126.pdf csikszentmihalyi, m. (1997). finding flow: the psychology of engagement with everyday life. new york, ny: harpercollins. de grez, l., valcke, m., & roozen, i. (2009). the impact of goal orientation, selfreflection and personal characteristics on the acquisition of oral presentation skills. european journal of psychology of education, 24(3), 293–306. doi:10.1007/bf03174762 duarte, n. (2008) slide:ology. sebastopol, ca: o’reilly. eriksson, t., adawi, t., & stöhr, c. (2016). “time is a bottleneck”: a qualitative study exploring why learners drop out of moocs. journal of computing in higher education, 24(1), 133–146. doi:10.1007/s12528-016-9127-8 jacobson, m., & ruddy, m. (2015). open to outcome: a practical guide for facilitating and teaching experiential reflection (2nd ed.). bethany, ok:wood n barnes. jay, r., & conners, l. (2005). telling tales: a peak into the world of digital story telling. the learning powerhouse: inaugural tafe nsw icvet conference on vet teaching and learning. retrieved from http://lrrpublic.cli.det.nsw.edu.au/lrrsecure/sites/web/13289/powerhouse_2 005/papers/telling_tales2.pdf jones, a. m. (2003). the use and abuse of powerpoint in teaching and learning in the life sciences: a personal overview. bioscience education, 2(1), 1─13. doi:10.3108/beej.2003.02000004 kernbach, s., bresciani, s., & eppler, m. j. (2015). slip-sliding-away: a review on the literature on the constraining qualities of powerpoint. business and professional communication quarterly, 78(3), 292–313. doi:10.1177/2329490615595499 powerpoint slide formatting skills journal of effective teaching in higher education, vol. 2, no. 2 41 klemm, w. r. (2010). computer slide shows: a trap for bad teaching. college teaching, 55(3), 121-124. doi:10.3200/ctch.55.3.121-124 kolb, d. a. (1984). experiential learning. englewood cliffs, nj: prentice hall. kvavik, r. b. (2018). convenience, communications, and control: how students use technology. twin cities, mn: educause center for analysis and research and university of minnesota. retrieved from https://www.educause.edu/research-and-publications/books/educating-netgeneration/convenience-communications-and-control-how-students-usetechnology lalley, j. p., & miller, r. h. (2007). the learning pyramid: does it point teachers in the right direction? education, 128(1). 64─79. lave, j., & wenger, e. (1991). situated learning legitimate peripheral participation. cambridge: cambridge university press. marton, g. e., mccullough, b., & ramnanan, c. j. (2014). a review of teaching skills development programmes for medical students. medical education in review, 49, 149─160. doi:10.1111/medu.12571 mitchell, p. (2012). digital literacy and the spaces of academic discourse. in p. p. trifonas (ed.), learning the virtual life: public pedagogy in a digital world (pp. 1–14). new york, ny: routledge. mollerup, p. (2014). slide presentations. seriously. visible language, 48(1), 5─21. retrieved from https://s3-us-west2.amazonaws.com/visiblelanguage/pdf/48.1/slide-presentations-seriously.pdf national institute for learning outcome assessment. (2016, may). higher education equality: why documenting learning matters. urbana, il: university of illinois and indiana university, author. ramaswarmy, s., harris, i., and tschirner, u. (2001). student peer teaching: an innovative approach to instruction in science and engineering education. journal of science education and technology, 10(2), 165–171. retrieved from https://www.jstor.org/stable/40186621 shuell, t. j. (1986). cognitive conceptions of learning, review of educational research, 56, 411─436. doi:10.3102/00346543056004411 van ginkel, s., gulikers, j., biemans, h., & mulder, m. (2015). towards a set of design principles for developing oral presentation competence: a synthesis of research in higher education. education research review, 14, 62─80. doi:10.1016/j.edurev.2015.02.002 vygotsky, l. (1978). mind in society. cambridge, ma: harvard university press. živković, s. (2014). the importance of oral presentations for university students. mediterranean journal of social sciences, 5(19), 468─475. doi:10.5901/mjss.2014.v5n19p468 journal of effective teaching in higher education, vol. 1, no. 1 testing effect: a further examination of open-book and closed-book test formats olesya senkova, hajime otani, reid l. skeel, and renée l. babcock central michigan university, mount pleasant, mi abstract. if assessment is the purpose of testing, open-book tests may defeat the purpose. however, a goal of education is to build knowledge, and based on the literature, open-book tests may not be inferior to closed-book tests in promoting long-term retention of information. participants studied swahili-english pairs and either re-studied or took an initial quiz, which was cued recall or recognition in an open-book or closed-book format. one week later, the final closed-book recognition test showed higher performance in the quizzed conditions than in the study-twice condition, replicating the testing effect. however, performance was similar across the quizzed conditions, indicating that testing promoted long-term retention regardless of test format (open-book versus closed-book) and test type (cued recall versus recognition). open-book tests are not inferior to closed-book tests in building knowledge and can be particularly useful in online classes because preventing cheating is difficult when closed-book tests are administered online. keywords: testing effect; open-book tests; long-term retention; learning in educational settings, tests are used as assessment tools because testing students is the primary method of determining how much students have learned. traditionally, tests are divided into two categories, recall-based and recognitionbased, with essay and short-answer questions representing the former and multiple-choice questions representing the latter. furthermore, tests can be administered with closed-book or open-book formats, with the former requiring students to rely entirely on memory and the latter allowing students to look up what they did not commit to their memory. choosing the right format for testing students presents a challenge particularly because of the recent popularity of online classes. in most online classes, it is difficult to know whether students are looking up answers to test questions. in fact, there is a debate over whether cheating on the test is more prevalent in online classes than in face-to-face classes, and if so how to prevent it (e.g., alessio, malay, maurer, bailer, & rubin, 2017; christe, 2003; cluskey, ehlen, & raiborn, 2011; grijalva, nowell, & kerkvliet, 2006; michael & williams, 2013; owens, 2016; rowe, 2004). although measures can be implemented to minimize cheating such as using a lock-down web browser with a monitor and imposing a time limit, one must realize that no method is completely foolproof. because a traditional closed-book test is difficult to implement in an online environment, the instructor may adopt an open-book test. however, the consequences of adopting open-book tests, instead of using traditional closed-book tests, are still not clear. testing effect: a further examination of open-book and closed-book test formats 21 journal of effective teaching in higher education, vol. 1, no. 1 in the present study, the two formats, closed-book and open-book, were compared to examine a commonly held assumption that the latter format is inferior to the former in achieving the goal of promoting long-term retention of studied materials. additionally, the type of the test (i.e., cued recall and recognition) was manipulated to investigate whether test format would interact with test type. if the purpose of testing is to assess learning, one may argue that an open-book test would defeat the purpose because when one is allowed to look up answers, it would be difficult to assess what one knows and does not know. however, because one of the goals of education is to develop knowledge (see bloom’s taxonomy of educational objectives, bloom, engelhart, furst, hill, & krathwohl, 1956; also see anderson et al., 2001, for a revised version), it is important to determine whether using openbook tests, rather than the traditional closed-book tests, would defeat this goal. reasons that open-book tests may be inferior to closed-book tests there are several reasons to assume that open-book tests are inferior to closedbook tests in building knowledge. first, when a test is offered in an open-book format, students may view the impending test as easy because the answers will be available during the test, and thus, they may devote less effort in studying. in fact, sparrow, liu, and wegner (2011) showed that when participants were informed that they would be allowed to look up the answers when they take the test later on, they tended to show on the test lower recall of the target information but enhanced recall of where to find the target information. accordingly, expecting an open-book test may discourage students from putting enough effort to build their own knowledge. the second reason that open-book tests might be inferior to closed-book tests in building knowledge is based on the notion that retrieving information from memory enhances learning and facilitates future retrieval. according to bjork and bjork (1992), the likelihood of remembering depends on both storage strength and retrieval strength. the storage strength, commonly referred to as memory strength, reflects the amount of learning and determines the availability of information in memory. the storage strength can be increased by repeatedly studying the material. another component of remembering is retrieval strength, which is how easily memory can be accessed. bjork and bjork assumed that building strong memory (storage strength) by repeatedly studying is not sufficient to guarantee successful memory retrieval because in addition to creating strong memory, one needs to practice retrieving memory in order to make it easy to access. take an old telephone number for example. it may be still available in memory due to repeated use in the past; however, one may experience difficulty retrieving it because it has not been used recently. another aspect of this theory is that these two types of strength are related such that increasing retrieval strength by repeatedly retrieving memory would also increase storage strength. in effect, retrieving memory acts as another opportunity to learn. what is critical to the issue of test format and building knowledge is that there is an inverse relationship between the ease of retrieval and the amount of increment in storage strength, such that easy retrieval would result in a small increment in storage strength, whereas, difficult retrieval would result in a large increment in storage strength. in testing effect: a further examination of open-book and closed-book test formats 22 journal of effective teaching in higher education, vol. 1, no. 1 line with this principle, bjork (1999) argued that training procedures with easy access to correct responses would slow down the growth of storage strength. based on these notions, it is reasonable to assume that open-book tests, with information readily available during retrieval, would be less efficient in building knowledge because looking up an answer is easier than retrieving an answer from memory, resulting in a smaller increment in storage strength. reasons that open-book tests may not be inferior to closed-book tests there are indications that making a test open-book may not defeat the purpose of building knowledge. first, the format of the test, open-book or closed-book, may not matter as long as the test questions are sufficiently difficult to promote elaborative or deep-levels of processing. based on the literature on the levels of processing model (craik, 2002; craik & lockhart, 1972; craik & tulving, 1975), it is clear that regardless of one’s intention to learn, processing information at a deep level (i.e., semantic level) would produce durable memory compared to processing information at a shallow level (i.e., perceptual level). for instance, recognition performance was higher when participants were asked to think whether a given word would fit into a sentence frame (e.g., “he met _____ in the street?”) than when participants were asked to judge whether a word was in capital letters (craik & tulving, 1975). these orienting questions can be conceptualized as questions on a test; and based on the results of these studies, it is reasonable to assume that if the questions on the open-book test direct one to process the answers at a deep level, durable memories can be formed. research on the testing effect also supports the notion that open-book tests may not be inferior to closed-book tests with regard to long-term retention of information. the testing effect is a phenomenon that simply taking tests increases long-term retention of information better than re-studying does. there is substantial evidence showing that the testing effect is a robust phenomenon (see roediger & karpicke, 2006a, for an extensive review) that can be observed with a variety of tests (such as free recall, cued recall, and recognition), materials (such as word lists, lists of paired-associates, and prose materials), and settings (such as laboratory and educational settings). there are at least three recently published meta-analyses on the testing effect (adesope, trevisan, & sundararajan, 2017; pan & rickard, 2018; rowland, 2014), and all confirmed that the testing effect is a powerful method of increasing learning. furthermore, pan and rickard (2018) showed that the testing effect is robust even when the format and information being tested on the initial test are different from those on the final test, showing the transfer of learning effect. in addition, some studies have shown that the testing effect is similar between open-book and closed-book tests, even though contrary results have also been reported. in a review of the literature comparing open-book and closed-book test formats, durning and colleagues (2016) reported that among the five studies that examined the testing effect between these formats, four showed that the testing effect was similar between these test formats (agarwal, karpicke, kang, roediger, & mcdermott, 2008; agarwal & roediger, 2011; gharib, phillips, & mathew, 2012; pauker, 1974), whereas, one showed that the testing effect was lower in the open-book format than in the closed book format testing effect: a further examination of open-book and closed-book test formats 23 journal of effective teaching in higher education, vol. 1, no. 1 (moore & jensen, 2007). however, the results are not as straight forward as it appears. although pauker (1974) reported that the testing effect was similar between the open-book and closed book conditions, for the students who started out the semester at the bottom third of the class, open-book tests resulted in lower final test scores than closed-book tests. agarwal and roediger (2011) also showed that expecting an open-book test reduced time spent on learning, resulting in lower performance on transfer questions on the final test that probed deep learning. furthermore, moore and jensen (2007) showed that performance on the final test was lower when previous tests were open-book relative to closed-book, and that expecting open-book tests resulted in poor academic behaviors, such as skipping lectures and help-sessions as well as not completing extra assignments. in sum, the literature on testing effect showed mixed results that indicate a complex interaction of various factors (such as motivation) that goes beyond the issue of test format. another reason that open-book tests may not be inferior to closed-book tests is based on the issue of information re-exposure. open-book tests re-expose students to information successfully retrieved as well as information that was not retrieved, providing opportunities for additional learning. moreover, an open-book test allows students to access correct answers, which would, in turn, minimize commission errors. butler, marsh, goode, and roediger (2006) showed that when participants made commission errors on the first test and did not receive feedback with the correct answers, they made the same errors on the final test. it is, therefore, possible that open-book tests are superior to closed-book tests by limiting the number of commission errors, preventing long-term retention of incorrect information. note, however, that a recent meta-analysis by adesope et al. (2017) showed that providing or not providing feedback was not a significant moderator of the effect size associated with the testing effect. their explanation was that taking a test itself is cognitively challenging enough to produce the benefit of testing, and therefore, availability of feedback may not produce an additional benefit. nevertheless, these researchers also cautioned that their finding may be due to unidentified confounding variables given that there are studies that showed the benefit of providing feedback in addition to testing (e.g., metcalfe, kornell, & finn, 2009; pashler, cepeda, wixted, & rohrer, 2005). with this caveat, adesope et al. concluded that further research is needed to investigate the effect of providing feedback on the testing effect. rationale and methods of present study it is intriguing to consider a possibility that open-book tests are as effective as closed-book tests in building knowledge, particularly for traditional educators who regard tests only as assessment tools. because this issue has an important implication for educational practice, it warrants further investigation. as noted above, the results of research are conflicted as to whether open-book and closedbook tests differ in promoting long-term retention of studied material, with some studies showing that these two formats are similar (e.g., agarwal et al., 2008; agarwal & roediger, 2011; pauker, 1974; gharib et al., 2012) and the other studies showing that open-book tests are less effective than closed-book tests, testing effect: a further examination of open-book and closed-book test formats 24 journal of effective teaching in higher education, vol. 1, no. 1 particularly, in promoting deep learning (e.g., agarwal & roediger, 2011; pauker, 1974). however, not all learning situations require deep learning, such as during the early stage of learning. the question, then, is whether the lack of difference between open-book and closed-book tests can be generalized to non-text-based materials. although text-based materials clearly have educational relevance, building knowledge also involves acquiring facts and building vocabularies. accordingly, in the present study, we decided to investigate the effect of test format using swahili-english word pairs in a testing effect paradigm similar to agarwal et al. (2008). it is possible that the lack of difference between open-book and closed-book tests reported by agarwal et al. and other researchers in the past was based on the use of text materials. because text materials are well-organized and meaningful, it is possible that participants could easily engage in elaborative processing regardless of whether the initial test was open-book or closed-book, thereby reducing the difference between the two formats. the question is whether the effect of test format would emerge when the study material is less organized and less meaningful. the present study consisted of three phases. during phase i, participants were asked to study a list of swahili-english word pairs. during phase ii, participants were provided with either additional opportunity to study (re-study) or they received an initial test of the material. during phase iii, which occurred one week later, participants received a final test that assessed their memory for the word pairs. the critical manipulation was that for half of the participants, the initial test was an open-book test, and for the other half of the participants, the initial test was a closed-book test. finally, in addition to the format of the test, the type of the test (cued recall versus recognition) was manipulated during phase ii when the initial test was administered. this manipulation was based on the assumption that the difference between open-book and closed-book tests may appear when an initial test is not sufficiently difficult to promote a deep level of processing (i.e., recognition). that is, when a test is sufficiently challenging to promote long-term learning, the test format may not matter, whereas when a test is not challenging enough, a closedbook test may show superiority over an open-book test. in sum, participants in the present study were asked to learn a list of swahili-english word pairs followed by a cued-recall or recognition initial test, which was administered in an open-book or closed-book format. also, there was a control condition in which participants were asked to re-study the list instead of taking the initial test. the final recognition test was administered one week later to examine whether the test format (open-book versus closed-book) as well as the test type (cued recall versus recognition) made a difference in long-term retention of the study material. methods participants participants were 39 male and 136 female undergraduate students attending introductory psychology courses at a public university in the midwestern region of testing effect: a further examination of open-book and closed-book test formats 25 journal of effective teaching in higher education, vol. 1, no. 1 the united states. they were offered extra course credit for their participation. an equal number (n = 35) of participants were randomly assigned to five betweensubjects conditions based on the format (i.e., open-book versus closed-book) and type (i.e., cued recall versus recognition) of the initial quiz, plus a study twice control condition: (1) closed-book recognition, (2) closed-book cued recall, (3) open-book recognition, (4) open-book cued recall, and (5) study 2x (twice). table 1 summarized the conditions and procedure. note that for the rest of this paper, the initial test will be referred to as the ‘initial quiz’ and the final test will be referred to as the ‘final test’ to make it easy to differentiate these tests. the study was conducted in accordance with approval given by the institutional review board at the university where participants were tested. materials fifty swahili-english word pairs were selected from the nelson and dunlosky (1994) norms (see appendix for examples). based on the normative proportion of correct recall, these pairs were high in difficulty (ranged from .07 to .18). a powerpoint presentation was used to present these pairs, one at a time in the middle of the computer screen in lowercase letters at the rate of one pair per 5 s. the order of the pairs was randomized once, and the same order was used for all participants. the initial quiz was either cued recall or recognition. these quizzes were constructed by randomly selecting 35 swahili words from the study list. using 35 words rather than 50 words left 15 words for assessing performance on the oneweek delayed final test when there was no initial quiz. for the cued recall quiz, these words were presented on a sheet of paper in a random order with a blank space next to each word for a response (e.g., theluji _____). for the recognition quiz, these words were presented with four alternative choices of possible english translation for each swahili word. the distractor choices were randomly selected from english translation of the other words in the study list, making it associativerecognition rather than item-recognition. associative recognition was used to increase retrieval effort because unlike item-recognition, associative recognition depends more on retrieval than familiarity (e.g., hockley & consoli, 1999; westerman, 2001). each swahili word was presented with four choices next to it, randomly ordered. table 1 conditions and procedure conditions session 1 session 2 (one week later) phase i phase ii (initial quiz or study) phase iii (final test) closed-book cued recall study cued recall (closed-book) recognition (closed-book) closed-book recognition study recognition (closed-book) recognition (closed-book) open-book cued recall study cued recall (open-book) recognition (closed-book) open-book recognition study recognition (open-book) recognition (closed-book) study 2x study study recognition (closed-book) testing effect: a further examination of open-book and closed-book test formats 26 journal of effective teaching in higher education, vol. 1, no. 1 note: during phase i, participants studied 50 swahili-english word pairs. during phase ii, participants were quizzed on 35 word pairs or re-studied 50 word pairs. during phase iii, participants were tested on all 50 word pairs. the one-week delayed final test consisted of all 50 swahili words from the study list, each presented with four alternative choices of possible english translation. a recognition test was used as the final test based on the assumption that recognition would provide more sensitive assessment of learning than recall. for the 35 swahili words that were initially quizzed, the recognition items were the same as those in the initial recognition quiz. for the 15 swahili words that were not initially quizzed, recognition items were constructed using english translation of other studied words as distractors. these items were randomly ordered once, and the same order was used across participants. in addition, a sheet of paper with the study list (in the order of presentation) was used for the open-book quiz. a sheet of paper with random two-digit number was used for the filler task (see below), with a stopwatch used to time the duration of the filler task. procedure small groups up to four individuals were tested in two sessions with one-week delay between the sessions. during session 1, phase i and ii of the study were administered. during phase i, participants were instructed to study a list of 50 swahili-english word pairs. the presentation of the study list was repeated three times to ensure that participants learned the list at a sufficiently high level to avoid a floor effect. they were not informed about the format of the initial quiz nor the final test they took after one-week delay. following the study phase, participants were asked to perform a filler task for 2 minutes, crossing out the numbers divisible by three. the filler task was administered to eliminate a recency effect. following the filler task, phase ii commenced, and participants in the initially quizzed condition completed the self-paced initial quiz. participants in the cued recall condition were asked to write an english equivalent of each swahili word, and participants in the recognition condition were asked to select the correct english equivalent of each swahili word among the four alternatives. these quizzes were administered in a closed-book or open-book format; that is, participants in the closed-book condition were asked to take the quiz without looking up the answers whereas participants in the open-book condition were given a sheet of paper with the study list and were allowed to look up the answers. no feedback was given after completing the quiz. in the study 2x condition, instead of taking the initial quiz, participants re-studied all 50 swahili-english word pairs printed on a sheet of paper one more time. at the end of the first session, participants in all conditions were told that at the second session seven days later, they would be asked about the swahili words they studied during the first session. phase iii of the study was administered in session 2, which was scheduled one week after session 1. during phase iii, participants took the self-paced final recognition test, with instruction to select the correct english equivalent for each testing effect: a further examination of open-book and closed-book test formats 27 journal of effective teaching in higher education, vol. 1, no. 1 swahili word. however, before taking the test, participants were asked to make a global judgment of learning (jol), predicting how many words they would be able to correctly recognize among 50 swahili words. participants were asked to make a jol rating because it is possible that the format and type of the initial quiz would influence their metacognitive judgments. results the significance level was set at .05, and unless otherwise specified, two-tailed tests were performed. the dependent measures were the proportion of correct responses on the initial quiz, the proportion of correct responses on the final recognition test, and jol, which was converted to a proportion. note that on the initial quiz, participants were quizzed on 35 word pairs out of 50 word pairs they studied whereas on the final recognition test and jol, participants were tested on all 50 word pairs. because the goal of the study was to investigate the effect of the initial quiz format (i.e., open-book versus closed-book) as well as the initial quiz type (i.e., cued recall versus recognition) on final test performance, the proportion of correct responses on the final recognition test was compared across the conditions. a one-way analysis of variance (anova) on the final test performance for all 50 word pairs indicated that the difference among the conditions was significant, f (4, 170) = 2.95, mse = 0.03, p = .02, ηp2 = .07. as shown in table 2, least significant difference (lsd) tests revealed that the study 2x condition (m = .45, sd = .13) showed significantly lower performance than the open-book cued recall (m = .57, sd = .18), open-book recognition (m = .54, sd = .19), and closed-book cued recall (m = .57, sd = .17) conditions. the difference between the study 2x condition (m = .45, sd = .13) and the closed-book recognition condition (m = .52, sd = .14) did not reach statistical significance with a two-tailed test (p = .09); however, based on a priori hypothesis that the initial testing would produce a testing effect, the difference was significant with a one-tailed test (p = .04). no other difference was significant, indicating that the testing effect was similar across the quizzed conditions.1 in order to gain insight as to how the testing effect had occurred, different groups of word pairs were analyzed. because 35 word pairs out of 50 studied word pairs were quizzed on the initial quiz, these quizzed word pairs should show a testing effect on the final test, and that the effect should be similar across the quizzed conditions. this expectation was confirmed. a one-way anova on the final test performance for 35 words that were quizzed on the initial quiz indicated that the difference among the conditions was significant, f (4, 170) = 4.36, mse = 0.03, p = .002, ηp2 = .09. as shown in table 2, lsd tests indicated that the study 2x condition (m = .44, sd = .14) showed significantly lower performance than the open-book cued recall (m = .59, sd = .18), open-book recognition (m = .55, sd = .19), closed-book cued recall (m = .57, sd = .18), and closed-book recognition (m = .53, sd = .15) conditions. no other comparison was significant, indicating that the testing effect was similar across the quizzed conditions. testing effect: a further examination of open-book and closed-book test formats 28 journal of effective teaching in higher education, vol. 1, no. 1 next, 15 word pairs that were not quizzed on the initial quiz were analyzed to test a possibility that these non-quizzed words also showed a testing effect. this expectation was not confirmed. a one-way anova on final recognition performance for 15 word pairs that were not quizzed on the initial quiz indicated that the difference among the conditions was not significant, f (4, 170) = 0.83, mse = 0.04, p = .51, indicating that the testing effect was not observed with these words (see table 2). table 2 mean proportion of correct responses on the initial quiz and on the final recognition test as a function of initially quizzed conditions and different groups of words on final test initial quiz final test conditions (35 word pairs) all 50 word pairs 35 quizzed word pairs 15 not quizzed word pairs correct on initial quiz incorrect on initial quiz jol closed-book cued recall m sd .36a .19 .58a .18 .57a .18 .58a .19 .82a .16 .48a .18 .29a .19 closed-book recognition m sd .73b .18 .52a .14 .53a .15 .51a .14 .61b .14 .28b .21 .33a .20 open-book cued recall m sd .99c .01 .57a .18 .59a .18 .53a .24 .59b .18 .28a .14 open-book recognition m sd .98c .04 .54a .19 .55a .19 .52a .24 .55b .19 .30a .17 study 2x m .45b .44b .50a .29a sd .14 .14 .18 .19 note: for each column, significant differences (p < .05) are indicated by different superscript letters. all comparisons were based on two-tailed tests except for the comparison between the closed-book recognition and study 2x conditions for the final test with all 50 word pairs, which was based on a one-tailed test. to investigate whether correctly responding on the initial quiz influenced the final test performance, the next analysis examined what proportion of the correct responses on the initial quiz was also correct on the final test. because the initial quiz was not administered in the study 2x condition, this condition was excluded from the analysis. a one-way anova indicated that the difference among the conditions was significant, f (3, 136) = 19.23, mse = 0.03, p < .001, ηp2 = .30. as shown in table 2, lsd tests revealed that the closed-book cued condition (m = .82, sd = .16) showed significantly higher performance than the open-book cued recall (m = .59, sd = .18), open-book recognition (m = .55, sd = .18), and closed-book recognition (m = .61, sd = .13) conditions, indicating that successfully recalling without looking up answers on the initial quiz (i.e., closed-book cued recall) led to a higher success on the final test. no other comparison was significant. an analysis was also performed to investigate whether the initial quiz had a positive testing effect: a further examination of open-book and closed-book test formats 29 journal of effective teaching in higher education, vol. 1, no. 1 effect on the final test performance for those word pairs for which incorrect responses (i.e., omission and commission errors) were given on the initial quiz. however, in the open-book cued recall and recognition conditions, the number of such errors was close to zero, and therefore, a t-test was conducted to compare the closed-book cued recall and recognition conditions. final recognition performance on the words participants failed to remember on the initial quiz indicated that the closed-book cued recall condition (m = .48, sd = .18) produced higher performance than the closed-book recognition condition (m = .28, sd = .21), t (66) = 4.23, p < .001, d = 1.02, on the final test. this finding shows that the participants were able to recognize items on the final tests that they did not remember on the initial quiz, and that cued recall on the initial quiz showed higher likelihood of such success than recognition. next, jol was analyzed to investigate whether metacognition was influenced by the format and type of the initial quiz. a one-way anova on jol indicated that the difference among the conditions was not significant, f (4, 170) = 0.45, mse = 0.03, p = .77. in all conditions, participants predicted that they would be able to correctly recognize 30 percent from 50 swahili words they studied one week earlier (m = .30, sd = .18). lastly, a one-way anova on the initial quiz performance showed that the difference among the conditions was significant, f (3, 136) = 175.01, mse = 0.02, p < .001, ηp2 = .79. as mentioned, the number of errors was closed to zero for the open-book cured recall (m = .99, sd = .01) and recognition tests (m = .98, sd = .04), and lsd tests showed that the difference was non-significant between these two conditions. all the other comparisons were significant, indicating that recognition was easier than cued recall when the initial quiz was closed-book. discussion the present study examined whether open-book and closed-book formats of an initial quiz would influence performance on a delayed final recognition test when swahili-english word pairs, as opposed to text materials, are used as study material. as mentioned in the introduction section, in the online learning environment, testing is challenging because it is difficult, if not impossible, to make a test cheat proof.2 however, such a concern may only arise when a test is considered only as an assessment tool, consistent with the traditional view of education. in contrast, if the focus is shifted toward a long-term goal of education (i.e., building knowledge), it may not matter whether a test is open-book or closedbook because what matters most is whether students will develop knowledge of whatever they are learning. in fact, the past studies investigating the effect of test format using the testing effect paradigm showed that both open-book and closedbook formats produced similar performance on the final closed-book test, indicating that both formats would promote long-term memory (e.g., agarwal et al., 2008; agarwal & roediger, 2011; pauker, 1974; gharib et al., 2012). however, a concern with these studies is that they used text-based materials, and therefore, it is possible that the lack of difference between open-book and closed-book formats was simply reflecting the fact that the materials were well-organized and testing effect: a further examination of open-book and closed-book test formats 30 journal of effective teaching in higher education, vol. 1, no. 1 meaningful, thereby conducive to elaborative processing regardless of the format of the initial quiz. the present study, therefore, used a simple material (i.e., swahilienglish word pairs) in order to reduce the possible influence of other factors (such as organization, meaningfulness, and familiarity). furthermore, the present study investigated the type of initial quiz (cued recall versus recognition) because the difference between the test formats might emerge when the initial quiz is not sufficiently difficult to induce a deep level of processing (i.e., recognition). there were several major findings. first, the final test performance was similar among the four quizzed conditions, regardless of whether all 50 words from the study list (i.e., the whole list) or 35 words that were quizzed on the initial quiz were examined. the results, therefore, indicated that neither the initial quiz format (i.e., open-book versus closed-book) nor the initial quiz type (i.e., cued recall versus recognition) influenced the final test performance. second, the study 2x control condition produced significantly lower final recognition performance than the four quizzed conditions for both the whole list and the quizzed words. this finding is consistent with a phenomenon referred to as the testing effect (e.g., roediger & karpicke, 2006), indicating that taking a quiz increases long-term retention of information relative to re-studying. this notion is further supported by the finding that the testing effect was only found with the quizzed words, as opposed to the non-quizzed words (i.e., 15 control words that were not quizzed on the initial quiz). third, when the final test performance was conditionalized on correct responses on the initial quiz, the closed-book cued recall condition showed higher performance than the other quizzed conditions. this finding indicates that there is an advantage in making the initial quiz closed-book cued recall in line with the notion of desirable difficulty by bjork (1994, 1999), which contends that difficult processing, whether it is at encoding or retrieval, benefits long-term retention. finally, jols of the final test was similar across conditions: in all conditions, including the study 2x condition, participants predicted that they would be able to correctly recognize about 30% of 50 swahili words on the final test. because actual performance was higher than 30% in all conditions, participants underestimated their performance. although jol ratings were not accurate in all conditions, it is important to note that jols were similar between the study 2x and quizzed conditions, indicating that jols did not show a testing effect. this finding is consistent with the result of other studies (e.g., agarwal et al., 2008; roediger & karpicke, 2006b) that participants are not sensitive to the beneficial effect of testing. furthermore, note that in many studies, participants show a tendency to overestimate rather than underestimate their performance (e.g., agarwal et al., 2008; ayton & mcclelland, 1997). it is not clear the reason that participants underestimated their performance in the present study. a possibility is that the material they learned in this study was unfamiliar foreign vocabularies, and therefore, participants did not have high confidence. overall, the results of the present study showed that although performance on the testing effect: a further examination of open-book and closed-book test formats 31 journal of effective teaching in higher education, vol. 1, no. 1 initial quiz was predictably higher in the open-book condition than in the closedbook condition, this advantage vanished after one-week delay, resulting in similar final test performance between the open-book and closed-book conditions. furthermore, the final test performance did not show the effect of initial quiz type (i.e., cued recall and recognition). taken together, an open-book test format is as effective as a closed-book test format in promoting long-term retention, even when the study material is not text materials. in addition, the initial quiz could be either cued recall or recognition, indicating that in building knowledge, the act of being quizzed is the critical factor across quiz formats and quiz types. why is it that the initial quiz format did not make a difference in the amount of testing effect? it is possible that even when the quiz was open-book, participants took it as if it was a closed-book quiz. if so, there was no functional difference between the two quiz formats. although this is plausible, it is unlikely based on the initial quiz performance, showing that performance was almost 100% on the open-book quiz whereas performance was much lower on the closed-book quiz. alternatively, it is possible that what is critical is the opportunity to process the words deeply regardless of whether it is done by open-book or closed-book quizzes. it appears that both open-book and closed-book quizzes acted as orienting questions in an incidental learning study (e.g., craik & tulving, 1975) inducing a deep level of processing. further studies are needed to examine the processes that are induced by open-book and closed-book formats, which led to higher final test performance for the quizzed conditions relative to the study 2x condition. although the final test performance was similar across the quizzed conditions, there was an indication that a closed-book format with cued recall on the initial quiz may yield some advantage. as mentioned, consistent with the notion of desirable difficulty (bjork, 1994, 1999), the final test performance was higher in the closedbook cued recall condition than in other quizzed conditions when the words that were correctly responded on the initial quiz were examined. this finding indicated that difficult retrieval would produce greater increment in storage strength than easy retrieval, in line with the notion that there is a negative correlation between storage and retrieval strength (bjork & bjork, 1992). it is possible that the difficulty of retrieval may ultimately prevail when memory is tested after a retention interval longer than one week. in fact, meta-analyses by pan and rickard (2018) and rowland (2014) showed that retrieval effort or elaborative retrieval was a moderating variable that increased the testing effect. note however that a metaanalysis by adesope et al. (2017) showed that the testing effect was greater with less effort such that the testing effect was greater with recognition tests than with cued-recall tests. accordingly, the role of retrieval difficulty in the testing effect is not clear, and therefore, further investigation is needed. another interesting finding was that when the words that were not correctly responded on the initial quiz (i.e., omission and commission errors) were examined, the closed-book cued recall condition showed higher final recognition performance than the closed-book recognition condition. however, this finding may simply reflect the fact that recognition is easier than recall, such that the increase in performance testing effect: a further examination of open-book and closed-book test formats 32 journal of effective teaching in higher education, vol. 1, no. 1 from the initial quiz to the final test was as a result of comparing a difficult initial quiz (i.e., cued recall) and an easy final test (i.e., recognition). another issue that needs to be investigated in the future is the nature of the final test. in the present study, a recognition test was used to test final performance. however, it is possible that the results might be different when a cued recall test is used as the final test. the type of final test might be important because in the present study, the advantage of initial cued recall over recognition may have been masked due to a mismatch between the initial and final tests in the cued recall conditions. that is, a transfer appropriate processing (morris, bransford, & franks, 1977) is confounded between the initial cued recall and recognition quiz conditions. however, ultimately how knowledge will be tested would be critically dependent on the type of knowledge and how it is used. for instance, learning a foreign language would require more than just recognizing vocabulary words and their english equivalents. in this sense, a decision to adopt a particular test type may require a domain specific approach. nevertheless, the present study showed that it is premature to assume that a test is inferior just because it is administered using an open-book test format. in conclusion, the results of the present study showed that the testing effect is similar between open-book and closed-book quizzes, even when the study material is an unrelated set of swahili-english pairs, as opposed to well-organized and meaningful text materials. furthermore, initial quiz type, cued recall or recognition, did not make a difference. these results, therefore, supported the notion that an open-book test is not necessarily inferior to a closed-book test in promoting longterm retention. however, there was an indication that making the initial quiz difficult, as in the closed-book cued recall condition, has an added advantage, which needs to be investigated in future research. practical implications for classroom practice given that testing promotes long-term retention of studied material, coupled with the present result that there is no difference between open-book and closed-book formats, how can these research findings be translated to classroom practice? on the one hand, it seems to be safe to replace traditional closed-book tests with open-book tests if the purpose of education is to build knowledge. on the other hand, such practice would represent a radical departure from the traditional method, and as such, it may be difficult to convince teachers in traditional face-toface classrooms to adopt such new practice. however, the situation may be different for teachers in online classes because these teachers may be more experienced with non-traditional methods. nevertheless, doing away with closedbook tests entirely may not be practical because it would be difficult if not impossible to document the outcome of education unless there is an assessment.3 based on these considerations, a preferable approach would be to mix open-book tests and closed-book tests within a particular course (or a curriculum) with the former being used for building knowledge and the latter being used for assessment. in line with this recommendation, the second author of this paper began using this hybrid approach in a 300-level course at his university. in this face-to-face class, testing effect: a further examination of open-book and closed-book test formats 33 journal of effective teaching in higher education, vol. 1, no. 1 he implemented open-book quizzes (multiple-choice questions), which he allowed students to take multiple times. he used these open-book quizzes for building knowledge, but for assessment, he used closed-book exams. the results were encouraging. for the semester when he implemented the quizzes, there was a modest increase in the average score of the closed-book exams (9% improvement overall) compared to prior semesters. although the increase was not dramatic, the results were encouraging, given that the quizzes were not mandatory and did not contribute to the course grade. with this modest success, this hybrid approach should be explored further, particularly in online classes. we argue that this hybrid approach is especially relevant for online classes for at least two reasons. first, in these classes, in-person test proctoring can be expensive and time-consuming, and second, as mentioned earlier, other methods of minimizing the potential for cheating on online closed-book tests have limitations. accordingly, by incorporating open-book quizzes, the number of closed-book tests can be reduced without jeopardizing the development of long-term knowledge. in conclusion, any laboratory finding requires extensive translational research before it becomes useful in practice. however, the results of the present study showed that open-book tests are a viable method of building knowledge, which we regard as one of the important goals of education. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references adesope, o. o., trevisan, d. a., & sundararajan, n. (2017). rethinking the use of tests: a meta-analysis of practice testing. review of educational research, 87, 659-701. anderson, l. w., krathwohl, d. r., airasian, p. w., cruikshank, k. a., mayer, r. e., pintrich, p. r., raths, j., & wittrock, m. c. (2001). a taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives (complete edition). new york: longman. alessio, h. m., malay, n., maurer, k., bailer, a. j., & rubin, b. (2017). examining the effect of proctoring on online test scores. online learning, 21, 146-161. agarwal, p. k., karpicke, j. d., kang, s. h. k., roediger, h. l. iii, & mcdermott, k. b. (2008). examining the testing effect with openand closed-book tests. applied cognitive psychology, 22, 861-876. agarwal, p. k., & roediger iii, h. l. (2011). expectancy of an open-book test decreases performance on a delayed closed-book test. memory, 19, 836-852. ayton, p., & mcclelland, a. g. r. (1997). how real is overconfidence? journal of behavioral decision making, 10, 279–285. bjork, r. a. (1999). assessing our own competence: heuristics and illusions. in d. gopher & a. koriat (eds.), attention and performance xvii. cognitive regulation of performance: interaction of theory and application (pp. 435459). cambridge, ma: mit press. testing effect: a further examination of open-book and closed-book test formats 34 journal of effective teaching in higher education, vol. 1, no. 1 bjork, r. a. (1994). memory and metamemory considerations in the training of human beings. in j. metcalfe & a. p shimamura (eds.), metacognition: knowing about knowing (pp. 185-205). cambridge, ma: mit press. bjork, r. a., & bjork, e. l. (1992). a new theory of disuse and an old theory of stimulus fluctuation. in a. f. healy, s. m. kosslyn, & r. m. shiffrin (eds.), from learning processes to cognitive processes: essays in honor of william k. estes (pp. 35-67). hillsdale, nj: lawrence erlbaum associates, inc. bloom, b. s., engelhart, m. d., furst, e. j., hill, w. h., & krathwohl, d. r. (1956). taxonomy of educational objectives. the classification of educational goals. handbook 1 cognitive domain. new york: logmans, green and co. butler, a. c., marsh, e. j., goode, m. k., & roediger, h. l., iii. (2006). when additional multiple-choice lures aid versus hinder later memory. applied cognitive psychology, 20, 941-956. christe, b. (2003). designing online courses to discourage dishonesty. educause quarterly, 26, 54-58. cluskey jr, g. r., ehlen, c. r., & raiborn, m. h. (2011). thwarting online exam cheating without proctor supervision. journal of academic and business ethics, 4, 1-7. craik, f. i. (2002). levels of processing: past, present... and future? memory, 10, 305-318. craik, f. i., & lockhart, r. s. (1972). levels of processing: a framework for memory research. journal of verbal learning & verbal behavior, 11, 671684. craik, f. i., & tulving, e. (1975). depth of processing and the retention of words in episodic memory. journal of experimental psychology: general, 104, 268294. durning, s. j., dong, t., ratcliffe, t., schuwirth, l., artino, a. r., boulet, j. r., & eva, k. (2016). comparing open-book and closed-book examinations: a systematic review. academic medicine, 91, 583-599. gharib, a., phillips, w., & mathew, n. (2012). cheat sheet or open-book? a comparison of the effects of exam types on performance, retention, and anxiety. psychology research, 2, 469-478. grijalva, t. c., nowell, c., & kerkvliet, j. (2006). academic honesty and online courses. college student journal, 40, 180-185. hockley, w. e., & consoli, a. (1999). familiarity and recollection in item and associative recognition. memory & cognition, 27, 657-664. metcalfe, j., kornell, n., & finn, b. (2009). delayed versus immediate feedback in children’s and adults’ vocabulary learning. memory & cognition, 37, 10771087. michael, t. b., & williams, m. a. (2013). student equity: discouraging cheating in online courses. administrative issues journal, 3(2). retrieved from https://www.swosu.edu/academics/aij/2013/v3i2/michael-williams.pdf moore, r., & jensen, p. a. (2007). do open-book exams impede long-term learning in introductory biology courses? journal of college science teaching, 36, 4649. morris, c. d., bransford, j. d., & franks, j. j. (1977). levels of processing versus transfer appropriate processing. journal of verbal learning and verbal behavior, 16, 519-533. https://www.swosu.edu/academics/aij/2013/v3i2/michael-williams.pdf testing effect: a further examination of open-book and closed-book test formats 35 journal of effective teaching in higher education, vol. 1, no. 1 nelson, t. o., & dunlosky, j. (1994). norms of paired-associate recall during multitrial learning of swahili-english translation equivalents. memory, 2, 325335. owens, h. s. (2016). cheating within online assessments: a comparison of cheating behaviors in proctored and unproctored environments (doctoral dissertation). available from proquest dissertations and theses database. (umi no. 3737143). pan, s. c., & rickard, t. c. (2018). transfer of test-enhanced learning: metaanalytic review and synthesis. psychological bulletin, 144, 710-756. pashler, h., cepeda, n. j., wixted, j. t., & rohrer, d. (2005). when does feedback facilitate learning of words? journal of experimental psychology: learning, memory, and cognition, 31, 3-8. pauker, j. d. (1974). effect of open book examinations on test performance in an undergraduate child psychology course. teaching of psychology, 1, 71-73. roediger, h. l. iii, & karpicke, j. d. (2006a). the power of testing memory: basic research and implications for educational practice. perspectives on psychological science, 1, 181-210. roediger, h. l. iii, & karpicke, j. d. (2006b). test-enhanced learning: taking memory tests improves long-term retention. psychological science, 17, 249255. rowe, n. c. (2004). cheating in online student assessment: beyond plagiarism. online journal of distance learning administration, 7. retrieved from http://www.westga.edu/~distance/ojdla/summer72/rowe72.html rowland, c. a. (2014). the effect of testing versus restudy on retention: a metaanalytic review of the testing effect. psychological bulletin, 140, 1432-1463. sparrow, b., liu, j., & wegner, d. m. (2011). google effects on memory: cognitive consequences of having information at our fingertips. science, 333, 476-478. westerman, d. l. (2001). the role of familiarity in item recognition, associative recognition, and plurality recognition on self-paced and speeded tests. journal of experimental psychology: learning, memory, and cognition, 27, 723-732. footnotes 1 we also conducted a 2 (quiz type: cued recall and recognition) x 2 (quiz format: openbook and closed-book) anova without the study 2x condition. the results showed that no effect was significant. 2 we acknowledge that the problem of cheating is not limited to online tests. 3 we also acknowledge that assessments can be conducted in a variety of ways including (and not limited to) using closed-book tests. http://www.westga.edu/%7edistance/ojdla/summer72/rowe72.html testing effect: a further examination of open-book and closed-book test formats 36 journal of effective teaching in higher education, vol. 1, no. 1 appendix examples of swahili-english word pairs swahili word english equivalent jani leaf chura frog lozi almond nira yoke wakili agent yatima orphan bahasha envelope chaza oyster fumbo mystery microsoft word cook et al final version.docx journal of effective teaching in higher education, vol. 1, no. 2 using text mining and data mining techniques for applied learning assessment jessica cook, university of north carolina wilmington cuixian chen, university of north carolina wilmington, chenc@uncw.edu angelia reid-griffin, university of north carolina wilmington abstract. in a society where first hand work experience is greatly valued many universities or institutions of higher education have designed their quality enhancement plan (qep) to address student applied learning. this paper is the results of a university’s qep plan, called experiencing transformative education through applied learning or eteal. this paper will highlight the research that was conducted using text mining and data mining techniques to analyze a dataset of 672 student evaluations collected from 40 different applied learning courses from fall 2013 to spring 2015, in order to evaluate the impact on instructional practice and student learning. text mining techniques are applied through the nvivo text mining software to find the 100 most frequent terms to create a document-term matrix in excel. then, the document-term matrix is merged with the manual interpretation scores received to create the applied learning assessment data. lastly, data mining techniques are applied to evaluate the performance, including random forest, k-nearest neighbors, support vector machines (with linear and radial kernel), and 5-fold cross-validation. our results show that the proposed text mining and data mining approach can provide prediction rates of around 67% to 85%, while the decision fusion approach can provide an improvement of 69% to 86%. our study demonstrates that automatic quantitative analysis of student evaluations can be an effective approach to applied learning assessment. keywords: text mining, data mining, applied learning assessment, short answer questions, student evaluation text mining, sometimes referred to as text data mining, is the action of obtaining patterns or interesting knowledge from text-based documents. text mining can become very complicated and time-consuming when original text documents lack structure (tan, 1999). the process of text mining consists of two main phases: refining the original text documents to some chosen form and extracting knowledge from the text documents through patterns (delgado, 2002). mining a text-based document after it has been refined to the chosen form finds critical patterns and relationships seen across all documents (tan, 1999). student evaluations of teaching (set) are seen from two different perspectives: informal and formal (scriven, 1967; stake, n.d.). formal evaluation is done by conducting standardized testing of students. this study will focus heavily on the informal perspective of student evaluations. an informal student evaluation is perceived as informal based on its casual observation and subjective bias/judgment. the reason for focus on informal perspective is provide a personalize approach to evaluating course and learning objective that educator using text mining and data mining techniques for applied learning assessment 61 journal of effective teaching in higher education, vol. 1, no. 2 emphasizes in the course. one study revealed that most educators feared that scorers would not pay adequate attention to the characteristics that the educator deems most important. the best teachers continually utilize what is learned from student evaluations to improve their teaching practices (ramsden, 2003). educational evaluations: sets student evaluation of teaching (set) is a common tool used in numerous institutions of higher education to provide evidence of teaching effectiveness and reflection of students’ learning (wagner, rieger, & voorvelt, 2016). in terms of evaluating effectiveness of teaching, students are positioned to be intuitively knowledgeable of information on actual effectiveness. oftentimes students lack information on how to assess teaching effectiveness which is problematic when set scores are used for promotions and contracts renewals (boring, 2017). because sets have a history of being biased in areas of race/ethnicity and gender (boring, 2017; wagner, rieger, & voorvelt, 2016) is the reason why this study focusing on the informal perspective of sets and how it measures the instructional practices and student learning in 40 different applied learning courses from fall 2013 to spring 2015. the sets are typically designed as a rating form for students to rank the instructor and/or course based on numerous specific characteristics of effectiveness (uttl, white, gonzalez, 2017). they are administered at the end of the semester and are often optional for student to complete. however, some higher education institutions have implemented required completion of sets to improve response rates of the instrument (boring, 2017). students have been reported of not showing any objection to filling out evaluations and are often honest (douglas & carroll, 1987; gal & gal, 2014). according to gal and gal’s (2014) study on knowledge bias of student evaluations in an economics course, students believed their role in evaluating courses is special, as it positions them to provide feedback that is reflective of the teaching quality. other claims of student evaluations being reliable than other teacher effectiveness measures, such as peer ratings and observations is supported by other researchers (heller & clay, 1993; fike, fike, & zhang (2015). the research by galbraith, merrill, & kline’s (2012) on the student evaluation of teaching effectiveness (sete) validity in measuring student outcomes in business classes, found “student rating of learning outcome problem from different statistical perspectives, resulted in a high degree of consistency with respect to validity (p. 368). recent studies on set indicate that students demonstrate some bias in terms of teacher background and behaviors rather than quality of course instruction (wagner, rieger, & voorvelt, 2016). often, students believe that evaluations are effective and that teachers value the input from student evaluations and do not rank based on personal biases or grade. students also believe that evaluations are a critical way to improve/adjust faculty teaching methods and improve quality of course (scriven, 1967; wagner, rieger, & voorvelt, 2016). a study found that students prefer mid-semester evaluations over those that take place at the end of the semester, because they are able to see the change being applied from the evaluations (abott et al., 1990). using text mining and data mining techniques for applied learning assessment 62 journal of effective teaching in higher education, vol. 1, no. 2 student evaluation is a strong measure of how effective a faculty’s teaching practices are and can reflect student learning (beleche, fairris, & marks, 2012). it is important that students are motivated to actively participate and provide honest input that contributes to the success of evaluation systems. research conducted by chen et al. (2003) found that students consider improvement in the implemented teaching practices to be the most attractive outcome of the evaluation system. the second most attractive outcome is seeing change to improve the course content. chen et al. (2003) finds that students are more motivated to participate in evaluations when they believe their feedback is seen as meaningful. the quality of student evaluations is essential in obtaining meaningful student feedback to provide areas of opportunity to improve teaching methods and effectiveness. teaching and learning in higher education are inextricably and elaborately linked. good teachers continually use what they learn from their students to improve their own practice. the assumption that the primary goal of teaching is to improve student learning and teaching, leads to the argument that a reflective approach would be effective. thus, student evaluation is an essential aspect to improve faculty teaching methods and course content leading to increased student learning (ramsden, 2003). the role sets have in providing feedback in higher education aids in student satisfaction of course and retention and completion at the institution. when student course evaluations are matched with student specific objectives for courses there can be positive, statistically significant associations between students’ learning and the course evaluation (beleche, fairris, & marks, 2012). as there are numerous studies that have been conducted on student evaluation of faculty instruction using quantitative, meta-analyses practices (evans, 2013; uttl, white, & gonzalez, 2017; zhao & gallant, 2012), this study provides a timely and unique approach to using text mining and data mining techniques in examining the validity and reliability of student evaluations in accessing teacher effectiveness and student learning. taking into account previous literature on student evaluation we are able to use this practice in providing a thorough critique of assessment and gain insight on the extent to which the classroom environment or other related factors affect student evaluation of faculty instruction in the applied learning courses (zhao & gallant, 2012). abd-elrahman et al. (2010) considers the automatic text mining techniques as a good method to investigate student course evaluation in a qualitative, open-ended manner. these techniques aim to identify unrevealed aspects affecting student learning process and develop a quantitative tool for these aspects. after preprocessing, each evaluation is categorized with the negative and positive comments made regarding the course. then text mining is utilized to create two major groups: one for positive words and one for negative words. this study shows that the written responses from the student’s courses can be analyzed through text mining to understand the effectiveness of teaching. using text mining and data mining techniques for applied learning assessment 63 journal of effective teaching in higher education, vol. 1, no. 2 applied learning assessment like many universities, the higher education institution in this study aims to engage students in the research process or in creative scholarly activity in meaningful ways. following such commitment, among the quality enhancement plan, the experiencing transformative education through applied learning (eteal) program has been initiated to have a positive impact on student learning with an applied learning experience in three areas: critical thinking, thoughtful expression, and inquiry. the eteal supported pedagogy initiatives offer many great opportunities, resource and funds for faculty to explore innovative pedagogies in applied learning, and/or implement high-impact pedagogies in new disciplines, promote the involvement of undergraduate students in faculties’ scholarly and creativity work, and enrich the interdisciplinary collaboration across campus. since fall 2013, over a hundred eteal-supported initiatives have been implemented campus wide. enormous efforts have been made to promote applied learning among departments of traditional sciences, social sciences, humanities, arts, etc. after three years since the eteal initiatives started, it is pressing to review the assessment data to evaluate its impact on instructional practice and student learning. such data includes faculty survey, student survey, and scores of student artifacts from eteal-supported initiatives, as well as from non-eteal supported exploration beyond the classroom (ebc) activities in classes, projects, internships, study-abroad and etc. therefore, it is critical to formulate and evaluate the influence of applied learning experiences to determine analytically whether the eteal-supported applied learning techniques are effective in comparison to noneteal exploration beyond the classroom experiences. the statistical analysis outcomes will provide scientific evidence of student learning and program effectiveness, with assessment foci on both student learning outcome and program outcome. by comparing the assessment data from eteal and non-eteal exploration beyond the classroom (ebc), we aim to determine whether there is any statistically significant difference among eteal and ebc in terms of student learning and program effectiveness, and discover the related factors if such a difference exists. specially, applied learning courses at the university are assessed by student evaluations completed throughout the length of the course. at the start of the semester, students complete an intention reflection articulating their expectations, the purpose, and/or goals of the experience in terms of personal educational development (ebc 1). upon completion of the course, students submit a final reflection synthesizing: (i) knowledge drawn from their coursework to address challenges involved in the experience (ebc 2), (ii) the impact of the experience on personal educational development (ebc 3a), and (iii) the impact of the experience in the profession or in the field of study (ebc 3b). a sample of guidance for both the initial reflection and the final reflection for eteal supported pedagogy initiatives is illustrated in appendix a. in order to evaluate the impact on instructional practice and student learning, all student evaluations are manually interpreted and scored on a scale 0 to 4 based on a provided scoring rubric by scorers who must first go through a mandatory training process. a sample of the scoring rubric is illustrated in appendix b. for the training, using text mining and data mining techniques for applied learning assessment 64 journal of effective teaching in higher education, vol. 1, no. 2 each scorer is required to participate in two parts of an event. the first part consists of a five-hour session during which the rubric is reviewed and each person begins scoring with a partner. the second part consists of completion of the scoring of student work on one’s own, this can last up to approximately 5 hours. at the end of the event, each scorer is asked to provide feedback regarding the process and rubric for continual improvement in the scoring process. it is mandatory that scorers attend at least one event, but are invited to attend as many as they like. scorers are allowed to pick from events covering topics including student critical thinking skills, student-written communication skills, and student evaluation skills. it is noted that the human manual scoring process is very complicated and time consuming. it is believed that the more in-depth evaluation leads to a better understanding of instructional practice and student learning outcomes. therefore, even though intensive human manual scoring to analyze student evaluations is important, automatic quantitative analysis of student evaluation can be an alternative efficient approach to analyze students’ text response. in this paper, both text mining and data mining techniques are investigated on students’ text-based course evaluation to identify unrevealed aspects of instructional practice and student learning and develop a quantification tool to formulate and evaluate the influence of applied learning experiences. data gathering and cleaning all original pdf files are provided by the institution’s general education assessment office. these pdf files cover student evaluations of applied learning experiences from both eteal and ebc courses, consisting of scanned handwritten documents and scanned typed documents. as a pre-processing step, the answers from the original scanned pdf files are transcribed into .txt files by three students and a faculty member, which proved to be a very time-consuming process. many issues come with the case of scanned handwritten files, including sloppy handwriting and faded handwriting. for some files, human judgment is used to best make out the writing that is illegible or has become extremely faded after being scanned in as a pdf file. in the case of scanned typed files, a pdf file converter is used to convert the pdf files into a document that could easily be copied and pasted into a .txt file. the pdf file converter can only convert one file at a time, so it is a time-consuming process. a drawback of using the pdf file converter is spelling and grammatical errors that are caused by the converter program being used. to fix these errors, each file is manually checked for spelling and grammar mistakes. a few of the original pdf files are not used because they are written in a different language (e.g. in french). our final dataset consists of 672 student evaluation .txt files. all student evaluations are collected from the cycle of two academic years (fall 2013-spring 2015). among them, part of the student evaluations are collected from 21 different courses for the academic year of fall 2013spring 2014, while the rest are from 19 different courses during the academic year of summer 2014 -spring 2015. these using text mining and data mining techniques for applied learning assessment 65 journal of effective teaching in higher education, vol. 1, no. 2 courses include traditional sciences, social sciences, humanities, and arts. all, but four, of the applied learning courses covered are eteal-supported courses. figure 1: each pie chart shows the distribution of the scores all student evaluations received for each category of ebc 1, ebc 2, ebc 3a, and ebc 3b. the notation used above shows the score received, and a count of the student evaluations that received that score. for example, (1, 97) represents 97 student evaluations receive a score of 1. as mentioned previously, student evaluations are scored on four separate criteria. in this study, pie charts for ebc 1, ebc 2, ebc 3a, ebc 3b are created respectively to better visualize the manual perceived scores, which are shown in figure 1. it is clear that most student evaluations are scored with a 1 or 2. it is noted that the student evaluations are scored based on human manual scoring of the provided scoring rubric. also, when a student evaluation is scored as 0, this can either imply the student evaluation was written poorly or that no student evaluation is ever received. using text mining and data mining techniques for applied learning assessment 66 journal of effective teaching in higher education, vol. 1, no. 2 methodology of text mining techniques text mining techniques are performed on the cleaned student evaluation data, by using both the statistical programming language of r and nvivo. the characteristics, including strengths and weakness of both software will be compared in detail below. challenges on text mining with r in our text mining investigation, we begin analyzing student evaluations in the statistical computing software r. in order to perform text mining analysis, 21 required packages must first be installed in r. a directory is set up where all original .txt files are loaded into r to begin analysis. next, the files are loaded from the directory as the source of the files making up the corpus. the function corpus in r uploads all the files. to begin with, these files are named original documents so they can later be used for comparison. to prepare for text analysis, more preprocessing of the documents needs to be done. first, all numbers and punctuation are removed from the original documents. when numbers, punctuations, and stop words are removed, they are replaced by a white space where the word, number, or symbol have originally been in the corpus. in order to remove this white space, we use a command in r that strips any extra remaining white space. all text characters in the documents are converted to lowercase characters. next, all english stop words are removed. english stop words are common words found in the english language. there exist 174 common stop words in the english language. before moving forward to stemming and stem completion, it is important to check all student evaluations for spelling errors. this may seem trivial, yet it is essential in order to yield an accurate result. correcting a spelling error in r requires a new line of code for each correction. to avoid this, all evaluations are manually checked for spelling errors and updated. table 1: term frequency table least frequent terms 1 2 3 4 5 6 7 8 9 10 3256 1127 682 444 334 244 195 172 138 114 most frequent terms 1690 1737 1743 1755 1956 2006 2216 2467 2625 3190 1 1 1 1 1 1 1 1 1 1 note: this table provides a brief summary into the frequency distribution of terms appearing in the student evaluations for the least frequent and most frequent terms by r. for example, this table is interpreted as there ae 3,256 terms that only appear once in the evaluations. on the other extreme end, there is one term that appears 3,190 times. lastly, in the pre-processing phase, stemming and stem completion is done on all documents. stemming is the process of reducing words to their base form. sometimes a word is stemmed to a phrase that is not a base form itself and stem completion completes the phrase back to a base form. stem completion uses a dictionary created by the original documents. for example, “argue”, “argued”, “argues”, and “arguing” reduce to the stem “argu”. then, r refers to the dictionary using text mining and data mining techniques for applied learning assessment 67 journal of effective teaching in higher education, vol. 1, no. 2 to stem complete “argu” back to a base form. at this stage, r tends to have difficulties with stemming and stem completion. to list a couple examples, “many” is stemmed into “maniac” and “really” is stemmed into “reallife.” outputting both the results after stemming and the stem completion into an excel file allows us to compare with the original documents and find where mistakes are made. figure 2: a bar graph of the 20 most frequent terms by r. this graph allows for a better visualization of the terms that are appearing most frequently throughout the evaluations. a document-term matrix (dtm) is obtained, as a matrix with the 672 student evaluations as the rows and the terms found in the student evaluations as the columns. each cell in the matrix is a frequency/count. inspecting the dtm shows the distribution of the terms and the percentage of sparsity found in the matrix. to obtain the distribution of term frequencies, the dtm must be converted into a regular matrix and then the sum of columns is taken. ordering the term frequencies allows a list to easily be created showing the least and most frequent terms, with a sample shown in table 1, for easier interpretation. at first inspection of the dtm, it is revealed that the dtm contains 98% sparsity. sparsity refers to infrequent terms occurring in the student evaluations. for example, in table 1, there are 3,256 terms that only appear once in the student evaluations. r has a function to remove using text mining and data mining techniques for applied learning assessment 68 journal of effective teaching in higher education, vol. 1, no. 2 a selection of sparse terms. after sparse terms are removed, the dtm now contains 37% sparsity, which is a huge improvement. figure 3: a visual representation of the word cloud produced by r. after all the pre-processing is done and the dtm is created, we can analyze the data to get better visual representations. figure 2 represents the 20 most frequent words found in all student evaluations. it appears that some of these words may be deemed insignificant for what we are interested in (e.g., will, also, and et al.). to better understand the significance of these terms, it is important to look at the context in which the terms are used. a better visualization of the most frequent terms is shown in the word cloud produced by r in figure 3. alternative text mining with nvivo as previously mentioned, r lacks an approach to efficiently looking at the context of a term and performed poorly in the pre-processing stage of stemming and stem completion. these fallbacks in r steer us away from the software, and introduce us to the qualitative analysis software called nvivo. nvivo has the ability to create a flowchart of a term over all the student evaluations. this allows a deeper look at the context of the term in question over that achieved by human manual interpretation, when categorizing a term as significant or insignificant. nvivo also has the ability to produce word clouds with a chosen number of significant terms faster and more efficiently than r. nvivo has the option to group together like terms (stemming and stem completion) by just simply clicking a button. this fixes using text mining and data mining techniques for applied learning assessment 69 journal of effective teaching in higher education, vol. 1, no. 2 the mistakes caused by r, after running stemming and stem completion on all student evaluations. hereafter, nvivo is used as the primary software for all textual based analysis. the nvivo option of word count query is used to produce the word cloud shown in figure 4, which includes the 100 most frequent words that appear in all student evaluations. in the word cloud, different words are depicted with different color and font size. the font size is directly related to the frequency of the 100 most frequent words found. nvivo also has a word count query that allows us to search for each of the most frequent words across all student reflections and provides a count of how many times these words appears in that reflection. this function was used to generate the document-term matrix for all data mining classification techniques conducted below. figure 4: a word cloud produced quickly and efficiently by nvivo that shows the 100 most frequent words found in all student evaluations. as mentioned above, we use the “stemmed words” option to group together the like terms so no one term appears more than once in the word cloud. we see very similar results when comparing the word clouds produced by r and nvivo. table 2 illustrates a count of the term and the terms that are grouped together under a given term to present a better idea on how nvivo performs in stemming and stem completion. it is important to note that the “experience” is included in the word cloud by r, whereas “experiments” is shown in the nvivo word cloud. note that in table 2, “experience” is grouped together with the term “experiments”. nvivo is able to quickly produce a flow chart of the context of the term used in all evaluations. however, it is a large flowchart that requires time to shift through. hereafter, it is assumed that all most frequent terms are used in a positive and significant context. using text mining and data mining techniques for applied learning assessment 70 journal of effective teaching in higher education, vol. 1, no. 2 table 2: word frequency provided by nvivo word count similar words learn 3181 learn, learned, learning, learns works 2641 work, worked, working, workings, works experiments 2288 experience, experiences, experiment, experimented, experimenting, experiments helps 2000 help, helped, helpful, helping, helps note: nvivo software has a “stemmed words” option that groups together like terms when calculating word frequency. r is able to produce a document-term matrix (dtm) which is a matrix including a count of the number of times a term appears in each of the student evaluations. nvivo has a similar function under its word search query option. this option allows the user to input the term and nvivo produces a list of all evaluations the term is located and a count of the term is located in that individual evaluation. a drawback of this option in nvivo is that nvivo does not include the evaluations where the term is not found in. as a result of this, a difficulty is created when generating a larger matrix that includes all evaluations as rows and the most frequent terms as columns. a count of the most frequent term is included in each cell. another drawback of this nvivo option is that it only allows the user to search for one word at a time. due to these drawbacks, the matrix had to be entered manually, which proves to be a time-consuming process. once this document-term matrix is created for 100 most frequent terms and a matrix is created in excel to lay out how many times these 100 most frequent words occur in each individual student evaluation, the data mining techniques are used to further analyze the student evaluations quantitatively. methodology of data mining techniques in this paper, after the document-term matrix (dtm) is obtained from the text mining techniques, we first consider four different classifiers to access the classification, including random forest, k-nearest neighbors (knn), and support vector machines (svm) with linear kernel and radial kernel. suppose there are n observations: (𝑥#,𝑦#), (𝑥(,𝑦(),…,(𝑥*,𝑦*), where 𝑥+ ∈ 𝑅., and 𝑦+ ∈ {0,1} representing a score of low or high. random forest is a statistical classifier developed by breiman (2001). random forest builds a number of decorrelated decision trees, and then uses the mode of the predictions from the decision trees as the model output. breiman (2001) suggests that as the number of the trees in the forest increases, the generalization error of random forest converges almost surely to a limit. thus, the weak but unbiased decision trees produce relative efficient using text mining and data mining techniques for applied learning assessment 71 journal of effective teaching in higher education, vol. 1, no. 2 predictions. in order to decorrelate the trees, a random sample of predictors is chosen from the full set of predictors at each split in a tree. let n be the number of data observations and let d be the number of predictors to be selected. suppose the number of decision trees to be built is 𝑁4, with minimum node size 𝑛*6.7. the algorithm for random forest for classification is as following: (1) draw a bootstrap sample of size n from the training observations. (2) with the bootstrapped data, grow a tree by repeating the following steps: i. select m variables at random from the d predictors. ii. find the best variable among the m selected variables, as well as the best split point for classification. iii. split the node into two descendent nodes with each node resulting from the classification. iv. stop growing the tree when the minimum node size 𝑛*6.7 is reached for all terminal nodes. (3) repeat steps (1) and (2) 𝑁4 times to obtain the a collection of trees {𝑇+}+9# :; . (4) for any input vector 𝑥, let 𝐺+(𝑥) be the class prediction from the 𝑖th random forest tree. the prediction from the random forest is 𝐺(𝑥)= mode of {𝐺+}+9# :; . the k-nearest neighbors classifier is memory-based. given a query point, say 𝑥>, assume we find k training points closest in distance to the given point 𝑥> among n observations, say (𝑥# ∗,𝑦# ∗), (𝑥( ∗,𝑦( ∗), (𝑥@ ∗ ,𝑦@ ∗ ), which satisfies that a| 𝑥# ∗ − 𝑥>|a ≤ a|𝑥( ∗ − 𝑥>|a ≤ ⋯ ≤ a|𝑥*∗ − 𝑥>|a, where a|∙|a represents the euclidean distance. let 𝐻(𝑥>) be the class prediction for the query point 𝑥>. then 𝐻(𝑥>) = mode of { 𝑦# ∗, 𝑦( ∗,… , 𝑦@ ∗ }, by the majority vote of its k nearest training points. k can take any integer within the sample size. to determine the best k for our experiments, 5-fold cross-validation (cv) is applied to choose a k value in order to minimize the cross-validation prediction error: 𝑚𝑖𝑛@𝐶𝑉𝑒𝑟𝑟𝑜𝑟(𝐾). the technique of support vector machines is considered as a method of classifying the data into the newly created high/low variable. in the binary setting, suppose there are n observations: (𝑥#,𝑦#), (𝑥(,𝑦(),…,(𝑥*,𝑦*), where 𝑥+ ∈ 𝑅., and 𝑦+ ∈ {−1,1}. svm aims to find a separable hyperplane that best separates the two classes and produces a lower error of classification. the optimal hyperplane is the hyperplane that passes the farthest from all training observations with a maximum margin separating hyperplane 𝑤 ∙ 𝑥 + 𝑏 = 0 in the feature space through a quadratic programming: 𝑚𝑖𝑛s,t # ( ||𝑤||( + 𝐶 ∑*+9# 𝜉+, subject to 𝑦+(𝑤 ∙ 𝑥 + 𝑏) ≥ 1 − 𝜉+ and 𝜉+ ≥ 0, ⩝ 𝑖, where || ∙ || represents the 𝑙( vector norm, 𝑤 is the normal vector to the hyperplane and the parameter t ‖s‖ determines the offset of the hyperplane from the origin. the constant 𝐶 > 0 is a “cost” parameter which must be carefully tune for the “counts” of feature points ∑ 𝜉+ * +9# which lie within the margin or on the wrong side of the hyperplane. in the case that the data is linearly separable, we select two parallel hyperplanes that separate the two classes of data. when selecting these two parallel hyperplanes, we want the distance between them maximized. using text mining and data mining techniques for applied learning assessment 72 journal of effective teaching in higher education, vol. 1, no. 2 geometrically, the distance between the two parallel hyperplanes defined above is represented as ( ‖s‖ . we want to maximize the distance between the two parallel hyperplanes which is achieved by minimizing ‖𝑤‖. to extend the method of svm to cases in which the data is not linearly separable, we can consider a kernel function 𝜅(𝑥,𝑥]): 𝜅(𝑥,𝑥]) = 𝛷(𝑥) ∙ 𝛷(𝑥]) = 𝑒𝑥𝑝 (−𝛾||𝑥 − 𝑥]||(), ∀ 𝑥,𝑥] ∈ 𝑅., where 𝛾 is a positive constant, 𝛷 is a function to map the training examples into some feature space ƒ such that 𝛷: 𝑅. ↦ ƒ. furthermore, 5-fold cross-validation is considered to evaluate the performance of these four different classifiers. in 5-fold cross-validation, the dataset is randomly divided into five folds with approximately equal size. then one fold is held out and treated as a validation set, while the remaining four folds are treated as a training set to build a classification system. this procedure is repeated five times, with a different fold of observations treated as a validation set until all folds have been used as a test dataset. ensemble learning to improve classification performance to further improve the overall performance, ensemble learning by fusing multiple predictive decisions to make a final decision could be a potential way to get a more robust decision (polikar, 2006; moreno-seco et al, 2006). for example, the classifier ensembles with different combination techniques have been widely explored in recent years. these methods have been shown to potentially reduce the error rate in the classification tasks compared to an individual classifier in a broad range of applications. in the decision fusion with ensemble-based systems, it is important to consider the diversity of decisions to be fused, with respect to diverse classifiers. in our analysis, we consider fusing independent classifiers among random forest, k-nearest neighbors, and support vector machines with radial kernel. for the 𝑖th observation 𝑥+, let 𝐺(𝑥+), 𝐻(𝑥+), 𝐽(𝑥+) be the class predictions from random forest, k-nearest neighbors, and support vector machines respectively. then the final class predictions for ensemble learning is given by 𝐹(𝑥+) = mode of {𝐺(𝑥+), 𝐻(𝑥+), 𝐽(𝑥+)}. results of data mining techniques using the document-term matrix (dtm), data mining techniques can now be applied to classify these student evaluations into two categories of high or low. all data mining techniques are performed in r. in order to achieve that, first, a new response variable of high and low is created for ebc1, based on both the distribution of scores shown in the pie charts in figure 1 and the criteria of the applied learning scoring rubric shown in appendix b. then repeat this procedure for the rest of ebc 2, ebc 3a, and ebc 3b, creating four new response variables. with these factors in mind, all student evaluations that received a score of 2 or below using text mining and data mining techniques for applied learning assessment 73 journal of effective teaching in higher education, vol. 1, no. 2 are classified in the low class and student evaluations receiving a score of a 3 or 4 are classified as high. the aforementioned document-term matrix (dtm) is then merged with the student evaluation corresponding ebc 1, ebc 2, ebc 3a, and ebc 3b scores with four response variables of high or low. random forest, k-nearest neighbors, and support vector machines (with either linear or radial kernel) are all considered as the classification techniques using the 5-fold cross-validation to analyze the free-style text of student evaluations. each classification method is run on all ebc category of high or low that a student evaluation receives (ebc 1, ebc 2, ebc 3a, and ebc 3b) respectively. the overall accuracies are shown in table 3. comparing to the ebc2, ebc3a and ebc3b, the overall prediction accuracy for student reflection ebc 1 scores indicates the lowest accuracy of around 65-68%. on the other hand, the ebc 2 student evaluation scores have a stronger accuracy of 78-82%. ebc 3a scores hold around a 70-73% overall prediction accuracy, and ebc 3b scores have the highest overall prediction accuracy of 83-85%. a graphical visualization of the overall accuracies produced from each method of classification is shown in figure 5. from figure 5, it is interesting that the classification results from ebc1 illustrate outliers consistently for all four classification methods applied. the possible reason why the overall accuracies for ebc 1 is lower than ebc 2, ebc 3a and ebc 3b is that students’ expectations, the purpose, and/or goals of the experience in terms of personal educational development can be at a larger range of terms used and/or less associated to the terms from the document-term matrix. table 3: prediction rates table. ebc 1 ebc 2 ebc 3a ebc 3b random forest 0.671 0.813 0.723 0.840 k-nearest neighbors 0.674 0.809 0.728 0.845 svm with linear kernel 0.657 0.784 0.701 0.838 svm with radial kernel 0.668 0.807 0.707 0.845 ensemble learning 0.685 0.817 0.707 0.856 note: 5-fold cross-validation is run on the four models of random forest, knearest neighbors, and support vector machines with either linear or radial kernel. each classification method is run four different times using each of the ebc scores among ebc1, ebc2, ebc2a, and ebc3b) as the response variable. the overall accuracies from the five folds is shown in the table. ensemble learning accuracies after the method of decision fusion is used to combine the classifier methods of knn, random forest, and svm with a radial kernel. note: for knn, a different k (the number of neighbors) is chosen each time after running 5-fold cross-validation to determine the best k. decision fusion is further considered as a method aiming to improve the classification performance. random forest, k-nearest neighbors, and support vector machines with radial kernel are used in the decision fusion approach. it is important in decision fusion that all methods are independent of one another and an odd number of methods are used so that there are no ties created. it is revealed using text mining and data mining techniques for applied learning assessment 74 journal of effective teaching in higher education, vol. 1, no. 2 that the majority of the misclassified observations in the data are the observations with a ground truth of high that are misclassified as low. decision fusion is once again run on all ebc scores. the accuracies are illustrated in table 4. overall prediction accuracies are improved slightly. it is shown that the decision fusion approach results in higher accuracy than any of individual classifiers for ebc1, ebc2, and ebc3b. for ebc3a, even though the decision fusion approach does not lead to the highest accuracy, the accuracy is still competitive comparing to the individual classifiers. these results indicate that decision fusion with ensemble is effective in this text mining task. figure 5: boxplots to the overall accuracies for the four classification methods of random forest, k-nearest neighbors (knn), and support vector machines with linear and radial kernel using ebc1, ebc2, ebc3a, and ebc3b as response variables. conclusion and recommendations our dataset from two academic years over fall 2013-spring 2015 is studied systematically to provide the preliminary analysis results. the results of our experiments show that text mining is a promising technique to analyze the openended free-style text based student reflections quantitatively, and automatically. text mining can be an effective way to analyze text responses and how a student evaluation will score quantitatively which reveals how well a course and/or instructor is performing. analyzing these text based student evaluations into using text mining and data mining techniques for applied learning assessment 75 journal of effective teaching in higher education, vol. 1, no. 2 quantitative information allows one to gain additional insights to evaluate student performance, instructor performance, and course performance. one can also gain a deeper understanding of individual schools at the university, departments, and majors as well, and eventually evaluate the impact on the implemented instructional practice and the student learning outcomes. data mining classification methods show promising overall prediction accuracies for all ebc scores of student evaluations. decision fusion is a method implemented to further improve classification accuracies, and while it does so, no strong change is made in the overall prediction accuracy of student reflections that are classified or misclassified by the three classification methods used. while accuracies held steady after decision fusion, the method does allow for a deeper understanding of the data being analyzed. decision fusion reveals the individual student evaluations that are misclassified which can reveal what departments or majors have the most incorrect classifications and the performance or motivation of the students in those departments or majors on evaluations. providing faculty and administrators with this information to be able to interpret results more critically and be able to make rational and fair decisions in terms of teaching effectiveness (hou,lee, & gunzenhauser, 2017). analyzing student evaluations by terms is a significant way to analyze the applied learning program as a whole, as well as the effectiveness of the applied learning program on overall student learning. analyzing text based student evaluations provide additional insights. for example, a higher ebc score is associated with greater student performance and/or understanding of the course. then we associate that these motivated students will provide a more in-depth and meaningful evaluation. on the other side, this well-trained system of text mining and data mining can be applied to the future applied learning student evaluations. in this case, the new student evaluations will be pre-processed in the same way of text mining as described previously and fed into the data mining system. consequently, the scores of ebc1, ebc2, ebc3a, and ebc3b will be produced automatically. this framework can be an efficient way to provide quick preliminary analysis on the program evaluation of instructional practice and student learning. abd-elrahman et al. (2010) support the claim of automatic text mining techniques as a good method to investigate open-ended student course evaluation. this study extends the original two categories of negative and positive comments made regarding the course for each evaluation into four categories of benchmark, milestone-i, milestone-ii, and capstone. data mining techniques are incorporated in our study for quantitative analysis. the promising overall prediction accuracies demonstrate that such automatic quantitative analysis of student evaluations can be an effective approach to applied learning assessment. hou, lee and gunzenhauswer (2017) support the claim of these evaluations as instruments that can support transformative decisions in improving quality of teaching. by valuing the contributions of students and faculty in this process could help in preventing erroneous decisions based on some biased student feedback. using text mining and data mining techniques for applied learning assessment 76 journal of effective teaching in higher education, vol. 1, no. 2 references abbott, r.d., wulff, d.h., nyquist, j.d., ropp, v.a. & hess, c.w. (1990). satisfaction with processes of collecting student opinions about instruction: the student perspective. journal of educational psychology, 82, 201-206. abd-elrahman, a., andreu, m., & abbott, t. (2010). using text data mining techniques for understanding free-style question answers in course evaluation forms. research in higher education journal, 9, 11–21. beleche, t., fairris, d., & marks, m. (2012). do course evaluations truly reflect student learning? evidence from an objectively graded post-test. economics of education review. 31, 709-19. boring, a. (2017). gender biases in student evaluations of teaching. journal of public economics. 145. 27-41. chen, y., & hoshower, l. b. (2003). student evaluation of teaching effectiveness: an assessment of student perception and motivation. assessment & evaluation in higher education, 28(1), 71-88. cohen, k. b., & hunter, l. (2008). getting started in text mining. plos computational biology, 4(1), e20. doi:10. 1371/journal.pcbi.0040020 cronbach, l. j. (1963). course improvement through evaluation. teachers college record, 64, 672-683. fike, d. s., fike, r., & zhang, s. (2015). teacher qualities valued by students: a pilot validation of the teacher qualities (t-q) instrument. academy of educational leadership journal, 19(3), 115-125. gal, y., & gal, a. (2014). knowledge bias: is there a link between students' feedback and the grades they expect to get from the lecturers they have evaluated? a case study of israeli colleges. journal of the knowledge economy, 5(3), 597-615. doi:10.1007/s13132-014-0188-5 galbraith, c. s., merrill, g. b., & kline, d. m. (2012). are student evaluations of teaching effectiveness valid for measuring student learning outcomes in business related classes? a neural network and bayesian analyses. research in higher education, 53(3), 353-374. doi:http://dx.doi.org.liblink.uncw.edu/10.1007/s11162-011-9229-0 delgado, m., matrín-bautista, m.j., sánchez, d., & vila, m.a. (2002, september). mining text data: special features and patterns. proceedings of eps exploratory workshop on pattern detection and discovery in data mining, london. douglas, p.d. & carroll, s.r. (1987). faculty evaluations: are college students influences by differential purposes? college student journal, 21(4). evans, c. (2013). making sense of assessment feedback in. higher education. review of educational research. 83(1), 70-120. doi: 10.3102/0034654312474350 hastie, t., tibshirani, r. & friedman, j. (2016). the elements of statistical learning: data mining, inference, and prediction. new york: springer. heller, h.w., & clay, r.j. (1993). predictors of teaching effectiveness: the efficacy of various standards to predict the success of graduates from a teacher education program. ers spectrum, 11, 7-11. hou, y., lee, c., & gunzenhauser, m.g. (2017). student evaluation of teaching as a using text mining and data mining techniques for applied learning assessment 77 journal of effective teaching in higher education, vol. 1, no. 2 disciplinary mechanism: a foucauldian analysis. the review of higher education, 40(3), 325-352. marsh, h.w. (1984). students’ evaluations of university teaching: dimensionality, reliability, validity, potential biases and utility. journal of educational psychology, 76(5), 707-754. marsh, h.w. (1987). students’ evaluation of university teaching: research findings, methodological issues and directions for future research. international journal of educational research, 11(2), 253-388. moreno-seco, f., inesta, j. m., de león, p. j. p., and micó, l. (2006). comparison of classifier fusion methods for classification in pattern recognition tasks. in structural, syntactic, and statistical pattern recognition (pp. 705-713). berlin, germany: springer-verlag. polikar, r. (2006). ensemble based systems in decision making. ieee circuits and systems magazine, 6(3), 21-45. ramsden, p. (2003). learning to teach in higher education (2nd ed.) london: routledge falmer. scriven, m. (1967). the methodology of evaluation. in r.w. tyler, r. m. gagné, & m. scriven (eds.), perspectives of curriculum evaluation. chicago, il: rand mcnally. stake, r. e. (n.d.). the coutenance of educational evaluation. center for instructional research and curriculum evaluation, university of illinois. tan, a. (1999). text mining: the state of the art and the challenges. in proceedings, pakdd ’99 workshop on knowledge discovery from advanced databases (kdad ’99). uttl, b., white, c., & gonzalez, d. (2017). meta-analysis of faculty’s teaching effectiveness: student evaluation of teaching ratings and student learning are not related. studies in educational evaluation, 54, 22-42. wagner, n., rieger, m., & voorvelt, k. (2016). gender, ethnicity and teaching evaluations: evidence from mixed teaching teams. economics of education review, 54, 79-94. zhao, j. & gallant, d. (2012). student evaluation of instruction in higher education: exploring issues of validity and reliability. assessment & evaluation in higher education, 37(2), 227-235. using text mining and data mining techniques for applied learning assessment 78 journal of effective teaching in higher education, vol. 1, no. 2 appendix a here is a sample of guidance for both the initial reflection and the final reflection for eteal supported pedagogy initiatives: (note: slo represents student learning outcome) intention reflection prompts (at the start of the semester): explains in depth the purpose for engaging in the experience and directly links it to personal educational development through expected educational outcomes. your intention reflection will be typed in 1 page, by answering the following questions. (slo1) a. articulate your expectation from, and the reason for participation in this project. (slo1) b. examine and explain what you hope to gain from this experience in terms of personal, educational, and/or career goals. (slo1) c. explain what statistical methods, presentation and communication skills, and use of technology you hope to learn from this project. (slo1) d. explain the impact (on others or on the field) that you hope to make through this project. final reflection prompts (upon completion of the course): (slo2) summarize the relevant theories, ideas and skills you were able to apply in this project. (slo2) demonstrate how you apply what you learnt from other courses to complete this project. (slo3) summarize your team work and/or leadership experience through this project. (slo3) over the several presentation occasions, explain how you address questions from people of different fields, and lessons you have learn to improve your oral presentation and communication skills. (slo3) summarize the significance of your work in the field from this project. (slo3) summarize a personal challenge and how you overcome it during this project. using text mining and data mining techniques for applied learning assessment 79 journal of effective teaching in higher education, vol. 1, no. 2 appendix b here is a sample of the scoring rubric for human manual scoring (revised october 2014): microsoft word 58-article text-854-1-6-20210407.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.58 understanding student perceptions of class participation kelly wonder, university of wisconsin-eau claire, wonderka@uwec.edu abstract: located in the philippines, this study explored the perceptions secondary english education majors at a public university in metro manila, philippines, have of classroom participation. through an open-ended questionnaire and a series of interviews, the researcher found participants defined class participation as “recitation,” such as student responses to teacher-generated questions with limited interaction with peers. participants also reported a strong power distance relationship between student and teacher and suggested a fear of failure restricted their class participation. despite these challenges, participants expected their participation to be assessed. when examined through the lens of willingness to communicate, student perceptions were contrary to the skills expected of 21st century learners. understanding the student perception of class participation has the potential to mediate perceptual mismatches and create more effective learning environments. keywords: participation, higher education, willingness to communicate, student engagement, english as a foreign language the development of 21st century skills is at the forefront of education professionals in the philippines due to the influence of the association of south east asian nations (asean). in 2017, the republic of the philippines department of education issued order do42, s. 2017 regarding the adoption of new professional standards of teachers “brought about by various national and global frameworks such as the k to 12 reform, asean integration, globalization and the changing character of the 21st century learners…”. the idea of nurturing human growth through critical thinking, collaboration, creativity, and motivation is present in several frameworks that define a 21st century education (lai & viering, 2012; wagner, 2010). however, ananiadou and claro (2009) suggested insufficient training and difficulty assessing contributions limit use. orhan göksü and kurt (2017) confirmed that in the context of 21st century skills, teachers often teach the way they learned. therefore, the schema of the teacher may not align with the schema of the learner and may be more likely to create generational conflict. orhan göksü and kurt (2017) also suggested cognitive skills were most widely used by pre-service teachers while autonomous skills were least used. knowledge based activities were less likely to transfer to external tasks. when the classroom environment intentionally builds skills and ways of thinking that are meaningful outside of the classroom, participation in class aids in the development of 21st century skills. whether problem-based, discovery-based, inquiry-based, project-based, or casebased, a constructivist environment requires learners to actively participate (hood cattaneo, 2017). learners “build on their prior knowledge, think critically, reflect, and present information individually and in a small group” (p. 146). effective teaching through active participation is created through intentional practices that understanding student perceptions of class participation 41 journal of effective teaching in higher education, vol. 4, no. 1 integrate content with 21st century skills while building an environment where learners are willing to communicate. students who participated in class went on to participate in society (quinlan & fogel, 2014). this study examined the student perception of participation through the lens of willingness to communicate (wtc) to learn how pre-service english teachers in metro manila, philippines, interpreted the instructional approach employed in the classroom. while prior wtc studies have taken a comprehensive look at communication (riasati & noordin, 2011), this study emphasizes oral production in a language classroom. the findings suggest participants viewed participation as recitation, were fearful of how their english language proficiency and their responses were perceived, and that despite this fear of failure, expected their participation in class to be graded. this article asserts that by understanding learner perceptions of class participation, educators can develop learners wtc, which in turn has implications for developing 21st century skills. relevant literature english plays an important role in the philippines. the medium of instruction transitions to english in fourth grade. it is an official language of the country as the well as the working language and lingua franca amongst asean nations. these roles necessitate that learners build english language skills for use outside of the classroom. for language learning to take place, all members of the community play a role in creating learning opportunities (kumaravadivelu, 2003). mustapha, rahman, and yunus (2010) found instructor ability to build a supportive environment, encourage participation, value a range of contributions, and help learners overcome fears stimulated participation. willingness to communicate and class participation wtc illustrates how a learner’s ability to enter a conversation at a given point in time is influenced by a series of variables and behaviors. a summary of wtc is presented in figure 1 (macintyre et al., 1998). figure 1 wtc layer 1: communication behavior second language (l2) use layer 2: behavioral intention willingness to communicate layer 3: situated antecedents desire to communicate with a specific person state communicative selfconfidence understanding student perceptions of class participation 42 journal of effective teaching in higher education, vol. 4, no. 1 layer 4: motivational propensities interpersonal motivation intergroup motivation l2 selfconfidence layer 5: affectivecognitive context intergroup attitudes social situation communicative competence layer 6: social and individual context intergroup climate personality note. modified from macintyre, dörnyei, clément, & noels (1998) several studies have tested the wtc model in the english language classroom. wood (2016) emphasized the connection between fluency and wtc, finding that as cognitive demands of a task changed, even within the task, wtc would also change. fatemi and choi (2014) used structural equation modeling to investigate the linguistic, psychological, and contextual variables used to predict wtc. their findings emphasized the role of the teacher in establishing a classroom environment where students were willing to engage in oral language activities. hsu and huang (2017) focused on the connection between teacher confirmation and learners willingness to talk. they determined teacher confirmation was positively connected to the classroom environment and improved the perception of language proficiency, reduced anxiety and increased oral participation. additionally, student centered discussions encouraged participation in the classroom. students were more likely to engage when they practiced in small groups prior to large group discussions and when large group discussions continued throughout the semester (yashima, macintyre & ikeda, 2018). developing wtc in the classroom is a means to build 21st century skills. in the philippines, the need for a wtc is twofold: 1) english is the working language of the country and the official language of asean and 2) english is the lingua franca of southeast asian nations (hill & fernandez-chung, 2017; kirkpatrick & bolton, 2010). participation in a class occurs on many levels. participation may be spontaneous, voluntary, compulsory, or forced; student-initiated or teacher-initiated; passive or active; positive or negative; student to student, student to teacher, or student to course (loftin et al., 2018; weaver & qi, 2005). sociocultural theory emphasizes oral contributions as “a fundamental component of language learning” (thoms, 2012, p. s10; swain, 1997). in the language classroom, participation is frequently associated with vocalized oral contributions (bernales, 2016; ellwood & ikuko, 2009; kim, 2008). wtc orally in class can be indicated through vocalized participation and evidence of a desire to participate by a student raising his/her hand. a desire to participate without a concrete act indicating intention on the part of the student has been more difficult to capture (bernales, 2016). the sociocultural view emphasizing the importance of oral contributions to classroom are of particular significance to this study. sfard (1998) illustrated the dichotomy of learning through the metaphors of acquisition and participation. the acquisition lens described the learner as one who understanding student perceptions of class participation 43 journal of effective teaching in higher education, vol. 4, no. 1 acquires knowledge and emphasizes individual cognitive growth. the participation lens described learning as community based where the learner takes an active role. when learners exchanged ideas with their peers and/or the teacher, they developed higher order thinking skills in addition to acquiring language. articulating insights led to negotiation, problem solving, analysis, evaluation, and creation building both communicative competence and 21st century skills. participation through active learning increased the responsibility of the learner. this shift from passive to active learning is what ohashi (2013) described as “transformational linguistic participation” (p. 33). students who are more engaged in a course perform better. handelsman et al. (2005) found, “the only significant predictor of final examination grades was participation/interaction engagement (β=.62 )” (p. 189). one form of class participation, active learning, improved the transfer and retention of knowledge (barkley, 2010) and has also been found to influence critical thinking skills (kim et al., 2012; niu et al., 2013). dallimore et al. (2008) found that students enrolled in classes where the instructor explicitly stated their expectations for participation and graded those contributions were more prepared for class, participated more frequently, and reported a significant effect on their oral communication skills. class participation is one way that english learners can develop their wtc with other asian speakers of english. lingua franca communication emphasizes intelligible english, acknowledges variations in pronunciation to allow for accented english, and accepts grammatical imperfections that do not interfere with comprehensibility (fatemi & choi, 2014). understanding participation as a type of student engagement and the effects of engagement on student performance provides insight into why the student perspective is significant. several studies found differences in what teachers and learners perceive (barkhuizen, 1998; block, 1994, 1996; kumaravadivelu, 1991; national survey of student engagement, 2017). kumaravadivelu (2003) terms these learner-teacher differences as “perceptual mismatches.” perceptual mismatches in class participation are no exception. fritschner (2000) found that definitions of participation vary widely and were influenced by age, gender, and level of course. if student engagement, particularly participation, has implications for learner achievement, then a clearer understanding of participation is necessary. methodology this exploratory case study aimed to understand the participants’ perspectives on class participation within the secondary english education major at one institution. research was conducted at a public university in metro manila, philippines, where english is the medium of instruction. data was collected through a questionnaire and interviews in accordance with human subjects guidelines. understanding student perceptions of class participation 44 journal of effective teaching in higher education, vol. 4, no. 1 the questionnaire using patton’s idea of purposeful sampling (glesne, 1999), the questionnaire was distributed to students enrolled in five classes at the start of the semester. data were collected from 139 students who voluntarily completed the open-ended questionnaire (see appendix a). the open-ended questions sought to understand how the participants view participation and elicit responses related to the classroom environment, their role as well as the role of the teacher and their classmates, and their feelings towards participation. twenty participants identified as male, 118 as female, and one did not to identify a gender. participants included second, third, and fourth year students with an average age of 18.5. at the time this survey was conducted, participants completed high school at grade 10. as a result, traditional college students began their first year at approximately 16 years of age. demographic information was collected for reference purposes only and limited to gender, year in school, and age to protect the identity of the participants. this sample is representative of students enrolled in the major. the interviews semi-structured interviews were conducted with 10 students. during the interviews, participants described their experiences as college students, the expectations of students, and the relationships students have with faculty and peers. interviews allowed the researcher to probe more deeply into themes that arose in the survey. seven participants were female and three were male. two participants were second-year students, four were third-year students, and two were fourth-year students. data analysis initial coding identified themes that emerged in the data (saldana, 2013). questionnaires were coded and analyzed first. data were coded across question to identify themes that arose across the data overall. interviews were transcribed and coded individually. a thematic hierarchy was used to group data, determine higher order codes, and understand relationships within the data. the themes identified in the survey were compared to those identified in the interviews. as an outsider immersed in the educational setting of the philippines, my lens is subject to cultural interpretation. to address validity, i used several verification procedures (glesne, 1999). data was collected through multiple methods to allow for triangulation. interview participants reviewed emerging findings to affirm their experiences were accurately represented. as a final measure, a filipino colleague familiar with the experiences of this sampling of participants conducted an external audit of the findings to verify that i had accurately reflected the perspectives of the participants. quotations from participants have not been edited. understanding student perceptions of class participation 45 journal of effective teaching in higher education, vol. 4, no. 1 findings the analysis of the data yielded three key findings. participants 1) described class participation as “recitation,” such as student responses to teacher generated questions with limited interaction with peers; 2) reported a strong power distance relationship between student and teacher and suggested a fear of failure restricted their class participation; and 3) expressed an expectation that class participation be assessed. understanding recitation participants defined participation in a number of ways. they used terms to describe oral and written activities, active and passive learning, and interactions with the content, students, and teachers. “class participation is involving oneself’s (sic) actively in the class through recitation, written output (papers, etc) evaluation (quizzes and exams), and attendance” (second year, female). formal assessment tools like research papers and oral reports were included in definitions. however, participation was more likely to be described as oral contributions to classroom. the most common theme amongst responses was the persistent use of the term “recitation” to describe of class participation. forty participants used the word “recite” in their survey responses. additionally, several participants described participation as recitation without using the term directly. class participation is how students interacts inside the classroom. it can be by how the response to the question the teacher throws, it can also be by how they cooperate within assigned groups and co-student. it is a way of letting them speak and respond. (third-year female) understanding the data requires an understanding of the word “recite.” merriamwebster (n.d.) includes in the definitions for the verb “to recite” as “to read from memory or read aloud publicly; to repeat or answer questions about (a lesson); to reply to a teacher’s question on a lesson.” an examination of the corpus of webbased global english (davies, 2018), shows 104 examples of the word “recite” being used in this way in the philippine context. the term “recitation” is used in the corpus in the same context defined by merriam-webster (n.d.): “the act of saying or repeating something out loud for the audience; the act of describing or listing many things in a series.” interviewees described three types of recitation that take place in the classroom. two formats are less frequently used: 1) formal, similar to an oral quiz and 2) reading aloud or recalling a poem/literature from memory. most commonly “the teacher asks questions. the student answers in relation to what the teacher has asked” (second-year female). a third-year female stated (recitation) it’s done when the teacher asks something then the students who knows the answers just raise their hand and the teacher will (call on) understanding student perceptions of class participation 46 journal of effective teaching in higher education, vol. 4, no. 1 whoever he/she thinks can answer. if the teacher wants more, the teacher will ask other students who raise their hand. participants indicated raising your hand and being called upon was more typical than cold calling. in defining participation and the roles of the classroom, the participants tend to perceive participation as “recitation,” a response to teachergenerated questions. students indicated their wtc through recitation for a variety of reasons. participants emphasized the need to be self-confident to participate. i think recitation is very common in the philippines because whenever a person recites that person is very bold inside the class. they are the kind of students that voice out their answers or answer questions from the teacher vocally. they are competent enough if they are able to do that. (third-year female) a second-year female added, “i recite when i think i have a point. when i talk i should make sense, or it’s wasting time.” students who do not recite run the risk of not being recognized by the teacher, which impacts their grade. a fourth-year female who described herself as a nontalker added, most of the time there are two to three persons who always speak. whatever happens they always speak. there are days i don’t get to participate, or i don’t get to speak in class because others consume the whole time. professors think i am an average student if i am not talking. participation is giving ideas, not just think of ideas, but give it out so others can receive. teachers think i am not participating because i don’t talk. a third-year female described a similar experience. “i don’t like it (recitation). my classmates told me i have to recite, so the professor will recognize me. (recognition) that’s one of the advantages of recitation. if i recite in class, i will earn more credit.” a third-year male reiterated, “you have to talk to be noticed”. a recitation model of class participation poses two potential problems. first, recitation limits interaction within the classroom and the development of 21st century skills. secondly, recitation creates competition amongst classmates by privileging large group oral contributions as class participation. the descriptions of recitation provided by the participants align with the initiationresponse-evaluation (ire) model of classroom discourse. the ire model emphasizes interaction between student and teacher that is initiated by the teacher (mehan, 1979; sinclair & coulthard, 1975; thoms, 2012). the participants described the pattern of recitation that begins with the teacher posing a question and students raising their hands to indicate their desire to answer. a student is called upon by the teacher, and the response is evaluated by the teacher. if the understanding student perceptions of class participation 47 journal of effective teaching in higher education, vol. 4, no. 1 answer is incorrect or requires elaboration, another student is called upon to recite. if the answer satisfies the teacher, a new question is posed, and the process repeats. in the ire model, the teacher who is posited as the expert, controls participation. the teacher determines who participates and evaluates the accuracy of the response. this pattern limits who speaks and interacts. for a student to participate, they must have personality, communicative competence, self-confidence, and desire. layers 3, 4, 5, and 6 of the wtc model are simultaneously determining whether or not the student will enter the conversation. in the model described by participants, the teacher must also be willing to enter the conversation with the student as indicated by selecting a specific student from those who have raised their hands to answer the question. wtc is determined by raising your hand, and vocalized oral participation is determined by the teacher. additionally, the ire model of class participation creates a competitive environment that privileges those who are willing to vocalize contributions to the large group rather than promoting collaboration amongst peers (hsu & huang, 2017). of course there is competition. they want to get high grades. they want to achieve something more than their classmates have. they just don’t know their eagerness is a type of competition. as for myself, i want to get the highest grade possible. (second-year male) in large classrooms, it is nearly impossible for all learners to contribute, therefore, only the most confident and assertive are heard. the competition transcends to small group presentations as well. even if the teacher is not good but the first group (that talks) produces a very good output, the second group will be pressured to do a better output. the students are intrinsically motivated to do more than expected. the teacher will give a better grade. (fourth-year male) the social situation described in layer 5 of the wtc model equates confidence with the experience within the community (macintyre et al., 1998). zhang et al. (2013) found english learners improve comprehension and production “through languagerich collaborative reasoning discussions” (p. 57). criticisms of the ire model have emphasized the limitations to interaction that inhibit higher order thinking skills in a teacher-student-teacher model of discourse. students rarely speak freely, pose questions, or respond to comments made by other students (fagan, 2017; rustandi, 2017). students volunteer to answer because they are confident in their response whereas those who are less motivated or anxious about either their content knowledge or language proficiency are unlikely to contribute. the recitation model described here creates an expectation of discourse in the classroom, which is contradictory to the development of 21st century skills for understanding student perceptions of class participation 48 journal of effective teaching in higher education, vol. 4, no. 1 learners at a high proficiency level. through the lens of bloom’s taxonomy, the word “recite” corresponds with the knowledge dimension. recitation can show factual knowledge, conceptual knowledge, procedural knowledge, and metacognitive knowledge (anderson & krathwohl, 2001). the word “recite,” while not used to describe the cognitive process dimension, is most closely associated with the process categories of remembering, where learners “retrieve relevant knowledge from long-term memory” (p. 31). however, to show understanding and “construct meaning from instructional messages,” learners transition to more complex tasks that include paraphrasing and inferring grammatical principals from examples (p. 31). the cognitive tasks shift moderately in the category of understanding but become more significant in the categories of applying, analyzing, evaluating, and creating. carrying out procedures, deconstructing parts, making judgments, and forming patterns are higher-level skills than recitation. even if the teacher poses a question that builds upon higher-order skills, the teacher posing of questions and the limited student interaction limits the development of 21st century skills. fritschner (2000) found student views of participation varied depending on if the student classified themselves as a “talker” or “non-talker.” faculty defined participation on a spectrum. at the lower end of the spectrum, participation included characteristics such as staying awake, coming to class, taking notes, and doing assignments, while higher levels of participation included asking questions, making comments, providing input, and coming to class with additional questions. listening is noticeably absent from student and teacher definitions of participation in fritschner’s study. sung (2017) found if students are not invested in the classroom activity, they are less likely to engage. the interviewees that described themselves as talkers were comfortable with the ire model, while non-talkers preferred a broader definition of class participation that included more small group work and valued active listening. a fear of failure the classroom is a dynamic environment with a broad range of characteristics that incorporate the physical space, teacher, and classroom climate (brown, 2007). the participants in this study expressed a traditional view of the classroom environment through their account of their roles and expectations in the classroom. “my role in class is a student, and as a student i am to conform with the tasks and requirements given by teachers” (third-year, female). a second-year female added, “i think it would be easy if the teacher was enthusiastic and friendly.” students expressed fear to recite in class because they are afraid their answers might be wrong and thus be ridiculed by both the teacher and the classmates. a second year male added, “i seldom participate because i’m afraid to commit mistakes in front of my classmates.” “i fear that if i don’t get to answer correctly, the people around might laugh at me which is too embarrassing or if in activities i showed vagueness or something; it is also embarrassing” (third-year, female). the fear of failure appears to be linked to the practice of recitation and the ire model of discourse. the tone of the environment set by “recitation” aligns with the understanding student perceptions of class participation 49 journal of effective teaching in higher education, vol. 4, no. 1 banking model of education, which alienates students from learning and impedes confidence. i feel inferior because i feel most of my classmates are ahead of me and they have better ideas than i do. they have more confidence than i do. i just keep my ideas to myself because if i say something wrong it will make me more inferior. so if i commit a mistake, i will feel more inferior. it is not just me making me feel inferior, but the others also. if i make a mistake, they will look down on me. (fourth-year female) a study of chinese english learners by mak (2011) suggested lack of preparation by students and fear of a negative evaluation by both teachers and peers are the main contributors to classroom speaking anxiety. this fear of giving an incorrect or incomplete answer is heightened by the expectation to respond quickly and having one’s performance negatively evaluated by either peers or the teacher (mak, 2011; ohashi, 2013). additionally, mystowska wiertelak (2016) found the power dynamics within the classroom and speaking in large group settings influenced the growth of wtc in learners. emphasizing wtc over accuracy by using mitigating techniques such as think-pair-share activities allows students the opportunity to process the question and practice their responses before speaking publicly; increasing wait time reduces anxiety and encourages more active participation (mak 2011; sung, 2017). a classroom that is intentionally democratic and inclusive helps learners develop language skills and reduces fear. the hofstede insights (n.d.) model of national culture identifies six dimensions that provide insight into the values and behaviors of a culture: power distance, individualism, uncertainty avoidance, masculinity, long-term orientation, and indulgence vs. restraint. the scores, out of 100, evaluate countries on a spectrum with higher scores indicating greater value within the culture. while it is necessary to acknowledge individual learner differences, hofstede’s value dimensions serve as a framework for understanding the culture of the classroom. table 1 shows how the philippines scores in each dimension. table 1 hofstede’s value dimensions for the philippines power distance individualism uncertainty avoidance masculinity long-term orientation indulgence vs. restraint 94 32 44 64 27 42 note. hofstede insights (n.d.). the value indicators for the philippines place teachers in a position of power. the ire model as an instructional approach is consistent with this lens in that the understanding student perceptions of class participation 50 journal of effective teaching in higher education, vol. 4, no. 1 teacher poses the questions, controls the pace, and evaluates the response. morales (2014) found the hierarchy presents itself in the classroom through the need to save face and avoid shame. a third year female stated, “when you question your professor, you are showing your classmates, you are deeper than your professor.” in a high power distance environment, the teacher would maintain a teacher-centered environment to maintain the appearance of “expert.” students aim to please the teacher through correct responses and obedience. in a low uncertainty avoidance environment, preference is given to one correct answer. the hesitation to participate is consistent with the need to avoid the uncertainty of how the teacher and classmates will respond. recitation provides structure which both students and teachers are familiar, while democratic discussion leaves more room for uncertainty. the collective nature of the culture leaves less room for individualization in terms of student learning needs and open-ended questions with the potential for subjective responses. the low individualism score suggests that a collective understanding is preferred over an individualized response that may challenge the ideas of the teacher, classmates, institution, or country. this perception of challenge may create additional uncertainty. lou and noels (2016) found a mindset such as the fear of failure can change if the learning environment is supportive. creating a democratic classroom environment required leveling of the playing field and reducing the distance between student and teacher (ayyash, 2011). for the teacher and students, this often means showing vulnerability in the classroom: relinquishing control to put the students at the center of the discussion and not knowing how the audience would respond. in contrast, participants viewed students who participated as “bold” and “confident” whereas vulnerability suggested weakness. it is also important to consider the impact english medium of instruction (emi) plays in fear. the mother tongue-based multilingual education program was implemented after the participants of this study entered college. they learned content in english, which is not the first or second language of most filipinos, yet proficiency in the first language or mother tongue is often a predictor of proficiency in other languages (jorolan-quintero, 2018). many of the mother tongue languages of the philippines are rooted in oral tradition, which should contribute to class participation. however, the power-distance relationship complicated by the limitations and inhibitions emi has on language development may stoke the fear of failure. in the filipino context, participation has the potential to enhance learning as it builds upon the oral traditions of the culture. one variable influencing wtc is the relationship between students and teacher and amongst students. layer 3 of the wtc model describes the role of anxiety and perceived self-confidence established by the classroom climate (macintyre et al., 1998). the “fear of failure” illustrated the distance between interlocutors and suggests the fear extends beyond the confidence individual students feel in independent situations. participants of this study expressed a desire to please the teacher, but anxiety overruled their wtc. educators should consider the possibility that students are self-actualizing their fear of failure, which can be mediated by classroom structure and variations in evaluation and feedback. understanding student perceptions of class participation 51 journal of effective teaching in higher education, vol. 4, no. 1 assessment of participation despite their fears, the majority of participants felt class participation should be assessed. “i feel it’s but proper to grade my participation because it’s effort. the teacher can see my interest and enthusiasm in learning though my participation. for me, my grade doesn’t have to rely only on exams, quiz, and projects” (fourthyear, male). participants viewed participation as motivating and rewarding, despite the challenges they face in the classroom. the students expressing concern about graded participation expressed a lack of confidence and concern about intergroup climate. “i feel unconfident or i feel that there is a discrimination because sometimes professors give good grades to those students they know by name and for the class officers that assist them” (third-year female). a fourth-year female added “i feel worried because i am not the participative type who expresses my opinion regularly.” the need or want for assessment of class participation is juxtaposed to the expressed fear of failure. however, students wanted to be rewarded and wanted to please the teacher. assessment provides opportunities to quantify what students know. formative assessment provides a layer of feedback that pre-service english teachers can use to develop their confidence and proficiency. when they are in front of their own classroom, the need to save face by speaking english accurately prevails. while not explicitly discussed in the wtc model, assessment offers motivation to learners. assessment can be tied to “behavior standards” that create a positive communicative environment (macintyre et al., 1998, p. 553). for class participation to be successful at any level, students need to prepare. assessment of participation holds learners accountable for their preparatory work and builds autonomous and self-directed learning (muldrow, 2013). however, woods (2007) cautioned that assessing participation could lead to an environment that promotes competition rather than cooperation amongst class members. creating an environment that allows for different voices to contribute should be a consideration for educators in a large classroom. layer 6 of the wtc model includes intergroup climate and personality which are the hesitations expressed by participants of this study (macintyre et al., 1998). the social and individual context, including intergroup climate and personality, keep those not predisposed to contributing from participating. implications in the 21st century, students are not only expected to master core subjects, but are also expected to be innovative, creative, and critical thinkers who are selfdirected and adept in information technology (greenhill, 2010). in february 2009, article 34 of the asean charter declared, “the working language of asean shall be english” (2008, p. 29). this declaration further increased the importance of being able to communicate effectively in english. this study finds the recitation model understanding student perceptions of class participation 52 journal of effective teaching in higher education, vol. 4, no. 1 described by the participants limits interaction in the classroom and has the potential to limit 21st century skill development. wtc through class participation can be inhibited by fear and lack of assessment. in recent years, the philippines experienced a series of education reforms spurred in part by the role of asean. in 2012, mother tongue-based multilingual education was implemented for the first four years of schooling (metila et al., 2016). the basic education act added three years of formal education in 2013 (abulencia, 2015). most recently the department of education adopted professional standards for teachers in 2017 (department of education, 2017). these reforms present challenges to educators and policy makers ranging from understanding the dynamics of the community to finding qualified teachers. while this study does not discuss the negative implications of english as a lingua franca on regional languages, the significance of asean’s influence on education policy and the role of english in the philippines is clearly indicated in the reform initiatives. the classroom presents a place for learners to develop their english language skills. however, for participation to be effective, the instructor must consider how students perceive participation as it impacts their wtc. these perceptions include intentionality of pedagogy to engage students, the power distance relationship between teacher and students, and the assessment practices used to motivate and measure student participation. martin (2014) found communicative language teaching, task based language teaching, and english for specific purposes to be the predominant pedagogical approaches to language teaching in the philippines. however, martin also found a perceptual mismatch between understanding and usage of these fluency and functionality based approaches as educators reported near equal time emphasizing form and function. this is consistent with barrot’s (2014) findings that novice teachers do not have the experience or the theoretical underpinnings to implement active learning strategies effectively. inconsistent use can result in unclear expectations of class participation. while the amount of scaffolding and teaching guidance varies based on student need, it is essential throughout the learning process. in a language classroom, the cognitive demands of a task may vary considerably. language proficiency and ability to use higher order skills may not develop at the same rate. to increase oral participation, increase wtc, and develop 21st century skills, educators will need to re-imagine classroom discourse beyond the ire model and vocalized oral contributions. this can be done by encouraging different forms of participation, integrating higher-order skills into the language classroom, and providing feedback that balances language proficiency, content, and skill sets. consistent as opposed to intermittent use of active learning and discussion-based practices can help learners make meaning and build 21st century skills. limitations this study of secondary english education majors focuses on participants with motivation to achieve, if not perfect, english language proficiency. their career paths necessitate use outside of the sheltered classroom environment. the datu understanding student perceptions of class participation 53 journal of effective teaching in higher education, vol. 4, no. 1 and valdez (2015) study into psychological capital as a predictor of academic engagement of filipino high school students supports this notion. students who are highly motivated by hope, optimism, resilience, and self-efficacy are more likely to be academically engaged. the motivation to communicate in the target language may influence the perception participants have on class participation and wtc. secondly, this study only surveyed the student perception, not the educators’ perceptions. research into common educational practices and challenges in the philippines is limited. much of the work in the area of participation, student engagement, and wtc is quantitative and takes place in the united states. additional research into both the student and educator perception is needed to better understand perceptual mismatches. conclusion the growth of english as lingua franca on a global scale, not just within asean, creates a need to blend both proficiency and usage. while not explicit, new professional standards adopted by the philippines in 2017 included domains that suggest teaching strategies should promote a higher standard of learning and provide safe and fair learning environments to support the “changing character of the 21st century learner” (department of education, 2017). the benefit to the philippines is two-fold. higher education in the asean region plays a vital role in “creating, disseminating, and applying knowledge” used to train individuals in leadership positions within the region and bring economic opportunity to the area (umemiya, 2008). additionally, the philippines is a place where learners from asean nations come to study english (ang, 2017). engaging students through classroom participation requires teachers to design activities as a response to student needs that is also highly influenced by the culture of asean. student perceptions of participation in the classroom provide insight into learners’ wtc. kumaravadivelu (1991, p. 107) found “the more we know about the learner’s personal approaches and personal concepts, the better and more productive our intervention will be.” if learners are unable to participate because of limited opportunities or are unwilling to speak in class because of a fear of failure, then the likelihood that they will engage in the target language outside of the classroom diminishes. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references abulencia, a. (2015). the unraveling of k-12 program as an education reform in the philippines. south–east asian journal for youth, sports, & health education, 1(2), 229–240. understanding student perceptions of class participation 54 journal of effective teaching in higher education, vol. 4, no. 1 ananiadou, k., & claro, m. (2009). 21st century skills and competences for new millennium learners in oecd countries. oecd education working papers, 41, oecd publishing. https://doi.org/10.1787/19939019 anderson, l., & krathwohl, d. r. (eds.). (2001). a taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives. longman. ang, m. (2017). responding to the educational challenges and opportunities of asean integration: a case analysis of st. paul university philippines. asian education studies, 2(4), 19–26. https://doi.org/10.20849/aes.v2i4.257 asean charter. (2008). http://www.asean.org/wp-content/uploads/2012/05/11.october-2015-the-asean-charter-18th-reprint-amended-updated-on-05_april-2016-ijp.pdf ayyash, a. (2011). towards a class-centered approach to efl teaching in the palestinian context. english language teaching, 4(4), 65–76. barkhuizen, g. p. (1998). discovering learners' perceptions of esl classroom: teaching/learning activities in a south african context. tesol quarterly, 32(1), 85–108. barkley, e. (2010). student engagement techniques. jossey-bass. barrot, j. s. (2014). a macro perspective on key issues in english as a second language (esl) pedagogy in the postmethod era: confronting challenges through sociocognitive-transformation approach. asia-pacific education research, 239(3), 435–449. https://doi.org/10.1007/s40299-013-0119-4 bernales, c. (2016). towards a comprehensive concept of willingness to communicate: learners' predicted and self-reported participation in the foreign language classroom. system, 56, 1–12. https://doi.org/10.1016/j.system.2015.11.002 block, d. (1994). a day in the life of a class: teacher/learner perceptions of task purpose in conflict. system, 22, 473–486. block, d. (1996). a window on the classroom: classroom events viewed from different angles. in k. m. bailey & d. nunan (eds.), voices from the language classroom: qualitative research in second language education (pp. 168–194). cambridge university press. brown, h.d. (2007). principles of language teaching. pearson. dallimore, e. j., hertenstein, j. h., & platt, m. b. (2008). using discussion pedagogy to enhance oral and written communication skills. college teaching, 56(3), 163–172. understanding student perceptions of class participation 55 journal of effective teaching in higher education, vol. 4, no. 1 datu, j.a., & valdez, j.p.m. (2015). psychological capital predicts academic engagement and well-being in filipino high school students. asia-pacific education research, 25(3), 390–405. https://doi.org/10.1007/s40299-0150254-1 davies, m. (2018) corpus of web-based global english: 1.9 billion words from speakers in 20 countries. https://www.english-corpora.org/glowbe/ department of education (2017). national adoption and implementation of the philippine professional standards for teachers. (deped order no. 42, s. 2017). ellwood, c., & nakane, i. (2009). privileging of speech in eap and mainstream university classrooms: a critical evaluation of participation. tesol quarterly, 43(2), 203–230. fagan, d. (2017). addressing learner hesitancy-to-respond within initiationresponse-feedback sequences. tesol quarterly, 52(2), 425–435. https://doi.org/10.1002/tesq.421 fatemi, h., & choi, c. (2014). willingness to communicate in english” a microsystem model in the iranian efl classroom context. tesol quarterly, 50(1), 154–180. fritschner, l. (2000). inside the undergraduate college classroom: faculty and students differ on the meaning of student participation. the journal of higher education, 71(3), 342–362. glesne, c. (1999). becoming qualitative researchers: an introduction (2nd ed.). longman. greenhill, v. (2010). 21st century knowledge and skills in educator preparation. https://files.eric.ed.gov/fulltext/ed519336.pdf handelsman, m., briggs, w., sullivan, n., & towler, a. (2005). a measure of college student course engagement. the journal of educational research, 88(3), 84– 191. hill, c., & fernandez-chung, r. (eds). (2017). higher education in the asian century: the european legacy and the future of transnational education in the asean. routledge. hofstede insights (n.d.). dimensions of national cultures. https://www.hofstedeinsights.com/ hood cattaneo, k. (2017). telling active learning pedagogies apart: from theory to practice. journal of new approaches in educational research, 6(2), 144–152. understanding student perceptions of class participation 56 journal of effective teaching in higher education, vol. 4, no. 1 hsu, c., & huang, i. (2017). are international students quiet in class? the influence of teacher confirmation on classroom apprehension and willingness to talk in class. journal of international students, 7(1), 38–52. jorolan-quintero, g. (2018). oral traditions: an aid to implementation of mother tongue-based multilingual education in the philippines’ basic education programme. international review of education, 64, 759–777. https://doi.org/10.1007/s11159-018-9743-9 kim, k., sharma, p., land, s. m., & furlong, k. p. (2012). effects of active learning on enhancing student critical thinking in an undergraduate general science course. innovative higher education, 38. kim, s. (2008). silent participation: east asian international graduate students’ views on active classroom participation. journal on excellence in college teaching, 19(2–3), 199–220. kirkpatrick, a., & bolton, k. (2010). english as a lingua franca in asean: a multilingual model. hong kong university press. kumaravadivelu, b. (1991). language-learning tasks: teacher intention and learner interpretation. elt journal, 45(2), 98–107. kumaravadivelu, b. (2003). beyond methods: macrostrategies for language teaching. yale university press. lai, e. r., & viering, m. (2012). assessing 21st century skills: integrating research findings. national council on measurement in education. loftin, c., davis, l., & hartin, v. (2010). classroom participation: a student perspective. teaching and learning in nursing, 5(3), 119–124. lou, n., & noels, k. (2016). changing language mindsets: implications for goal orientations and responses to failure in and outside of the second language classroom. contemporary educational psychology, 46, 22–33. https://doi.org/10.1016/j.cedpsych.2016.03.004 macintyre, p., dörnyei, z., clément, r., & noels, k. (1998). conceptualizing willingness to communicate in a l2: a situational model of l2 confidence and affiliation. modern language journal, 82(4), 545–62. mak, b. (2011). an exploration of speaking-in-class anxiety with chinese esl learners. system, 39(2011), 202–214. https://doi.org/10.1016/j.system.2011.04.002 martin, i. p. (2014). english language teaching in the philippines. world englishes, 33(4), 472–485. understanding student perceptions of class participation 57 journal of effective teaching in higher education, vol. 4, no. 1 mehan, h. (1979). learning lessons: social organization in the classroom. harvard university press. merriam-webster. (n.d.). recite. https://www.merriamwebster.com/dictionary/recite metila, r. a., padilla, l. a. s., & williams, a. b. (2016). the challenges of implementing mother tongue education in linguistically diverse contexts: the case of the philippines. asia-pacific education researcher, 25(5), 781–789. https://doi.org/10.1007/s40299-016-0310-5 morales, m. (2014). cultural and epistemological profile of filipino learners. electronic journal of science education, 18(6), 1–25. mystowska-wiertelak, a. (2016). dynamics of classroom wtc: results of a semester study. studies in second language learning and teaching, 6(4), 651–676. https://doi.org/10.14746/ssllt.2016.6.4.5 muldrow, k. (2013). a new approach to language instruction—flipping the classroom. the language educator, 11, 28–31. mustapha, s., rahman, n., & yunus, m. (2010). factors influencing classroom participation: a case study of malaysian undergraduate students. procedia social and behavioral sciences, 9, 1079–1084. https://doi.org/10.1016/j.sbspro.2010.12.289 national survey of student engagement. (2017). engagement insights: survey findings on the quality of undergraduate education. eric. https://eric.ed.gov/?id=ed590611 niu, l., behar-horenstein l., & garven, c. (2013). do instructional interventions influence college students’ critical thinking skills? a meta-analysis. educational research review, 9, 114–128. https://doi.org/10.1016/j.edurev.2012.12.002 ohashi, y. (2013). linguistic participation as a manifestation of second language learning in the primary l2 classroom. innovation in language learning and teaching, 7(1), 19–39. https://doi.org/10.1080/17501229.2012.665214 orhan göksün, d., & kurt, a. (2017). the relationship between pre-service teachers’ use of 21st century learner skills and 21st century teacher skills. ted eğitim ve bilim, 42. https://doi.org/10.15390/eb.2017.7089 quinlan, a., & fogel, c. (2014). transcending convention and space: strategies for fostering active learning in large post-secondary classes. higher education studies, 6(6), 43–48. understanding student perceptions of class participation 58 journal of effective teaching in higher education, vol. 4, no. 1 riasati, m. j., & noordin, n. (2011). antecedents of willingness to communicate: a review of literature. studies in literature and language, 3(2), 74–80. https://doi.org/10.3968/j.sll.1923156320110302.326 rustandi, a. (2017). an analysis of irf (intiation-response-feedback) on classroom interaction in efl speaking class. edulite: journal of english education, literature, and culture, 2(1), 239–250. saldana, j. (2013). the coding manual for qualitative researchers. (2nd ed.). sage. sinclair, j., & coulthard, r. (1975). toward an analysis of discourse. oxford university press. sfard, a. (1998). on two metaphors for learning and the dangers of choosing just one. educational researcher, 27(2), 4–13. https://doi.org/10.3102/0013189x027002004 sung, c. (2017). understanding classroom participation and identity negotiation: an undergraduate student’s l2 learning experiences in an english-medium university in hong kong. applied linguistics review, 8(4), 375–399. swain, m. (1997). collaborative dialogue: its contribution to second language learning. revista canaria de estudios ingleses, 34, 115–132. thoms, j. (2012). classroom discourse in foreign language classrooms: a review of literature. foreign language annals, 45(s1), s8–s27. https://doi.org/10.1111/j.1944-9720.2012.01177.x umemiya, n. (2008). regional quality assurance activity in higher education in southeast asia: its characteristics and driving forces. quality in higher education, 14(3), 277–290. https://doi.org/10.1080/13538320802507679 wagner, t. (2010). the global achievement gap: why even our best schools don't teach the new survival skills our children need--and what we can do about it. basic books. weaver, r., & qi, j. (2005). classroom organization and participation: college students' perceptions. the journal of higher education, 76(5), 570–601. https://doi.org/10.1080/00221546.2005.11772299 wood, d. (2016). willingness to communicate and second language speech fluency: an idiodynamic investigation. system, 60, 11–28. https://doi.org/10.1016/j.system.2016.05.003 woods, c. (2007). researching and developing interdisciplinary teaching: towards a conceptual framework for classroom communication. the international journal of higher education and educational planning, 54(6), 853–866. understanding student perceptions of class participation 59 journal of effective teaching in higher education, vol. 4, no. 1 yashima, t., macintyre, p., & ikeda, m. (2018). situated willingness to communicate in an l2: interplay of individual characteristics and context. language teaching research, 22(1), 115–137. zhang, j., anderson, r., & nguyen-jahiel, k. (2013). language-rich discussions for english language learners. international journal of educational research, 58, 44–60. appendix a survey questions part 1: demographics circle the response that best describes you. 1. gender: male female prefer not to respond 2. year in school: 1st 2nd 3rd 4th 3. age: part 2: perceptions of participation answer the questions below to the best of your ability. 1. how do you define class participation? 2. what is the role of the teacher in the classroom? 3. how should a teacher encourage student participation in the classroom? 4. how would you describe your role in the classroom? 5. how would you describe the role of your classmates in the classroom? 6. what does effective classroom participation look like? 7. what behavior is unacceptable in the classroom? 8. what do you find to be the benefits of participating in class? 9. what challenges do you face when participating in class? 10. how do you feel about your class participation being graded? journal of effective teaching in higher education, vol. 1, no. 1 handwriting to learn: embedding a crucial study skill in a gateway science course shylaja akkaraju cuny bronx community college, bronx, ny abstract. handwriting is a multisensory process known to enhance memory, focus, engagement, and cognition in the learner, thereby making it a valuable study skill. in an attempt to embed handwriting as a study skill in a gateway science course, i used a combination of scaffolding and individual student consultations. scaffolding emphasized the use of desirable difficulties and retrieval practice through the implementation of techniques such as the flipped learning approach and take-home in-class hybrid essay exams. while the former helped to increase overall student success, the latter technique was aimed at enhancing written fluency in the subject, curbing procrastination, and providing opportunities for high impact retrieval practice during the in-class essay exams. one-on-one student consultations that emphasized rewriting and maintaining good notes were used as interventions throughout the semester to promote positive study behavior and also as a crucial stage in preparing for each unit exam. the level of student engagement in using handwriting as a study skill was traced within the context of a typical threshold experience. handwriting as a study skill was successfully embedded in this course and showed positive results in terms of student retention, engagement, and overall success. keywords: handwriting, threshold experience, retrieval practice, desirable difficulties, flipped learning writing by hand employs many more senses in comparison to reading. handwriting involves tactile, proprioceptive (sense of movement), visual, and spatial senses; whereas, reading employs the visual and spatial senses only. when learning new information, handwriting engages learners by directing visual focus to the tip of their pens (fogassi & gallese, 2004). this stimulates the reticular activating system in the brain, whose function is to prioritize sensory information that enters the consciousness and help in focusing on what is important. therefore, when turned on during handwriting, the reticular activating system can sharpen focus and heighten attention (kinomura, 1996). handwriting is, in general, a slow process that involves the hand and the brain in a sensory-motor feedback loop allowing better factual recall and deeper cognitive processing (mangen & velay, 2010; mueller & oppenheimer, 2014). the idea that learning occurs not only via cerebral processing but also via muscle/motor activity is called embodied cognition, a concept that is gaining enormous importance (borghi & cimatti, 2010; engel, maye, kurthen, & konig, 2013; foglia & wilson, 2013). a recent study has shown that when children used handwriting to learn 56 journal of effective teaching in higher education, vol. 1, no. 1 finger tracing on an ipad to study graphs they did significantly better when tested on the understanding of these graphs when compared to children who studied the same graphs without finger tracing (agostinho et al., 2015). finger tracing of graphs takes advantage of embodied cognition to deepen comprehension in a way that reading graphs off a screen does not. when writing by hand, the writer graphomotorically forms each letter (mangen & velay, 2010) much like in finger tracing. writing by hand is a motor activity that uses multiple sensory inputs (tactile, proprioceptive, etc.) thereby making it a readily available mode of embodied cognition for students in higher education. in contrast, keyboarding entails hitting ready-made alphabets and is therefore faster; yet, it does not appear to benefit the learner in terms of deeper cognitive processing. in a recent note-taking study (mueller & oppenheimer, 2014), students who used longhand tended to retain more information and demonstrated a deeper understanding of the lecture material when compared to those who used keyboarding to take notes. they found that although students using keyboarding take more notes than those using longhand, they tend to type what they hear verbatim without deeply processing what they type (mueller & oppenheimer, 2014). note-taking has two major—storage of information that can be reviewed later and encoding of information in the brain that results in higher cognitive processing (kiewra, 1985). this encoding function of note-taking has a positive impact on student performance with students who take notes performing significantly better than those who do not (boyle, 2007; boyle, 2010; kiewra, 1985; kiewra & benton, 1988; piolat, olive, & kellogg, 2004; titsworth, 2001). yet, note-taking is a very complex activity that places enormous demands on a very limited working memory. our working memory can only hold about four units of newly acquired information at any given time and it decays within seconds if it is not rehearsed or reinforced (cowan, 2001). if handwriting speed is slow, then information held in the working memory is quickly forgotten and the quality of note taking becomes greatly hampered. transcription fluency or handwriting speed has been shown to have a positive effect on note-taking (peverly et al., 2007) by lightening the load on working memory resources, thus freeing up some of the working memory space to address higher order cognitive processes such as comprehension and application of newly acquired information. note-taking requires a great deal of cognitive effort because it involves listening, remembering, writing, organizing, selecting, and making connections with prior knowledge all at the same time (piolat et al., 2004). a subject that is complex with many interacting elements (human physiology for example) is said to have a high level of intrinsic cognitive load, placing high demands on the working memory of the learner (mayer, 2005; sweller, 1994). coupling the cognitive demands with note-taking can hamper both comprehension and transcription. this is because the note-taker often ends up either limiting transcription to allow for comprehension or limiting comprehension to allow for better transcription (kiewra, 1985). this predicament introduces yet another type of cognitive load called extrinsic cognitive load that can impede learning (sweller, 1994). instructors have addressed this problem by providing students with handwriting to learn 57 journal of effective teaching in higher education, vol. 1, no. 1 readymade notes in the form of handouts or powerpoint slides, a practice that deprives students of the opportunity to generate their own study materials. students perform better when they generate their own study materials in the form of handwritten notes when compared to students who have been given readymade study materials (mueller & oppenheimer, 2014; piolat et al., 2004). alternatively, making students copy notes off a powerpoint slide or the board does not lead to comprehension of the material (piolat et al., 2004). therefore, to maximize the benefit to students, we need to uncouple, at least temporarily, the comprehension and transcription aspects of note-taking. one way to accomplish this would be to scaffold the learning process into several stages to provide adequate time and working memory resources for both comprehension and transcription to occur. scaffolding can help to minimize extrinsic cognitive load by presenting concepts in a way that is not distracting or overwhelming to the learner and manage intrinsic cognitive load by providing multiple opportunities for students to grasp the material in stages while monitoring their understanding. adopting the flipped learning format can effectively address intrinsic cognitive load (akkaraju, 2016). apart from effective note taking in the classroom, it is equally important to maintain accurate and complete notes aligned to each learning outcome listed for a particular topic. rewriting class notes by hand in a separate study notebook can serve both as an excellent external storage of study materials and as a highly effective learning tool. adopting a take-home in-class hybrid essay exam format is one way to motivate students to rewrite class notes, align them to learning outcomes for a specific topic, practice drawing diagrams and flow charts wherever appropriate, and feel confident about their understanding of the material. the flipped learning approach and the take-home in-class hybrid essay exam introduce a number of desirable difficulties for the learner. a desirable difficulty is a strategy that helps the learner to experience deeper cognitive processing and achieve lasting retention by initially slowing down the learning process and making it deliberately harder for the learner. however, the learning strategy cannot make it so difficult as to discourage the learner (bjork, 1994; brown, roediger, & mcdaniel, 2014; marsh & butler, 2013). the task must be difficult, but accomplishable. for example, re-reading one’s notes is easier than re-writing one’s notes with the latter being a desirable difficulty because it is both more time-consuming and memory enhancing than the former. with the in-class portion of the essay exam, students are expected to provide detailed and often lengthy answers to essay questions. retrieval practice is a type of desirable difficulty, known to significantly increase long term retention and meaningful learning. answering essay questions on an exam is a powerful form of retrieval practice (karpicke, 2012; roediger & butler, 2011; roediger & pyc, 2012). to successfully embed a study skill in a course, it helps to track student experience and uncover any problems that the student might be facing. one way to approach this would be to view this skill acquisition process in terms of a threshold experience, which is a fundamental aspect of the threshold concept theory (meyer & handwriting to learn 58 journal of effective teaching in higher education, vol. 1, no. 1 land, 2003). a threshold concept is a doorway that leads to an entirely new way of thinking about something, resulting in a shift in perception. a threshold concept is both troublesome and transformational. it is troublesome because it is difficult or perhaps counter-intuitive; it is transformational because it results in a paradigm shift in perception. according to meyer and land, in order to master a threshold concept in any subject, the student must enter a liminal space in which the student deeply engages with the concept to master it. however, this learning process is rarely linear as the learner oscillates between excursive and recursive journeys (land, cousin, meyer, & davies, 2005). during the excursive journey, the learner takes bold steps towards tackling the challenging concept; whereas during the recursive journey, the learner gets stuck or returns to areas of confusion. the learning process involved in mastering a threshold concept is called a threshold experience (meyer & land, 2005). even though threshold experiences were originally meant to only describe the mastery of threshold concepts that are theoretical, more recently some authors have proposed the inclusion of threshold skills in this category (thomas et al., 2017). in this article, i refer to a threshold experience set in the context of students acquiring a threshold skill. when observing students undergoing a threshold experience, those students who are unwilling or not yet ready to engage with the study skill are said to be in the preliminal stage; those that engage with the study skill are said to be liminal; and those that have truly mastered the skill are said to be post-liminal. therefore, it is very useful to regard student experience of acquiring a study skill as a threshold experience. in this way, it is easy to identify the pre-liminal learners and reach out to them. consultations with students can help to emphasize the importance of rewriting notes and positively influence future study behavior (huijser, kimmins, & gallager, 2008). in this article, i describe: (a) how i scaffold the learning process to maximize comprehension and allow opportunities for unhurried, effective note-taking by hand; (b) how i scaffold the take-home in-class hybrid exams to promote the use of handwriting as a study skill, increase transcription fluency, curb procrastination, and provide opportunities for powerful retrieval practice; (c) how i track student progress in the context of a threshold experience and use student consultations to reinforce positive study behavior; and (d) how embedding handwriting as a study skill can significantly improve student engagement and overall performance. methods this study was conducted in bronx community college of the city university of new york, where 98% of the students are ethnic minorities with 90% of the students needing at least one remedial course. handwriting to learn was implemented for four consecutive semesters in the lecture session of an undergraduate human anatomy and physiology i course with a typically low retention and passing rate. the overall pass rate (passing with at least a dor 60%) for this gateway science course is typically 60% and the retention rate for this course is 70%. a total of 90 students were enrolled in all four sections. however, a total of 10 students withdrew and therefore the overall results presented here are based on a sample handwriting to learn 59 journal of effective teaching in higher education, vol. 1, no. 1 size of 80 students. on the first day of class, all students were asked if they used handwriting as a study method, and at mid-semester they were asked to respond to a survey on their study methods in general and how they view the use of handwriting to learn in particular. handwriting to learn was embedded in the course by meticulously scaffolding the learning process and by viewing the skill acquisition process as a threshold experience. scaffolding the learning process the scaffolded design of the flipped learning format was aimed at enhancing comprehension and note-taking (table 1). the scaffolded design of the take-home in-class hybrid exam was aimed at enhancing transcription fluency, creation of good study notes, curbing procrastination, and providing opportunities for high impact retrieval practice (table 2). table 1. a scaffolded design of flipped learning to maximize comprehension and note-taking rationale student learning outcome “student will be able to…” teaching technique learning opportunity enhances comprehension demonstrate mastery of factual knowledge and basic conceptual knowledge prior to engaging in higher order information processing flipped learning learning module made available to student before class followed by a pre-lecture quiz at the start of each lecture session demonstrate understanding of conceptual knowledge prior to note-taking socratic method students answer questions throughout the lecture and are instructed to not take notes apply conceptual knowledge to solve problems active learning students are asked to solve problems in groups enhances effective note-taking take notes that are complete and accurate. socratic method the concept is developed again on the chalk board in a step wise manner and the students are actively engaged by providing answers to “what comes next?” or “what happens if?” before writing down these steps by hand. handwriting to learn 60 journal of effective teaching in higher education, vol. 1, no. 1 during a typical lecture session, the students would take a pre-lecture quiz at the beginning of class. the pre-lecture quiz, which was worth 20 points, was simple and straightforward with the goal of assessing the students’ grasp of factual knowledge. together, we then reviewed the correct responses for all 20 questions. the students were now primed to receive conceptual knowledge. table 2. a scaffolded design of the take-home in-class hybrid exams to curb procrastination, generate study guides, enhance transcription fluency, and provide opportunities for retrieval practice. stages of the hybrid essay exam rationale student learning outcome assessment vehicle stage i: students are given about 12 essay questions as a take-home exam. they are encouraged to work in groups curb procrastination student will be able to rewrite, organize, and align lecture notes with the essay questions listed in the take-home exam. handwritten early drafts of take-home exam stage ii students present their answers for feedback before the actual deadline. generate study guides stage iii: students are instructed to practice rewriting their essays and re-drawing illustrations before they sit for the unit exam enhance transcription fluency students will be able to demonstrate a high degree of transcription fluency in-class exam stage iv: students take the unit exam, which is a combination of multiplechoice questions (50%) and essay questions (50%) provide opportunity for high-impact retrieval practice students will be able to provide answers to essay questions that are relevant, accurate, sufficiently detailed, and accompanied by illustrations wherever appropriate. using the socratic method of questioning, waiting for student response, and monitoring student understanding, the concept was developed in the students’ minds using one or two visuals. during this period, the students were instructed to not take any notes; instead, they were asked to “just relax and listen”. more precisely, they practiced “active listening” as they were continuously prompted to answer questions and encouraged to clarify understanding with questions of their own. the concept was repeated two or three times in this way using an increasing number of prompts in the second and third rounds. at this point, the students were ready for note taking. rather than merely copying or transcribing notes off the chalk board, we developed the concept or physiological process on the board together. the students paused to answer questions such as “what comes next?” or “what would happen if?” before the responses to these questions were written on the board to allow for both comprehension and transcription to occur. a total of four handwriting to learn 61 journal of effective teaching in higher education, vol. 1, no. 1 take-home in-class hybrid essay exams were given during the course of the semester. three out of four of these exams were given as unit exams and served as benchmark assessments to provide ample feedback to the student and allow for metacognitive thinking regarding the students’ own performance. the last exam was comprehensive and served as a summative assessment for the lecture portion of the course. table 3. a sample of essay questions that appear on a take-home exam essay question designed to assess student ability to…. skills involved draw two flowcharts showing how the body adjusts to maintain constant body temperature specifying the stimulus, receptor, control center, effector, and response—(a) when the body temperature rises above normal on a hot day; (b) when the body temperature drops below normal on a cold day 1. describe a process 2. use key terms correctly 3. recognize subtle yet important differences in the two examples drawing a flow chart draw a table and list the four biological macromolecules, specify the functional groups found in each, building blocks for each, and list the subclasses for each macromolecule or draw a detailed mind map to represent the same information. 1. classify and organize information 2. recognize subtle yet important differences between molecules 3. use key terms correctly making a table or creating a mind map show the formation of the bonds that hold nacl and cacl2 together using drawings. specify the type of bond that is formed here and the changes in charge that occurs. 1. describe a process 2. display logical reasoning 3. use key terms correctly drawing molecular structures to illustrate a process draw diagrams to explain what happens to kcl in the presence of water 1. describe a process 2. display logical reasoning 3. recognize subtle yet important differences between molecules 4. use key terms correctly drawing diagrams to illustrate a process draw the electron arrangement in the following molecules: h2o, ch4, o2, and h2. specify the type of bond that is formed in each case. 1. describe a process 2. display logical reasoning 3. recognize subtle yet important differences between molecules 4. use key terms correctly drawing molecular structures to illustrate a process a typical unit exam consisted of both multiple choice and essay questions, with each portion taking up 50% of the total points for the exam. to stay within the topic of handwriting to learn, i will only be referring to the essay portion of these handwriting to learn 62 journal of effective teaching in higher education, vol. 1, no. 1 unit exams. each exam was scaffolded into separate stages (table 2) to allow for maximum preparation on the part of the student. a list of 12 to 15 essay questions are provided to the student one month prior to the unit exam and the students are expected to complete answering these questions prior to the actual date of the unit exam (table 3). students were encouraged to work in groups and help one another in this process. on the exam day, the students were expected to submit their takehomes and sit for the exam. four to six questions from this take-home question set appear on the actual exam, and the student is expected to answer them without the aid of their notes or text. hence the term take-home in-class hybrid exam. the expectation is that the answers would be relevant, accurate, possess the right amount of detail, and include illustrations, graphs, flow charts, or concept maps wherever appropriate. table 4. rubric used to assess take home and in-class exams typically, the essay questions on a take-home exam cover all the major concepts within the topics assigned for that unit exam. therefore, answering these questions in a detailed and complete manner would automatically create self-generated study notes for the student. the essay questions were designed to assess student ability to describe a process, display logical reasoning, use key terms correctly, classify and organize information, and recognize subtle yet important differences between criteria ready for the final needs a little work needs a lot of work completion student has successfully completed all 12 essays subparts of all questions have been answered to completion student has successfully completed 8–10 essays subparts of most of the questions have been answered to completion student has successfully completed 0–6 essays subparts of many questions have not been addressed to completion comprehension & relevance demonstrates firm understanding of main point in all essays demonstrates firm understanding of the main point in most essays appears to have missed the point of many of the essays accuracy & logical reasoning relevance sequence of events in flow charts or logical reasoning is correct in all essays there are fewer than three factual errors there is no inclusion of unnecessary or distracting information correct use of arrows throughout sequence of events in flow charts or logical reasoning is correct in most essays there are less than five factual errors there is some inclusion of unnecessary or distracting information correct use of arrows for the most part sequence events or logical reasoning is incorrect in several essays. there are four or more factual errors there is a lot of inclusion of unnecessary or distracting information incorrect use of arrows in many places attention to detail graphs, illustrations, and keys contain no errors and they are complete graphs, illustrations, and keys contain minor errors and/or exclusions graphs, illustrations, and keys are incorrect or incomplete. handwriting to learn 63 journal of effective teaching in higher education, vol. 1, no. 1 molecules, processes, concepts, etc. (table 3). the take-home and in-class exams were graded using a rubric that assessed relevance, logical reasoning, comprehension, accuracy, attention to detail, and completion (table 4). viewing the skill acquisition process as a threshold experience for students in the context of the threshold experience, learning occurs within the liminal environment, which is a nurturing environment that is created and maintained by the instructor. within this environment is the liminal space in which the student may struggle with a new concept or skill and may even get stuck during this process (land, meyer, & baillie, 2010). visualizing students as being within this liminal space is very useful in identifying students that are stuck, struggling, or hesitant to engage in the learning process. students who are hesitant to participate in the learning process are said to be pre-liminal. the rest of the students who are engaged, but face difficulties in the normal course of learning, are said to be liminal. they could be taking an excursive journey by fully embracing the task at hand or they could be taking a recursive journey by retreating from the task at hand. a student who has come out of the threshold experience after having been transformed by it is said to be post-liminal (cousin, 2006). a variety of assessments (formative, benchmark, and summative) were used to identify pre-liminal, liminal, and post-liminal students (table 5). the pre-lecture quiz served as a weekly formative assessment. it was used to assess the quality of preparation and mastery of factual and basic conceptual knowledge at the beginning of each lecture session. results were used to identify pre-liminal students who were either unwilling to put in the effort or hesitant to engage with the material. the take-home exam served as a monthly formative assessment. students presented their handwritten take home essays for review and feedback throughout the four-week period leading up to the day of the unit exam. this practice ensured that their answers were complete, accurate, well-organized, and germane to the essay question. results from this assessment were used to identify pre-liminal students who were hesitant to put in the time and effort necessary to prepare for the unit exam. the in-class exam served as a benchmark assessment. results from this assessment were used to identify both pre-liminal students and those liminal students who were not fully engaging in the habit of using handwriting as a study skill. one-on-one student meetings were designed to address the affective, behavioral, and cognitive dimensions of student engagement (fredricks, blumenfeld, & paris, 2004) with the goal of positively influencing study behavior (table 5). particular emphasis was placed on the importance of rewriting notes and aligning them to each essay question, and student accountability was built in by arranging short follow-up consultations to make sure that the student was staying on task. to get a better sense of the student threshold experience, students were asked to respond to a survey about their study habits and perceptions of using handwriting as a study skill (see table 7). handwriting to learn 64 journal of effective teaching in higher education, vol. 1, no. 1 table 5 student consultations addressed different aspects of student engagement. results overall student performance by the end of each semester, benchmarks were met for all major desired outcomes (figure 1). benchmarks were set at 80% for the take home-exams and 70% for the in-class exams reflecting the level of difficulty involved in each assessment. figure 1. students achieving benchmark in all four sections combined (n=80) addressing engagement consultation process purpose instructor prompts affective (emotional) communicate empathy towards the student who is struggling in the course to make student feel supported in the learning process • what do you think is the problem? • let’s see what we can do to help you ace the next quiz… • are you writing as you learn for the lecture quizzes? encourage student to talk about longterm goals to give the student the opportunity to self-assess level of motivation to succeed • what is your major? • how did you choose it? behavioral ask student to show me their study notes to assess student’s preparation and their use of handwriting as a study skill • let’s have a look at your study notes…. cognitive emphasize the importance of rewriting notes and aligning them with learning outcomes for better retention to help the student understand the importance of handwriting as a valuable study skill • do you have the essay questions for this topic? • can i see your class notes? • can you find the answers to these questions? • how many times have you practiced re-writing these essays? build accountability into the student’s study behavior to help the student beat procrastination • i want you to show me your completed take home essays next week…. 81% 89% 77% 83% 90% 70% 75% 80% 85% 90% 95% pre-lecture quizzes take-home exams in-class essay exams passing rate c or above retention rate handwriting to learn 65 journal of effective teaching in higher education, vol. 1, no. 1 students met benchmarks for mastery of factual and basic conceptual knowledge, note-taking, generation of study notes, transcription fluency, high impact retrieval practice, retention rates, and pass rates (table 6). table 6 assessment summary for embedding handwriting as a study skill for all four sections desired outcomes assessment vehicle benchmark result students will be able to demonstrate mastery of factual knowledge and basic conceptual knowledge prior to engaging in higher order information processing pre-lecture quizzes overall 70% of students will score 80% or higher on the pre-lecture quizzes 81% of the students scored 80% or better on the prelecture quizzes students will take notes that are complete and accurate student notes 80% of the students will maintain complete and accurate lecture notes 90% of the students maintained complete and accurate notes student will be able to rewrite, organize, and align lecture notes with the essay questions listed in the take-home exam. take-home exams overall 80% of the students will score 80% or better on take home exams 89% of the students scored 80% or better on the take home exams students will be able to demonstrate a high degree of transcription fluency in-class exams overall 70% of the students will score 70% or better on in-class exams 77% of the students scored 70% or better on the in-class exams students will be able to provide answers to essay questions that are relevant, accurate, with the appropriate amount of detail and accompanied by illustrations wherever appropriate in-class exams overall 70% of the students will score 70% or better on in-class exams 77% of the students scored 70% or better on the in-class exams improve pass rates for this course final grades for this course at least 70% of the students will pass the course with a c or better 83% of the students passed the course with a c or better improve retention rates for this course final grades for this course at least 80% of the students will remain in the course 90% of the students remained enrolled in the course handwriting to learn 66 journal of effective teaching in higher education, vol. 1, no. 1 examination of the threshold experience in a single section the largest section with a total of 24 students was examined closely to exemplify the threshold experience. in this section, students had trouble with exam 2 for which they did not meet the benchmarks for the take-home and in-class portions (figures 2 and 3). only 78% of the students were able to score 80% or better on the take-home exam and a corresponding 61% of the students were able to score 70% or better on the in-class exam. however, the students appeared to regain their confidence and perform significantly better in the take-home and in-class essay portions of exam 3 and the cumulative final. nearly the entire class scored 80% or better on take-home portion of exam 3, and 91% of the students scored 70% or better on the in-class portion of exam 3 (figures 2 and 3). figure 2. student performance on take-home exams from a single section (n=24) figure 3. student performance on in-class exams from a single section (n=24) 83% 61% 91% 96% 0% 20% 40% 60% 80% 100% 120% in class exam 1 in class exam 2 in class exam 3 final in class exam students scoring 70% or better on inclass essay exams 87% 78% 96% 83% 0% 20% 40% 60% 80% 100% 120% take home exam 1 take home exam 2 take home exam 3 final take home students scoring 80% or better on take-home exams handwriting to learn 67 journal of effective teaching in higher education, vol. 1, no. 1 the grade distribution in each of these exams helped to classify the students into pre-liminal and liminal. there were those who were yet unaware of the importance of handwriting to learn or hesitant to put in the effort; these students can be viewed as pre-liminal. within the liminal group, there were those who were clearly stuck or seriously struggling; these students can be viewed as being liminal recursive. there were those who were coping well with this study strategy and producing good results; these students can be viewed as being liminal excursive. this classification was extremely useful in understanding and being empathetic towards the learner’s struggle in acquiring a new skill (figure 4). students scoring 80% or better on the in-class essay exam were classified as liminal excursive, students scoring 70-79% as liminal recursive, and those scoring below 70% as preliminal. figure 4. the percentage of liminal excursive students was low in the first two exams in this section (n=24) figure 5. the percentage of liminal excursive students rose significantly on the last two exams in this section (n=24) handwriting to learn 68 journal of effective teaching in higher education, vol. 1, no. 1 following numerous interventions in the form of one-on-one student consultations, student engagement and performance significantly improved in the latter part of the semester (figure 5). on the first day of class, when asked about their study habits, only 17% of the students stated that they used handwriting to learn. and even those that did use handwriting to learn stated that they used it only on rare occasions. by the end of the semester, 83% of the students were successfully using handwriting to learn. when asked about the usefulness of the take-home exams in their performance, 100% of the students stated that they found it useful. this number does not include those who withdrew from the course, which was about 11%. for each lecture exam, at least 80% of the students submitted drafts of their take-home exams for feedback a week before the actual exam, thereby demonstrating their ability to curb procrastination in preparing for the lecture exam. these students also stated that they used their take home essays to study for the lecture exam. table 7. sample student reflections on handwriting to learn recorded at mid-semester reflection prompt student reflections i believe that the take home exam helped me because… “those are usually hard questions, which then helps me understand them more” “i learn better by writing” “it forces me to really take the time out to study throughout my busy schedule between work and school. if i didn’t have the take home, i would probably be cramming rather than taking the time needed out” “it helped me understand the concept we were learning much better than just writing it in class” until i took these take home exams i never realized that… “studying could be so helpful by drawing and learning everything twice” “it will help me a lot. it helps to organize study notes and be prepared for the exam” “the time and patience it takes to review all the notes and power points to figure out and answer the take home questions” “actually, rewriting your notes and diagrams and definitions is a lot more effective than re-reading and highlighting things of importance” what i don’t like about take-home exams is that… “although rewriting is tedious, it’s so helpful for memory” “certain questions require a lot of information so it takes a little more time to understand and to be able to translate it back onto the exam” “it’s very lengthy and i always have to take 2-3 hours to complete it effectively” “sometimes it is stressful finding some answers” discussion and conclusions handwriting to learn as a study skill was successfully embedded in a flipped learning section of a gateway science course. results show that students took advantage of the desirable difficulties presented to them in the form of weekly handwriting to learn 69 journal of effective teaching in higher education, vol. 1, no. 1 quizzes, take-home, and in-class essay exams. as a result, they were able to develop their note-taking ability, beat procrastination, generate their own study materials, and enhance transcription fluency. the latter was demonstrated by the generally good performance on the in-class essay exams, which were essentially high impact retrieval practice opportunities. at the beginning of the semester only 17% of the students stated that they sometimes used handwriting as a study skill; at the end of the semester that number increased to 87%. students not only used handwriting to learn but they were all in favor of it by the end of the semester. even those students who were at the pre-liminal stage until mid-term were encouraged to step into the liminal space and take full advantage of this threshold experience (see figures 4 and 5). this was accomplished by regular one-on-one consultations with the pre-liminal students to influence positive study behavior (see table 5). students certainly struggled with the threshold skill of handwriting to learn. they found it to be troublesome due to the time commitment involved (see table 7). student performance in the second unit exam shows a definite recursive period during this threshold experience. yet, in the end, 83% of the students crossed the threshold and appeared to be transformed by this experience (see figure 5). their excursive journey began towards the end of the semester as shown by their strong performance in unit exam 4 and continued to remain steady for the cumulative final exam (see figure 5). both the flipped learning approach and the take-home in-class hybrid essay exam format appear to have had a positive effect on overall student performance and general morale of the students (see table 6). students met all the benchmarks set for this project including significantly improved retention and pass rates with 83% students passing the course with a grade of c or better and 96% of the students remaining enrolled throughout the course of the semester. these success rates are significantly higher than that of average pass and retention rates for this course. individual student consultations were not just meant for students at-risk. they also helped to curb procrastination. before the actual date of the unit exam, about 80% of the students regularly submitted their work for feedback well before the day of the lecture exam to make sure that they were on the right track (see table 2). this served as a sort of guarded threshold in which the instructor does not allow the student to enter the next stage of this threshold experience until they can demonstrate their ability to correctly answer the essay questions on the take-home exam (akkaraju, 2015). these individual student consultations appeared to significantly improve student performance when you compare student performance in the first essay exam with the final essay exam (see figures 4 and 5). student written reflections and one-on-one consultations on handwriting to learn were very encouraging. although they found this process to be time-consuming, a few appeared to be transformed by this learning experience (see table 7) thereby fulfilling the troublesome and transformational aspects of a typical threshold experience. in future semesters, i plan to focus more on the troublesome and handwriting to learn 70 journal of effective teaching in higher education, vol. 1, no. 1 transformational aspects of adopting handwriting to learn as a study skill to understand the extent to which this experience impacts both immediate and longterm study behavior. as an added note, a comparison of student performance in the multiple-choice portion of the exams versus the essay portion of the exams showed no correlation. this may have been because the learning opportunities provided for the multiplechoice portion of the exam were unlike the learning opportunities presented here for the essay portion of the exam. in the future, it is important to address whether or not students can demonstrate their understanding of conceptual information and apply these concepts to real life situations as a direct result of using handwriting to learn as a study skill. conflicts of interest the author(s) declare(s) that there is no conflict of interest regarding the publication of this article. references agostinho, s., tindall-ford, s.k., ginns, p., howard, s.j., leahy, w. & paas, f. (2015). giving learning a helping hand: finger tracing of temperature graphs on an ipad. educational psychology review, 27(3), 427–443, doi: 10.1007/s 10648-015-9315-5. akkaraju, s. (2015). writing creatively about evolution: overlapping threshold experiences. double helix, 3, 1–15. retrieved from http://qudoublehelixjournal.org/index.php/dh/article/view/41 akkaraju, s. (2016). the role of flipped learning on managing cognitive load in a threshold concept in physiology. journal of effective teaching, 16(3), 28–43. bjork, r.a. (1994). institutional impediments to effective training, in r.a. bjork and d. druckman (eds.), learning, remembering, believing: enhancing human performance (pp. 295–307). washington, dc: national academy press. borghi, a.m., & cimatti, f. (2010). embodied cognition and beyond: acting and sensing the body. neuropsychologia, 48(3), 763–773. boyle, j.r. (2007). the process of note-taking: implications for students with mild disabilities. the clearing house, 80(5), 227–232. boyle, j.r. (2010). strategic note-taking for middle school students with learning disabilities in science classes. learning disability quarterly, 33(2), 93–109. brown, p.c., roediger, h.l., and mcdaniel, m.a. (2014). make it stick: the science of successful learning. cambridge, massachusetts: belknap press. butler, a.c., & roediger, h.l., iii. (2007). testing improves long-term retention in a simulated classroom setting. european journal of cognitive psychology, 19, 514–527. cousin, g. (2006). an introduction to threshold concepts. planet, 17, 4–5. cowan, n. (2001). the magical number 4 in short term memory: a reconsideration of mental storage capacity. behavioral and brain sciences, 24, 87–114. http://qudoublehelixjournal.org/index.php/dh/article/view/41 handwriting to learn 71 journal of effective teaching in higher education, vol. 1, no. 1 engel, a.k., maye, a., kurthen, m., & konig, p. (2013). where’s the action? the pragmatic turn in cognitive science, trends in cognitive science, 17(5), 202– 209. fogassi, l., & gallese, v. (2004). action as a binding key of multisensory integration. in g.a. calvert, c. spense, & b.e. stein (eds.), the handbook of multisensory processes, 425–441. cambridge, massachusetts: mit press. foglia, l., & wilson, r. (2013). embodied cognition. wires: cognitive science, 4(3), 319–325. fredricks, a.f., blumenfeld, p.c., & paris, a.h. (2004). school engagement: potential of the concept, state of the evidence. review of educational research, 74(1), 59–109. huijser, h., kimmins, l., & galligan, l. (2008). evaluating individual teaching on the road to embedding academic skills. journal of academic language & learning, 2(1), a23-a38. karpicke, j. (2012). retrieval-based learning: active retrieval promotes meaningful learning. current directions in psychological science, 21(3), 157–163. kiewra, k.a. (1985). investigating note-taking and review: a depth processing alternative. educational psychologist, 20, 23–32. kiewra, k.a., & benton, s.l. (1988). the relationship between information processing ability and note-taking. contemporary educational psychology, 13,13–44. kinomura, s., larsson, j., gulyas, b., & roland, p.e. (1996). activation by attention of the human reticular formation and thalamic intra-laminar nuclei. science 271(5248), 512–515. land, r., cousin, g., meyer, j. h. f., & davies, p. (2005). threshold concepts and troublesome knowledge: linkages to ways of thinking and practicing (3): implications for course design and evaluation. in c. rust (ed.), improving student learning diversity and inclusivity, proceedings of the 12th improving student learning conference (pp. 53–64). oxford: oxford centre for staff and learning development (ocsld). mangen, a., & velay j.-l. (2010). digitizing literacy: reflections on the haptics of writing. advances in haptics. retrieved from http://www.intechopen.com/books/advances-in-haptics/digitizing-literacyreflections-on-the-haptics-of-writing marsh, e.j., & butler, a.c. (2013). memory in educational settings. in de. reisberg (ed.). the oxford handbook of cognitive psychology (pp. 299–317). oxford: oxford university press. mayer, r.e. (2005). cognitive theory of multimedia learning. in r.e. mayer (ed.), the cambridge handbook of multimedia learning (pp. 31–48). new york: cambridge university press. meyer, j. h. f., & land, r. (2003). threshold concepts and troublesome knowledge: linkages to ways of thinking and practicing. in c. rust (ed.), improving student learning—theory and practice ten years on (pp. 412–424). oxford: oxford centre for staff and learning development (ocsld). meyer, j. h. f., & land, r. (2005). threshold concepts and troublesome knowledge (2): epistemological considerations and a conceptual framework for teaching and learning. higher education, 49(3), 373–388. http://www.intechopen.com/books/advances-in-haptics/digitizing-literacy-reflections-on-the-haptics-of-writing http://www.intechopen.com/books/advances-in-haptics/digitizing-literacy-reflections-on-the-haptics-of-writing handwriting to learn 72 journal of effective teaching in higher education, vol. 1, no. 1 meyer, j. h., land, r., & baillie, c. (2010). threshold concepts and transformational learning. in land, r., meyer, j.h., baillie, c. (eds.), threshold concepts and transformational learning (pp ix–xii). rotterdam: sense publishers. mueller, p.a., & oppenheimer, d.m. (2014). the pen is mightier than the keyboard: advantages of longhand over laptop notetaking. psychological science, 25, 1159–1168. doi:10. 1177/0956797614524581. peverly, s.t., ramaswamy, v., brown, c., sumowsky, j., alidoost, m., & garner, j. (2007). what predicts skill in lecture note-taking? journal of educational psychology, 99(1), 167–180. doi: 10.1037/0022-0663.99.1.167 piolat, a., olive, t., & kellogg, r.t. (2004). cognitive effort of note-taking. applied cognitive psychology, 18, 1–22. doi:10.1002/acp.1086 roediger, h.l. iii, & pyc, m.a. (2012). inexpensive techniques to improve education: applying cognitive psychology to enhance educational practice. journal of applied research in memory and cognition, 1(2012), 242–248. roediger, h.l., iii, & butler, a.c. (2011). the critical role of retrieval practice in long-term retention. trends in cognitive sciences, 15, 20–27. sweller, j. (1994). cognitive load theory: learning difficulty and instructional design. learning and instruction, 4, 295–312. thomas, l., boustedt, j., eckerdal, a., mccartney, r., mostrom, j.e., sanders, k., & zander, c. (2017). in the liminal space: software design as a threshold skill. practice and evidence of scholarship of teaching and learning in higher education, 12(2), 333–351. table 2. a scaffolded design of the take-home in-class hybrid exams to curb procrastination, generate study guides, enhance transcription fluency, and provide opportunities for retrieval practice. assessment vehicle student learning outcome rationale stages of the hybrid essay exam handwritten early drafts of take-home exam student will be able to rewrite, organize, and align lecture notes with the essay questions listed in the take-home exam. curb procrastination stage i: students are given about 12 essay questions as a take-home exam. they are encouraged to work in groups generate study guides stage ii students present their answers for feedback before the actual deadline. in-class exam students will be able to demonstrate a high degree of transcription fluency enhance transcription fluency stage iii: students are instructed to practice rewriting their essays and re-drawing illustrations before they sit for the unit exam students will be able to provide answers to essay questions that are relevant, accurate, sufficiently detailed, and accompanied by illustrations wherever appropriate. provide opportunity for high-impact retrieval practice stage iv: students take the unit exam, which is a combination of multiple-choice questions (50%) and essay questions (50%) skills involved designed to assess student ability to…. essay question drawing a flow chart 1. describe a process 2. use key terms correctly 3. recognize subtle yet important differences in the two examples making a table or creating a mind map 1. classify and organize information 2. recognize subtle yet important differences between molecules 3. use key terms correctly drawing molecular structures to illustrate a process 1. describe a process show the formation of the bonds that hold nacl and cacl2 together using drawings. specify the type of bond that is formed here and the changes in charge that occurs. 2. display logical reasoning 3. use key terms correctly drawing diagrams to illustrate a process 1. describe a process 2. display logical reasoning 3. recognize subtle yet important differences between molecules 4. use key terms correctly drawing molecular structures to illustrate a process 1. describe a process 2. display logical reasoning 3. recognize subtle yet important differences between molecules 4. use key terms correctly table 4. rubric used to assess take home and in-class exams viewing the skill acquisition process as a threshold experience for students in the context of the threshold experience, learning occurs within the liminal environment, which is a nurturing environment that is created and maintained by the instructor. within this environment is the liminal space in which the student may strugg... a variety of assessments (formative, benchmark, and summative) were used to identify pre-liminal, liminal, and post-liminal students (table 5). the pre-lecture quiz served as a weekly formative assessment. it was used to assess the quality of preparat... one-on-one student meetings were designed to address the affective, behavioral, and cognitive dimensions of student engagement (fredricks, blumenfeld, & paris, 2004) with the goal of positively influencing study behavior (table 5). particular emphasis... figure 1. students achieving benchmark in all four sections combined (n=80) figure 2. student performance on take-home exams from a single section (n=24) figure 3. student performance on in-class exams from a single section (n=24) the grade distribution in each of these exams helped to classify the students into pre-liminal and liminal. there were those who were yet unaware of the importance of handwriting to learn or hesitant to put in the effort; these students can be viewed ... figure 4. the percentage of liminal excursive students was low in the first two exams in this section (n=24) figure 5. the percentage of liminal excursive students rose significantly on the last two exams in this section (n=24) following numerous interventions in the form of one-on-one student consultations, student engagement and performance significantly improved in the latter part of the semester (figure 5). on the first day of class, when asked about their study habits, only 17% of the students stated that they used handwriting to learn. and even those that did use handwriting to learn stated that they used it only on rare occasions. by the end of the se... table 7. sample student reflections on handwriting to learn recorded at mid-semester discussion and conclusions handwriting to learn as a study skill was successfully embedded in a flipped learning section of a gateway science course. results show that students took advantage of the desirable difficulties presented to them in the form of weekly quizzes, take-ho... at the beginning of the semester only 17% of the students stated that they sometimes used handwriting as a study skill; at the end of the semester that number increased to 87%. students not only used handwriting to learn but they were all in favor of ... students certainly struggled with the threshold skill of handwriting to learn. they found it to be troublesome due to the time commitment involved (see table 7). student performance in the second unit exam shows a definite recursive period during this... both the flipped learning approach and the take-home in-class hybrid essay exam format appear to have had a positive effect on overall student performance and general morale of the students (see table 6). students met all the benchmarks set for this p... individual student consultations were not just meant for students at-risk. they also helped to curb procrastination. before the actual date of the unit exam, about 80% of the students regularly submitted their work for feedback well before the day of ... student written reflections and one-on-one consultations on handwriting to learn were very encouraging. although they found this process to be time-consuming, a few appeared to be transformed by this learning experience (see table 7) thereby fulfillin... as an added note, a comparison of student performance in the multiple-choice portion of the exams versus the essay portion of the exams showed no correlation. this may have been because the learning opportunities provided for the multiple-choice porti... microsoft word ricci_final.docx journal of effective teaching in higher education, vol. 6, no. 1 https://doi.org/10.36021/jethe.v6i1.67 aiming for alignment: faculty and student perspectives on effectively teaching first-generation students leila ansari ricci, california state university, los angeles, lricci@calstatela.edu rebecca french, california state university, los angeles abstract. first-generation college students comprise a sizable number of the student population in colleges and universities across the nation. as faculty are tasked with teaching and supporting these students in their courses, it is important to consider the alignment of faculty perspectives and the needs of first-generation students. this study explored the perspectives of 176 post-baccalaureate students and 54 faculty at a large, urban hispanic-serving university in california, with nearly 77% of the campus population consisting of first-generation students and 65% of latinx students. data collection consisted of surveys completed by faculty as well as first-generation and non-first-generation students. although firstgeneration and non-first-generation students did not differ significantly in their ratings of faculty’s teaching practices, the means were lower for more items among the first-generation students. findings also indicate that greater alignment is needed between how faculty teach and what first-generation students need to succeed, most notably in the lower ratings of faculty teaching by students versus faculty themselves and the differing themes emerging from qualitative analyses of participants’ responses. keywords: first-generation college students; faculty practices; student perspectives nationwide, higher education faculty and leaders are tasked with teaching and ensuring the success of significant numbers of first-generation college students. first-generation refers to students who are the first in their family to attend college, with neither parent having obtained education levels beyond high school. in 2017, nearly 18% of those enrolled at four-year universities were first-generation students (stolzenberg et al., 2019). many first-generation students tend to be members of racially or ethnically underrepresented groups (hutchens et al., 2011). in 2016, hispanic students (from here on, referred to as latinx) comprised 18% of all students in degree-granting postsecondary institutions in the united states, with 39% of students between the ages of 18 to 24 being of latinx backgrounds (snyder et al., 2019). it is increasingly important to consider the needs and experiences of first-generation latinx students along with the perspectives of faculty who teach these students. while first-generation students are not a monolithic group—e.g., varying in race, ethnicity, and socioeconomic status—research points to several themes in the characteristics and needs of first-generation students. these students may experience more academic struggles; work more hours in college; have fewer financial resources; have lower grades and rates of college graduation; and may experience more conflict between the demands of school and community/family aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 26 priorities (hutchens et al., 2011; soria & stebleton, 2012; stephens et al., 2012; ward et al., 2012). these students often have less parental support and less information about higher education (sy et al., 2011). first-generation students may be less academically engaged, less likely to interact with professors, and contribute less frequently to class discussions (soria & stebleton, 2012). furthermore, these students tend to rely on themselves and often believe that they should be figuring out course expectations on their own rather than reaching out for support (yee, 2016). wang and castañeda-sound (2008) found that first-generation students scored lower on measures of self-esteem and self-efficacy than non-first-generation peers, particularly those of ethnic minority backgrounds. first-generation students may also lack the social and cultural capital that are needed to succeed in the university, including navigating the explicit and implicit norms and expectations of higher education (banks-santilli, 2014; stephens et al., 2012; ward et al., 2012). like other first-generation students, latinx students—who are part of the nation’s largest underrepresented group at 18.3% of the population (u.s. census bureau, 2018)—have intrinsic motivation and an internal drive to attend college (blackwell & pinder, 2014; trevino & defreitas, 2014). however, conflicts between college and family expectations can negatively affect their well-being and achievement (vasquez-salgado et al., 2015). this challenge of family obligations or families not understanding the demands of college can negatively impact these students’ choices and progress, particularly for latinas experiencing cultural gender role expectations (leyva, 2011; mitchall & jaegar, 2018). further, valencia (2002) writes that families are expected to show care according to white, middle-class standards; however, many first-generation families face structural barriers that do not allow them to show care in this manner. these identities living separately from each other creates a unique perspective and experience for female students who identify as latinx (crenshaw, 1991). banda (2020) found that even small overtures from faculty that demonstrate interest in students could be very important for feelings of belonging among latinx female students, who may feel alienated in their departments. faculty attention plays a critical role in the academic experiences of first-generation students. for example, of students who were satisfied to very satisfied at college overall, 86.1% reported close interactions with at least one faculty member who had taken an interest in their success (couch, 2019). faculty often have differing viewpoints about first-generation students: one view is that these students are underprepared and difficult to reach and the other view is that faculty can meet these students where they are and play a significant role in bolstering their academic skills (schademan & thompson, 2016). the perspective that faculty adopt appears to have a great impact on how first-generation students view their potential and belonging as well as their academic persistence and learning (irlbeck et al., 2014; kim & sax, 2009; watt et al., 2013; wirt & jaegar, 2014). while firstgeneration students may be reluctant to seek out interactions with their professors, researchers have found that positive relationships with faculty lead to more persistence in college. for example, demetriou et al. (2017) found that the following promoted first-generation students’ success: active engagement in aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 27 coursework; participating in faculty-mentored research; and involvement with campus organizations that emphasize close interactions with faculty. through these relationships, professors can help first-generation students navigate the hidden norms and expectations of the university (longwell-grice et al., 2016). watt et al. (2013) found that relationships with faculty were more important to first-generation students than even the teaching strategies used in classes. increasing first-generation students’ sense of belonging is also positively associated with their engagement in their courses and progress toward graduation (hausmann et al., 2007). establishing meaningful connections with students, reducing feelings of alienation, and mentoring students can help first-generation students succeed in college (jehangir, 2010; soria & stebleton, 2012). faculty can support firstgeneration students by having high expectations for them, being clear and transparent in their instruction, actively encouraging students to participate in classes, and helping them navigate norms and rules of campus (demetriou et al., 2017; kim & sax, 2009; longwell-grice et al., 2016; macias, 2013; tello & lonn, 2017). banks-santilli (2014) described supportive faculty advisors as those who ensure that students clearly understand the syllabus, give verbal reminders, and encourage students to make individual appointments with them. in addition, hutchen et al. (2022) found that first-generation graduate students felt they needed faculty to understand that many are working full time or have additional commitments outside of the university, making it difficult to take advantage of resources offered. means and pyne (2017) found that faculty were one of the most important factors for these students’ sense of belonging on campus. first-generation students benefited when faculty recognized and filled any gaps in prior knowledge; used multiple modalities during instruction (visual, auditory, kinesthetic); provided visuals and lecture notes; developed students’ self-efficacy skills; engaged in outreach, relationship building, and one-on-one conversations with students; provided feedback; and were approachable in their demeanor. these actions can mitigate the feelings of disconnectedness or invisibility that first-generation students can feel on college campuses. according to kim and sax (2009), “more contact between students and faculty, both inside and outside the classroom, enhances college students’ development and learning outcomes” (p. 438), even increasing the chances of these students remaining in college by up to 13% for each meeting between faculty and students (swecker et al., 2013). exploring faculty and first-generation student perspectives, collier and morgan (2008) found that faculty felt they provided clear and explicit standards and assignments and were concerned about students meeting their expectations in terms of managing course workload, prioritizing academics, following directions, and communicating with faculty about their needs. winkelmes et al. (2016) identified why students were completing an assignment, how students would complete the assignment, and how the assignments would be graded as connected to both student success in the course and linked this to success in future courses. students wanted faculty to be explicit in their expectations, provide detailed explanations of syllabi, and understand that they had both a different set of aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 28 demands on their time outside of courses as well as fewer resources. the way that professors spoke to the class influenced whether first-generation students approached faculty for questions or clarification. macias (2013) advocates for a strength-based approach to supporting first-generation students, focusing on what they can do and having high expectations rather than focusing on deficits. several factors can influence success in college for first-generation latinx students, including feelings of cultural incongruence (a lack of cultural fit between the student’s values and the culture of the university) as well as institutional or interpersonal microaggressions occurring on campus (tello & lonn, 2017). these factors emphasize the importance of creating safe, welcoming spaces on campus and promoting the social capital and feelings of cultural belonging for these students. values of latinx culture include interdependence, harmonious personal interactions, and personalized communication styles, which comprise the funds of knowledge (or the lived experiences and backgrounds) that first-generation latinx students bring with them to higher education (castillo-montoya, 2017). faculty can promote the success of these students by validating, making space for, and modifying their instruction to include these funds of knowledge. faculty can facilitate equity-based learning through having high expectations and showing confidence in students’ ability to succeed; co-constructing knowledge with their students; and developing students’ understanding of the ways in which social factors can affect their lived experiences (castillo-montoya, 2017; paris, 2012; paris & alim, 2017). using culturally responsive pedagogy or engaging with the cultural backgrounds of students has been shown to improve student outcomes (ladson-billings, 2014; museus & quaye, 2009) and can be particularly important for first-generation, latinx students. in addition, montero-hernandez and drouin (2021) found that faculty and universities should lean into the strengths and skills first-generation graduate students bring with them to the classroom. since developing faculty’s skills in teaching first-generation students can improve students’ learning outcomes (verschelden, 2017), it is important to consider the alignment of faculty perspectives with the needs identified by first-generation students as critical for their academic success. the alignment, or misalignment, of these perspectives and needs can yield lessons for increasing the academic retention and success of potentially vulnerable students. to this end, the present study examined the perspectives of faculty and both first-generation and non-firstgeneration students, more than half of whom were of latinx backgrounds, about effective teaching to support student needs. methods participants the participants were 176 students and 54 professors at a comprehensive, urban hispanic-serving university in california, with first-generation students comprising nearly 77% and latinx students 65% of the campus student population. the students consisted of 93 first-generation students and 83 non-first-generation students; 134 were females, 38 were males, and four individuals identified as nonaiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 29 binary. more than half (57%) were of latinx backgrounds; 16% were white; 16% were asian american; and 7% were black/african american. sixty-four percent of the students were ages 34 and under. ten percent were in undergraduate programs, while 90% were enrolled in master’s degree programs or postbaccalaureate credentials/certificates. of the 54 faculty, 22 were males, 31 were females, and one identified as nonbinary. seven were assistant professors, 7 were associate professors, 16 were full professors, and 24 were lecturers. forty-three percent were white, 35% were latinx/hispanic, 2% were black/african american, 15% were asian american, and 5% identified as other or declined to state their ethnicity. all faculty, except for one, taught post-undergraduate students. measures the measures consisted of two surveys developed by the first author that were based on teaching practices described in prior research as benefiting firstgeneration students (the practices are described in appendix a, which contains survey items). the students completed a 23-item survey, consisting of 20 likerttype questions rated on a 5-point scale (from 1 = strongly disagree to 5 = strongly agree) and three open-ended questions about students’ needs. for the survey items, cronbach’s alpha was .96, indicating strong or excellent reliability. the faculty survey consisted of 23 questions, three of which were open-ended questions and 20 were likert-type questions rated on a 5-point scale (1 = strongly disagree to 5 = strongly agree). the questions asked faculty about their current teaching practices and students’ needs. cronbach’s alpha was .94 for this survey. procedures approval to conduct this research was obtained by the institutional review board (irb) of the university in which the first author serves as a faculty member and the second author is a doctoral student. the study was supported by a small internal project grant awarded to the first author. the first author created the research surveys in qualtrics and sent an invitation with an anonymous survey link via faculty and student email lists. the quantitative data from the surveys was analyzed with spss data analysis software. the qualitative data in this study was coded using the constant comparative method used to develop a grounded theory (glaser & strauss, 1967) as well as procedures recommended for sound analysis in qualitative research (brantlinger et al., 2005) to ensure validity of this study. first, both the open-ended responses to the student and faculty surveys were separately coded using initial codes. every sentence or thematically organized cluster of phrases that contained a complete thought was assigned a code. for the student survey, both authors independently coded 100% of the data, generating a total of 63 initial codes, with 95% agreement. for the faculty survey, both authors also independently coded 100% of the data, creating a total of 45 codes, with 89% aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 30 initial agreement. after further discussion, the authors reached full agreement on all the initial codes. in the next step of data analysis, each author independently analyzed the initial codes for emerging themes for each survey. for the student survey, the 63 initial codes were collapsed into six central themes. for the faculty survey, the 45 initial codes were synthesized into six themes. both authors generated themes for all the data, regularly discussing the process of data coding to increase the credibility of the qualitative analysis. the system of codes and themes was formed and finalized based on these discussions, both during initial coding and when the central themes were identified, thus ensuring coding validity and interrater reliability. results for both first-generation and non-first-generation students, the means of the 20 survey items were all above 3.50, and in most cases, reflected students’ agreement with the statements related to faculty effectiveness in supporting their academic success. there were no significant differences between students who are firstgeneration versus non-first-generation students on the survey items, except for “my professors make an active effort to learn my name and get to know me” (firstgeneration m = 3.94, sd = .91; non-first-generation m = 4.22, sd = .81; t (174) = -2.16, p < .05). however, the means for 16 out of 20 survey items were lower for students who were first-generation students. for the 20 items on the survey, nonfirst-generation students agreed (means of above 4.00) with 16 of the statements regarding faculty’s teaching practices; first-generation students agreed with only 10 of the survey items. for example, the means for first-generation students were lower for the following items: “my professors value my cultural background, knowledge, and experiences in their courses” and “my professors encourage my sense of belonging and value my presence in their courses.” the highest mean scores for first-generation students were on the items such as “my professors make sure that i understand the purpose, tasks, and grading criteria of all course assignments” and “my professors make an active effort to encourage my participation in their courses.” for non-first-generation students, the highest means were for items including “my professors are available and accessible if i have questions or need support” and “my professors make an active effort to learn my name and to get to know me.” the lowest survey means for both groups of students was “my professors ask me for my feedback on how i am learning and what supports i need to succeed in their courses.” the means for the faculty survey items were all above 4.00, indicating that the professors agreed that they implemented these teaching practices in their courses. the three most frequently reported teaching practices were making an active effort to encourage students’ participation in their courses; encouraging students’ sense of belonging and valuing their presence in courses; and actively offering support, advising, and mentorship to help their students succeed in courses (means ranging from 4.67 to 4.76). although the faculty agreed that they did engage in these teaching practices, the three behaviors with the lowest means for faculty were being confident in their ability to effectively teach first-generation students; asking aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 31 students for feedback on how they are learning and what they need to succeed in their courses; and giving students timely, high-quality feedback on their assignments (means ranging from 4.11 to 4.30). across all items related to faculty’s teaching practices, the means for all the students (as well as the sample of only first-generation students) were significantly lower than those of the faculty (all with p values of < .01), indicating that students were less likely to rate their faculty as implementing these teaching behaviors. the most significant discrepancies between the ratings of faculty and those of the firstgeneration students were found in the following survey items: actively working to minimize students’ feelings of alienation in courses, with first-generation student m = 3.63 (.98), faculty m = 4.49 (.61); being understanding of any personal challenges students experience that can influence academic success in courses, with first-generation m = 3.77 (1.05), faculty m = 4.51 (.61); encouraging students’ sense of belonging and valuing their presence in courses, with firstgeneration m = 3.95 (.84), faculty m = 4.66 (.48); and actively offering support, advising, and mentorship to help students succeed in courses, with first-generation m = 3.95 (.98), faculty m = 4.66 (.48). for the themes emerging from the student survey data, the students’ greatest needs were the following: effective teaching, understanding, advising, communication, and no needs. effective teaching students described a variety of examples of how professors could further support their learning in coursework, including clarity in assignments, timely and clear feedback, checking for understanding, awareness of the amount of coursework, and use of real-world scenarios in class. some students asked that professors review assignments at least one week before due dates to ensure student understanding. several students also responded that assignments should be meaningful and relevant to their current or future jobs. both first-generation and non-firstgeneration students also expressed the need for more timely feedback for their assignments. this was particularly relevant for the first-generation students who asked for a guide or examples of the level and type of work needed to successfully complete assignments. the amount of reading given in courses was also a concern. students wanted professors to dive more deeply into readings during class time to ensure understanding. in addition, first-generation students were particularly interested in having professors check in with them about their understanding of course content. several students explained that they were able to learn more when professors shared stories instead of lecturing and requested that professors use more visuals and real-world examples instead of reading from powerpoint slides during class. understanding the students in this study stated that they needed their professors to understand who they were and where they were coming from to fully support their learning. for aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 32 example, the students wanted professors to understand their outside responsibilities while also expressing empathy, showing flexibility, and being culturally sensitive by getting to know the students and their backgrounds. attending classes full time and working full-time jobs affected the students’ ability to complete and even understand coursework because they were dividing their time between both commitments. for students, this flexibility showed an understanding of what students were going through and what their needs were. students were grateful when professors offered options to attend classes or office hours through virtual means (e.g., skype or zoom). the students wanted their professors to get to know them beyond only name recognition. one first-generation student said, “find out at least 1-2 passions or interests, create opportunities for students to connect outside of class based on shared interests, goals, and course related matters.” advising for many students, some of their needs were outside of the classroom, but they believed that faculty could support and guide them in accessing college and campus services. specifically, students wanted clear information about program and graduation requirements, unified online portals, career and job placement guidance, and support for identifying and understanding campus-wide resources. several students wanted roadmaps for coursework, as they had taken unnecessary courses or had taken courses out of sequence. they asked for specific contact people for information and for professors to be knowledgeable about all aspects of their programs. students wanted faculty mentorship programs, hoping that professors would talk about job placement and career paths after graduation. communication students reported needing clear, transparent, and timely communication from faculty through both electronic means and face-to-face interactions. students appreciated when professors sent reminders of due dates and assignment requirements because students are very busy and often work full time. professors who made themselves available to clarify expectations were important to many students. for some students, effective communication was also a means to get to know one another and feel less intimidated by professors. overall, the faculty indicated that first-generation students’ greatest needs were the following: academic support; advising and mentoring; social and emotional support; and time management. academic support according to faculty, the greatest need of first-generation students was the need for academic support. these areas in need of support included content knowledge, writing and grammar, analytic thinking, reading, language, and study skills. faculty expressed concern that students did not have enough background knowledge to be aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 33 successful in classes, but several professors stated that it was their responsibility to reteach as needed. advising and mentoring faculty noted that first-generation students need support to understand, navigate, and use campus systems and resources, including financial aid advice and detailed roadmaps for completing their programs. furthermore, faculty stated that these students need role models in and outside of the university and feedback to improve performance and understanding. faculty emphasized that first-generation students need guidance to understand the unwritten rules of higher education and to know that their struggles are normal. social and emotional support faculty stated that first-generation students need to feel welcome and safe both on campus and in their classes, including a sense of belonging and a belief in themselves. cultural sensitivity to their specific needs would serve as an additional support. some professors commented on first-generation students needing to feel that it is acceptable for them to surpass their family members and peers in their educational levels. time management how first-generation students manage their time in meeting all their obligations and responsibilities was mentioned by the faculty in this study. faculty were aware that first-generation students often have financial and family obligations outside of their coursework that create obstacles to completion of course assignments and requirements. faculty stated that having so many obligations prevented students from focusing in any one area to the detriment of other aspects of their lives. discussion given that first-generation students of latinx backgrounds comprise a sizable amount of the student population in colleges and universities across the nation (snyder et al., 2019; stolzenberg et al., 2019), faculty are tasked with effectively teaching and supporting these students in their courses. this study of 176 students and 54 faculty at a large, urban hispanic-serving university in california examined the alignment of faculty and student perspectives. the findings of this study expand understanding of the needs of first-generation students and how faculty can more effectively teach these students. first-generation and non-first-generation students’ perceptions of teaching behaviors since the means for all student survey items were 3.50 or greater, it is encouraging that both first-generation and non-first-generation students in this study tended to agree that their faculty’s teaching behaviors were supportive of their learning. for aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 34 example, both groups of students agreed that their faculty clearly communicated the expectations and requirements for their courses, made an active effort to encourage their participation in courses, were available and accessible if the students had questions or needed support, and had high expectations of them—all of which are aligned with prior research about faculty teaching behaviors that support first-generation students (banks-santilli, 2014; collier & morgan, 2008; demetriou et al., 2017; kim & sax, 2009; longwell-grice et al., 2016; macias, 2013; tello & lonn, 2017). given that first-generation students have been shown to be more satisfied with college when they had close interactions with faculty (couch, 2019), it is also encouraging that the participants agreed that they had a positive relationship with at least one professor. overall, first-generation and non-first-generation students did not differ much in their ratings of faculty’s teaching. however, it is important to acknowledge the significant difference between the two groups regarding faculty making an active effort to learn their names and get to know them. given that research has shown that rapport and connectedness with professors promotes students’ learning and motivation (frisby & martin, 2010; legg & wilson, 2009), it is imperative that professors of first-generation students actively attempt to learn about these students in their courses. the lower mean ratings by first-generation students on 16 out of the 20 survey items should be considered. first-generation students had lower means on other social and emotional aspects of teaching, such as faculty actively working to minimize any feelings of alienation that students might feel in their courses, encouraging students’ sense of belonging and valuing their presence in courses, and being understanding of any personal challenges that students experience that can influence their academic success in courses. because feelings of belonging and connection are positively correlated with firstgeneration latinx students’ engagement in courses and progress toward graduation (castillo-montoya, 2017; hausmann et al., 2007; means & pyne, 2017; tello & lonn, 2017), it is important for professors to consider how they can enhance students’ sense of belonging and show that they value their presence. since the findings related to the differences between first-generation and non-first-generation students were largely not significant, recommendations can provide guidance but should be implemented with an eye to the needs of the specific set of students in a class. the question that produced the lowest means for all students focused on faculty asking students for feedback on how they are learning and what supports they need to succeed in the courses. faculty’s perceptions of teaching behaviors this study found that faculty agreed or strongly agreed, more so than students in this study did, that they used effective teaching behaviors as reflected in the means of 4.00 and above on the faculty survey items. for example, faculty rated themselves highest in making an active effort to encourage students’ participation in their courses, encouraging students’ sense of belonging and valuing their presence in courses, and offering their support, advising, and mentorship to help their students succeed in courses. however, across all items related to faculty’s aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 35 teaching practices, the means for all students, and particularly for the firstgeneration latinx students, were significantly lower than those of faculty. for example, the greatest discrepancies between faculty and students’ perceptions were related to faculty actively working to minimize students’ feelings of alienation, being understanding of any personal challenges students experience that can influence academic success in courses, encouraging students’ sense of belonging and valuing their presence in courses, and actively offering support, advising, and mentorship to help students succeed in courses. the faculty did seem to realize that they could be more effective at teaching first-generation students and rated themselves lower in asking students for their feedback on how they are learning in their classes. however, these findings may still indicate that faculty perceptions about their teaching practices may not be entirely accurate or match their students’ perceptions or may reflect deficit thinking, thus emphasizing the importance of twoway communication and feedback for faculty to develop a greater understanding of students and their needs. student and faculty perceptions of first-generation student needs these findings showed that there were differences between what faculty reported as students’ needs versus what students, particularly first-generation latinx students, described as their greatest needs. faculty described students’ top needs as that of academic support (e.g., study skills or writing support) and also mentioned time management as a need, neither of which were mentioned by students, who instead identified advising and effective teaching as their most important needs. both faculty and students agreed on the first-generation students’ needs for advising and social/emotional support or understanding. faculty’s description of first-generation students’ lack of knowledge and skills could indicate a focus on deficit thinking in relation to these students; thus, faculty should be encouraged to focus on student strengths and a growth-based approach toward teaching. faculty should be examining their own beliefs about first-generation students for ingrained biases that could impact their perceptions of students (gorski, 2018; gutierrez keeton, 2022). this highlights the importance of faculty meeting students where they are and helping them develop the skills to move forward (montero-hernandez & drouin, 2021; schademan & thompson, 2016). although first-generation students may need to build prerequisite skills, it is imperative that faculty become more effective, responsive teachers committed to increasing the learning outcomes of all students and leaning into the strengths students bring with them. practices to increase student learning and success the students’ responses, which confirm earlier research findings, revealed several practices that faculty can adopt to increase students’ learning and success. for example, it is beneficial for faculty to be clear and transparent in their expectations and assignments and give timely feedback to their students on their work. providing samples of assignments, reviewing course readings, checking for understanding frequently, and providing more in-class practice is helpful for these students. other practices that benefit first-generation students include assigning a aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 36 realistic amount of course readings and assignments, being flexible with deadlines, offering different options for office hours, promptly responding to emails, and providing roadmaps and advising about program requirements. further, student responses show a clear need for effective teaching while faculty responses reveal a lack of knowledge about effective teaching practices to reach first-generation students. institutions have a responsibility to provide faculty with ways to develop a growth mindset approach to learning, especially among first-generation latinx students, along with effective teaching practices. for example, aguilara et al. (2020) offered a reimagining of a first-year writing course as a model in which the course was tailored specifically to the strengths and needs of first-generation students, including but not limited to analyzing the cost of books, using universal design for learning to frame the course, and using new and different media throughout the course. targeting professional development in this area may increase effective teaching and therefore student success. limitations it is important to consider the limitations of this study. first, the sample size may limit the generalizability of the findings. also, student participants had already received their bachelor’s degrees and therefore were not able to speak to the needs of current first-generation undergraduate students, which is a weakness. however, these students remain first-generation students even as they navigate higher levels of education. another limitation of this study is that faculty were not specifically asked about undergraduate versus graduate students. this research also consisted of the survey method for data collection. observations of faculty teaching and interviews with faculty and students could have strengthened the findings. further research using other methods of data collection and with a higher number of participants will further validate these findings. conclusion this study is one of few exploring the perspectives of faculty, along with that of both first-generation and non-first-generation students, and including a significant number of first-generation, latinx students. given the high numbers of firstgeneration students across the nation and efforts to increase retention and graduation rates of this population, this study contributes toward better understanding of first-generation students’ needs and faculty teaching practices. greater alignment and consistency between faculty and student perspectives can further promote the academic success of these students in higher education. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 37 references aguilera, e., lopez, g., frankel, k. k., & brooks, m. d. (2020). centering firstgeneration college students’ lived experiences through critical digital storytelling. journal of adolescent & adult literacy, 63(5), 583–587. https://doi.org/10.1002/jaal.1037 banda, r. m. (2020). from the inside looking out: latinas intersectionality and their engineering departments. international journal of qualitative studies in education, 33(8), 824–839. banks-santilli, l. (2014). first-generation college students and their pursuit of the american dream. journal of case studies in education, 5, 1–32. blackwell, e., & pinder, p. j. (2014). what are the motivational factors of first generation minority college students who overcome their family histories to pursue higher education? college student journal, 48(1), 45–56. brantlinger, e., jimenez, r., klingner, j., pugach, m., & richardson, v. (2005). qualitative studies in special education. exceptional children, 71(2), 195– 207. https://doi.org/10.1177/001440290507100205 castillo-montoya, m. (2017). deepening understanding of prior knowledge: what diverse first-generation college students in the u.s. can teach us. teaching in higher education, 22(5), 587–603. https://doi.org/10.1080/13562517.2016.1273208 collier, p., & morgan, d. (2008). “is that paper really due today?”: differences in first-generation and traditional college students’ understandings of faculty expectations. higher education, 55(4), 425–446. https://doi.org/10.1007/s10734-007-9065-5 couch, v. (2019). findings from the 2018 administration of your first college year (yfcy) survey: technical report. higher education research institute. crenshaw, k. (1991). mapping the margins: intersectionality, identity politics, and violence against women of color. stanford law review, 43(6), 1241–1299. https://doi.org/10.2307/1229039 demetriou, c., meece, j., eaker-rich, d., & powell, c. (2017). the activities, roles, and relationships of successful first-generation college students. journal of college student development, 58(1), 19–36. doi:10.1353/csd.2017.0001 frisby, b. n., & martin, m. m. (2010). instructor–student and student–student rapport in the classroom. communication education, 59(2), 146–164. https://doi.org/10.1080/03634520903564362 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 38 glaser, b., & strauss, a. (1967). the discovery of grounded theory: strategies for qualitative research. sociology press. gorski, p. (2018). poverty ideologies and the possibility of equitable education: how deficit, grit, and structural views enable or inhibit just policy and practice for economically marginalized students. counterpoints (new york, n.y.), 523, 95–120. gutierrez keeton, r., mendoza, y., zarate, m. e., & magruder, e. d. (2022). teaching faculty to reach first-generation college students: the impact of a professional development course on faculty. college teaching, 70(2), 152– 160.https://doi.org/10.1080/87567555.2021.1909526 hausmann, l. r., schofield, j. w., & woods, r. l. (2007). sense of belonging as a predictor of intentions to persist among african american and white first year college students. research in higher education, 48(7), 803–839. https://doi.org/10.1007/s11162-007-9052-9 hutchens, k., deffendall, m., & peabody, m. (2011). supporting first generation college students. kentucky journal of higher education policy and practice, 1(1), article 4. hutson, j., nasser, r., edele, s., parrish, g., rodgers, c., richmond, s., marzano, m., & curtis, r. (2022). predictors of persistence, retention & completion for first-generation graduate students. journal of organizational psychology, 22(1), 99–114. irlbeck, e., adams, s., akers, c., burris, s., & jones, s. (2014). first generation college students: motivations and support systems. journal of agricultural education, 2, 154–166. jehangir, r. (2010). stories as knowledge: bringing the lived experiences of firstgeneration college students into the academy. urban education, 45(4), 533– 553. http://dx.doi.org/10.1177/0042085910372352 kim, y. k., & sax, l. j. (2009). student-faculty interaction in research universities: differences by student gender, race, social class, and first-generation status. research in higher education, 50(5), 437– 459. https://doi.org/10.1007/s11162-009-9127-x leyva, v. l. (2011). first-generation latina graduate students: balancing professional identity development with traditional family roles. new directions for teaching and learning, 127, 21–31. ladson-billings, g. (2014). culturally relevant pedagogy 2.0: a.k.a. the remix. harvard educational review, 84(1), 74–84. https://doi.org/10.17763/haer.84.1.p2rj131485484751 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 39 legg, a.m., & wilson, j. h. (2009). e-mail from professor enhances student motivation and attitudes. teaching of psychology, 36(3), 205–211. http://doi.org/10.1080/00986280902960034 longwell-grice, r., adsitt, n. z., mullins, k., & serrata, w. (2016). the first ones: three studies on first-generation college students. nacada journal, 36(2), 34–46. https://doi.org/10.12930/nacada-13-028 macias, l. v. (2013). choosing success: a paradigm for empowering firstgeneration college students. about campus, 18(5), 17–21. https://doi.org/10.1002/abc.2113 means, d. r., & pyne, k. b. (2017). finding my way: perceptions of institutional support and belonging in low-income, first-generation, first-year college students. journal of college student development, 58(6), 907–924. https://doi.org/10.1353/csd.2017.0071 mitchall, a. m., & jaeger, a. j. (2018). parental influences on low-income, firstgeneration students’ motivation on the path to college. the journal of higher education, 89(4), 582–609. https://doi.org/10.1080/00221546.2018.1437664 montero-hernandez, v., & drouin, s. (2021). the journey into a new me: narratives of self-development and community revitalization among firstgeneration, latinx, adult graduate students. journal of adult & continuing education, 27(2), 175–204. https://doi.org/10.1177/1477971420913911 museus, s. d., & quaye, s. j. (2009). toward an intercultural perspective of racial and ethnic minority college student persistence. the review of higher education, 33(1), 67–94. paris, d. (2012). culturally sustaining pedagogy: a needed change in stance, terminology, and practice. educational researcher, 41(3), 93–97. https://doi.org/10.3102/0013189x12441244 paris, d., & alim, h. s. (eds.) (2017). culturally sustaining pedagogies: teaching and learning for justice in a changing world. teachers college press. schademan, a. r., & thompson, m. r. (2016). are college faculty and first generation, low-income students ready for each other? journal of college student retention: research, theory & practice, 18(2), 194–216. https://doi.org/10.1177/1521025115584748 soria, k. m., & stebleton, m. j. (2012). first-generation students’ academic engagement and retention. teaching in higher education, 17(6), 673–685. https://doi.org/10.1080/13562517.2012.666735 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 40 stephens, n. m., fryberg, s. a., markus, h. r., johnson, c. s., & covarrubias, r. (2012). unseen disadvantage: how american universities’ focus on independence undermines the academic performance of firstgeneration college students. journal of personality and social psychology, 102(6), 1178–1197. https://doi.org/10.1037/a0027143 stolzenberg, e. b., eagan, m. k., aragon, m. c., cesar-davis, n. m., jacobo, s., couch, v., & rios-aguilar, c. (2019). the american freshman: national norms fall 2017. higher education research institute, ucla. swecker, h. k., fifolt, m., & searby, l. (2013). academic advising and first generation college students: a quantitative study on student retention. nacada journal, 33(1), 46–53. https://doi.org/10.12930/nacada-13-192 sy, s. r., fong, k., carter, r., boehme, j., & alpert, a. (2012). parent support and stress among first-generation and continuing-generation female students during the transition to college. journal of college student retention: research, theory & practice, 13(3), 383–398. https://doi.org/10.2190/cs.13.3.g snyder, t. d., de brey, c., & dillow, s. a. (2019). digest of education statistics 2017 (nces 2018-070). national center for education statistics, institute of education sciences, u.s. department of education. tello, a. m., & lonn, m. r. (2017). the role of high school and college counselors in supporting the psychosocial and emotional needs of latinx first-generation college students. professional counselor, 7(4), 349–359. doi:10.15241/amt.7.4.349 trevino, n. n., & defreitas, s. c. (2014). the relationship between intrinsic motivation and academic achievement for first generation latino college students. social psychology of education, 17(2), 293–306. https://doi.org/10.1007/s11218-013-9245-3 u.s. census bureau. (2018). vintage 2018 data retrieved from https://www.census.gov/quickfacts/fact/table/us/rhi725218 valencia, r. (2004). “mexican americans don’t value education!” on the basis of the myth, mythmaking, and debunking. journal of latinos and education, 1, 81– 103. vasquez-salgado, y., greenfield, p. m., & burgos-cienfuegos, r. (2015). exploring home-school value conflicts: implications for academic achievement and well-being among latino first-generation college students. journal of adolescent research, 30(3), 271–305. https://doi.org/10.1177/0743558414561297 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 41 verschelden, c. (2017). bandwidth recovery: helping students reclaim cognitive resources lost to poverty, racism, and social marginalization. stylus publishing. wang, c. d. c., & castañeda-sound, c. (2008). the role of generational status, self-esteem, academic self-efficacy, and perceived social support in college students' psychological well-being. journal of college counseling, 11(2), 101–118. https://doi.org/10.1002/j.2161-1882.2008.tb00028.x ward, l., siegel, m. j., & davenport, z. (2012). first-generation college students: understanding and improving the experience from recruitment to commencement. jossey-bass. watt, k. m., butcher, j., & ramirez, e. f. (2013). advancement via individual determination (avid) at a postsecondary institution: support for first– generation college-goers. journal of latinos & education, 12(3), 202–214. https://doi.org/10.1080/15348431.2013.765804 winkelmes, m., bernacki, m., butler, j., zochowski, m., golanics, j., & weavil, k. h. (2016). a teaching intervention that increases underserved college students’ success. peer review, 18(1-2), 31–36. wirt, l. g., & jaeger, a. j. (2014). seeking to understand faculty-student interaction at community colleges. community college journal of research and practice, 38(11), 980–994. https://doi.org/10.1080/10668926.2012.725388 yee, a. (2016). the unwritten rules of engagement: social class differences in undergraduates’ academic strategies. journal of higher education, 87(6), 831–858. https://doi.org/10.1353/jhe.2016.0031 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 42 appendix a student and faculty survey means student survey items student means faculty survey items faculty means faculty survey items faculty means my professors make an active effort to learn my name and get to know me. 4.07 i make an active effort to learn my students’ names and get to know them. 4.43 i need to improve in my ability to learn my students’ names and get to know them. 3.02 my professors value my cultural background, knowledge, and experiences in their courses. 4.02 i show that i value my students’ cultural background, knowledge, and experiences in my courses. 4.49 i need to improve in my ability to show that i value my students’ cultural background, knowledge, and experiences in my courses. 3.11 my professors encourage my sense of belonging and value my presence in their courses. 4.05 i encourage my students’ sense of belonging and value their presence in my courses. 4.67 i need to improve in my ability to encourage my students’ sense of belonging and value their presence in my courses. 3.11 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 43 student survey items student means faculty survey items faculty means faculty survey items faculty means my professors make an active effort to learn my name and get to know me. 4.07 i make an active effort to learn my students’ names and get to know them. 4.43 i need to improve in my ability to learn my students’ names and get to know them. 3.02 my professors make an active effort to encourage my participation in their courses. 4.17 i make an active effort to encourage my students’ participation in my courses. 4.76 i need to improve in my ability to make an active effort to encourage my students’ participation in my courses. 3.02 my professors help me learn about and handle the requirements, norms, and expectations of the university. 4.01 i help my students learn about and handle the requirements, norms, and expectations of the university. 4.35 i need to improve in my ability to help my students learn about and handle the requirements, norms, and expectations of the university. 3.17 my professors actively work to minimize any feelings of alienation i might experience in their courses. 3.73 i actively work to minimize any feelings of alienation students might experience in my courses. 4.50 i need to improve in my ability to minimize any feelings of alienation students might experience in my courses. 3.20 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 44 student survey items student means faculty survey items faculty means faculty survey items faculty means my professors make an active effort to learn my name and get to know me. 4.07 i make an active effort to learn my students’ names and get to know them. 4.43 i need to improve in my ability to learn my students’ names and get to know them. 3.02 my professors actively check for my understanding of their course content. 3.95 i actively check for my students’ understanding of my course content. 4.43 i need to improve in my ability to actively check for my students’ understanding of my course content. 3.17 my professors provide me with clear, easy-tounderstand, and student-friendly instruction. 4.00 i provide my students with clear, easy-to-understand, and student-friendly instruction. 4.56 i need to improve in my ability to provide my students with clear, easy-tounderstand, and student-friendly instruction. 2.96 my professors make sure their classrooms are a safe and welcoming space for discussion. 4.14 i make sure my classroom is a safe and welcoming space for discussion. 4.61 i need to improve my ability in making sure my classroom is a safe and welcoming space for discussion. 2.92 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 45 student survey items student means faculty survey items faculty means faculty survey items faculty means my professors make an active effort to learn my name and get to know me. 4.07 i make an active effort to learn my students’ names and get to know them. 4.43 i need to improve in my ability to learn my students’ names and get to know them. 3.02 my professors actively offer their support, advising, and mentorship to help me succeed in their courses. 4.04 i actively offer my support, advising, and mentorship to help my students succeed in my courses. 4.67 i need to improve in my ability to offer my support, advising, and mentorship to help my students succeed in my courses. 2.77 my professors clearly communicate their expectations and requirements for their courses. 4.23 i clearly communicate with students my expectations and requirements for my courses. 4.61 i need to improve in my ability to clearly communicate with students my expectations and requirements for my courses. 2.89 my professors are available and accessible if i have questions or need support. 4.17 i am available and accessible if my students have questions or need support. 4.66 i need to improve in my ability to be available and accessible if my students have questions or need support. 2.66 aiming for alignment journal of effective teaching in higher education, vol. 6, no. 1 46 student survey items student means faculty survey items faculty means faculty survey items faculty means my professors make an active effort to learn my name and get to know me. 4.07 i make an active effort to learn my students’ names and get to know them. 4.43 i need to improve in my ability to learn my students’ names and get to know them. 3.02 my professors are understanding of any personal challenges i experience that can influence my academic success in their courses. 3.85 i am understanding of any personal challenges that my students experience that can influence their academic success in my courses. 4.52 i need to improve in my ability to understand any personal challenges that my students experience that can influence their academic success in my courses. 2.85 microsoft word baker_tukhvatulina_final.docx journal of effective teaching in higher education, vol. 6, no. 1 https://doi.org/10.36021/jethe.v6i1.344 reflections of adult learners in asynchronous online degree programs jonathan tyler baker, miami university, bakerjt@miamioh.edu sophya tukhvatulina, independent researcher abstract. the growing prevalence of online higher education courses has attracted millions of students, many of whom fall outside the traditional student demographic. this investigative study aims to close the existing knowledge gap about the experiences and preferences of adult asynchronous learners. the study was conducted as a series of semi-structured interviews with adult students enrolled in free-tuition online asynchronous programs (n = 26). the interviews were then coded using the content analysis technique. the findings suggest that the lack of consistent communication with faculty and university staff is the biggest issue faced by adult students online. respondents also reported missing the sense of community. despite that, it is the ability to do work on one’s own schedule and flexibility in choosing the format for learning materials and assignments that are valued the most by adult students. other findings and their implications are discussed further in the article. keywords: adult education; online degree program; asynchronous learning the aim of this paper is to provide insights into the experiences of adult learners enrolled in asynchronous online degree programs at both the associates and baccalaureate levels as well as provide recommendations from adult learners for faculty, university support staff, and institutions currently offering, or considering the option of offering, online asynchronous degree programs. whereas some research focuses on the medium of online instruction, best practices for online course delivery, or the needs of adults in continuing online higher education, no study has documented the reflections of adult learners enrolled in asynchronous online degree programs or their suggestions to improve asynchronous online learning. to define adult learners, we relied on the definition of nontraditional students provided by the national center for education statistics. in this study, we define an adult learner as a student over the age of 24 (“nontraditional undergraduates/definitions and data,” n.d.). though age is the main criterion for identifying an adult student, students who fall into this category often share similar circumstances in their educational journey: they return to education after taking a break, they have a fullor part-time job, and/or they have family responsibilities while being enrolled in educational institutions (“nontraditional undergraduates/definitions and data,” n.d.). this project seeks to provide insights into the preferences adult learners in asynchronous online degree programs have about assignments, course organization, communication with faculty and university staff, methods of learning and evaluation, and interaction with classmates. this project also highlights the challenges and difficulties that adult learners have in asynchronous online degree programs, which include isolation from university resources, faculty, and staff as well as a longing for a sense of community and reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 48 inclusion. the students who participated in this study also offered key insights about how to improve the effectiveness of asynchronous online courses and degree programs for adult learners. since 2020, the demand for and relevance of distance education—namely, courses and degree programs delivered fully online where teachers and students are separated by time and distance—is noticeably higher than pre-pandemic levels. presently, 12.5 percent of all undergraduate students in the united states take all their courses online (smalley, 2021). the coronavirus pandemic also accelerated the pace at which american colleges and universities offer online degree programs; the number of such programs now stands at roughly 2,500 (gallagher & palmer, 2020). the most recent numbers indicate that nearly 16.8 million adults over the age of twenty are enrolled in at least one course in an online degree program (u.s. department of education, 2021). review of literature compared to the traditional in-person experience on brick-and-mortar campuses or in synchronous online courses, asynchronous online education is not dictated by time, and distance students are not required to be on campus and are responsible for completing their coursework when it is most convenient for them. for adult learners, especially those with full-time jobs or family commitments, asynchronous online college programs are an ideal way to earn a degree (hannon, 2021; liu, 2012). it is important to note that online degree programs, and even courses within those programs, can be offered either synchronously—with scheduled online virtual meeting times—or asynchronously wherein students and teachers do not have required meeting times. research shows that the method of online course delivery—synchronous or asynchronous—does not make a statistically significant difference in student learning outcomes (johnston et al., 2005; means et al., 2010; nguyen, 2015). although there is a substantial amount of literature addressing topics within adult online education, such as retention, motivation practices within specific subjects, self-esteem, and resilience, no study has focused exclusively on the experiences of adult learners enrolled in asynchronous online degree programs and the suggestions adult learners have for the improvement of asynchronous online degree programs (allen & zhang, 2018; essary, 2014; rapchak et al., 2015; rogers, 2018). there is also little research that discusses adult learners’ opinions on modes of delivery for their coursework—whether face-to-face, hybrid, synchronous online, or asynchronous online—or preferences on receiving live instruction from their professors (gysbers et al., 2011). an extensive body of research exists covering the benefits of taking online courses, which include better access to college degree programs, the convenience of scheduling, the flexibility to study and learn at their own pace, and the potential to earn a credential for employment or promotion within their occupation (conceiçâo, 2006; michael, 2012). this study is, in a way, building on the work of kellogg and smith (2009) who argue that the needs and expectations of adult learners online reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 49 are different than those of typical college-aged students, especially those oncampus. the reported experiences of participants of this study, particularly their thoughts on how to improve asynchronous online learning, parallel riggs and linder’s (2016) argument that adult learners want to see active learning in their coursework and want to see more freedom in how they complete their assignments. despite the significant number of adult learners enrolled in online degree programs and the growing popularity of using online distance education to earn a college degree, there is little research about their experiences in asynchronous online degree programs or what they would like to see changed in asynchronous online distance education to make it more advantageous to them. as more adults enroll in asynchronous online degree programs, gaining insights into their experiences and reflections is imperative to better the overall quality of coursework, best instructional methods, communication strategies, and outcomes of asynchronous online degree programs. methods and procedures research participants were all adults over the age of 30 and were enrolled in online degree programs at multiple institutions of higher education in the american midwest. all participants were receiving their education at no cost due to their labor union having a partnership with these institutions, which provided them access to the no-cost asynchronous online associate or bachelor’s degree programs. to ensure the privacy of the participants, the names of the institutions they attended will not be shared. for the same reason, participants’ demographic information will not be shared individually. participation in this study was voluntary as participants were invited through email to answer a questionnaire about their experiences in an online asynchronous degree program. the interviews were conducted either over the phone or through a video conferencing platform at the participants’ convenience and lasted between 35–75 minutes. each participant received and signed an irb-approved consent form prior to the interview. the interviews were transcribed, coded, and represent the present research. data collection for this study served to answer the following research questions: (1) how would adult learners describe their experiences with asynchronous online degree programs? (2) what aspects of an asynchronous online curriculum were most helpful to adult learners? and (3) what would adult learners want to change about their asynchronous online degree programs? the total sample consisted of 26 participants who were either currently enrolled in midwest university’s no-cost asynchronous online degree program or had recently graduated from the program. nineteen participants (73%) identified as women while seven (27%) identified as men. there were 16 participants (61%) who identified as white (non-hispanic), seven participants (27%) who identified as hispanic/latinx, two participants (8%) who identified as african american, and one participant (3%) who identified as asian-american. twenty-five participants (96%) indicated that they had a full-time job while one participant (3%) was a stay-atreflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 50 home parent to a child with special needs. combined, 22 participants (84%) indicated that they had both a full-time job and children living in the home with them. twenty-one participants (80%) indicated that they previously attended college either in-person or online in the last two decades. there were also 20 participants (73%) who indicated that they were first-generation college students. the present study is built on data collected from a 26-question semi-structured indepth interview where questions were grouped into three categories: demographic information, background in higher education, and general reflections on asynchronous online learning. within the category of general reflections, participants were asked about their expectations of asynchronous online learning, the benefits and detriments of asynchronous online learning, coursework (e.g., written papers, tests, discussion posts) that they found to be the most helpful to their learning, their experience with faculty assigned to teach their courses, experiences communicating with university staff, and the changes that could be made to asynchronous online learning to make it more advantageous to them. due to the nature of both semi-structured and in-depth interviews, both the researcher and participant asked follow-up questions or requested further elaboration on the topic. participants were also given the opportunity to share any reflections not touched upon within the interview. data collected from the interviews were then coded and categorized into three descriptive themes that run throughout the first section of this study: reflections on the effectiveness of coursework, experiences communicating with faculty and staff, and the extent to which asynchronous online learning provides a sense of community and belonging to adult learners. the final section of this paper highlights a fourth and more analytical theme—suggestions for the improvement of asynchronous online degree programs and courses for adult learners. the experience of adult learners in asynchronous online degree programs difficulties of asynchronous online learning participants were asked what they believed to be the detriments of asynchronous online learning. their responses fell into five categories: not feeling as prioritized as in-person students, communication with faculty and the university, course organization, time zone differences, and internet connectivity/reliability. lack of human connection with professors there are a myriad of studies highlighting the pivotal role of instructors’ support and involvement in facilitating online learning and increasing student satisfaction among the typically-aged student population (de smet et al., 2008; richardson & swan, 2003; rovai, 2007; sun & chen, 2016; shea et al., 2010). our participants indicated that the lack of connectivity with professors was one of the detriments of asynchronous online learning. when asked what they believed was the biggest impediment to asynchronous learning, participants remarked that they “can’t walk up and knock on a door and get a human being,” or weren’t “able to actually go see reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 51 the professor in person in the office.” most participants could contact their professors virtually but lamented that “it’s still not the same because the person’s not there with you to help you figure certain things out” and that the ability to build relationships with professors was “kind of lost.” trouble receiving responses from faculty or staff the availability of faculty and university staff and the presence of timely communication are integral parts of student support (croxton, 2014; lee et al., 2011). participants reported that the distance created by the geographic separation from their campuses and contact with universities being limited to telephone or email was further aggravated by communication breakdowns from faculty or staff. four participants (15.4%) said their professors were not putting any effort into communication and expected that communication with professors each semester, as one participant put it, was “gonna be a disaster.” when asked how long they usually waited for a response from faculty, 3 participants (11.5%) indicated that they would usually hear from their professors on the fourth or fifth day after they sent their initial email. participants also pointed out that “it’s really tricky to get an answer” from staff members from various offices at their universities. a total of 17 participants (65.4%) reported having issues getting a response from the universities’ staff. their universities’ failure to communicate made participants feel like they were “just a number” or were “not valued as much as an on-campus traditional student.” one respondent said, “i’m a real person, a real student. please care about me for five minutes.” issues with technology and internet despite the advances in technology surrounding learning management systems where online courses are delivered, wireless internet access, and modes of communicating from a distance, 3 of the participants (11.5%) in this study reported that technology was the biggest detriment of asynchronous online learning and that internet access “is the most important thing” because, as one participant stated, “if you don’t have the internet,” the online coursework “is not gonna go really well for you.” another 6 respondents (23.1%) cited problems including “email is down” or when “the learning platform is down” as major barriers to completing their coursework. while some technical issues, like access to the internet, cannot be resolved with the availability of technical support, technical support is vital for effective asynchronous online learning, especially for the older population of students (netanda et al., 2017). experience with coursework participants were asked what type of coursework (e.g., tests, discussions, essays) they found most helpful and unhelpful to their learning. their responses indicate that adult learners find all types of assignments helpful—it just depends on the preferences of the student. what was more certain, though, is that adult learners want the following: (1) clear connections between assignments and a course’s learning objectives; (2) constructive feedback from professors; (3) consistent and reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 52 reasonable amount of assignments each week; (4) assignments that allow them to reflect on their career or personal experiences; and (5) professor-created content, like pre-recorded lectures, as an alternative or supplemental assignment to reading a textbook or watching content not created by the professor online. malcolm knowles (1984), who worked in the field of andragogy and adult learning theory, allocated five characteristics of adult learners differentiating them from child learners: self-concept, experience, readiness to learn, orientation on learning, and motivation to learn. participants’ preferences in coursework content and organization are partly consistent with knowles’ view on adult learners. the main source of participants’ dissatisfaction with coursework came from a lack of understanding of how assigned tasks can be applied outside of the classroom setting. the life experience of learners plays a great role in their learning—so do the readiness, orientation, and motivation to learn. more than half of adult students who participated in the study (15 participants, 57.7%) want the coursework and the way it is organized to reflect their life experiences, responsibilities, and roles outside of the classroom. for 22 participants (84.6%) the type of assignments given often did not play a significant role if the assignments were perceived as “relevant,” which meant that the tasks required critical reflection and/or were problem-based. the responses of our participants stand in contradiction with adult learning theory when it comes to adult learners’ self-concept. based on the interviews analyzed in this paper, nontraditional online students might need more instructor support than previously thought (el-amin, 2020; saudelli et al., 2015). being enrolled in a formal education institution, adult learners in this study were expecting to be evaluated by the faculty assigned to teach their courses. the overwhelming majority of participants in the study reported the importance of timely constructive feedback from their instructors, which was viewed both as an important part of acquiring knowledge and positive communication with faculty. connecting assignments to course goals and learning outcomes participants often did not care about the type of assignment they were asked to complete for a course, but they did care about how an assignment aided their learning. eleven participants (42.3%) believed assignments aided their learning most when they felt like they were “actually participating,” “putting tools to use” to “create,” “build,” and “present” assignments rather than doing coursework that’s “not interesting” or “pointless” such as when professors ask students to “read and digest the textbook and spit it all back out.” twelve participants (46.2%) wanted their professors to make the requirements and evaluation criteria for course learning objects to be “one hundred percent clear,” and 9 participants (34.6%) believed that the goal of each course should be to do assignments that are “meaningful” to students. reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 53 receiving feedback from professors respondents expressed their preference for receiving written feedback from professors as opposed to a numerical grade. participants explained that a “professor’s feedback on [assignments] is paramount” and do not like when their only feedback “is just a number” professors put into “the grade book.” participants “really appreciate faculty members that give feedback on assignments,” want to understand what they “got wrong,” what “they did right,” or what they “can improve upon in the future.” as one respondent put it, “as opposed to not telling me what i got wrong, help me learn…i like to know what i can improve on or what i can keep working at.” fourteen respondents (53.8%) reported feeling “kind of upset” if “there’s no feedback.” twelve participants (46.2%) report feeling “unmotivated” when “professors are not giving feedback,” and 6 participants (23%) indicated that they were “gonna get a 100% no matter what,” which they described as “disappointing.” the amount of coursework and deadlines adult learners in this study, most of whom had jobs and children, believed that coursework should accommodate those realities and that professors should be mindful of the demands on the time of adult learners outside of coursework. participants felt that their courses were “very much designed for students who are only going to school as opposed to people like [adult learners] who are obviously working full-time.” participants wondered if their professor assumed participants were “stay-at-home moms” or unemployed and “online at home doing [courses],” not students who were “working full-time” with “families and other commitments” and should be considered as “working professionals.” what tended to aggravate a number of the participants were professors who “expected a lot of things” to be submitted during the work week like discussion boards, exams, and quizzes, which, according to participants, “makes no sense” because they are “working professionals.” a total of 8 participants (30.8%) reported taking at least one course in which their work responsibilities during the week and expectations of online education came into conflict with their professors’ expectations for assignments to be submitted between monday and friday. things like “having discussion questions due on a wednesday” or “having to log in to the class three times a week” were “upsetting” to participants because “the beginning of the week…is the busiest” time of the week making it hard to schedule time to work on classes. others asked questions like, “as a working parent, when do i do my homework?” seven out of 8 participants (26.9%) who had an experience of attending classes with mid-week deadlines said that if they wanted multiple assignments due throughout the week, they “would’ve gone to a brick and mortar” school. all participants agreed that assignments should be due on sundays at 11:59pm because it is assumed by adult online learners to be “the standard.” as one respondent said, “it’s just easier and it makes the most sense for people. the way that we live our lives—we’re busy.” reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 54 assignments that connect to careers or personal experiences the participants in this study found value in assignments that connect to what they are doing in their careers or allow them to highlight a personal academic interest or personal experiences. eleven adult learners (42.3%) who have spent years in their fields found value in being able to elaborate on their personal experiences because they could share “a different view of the world” with their classmates. roughly 10% of participants felt that since many of them already had field/job experience in their degree program they could share a perspective not reflected in a book or a classroom setting. the same share of participants—around 10%—also wanted to know how they were going to use course information in their jobs or careers and wanted practical, helpful information and skills they could use in their field. professor-created content professor-created content was cited as one of the main types of coursework or assignments that the adult learners in this study found to be the most beneficial to their learning. although the most preferred type of professor-created content for the participants in this study is a lecture (expressed by 20 participants, 76.9%), 2 participants (7.7%) found value in voice-over powerpoints because they, as one participant pointed out, “condensed what [participants] had to learn” each week and “made it easier … to do the assignments.” however, only 3 participants said they were offered the option to learn via voice-over powerpoints. nonetheless, this sample of adult learners found that their learning was most benefited by professors who recorded lectures and required students to view them because it made them feel as if they were in a college course. participants describe recorded lectures as “awesome” and “great” because lectures “actually break [the content] down” (17 participants, 65.4%) and serve as an “eye-opener” to information students “initially didn’t understand” (11 participants, 42.3%). some preferred recorded lectures because they made them feel a connection with their professors. participants deeply enjoyed when professors recorded themselves exposing their thought processes behind the material (18 participants, 69.2%). in the words of one participant, they valued their professors “sitting down and going over the coursework” because students “could see [professors] talking, …their inflection, how they’re thinking about what it is they’re trying to portray to you.” respondents argued that the recorded lectures “really reach people” because they feel like they “have a personal connection with the professor.” communication is key the theme of communication, namely the speed in which students receive a response and how helpful the response is to the student, stood out in this sample of adult learners enrolled in asynchronous online education. participants in this study emphasized their insecurity caused by being geographically separated from their campuses and indicated that communication is a major difficulty of asynchronous online learning. as a result, identifying how communication can be improved is key to understanding how to make asynchronous online learning more effective for reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 55 adult learners. respondents were asked to describe their experiences with communication, and most of their responses focused on various aspects of communication within the professor-student relationship. all faculty interaction is viewed as communication the adult learners participating in this study classified several types of interactions with professors as “communication,” including feedback on assignments, response time for emails, the personalization of their courses, and weekly announcements. in other words, participants had a broad view of communication and viewed communication with their professors as much more than individual correspondence via email. asked what makes a professor a good communicator, respondents described a professor that will interact (25 participants, 96.2%), try to make [students] think outside the box (11 participants, 42.3%), and set very clear expectations (14 participants, 53.8%) with rubrics, responses, and feedback. some participants reported feeling isolated from their professor’s lack of communication (10 participants, 38.5%), with one of them saying that they were left wondering if their “class is being taught by a.i.” because they did not receive “any kind of human response or interaction” from their professor. a total of 18 students (69.2%) reported having a similar experience in at least one of their classes. in all instances, professors either would not respond or would refer students to the syllabus if they had any questions. nine respondents (34.6%) also told of courses where they felt abandoned by their professor. one participant said they “just submit assignments, get a grade and that’s it,” and questioned if the professor “actually looked” at their assignments “or just ran on autopilot all semester.” inconsistent grading is inconsistent communication other participants pointed to a lack of grading as sign of poor communication. ten participants (38.5%) explained how they had courses, sometimes multiple, wherein the professor would, as one participant said, “grade for the 1st week” and then “not grade again ‘till the last week,” leaving the students to reach out to classmates asking, “has anyone heard from the professor?” when a professor did not grade in a timely manner, participants describe feeling isolated and without direction, asking questions like “is anybody even hearing us?” or “what are we even doing here?” not having consistent communication about grades from professors was identified as a huge issue by 13 students (50%) because it left them with “no idea of what kind of progress they were making and how to correct any of their work along the way.” five participants (19.2%) indicated that they had “resorted to reaching out to other peers” for help with assignments in courses, like math, because they knew they were not “gonna hear from the professor” and needed to know if they were doing assignments correctly. professor communication is key in cases when faculty did provide communication, or made themselves available for students to contact them, participants described their experiences with communication in a positive light. overall, communication with professors was reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 56 deemed as “good” when professors responded in a timely manner (22 participants, 84.6%) and directly addressed students’ questions or concerns (17 participants, 65.4%). when professors did communicate effectively, participants reported feeling supported by their professors (23 participants, 88.5%). one of the participants said that professors’ consistent presence in the course left them feeling “like [professors] actually wanted [students] to learn something” and that their professors were “here to help.” communication from professors left participants wanting to “pay more attention” (20 participants, 76.9%) in their courses because their professors were “actually helping” and made students feel like professors “care.” sense of community and belonging participants in this study were not directly asked about their relationships with classmates or if they felt a sense of belonging while enrolled in an asynchronous online degree program, but respondents continually referenced a lack of community within their courses and degree programs. roughly 40% of respondents either brought up their isolation from fellow classmates, a sense of not belonging to the institution that would potentially be their alma mater, or the benefits of interacting with classmates and/or professors through video conferencing platforms. existing research points to the positive effects the sense of community has on students’ satisfaction with and engagement in courses (farrell & brunton, 2020; ritushree & correia, 2020). in online classrooms where all communication is mediated by technology, technological self-efficacy plays an important role in promoting a sense of community among students (macleod et al., 2017). thus, it is crucial that the responsibility for creating a community is not carried only by the students. this principle becomes ever more relevant in relation to adult students as they may experience a greater need for support with technologies involved in online education (netanda et al., 2017). no community, no problem the lack of interaction with classmates, while noticeable, did not present a problem for some participants who preferred the individualized nature of asynchronous online learning and focusing solely on their coursework. eight participants (30.8%) reported seeing some of the same names’ of classmates a few times but were primarily, as one participant stated, “just focused on doing what [they] had to do” to pass the courses. some respondents would send private messages or emails back and forth, yet, as one participant said, “with a lot of asynchronous learning, everybody is just trying to get it done; [they] all have families [and] all have jobs [and are] just trying to get it done.” across all interviewees, 9 (34.6%) expressed the idea of non-traditional students not having enough time to seek peer-to-peer communication outside of class interactions. perhaps this sentiment is best illustrated by a participant who said, “i mean, not to sound like i wouldn’t wanna foster relationships, but i kind of wanna come in and, and do the work and get it done.” reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 57 isolation from peers and institutions other respondents (10 participants, 38.5%) felt isolated from their classmates and wished they had more chances for peer-to-peer interaction. for some participants, more interaction with classmates would have made them feel more like they were an active member of a college classroom and filled in for the missed ability to get together and work on projects (9 participants, 34.6%). the lack of interaction with classmates made some students (8 participants, 30.8%) miss exchanging ideas and learning from other students. one participant put it this way: “[i] miss exploring [my] personality in a way where [i could] experience a group dynamic and learn from other people.” instead, these participants said that they “don’t make friends” in their courses. despite being enrolled to receive an associate or bachelor’s degree, some participants of this study also reported not feeling a sense of belonging or connection to their potential alma mater. their argument is that they are “not there” on-campus, so the institution “doesn’t mean anything” to them. these participants said they felt isolated from their potential alma maters because the communication they receive from the institutions “really only pertain to the oncampus students,” which makes some participants feel like they are “missing out” and “don’t get the feeling that” they “belong to a school or an organization” but are instead “just in the wind” by themselves. voluntary virtual class meetings respondents often brought up the benefits of professors holding voluntary online meetings to discuss course content or answer questions (8 participants, 30.8%). for the participants in this study, optional online meetings with professors were described as fun, increased their sense of belonging, and provided them an opportunity to bond with their peers. participants reported that having even one voluntary virtual meeting at the beginning of the semester would make them feel like “they’re part of something.” one participant said they would be happy if every professor started the semester with a voluntary virtual meeting wherein the professor said, “hi everybody. i’m the teacher. this is the course. these are your classmates.” ten participants (38.5%) reported that voluntary virtual meetings do not necessarily have to be a curriculum-focused thing, but something where students are getting that real-time in-person interaction. or, as one student said, they would like their professors to hold voluntary meeting where they can put “faces with names” and get “a little bit of background other than the introductory post,” which helps them learn “a lot about different classmates.” some adult learners want companionship in courses almost half of participants (11 participants, 42.3%) in this study reported that they wanted to make friendships with classmates and often initiated conversation with classmates to make connections. almost all these student-initiated friendships were cited as a benefit to their learning because they can get together offline, chat, and talk about classes or compare notes about different classes. these participants wanted to make friendships with their online classmates because, as one student noted, “learning from other people, their experiences, their subjects, their topics” is reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 58 “very interesting.” but much of the onus to make relationships with peers rested on students initiating the interaction on their own time and said they would not get those out-of-class friendships unless the students themselves initiated it. making asynchronous online education more effective students in this study were asked, “what could be changed about asynchronous learning to make it more advantageous for you?” their responses highlight the specific areas where professors, university administrators and staff, and course designers can make changes to their online degree program to make them more effective for adult learners taking asynchronous courses. recording classroom material participants in this study were vocal about their desire for professors to record voiceover powerpoints, demonstrations, or lectures because the material was perceived as beneficial to their learning. asked what could be changed about asynchronous learning, respondents (18 participants, 69.2%) brought up their hope to see more instructors who recorded videos using the whiteboard or various recorded lectures for the lessons each week so students could follow each section of the course through 30-minute lecture videos each week. eight students (30.8%) said they liked the recorded lectures/demonstrations because the video format allowed them to go back to the material at any time. one of these students even suggested that having a recording may be more beneficial than attending a traditional lecture as they “could go back and look at those [lectures] and do those” at their “own pace as opposed to sitting in class” where, if they “missed something,” that “was it.” a study conducted by gysbers et al. (2011) with traditional college students also suggests that recorded lectures can be a good substitute for on-campus lectures for students who cannot attend in-person. allowing for multimedia assignment submissions being able to submit certain assignments, like discussion posts, reflections, and large projects, through the adult learner’s desired medium—voice recording, written response, or visual presentation—is important to this sample of adult learners. that is not to say participants wanted to record a video in lieu of writing a paper. in fact, 2 out of 6 students who mentioned having assignments which required recording a video disliked the idea of personal recordings being the main format by which they completed assignments because they did not like being on camera. additionally, one of them felt video assignments took even more time away from their family and created unnecessary complications of finding space and time to make the recordings without being interrupted or having to expose their family members. almost 40% of the participants conveyed a desire for freedom in how they completed their assignments, with professors giving them the option to write or record. students expressed their frustration that discussion boards could not include recorded content and blamed professors for their lack of creativity. five participants (19.2%) reported feeling like their professors had no experience reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 59 learning online and could not relate to the experiences of online students, which was reflected in the requirements that faculty have for their students. participants couched their rationale for multiple assignment submission options in the fact that asynchronous online courses for adult learners should provide more freedom, allowing students to exercise their agency by getting the work completed on a timetable that best fits their schedule (8 participants, 30.8%). thirteen participants (50%) want professors to provide them with the assignments necessary to demonstrate comprehension. establishing and maintaining a presence respondents indicated that they wanted to see weekly interactions from professors in their courses, such as recorded announcements, replying to discussion posts, providing feedback, or organizing voluntary meetings. participants were not happy that there were professors who are not posting real announcements (9 participants, 34.6%) and not giving feedback on homework (17 participants, 65.4%). ten students (38.5%) wanted to see a weekly video from the professor so that, as one student said, they know the professor “is a human being and alive” or doing “a check-in.” it seems that there is no overdoing the amount of interaction a professor has with adult learners in asynchronous online education as participants wanted to see constant updates from their professors. the adult learners in this study also appreciated any level of interaction from their professors. when asked how asynchronous learning could be improved, a respondent replied, “for me, a pretty important part of school is to get feedback from my professors in terms of how i’m doing in the class, because i can be really self-critical…and when you don’t get any feedback for all that work that you’re doing and all that time and effort that you’re putting into this, it just feels like a waste of time.” arranging voluntary meetings with students and professors was also a priority for this sample of adult learners. reflecting on what they would change about asynchronous learning, one respondent wanted to see “some more interaction with the instructor, some legitimate interaction, like at least once a week, having a video call with them” or “just to talk to them and sort of see where we’re at.” overall, 7 respondents (26.9%) wanted to have an opportunity to meet with their professors and ask them questions at least once every one or two weeks, and 13 participants (50%) expressed their desire to see their professors at least a few times during a course to get an update on the course and work that they are doing. another participant formulated a similar response, “i know that it’s hard for professors to do that, to carve that time, but if it’s just an hour every other week or something just to like do a check-in, especially if you’re about to have a midterm, you know? like, it does make a difference.” sixteen participants (61.5%) said that they didn’t care when or how professors showed their faces, they just wanted to see them because it brought “humanity into the conversation,” which meant students were “not just looking at [the professor’s] little picture and name and bio.” instead, they said they “know that [the professor] has a heart” and “really cares about my education.” reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 60 allowing students to work ahead eight participants (30.8%) reported that they liked being able to move ahead on coursework during the semester and expressed frustration when they had to move week-to-week or couldn’t move ahead because the instructor had not opened future modules. respondents argued that having all modules or weeks of a course open for the entirety of the semester was “really beneficial” because they are adults that are juggling kids, full-time jobs, a commute to work, and when all of the course content is available throughout the semester, students reported being able to work ahead, sometimes “two to three weeks ahead, so they would be able to tend to other areas of their life, like having a weekend with the kids.” this assessment of the obstacles faced by non-traditional students online appears to be relevant to the sample for this study. twenty-five out of 26 respondents (96.2%) who participated in the study reported having a job during the time they were enrolled in the online program. fifteen respondents (57.7%) reported having at least one child under the age of 18, and 13 of these 15 participants (50%) said that the child (children) were living in the same household as them. ensuring consistency between courses nearly 50% of respondents brought up that they wanted more consistency between the way courses are delivered and organized. while it is a difficult task to have all faculty teach in a similar pedagogy, the participants raised a good point about following a template or using a set of standards that faculty follow for all online courses. participants hoped that there is some sort of standardization of the requirement for getting credit or getting through a class because the rigor of their courses varied wildly from classes where all they did was watch youtube videos to course where students, as one participant said, “really had to read and digest and, and critically think through material and really engage in the content in order just to do well.” students reported that from “the outside, it feels like there’s no real standardization” and wished to see more solid online course design so that assignments, grading standards, and the rigor of courses would be consistent. overcoming the reluctance to ask for help a theme that came through the interviews was the reluctance of some adult learners to ask for academic help despite resources being readily available to them. to some, the reluctance came from ingrained beliefs about being a self-sufficient learner and an adult, which made them “feel like [they] should already know [the answer]” to their question.” two participants (7.7%) believed there is a stigma surrounding adult learners who need academic assistance. as one participant said, “i’m an adult i will get through this myself.” three other participants (11.5%) stated that they tend to be stubbornly self-sufficient” or, as one respondent said of their situation, “you’re a grown-up, your son’s going through math at fifteen or sixteen [years old]. you can do it as an adult. you can figure this out.” nonetheless, 12 participants (46.2%) indicated that they would be more likely to ask for help if their professors initiated the contact. as one participant said, “put that ask out there, like, if you need help or if you have questions, please reach out” to me. reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 61 providing assistance with mathematics despite the fact that the questionnaire used in the study did not include any questions regarding specific disciplines, 7 interviewees (26.9%) brought up the dislike of mathematics courses. more than any other subject, participants bemoaned having to take mathematics courses and cited their inability to understand or perform mathematics as their primary reason for avoiding or disliking the subject. seven participants claimed that they were bad at mathematics in general. one participant called mathematics courses “super challenging,” while another proclaimed that they “never, ever could do math” or felt like mathematics courses were something that they did not understand. none of the participants could explain why they felt they struggled with mathematics, but their dislike of the subject indicates that mathematics may be an area of academic struggle for adult students in asynchronous online learning, which may require providing students with more support. conclusions findings from this study highlight the experiences of adult learners in online asynchronous degree programs and provide useful implications for institutions of higher education already offering, or looking to offer, distance learning to adults seeking a college degree. when institutions evaluate the effectiveness of their asynchronous distance education degree programs, consideration should be given to the types of students enrolling in their courses. adult learners often have competing interests for their time, such as children and full-time jobs, which change the characteristics of online learning. evaluation of any online degree program that caters to non-traditional learners should consider the areas of course organization, expectations for faculty communication with students, the type of coursework assigned, and the need for alternative instructional methods. this study suggests that online programs for adult learners should be more focused on self-directed learning. the major finding of this study is that adult learners want access to as much material as possible and interactions with their professors but want to work on assignments at a pace that fits their schedule. moreover, adult students need to see how the work they are doing in college relates to their life and work outside of the classroom. the experience of adult learners is crucial for their education and should direct the development of the coursework for this population of students. as 15 participants (57.7%) stated, adult learners also need their education to coordinate with the work and family responsibilities they have. however, in contradiction to existing research on adult learners, despite their high degree of independence, adult students in asynchronous online programs require support from their instructors and institutions to guide their self-learning and help them overcome the obstacles created by the often-unfamiliar environment of an online classroom and the system of formal higher education in large (glowackidudka, 2019). non-traditional students are not expecting their instructors to organize and structure their work; if anything, they expect their instructors to give reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 62 much freedom to schedule and organize their own work to students. nevertheless, adult students are expecting the instructors to provide them with study materials and consistent and constructive feedback to guide their learning. this does not mean that this population of students is dependent on the instructor but highlights that self-directed education should be done in student-instructor collaboration (garrison, 1992). non-traditional students online also need consistent support from university staff to compensate for the geographical distance and obstacles created by constantly reemerging digital technologies. the rapid change in digital technologies implemented in asynchronous online programs also requires support for faculty members. as the number of adult learners enrolled in online education continues to grow, it is crucial to ensure that the quality of that educational experience is in sync with the reality of the students’ lives enrolled in online degree programs. future researchers, especially those interested in providing more accessible and quality education from a distance, should focus on ways in which asynchronous online coursework is best delivered for adult learners and what barriers remain to providing quality instruction suitable for non-traditional students. researchers could also focus on how to build and deliver quality education for faculty teaching online courses, especially those enrolling adult learners. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references allen, t., & zhang, y. (2015). dedicated to their degrees: adult transfer students in engineering baccalaureate programs. community college review, 44(1), 70– 86. https://doi.org/10.1177/0091552115617018 conceição, s. (2006). faculty lived experiences in the online environment. adult education quarterly, 57(1), 26–45. https://doi.org/10.1177/1059601106292247 croxton, r. (2014). the role of interactivity in student satisfaction and persistence in online learning. merlot journal of online learning and teaching, 10(2), 314–325. de smet, m., van keer, h., & valcke, m. (2008). blending asynchronous discussion groups and peer tutoring in higher education: an exploratory study of online peer tutoring behaviour. computers & education, 50(1), 207–223. https://doi.org/10.1016/j.compedu.2006.05.001 el-amin, a. (2020). andragogy: a theory in practice in higher education. journal of research in higher education, 4(2), 54–69. https://doi.org/10.24193/jrhe.2020.2.4 reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 63 essary, m. l. (2014). key external factors influencing successful distance education programs. the academy of educational leadership journal, 8(3), 121–136. farrell, o., & brunton, j. (2020). a balancing act: a window into online student engagement experiences. international journal of educational technology in higher education, 17(1). https://doi.org/10.1186/s41239-020-00199-x gallagher, s., & palmer, j. (2020, september 29). the pandemic pushed universities online. the change was long overdue. harvard business review. https://hbr.org/2020/09/the-pandemic-pushed-universities-online-thechange-was-long-overdue garrison, d. (1992). critical thinking and self-directed learning in adult education: an analysis of responsibility and control issues. adult education quarterly, 42(3), 136–148. https://doi.org/10.1177/074171369204200302 glowacki-dudka, m. (2019). how to engage nontraditional adult learners through popular education in higher education. adult learning, 30(2), 84–86. https://doi.org/10.1177/1045159519833998 gysbers, v., johnston, j., hancock, d., & denyer, g. (2011). why do students still bother coming to lectures, when everything is available online? international journal of innovation in science and mathematics education, 19(2), 20–36. hannon, k. (2021, february 9). remote learning isn’t just for kids. the new york times. https://www.nytimes.com/2021/02/09/education/online-learningolder-adults.html johnston, j., killion, j., & oomen, j. (2005). student satisfaction in the virtual classroom. internet journal of allied health sciences and practice, 3(2). https://doi.org/10.46743/1540-580x/2005.1071 kellogg, d., & smith, m. (2009). student-to-student interaction revisited: a case study of working adult business students in online courses. decision sciences journal of innovative education, 7(2), 433–456. https://doi.org/10.1111/j.1540-4609.2009.00224.x knowles, m. (1988). the modern practice of adult education: from pedagogy to andragogy. cambridge adult education. lee, s., srinivasan, s., trail, t., lewis, d., & lopez, s. (2011). examining the relationship among student perception of support, course satisfaction, and learning outcomes in online learning. the internet and higher education, 14(3), 158–163. https://doi.org/10.1016/j.iheduc.2011.04.001 liu, o. (2012). student evaluation of instruction: in the new paradigm of distance education. research in higher education, 53(4), 471–486. https://doi.org/10.1007/s11162-011-9236-1 reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 64 macleod, j., yang, h., zhu, s., & shi, y. (2017). technological factors and studentto-student connected classroom climate in cloud classrooms. journal of educational computing research, 56(6), 826–847. https://doi.org/10.1177/0735633117733999 means, b., toyama, y., murphy, r., bakia, m., & jones, k. (2010). evaluation of evidence-based practices in online learning: a metanalysis and review of online learning studies. u.s. department of education. https://www2.ed.gov/rschstat/eval/tech/evidence-basedpractices/finalreport.pdf michael, k. (2012). virtual classroom: reflections of online learning. campus-wide information systems, 29(3), 156–165. https://doi.org/10.1108/10650741211243175 national center for education statistics. (2016, april). digest of education statistics, 2014. national center for education statistics. https://nces.ed.gov/programs/digest/d14/tables/dt14_303.40.asp?referrer=r eport netanda, r., mamabolo, j., & themane, m. (2017). do or die: student support interventions for the survival of distance education institutions in a competitive higher education system. studies in higher education, 44(2), 397–414. https://doi.org/10.1080/03075079.2017.1378632 nguyen, t. (2015). the effectiveness of online learning: beyond no significant difference and future horizons. merlot journal of online learning and teaching, 11(2), 309–319. https://jolt.merlot.org/vol11no2/nguyen_0615.pdf rapchak, m., lewis, l., motyka, j., & balmert, m. (2015). information literacy and adult learners. adult learning, 26(4), 135–142. https://doi.org/10.1177/1045159515594155 richardson, j., & swan, k. (2019). examining social presence in online courses in relation to students’ perceived learning and satisfaction. online learning, 7(1). https://doi.org/10.24059/olj.v7i1.1864 riggs, s., & linder, k. (2016, december). actively engaging students in asynchronous online classes. idea. https://www.ideaedu.org/idea_papers/actively-engaging-students-inasynchronous-online-classes/ ritushree, c., & correia, a. (2019). online students’ attitudes toward collaborative learning and sense of community. american journal of distance education, 34(1), 53–68. https://doi.org/10.1080/08923647.2020.1703479 reflections of adult learners in asynchronous online degree programs journal of effective teaching in higher education, vol. 6, no. 1 65 rogers, k. (2018). retaining adult learners: what academic leaders and faculty need to know. recruiting & retaining adult learners, 20(5), 6–7. https://doi.org/10.1002/nsr.30318 rovai, a. (2007). facilitating online discussions effectively. the internet and higher education, 10(1), 77–88. https://doi.org/10.1016/j.iheduc.2006.10.001 saudelli, m. (2015). adult learners, sociology of education and change theories. in the balancing act (pp. 27–28). sensepublishers. https://doi.org/10.1007/978-94-6300-016-1_3 shea, p., hayes, s., vickers, j., gozza-cohen, m., uzuner, s., mehta, r., valchova, a., & rangan, p. (2010). a re-examination of the community of inquiry framework: social network and content analysis. the internet and higher education, 13(1-2), 10–21. https://doi.org/10.1016/j.iheduc.2009.11.002 smalley, s. (2021, october 13). new u.s. data show jump in college students’ learning online. inside higher ed. https://www.insidehighered.com/news/2021/10/13/new-us-data-showjump-college-students-learning-online sun, a., & chen, x. (2016). online education and its effective practice: a research review. journal of information technology education: research, 15, 157– 190. https://doi.org/10.28945/3502 u. s. department of education. (2021). national center for education statistics, integrated postsecondary education data system (ipeds), fall enrollment component. see digest of education statistics 2021, table 311.15. microsoft word 156-article text-863-1-15-20210411.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.156 teacher-centered versus student-centered teaching: preferences and differences across academic majors laurie murphy, saint joseph's college of maine, lmurphy@sjcme.edu nina b. eduljee, saint joseph's college of maine karen croteau, saint joseph's college of maine abstract. this empirical study examined preferences between teacher-centered and student-centered teaching methods and academic major with 507 undergraduate college students. surveys were administered to the students that assessed their level of agreement with teaching methods utilized in the classroom. the results indicate that across all academic majors, students’ preferences included a mix of teacher-centered and student-centered approaches, some of which include lecture with student interaction, demonstrations and practice, lecture with use of powerpoint, free flowing classroom discussion, guest speakers, and games in the classroom. the least preferred teaching methods were predominantly teachercentered and included the use of unscheduled quizzes, lecture with no visuals, lecture with handwritten notes, and watching a long film. significant differences were obtained for preferred teaching methods between academic majors. keywords: teaching methods, teacher-centered, student-centered, academic major, college students students in college encounter different teaching techniques and strategies in their classes (e.g., lecture, group work, and demonstrations) that are aimed at motivating students and improving learning and academic performance (dunlosky et al., 2013; friesen, 2011; tanner, 2013). while there is considerable debate about which teaching methods or strategies are best used in a college classroom, professors may choose how to deliver the content and establish learning outcomes for their courses (crookes et al., 2013; khalil & elkhider, 2016; shreffler et al., 2019). these teaching methods can differ significantly and take on different formats based on class size and level, educational philosophy, learning outcomes, student characteristics like personality and gender, content being taught, and even academic major (akdemir & özçelik, 2019; carpenter, 2006; i̇lçin et al., 2018; murphy et al., 2020). given the wide variety of teaching methods, it is crucial that teachers focus on selecting methods that will engage students as well as increasing learning and retention (alvarez-bell et al., 2017; bidabadi et al., 2016; granjeiro, 2019; tews et al., 2015). this study examined student preferences for teacher or student-centered teaching methods as well as differences in methods utilized in different academic majors. teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 19 review of the literature teaching methods the teaching methods utilized in the undergraduate classroom are frequently classified into two distinct and separate pedagogical categories: teacher-centered methods and student-centered methods (garrett, 2008; lightweis, 2013). the key difference between teacher-centered and student-centered pedagogy is the focus. teacher-centered activities rely upon teacher input or output (lecture) and students work alone. the teacher provides the required information and students passively receive that information. the teacher controls the learning experience, and students tend to have fewer opportunities to interact with other students or think out loud (emaliana, 2017; serin, 2018). activities that are oriented toward the student (collaborating, communicating, interacting) are considered student-centered (janor et al., 2013). the teacher is involved in the learning process and directs their learning. students can work in pairs or groups, and they can interact and help others learn (emaliana, 2017). in the classroom, the teaching modality varies depending on the professor’s level of comfort, class size, academic level, and discipline. some disciplines like philosophy or theology may utilize teacher-centered methods like lectures to ensure retention of material, while other disciplines like nursing or exercise science may utilize student-centered methods like hands-on experiences or simulations to help students acquire knowledge and skills (harder, 2010). in larger classes, lecture may be the chosen instructional strategy. despite the academic discipline, understanding the differences between teacher and student-centered methods provides professors an opportunity to make informed decisions about how to deliver course content, increase student engagement and learning, and facilitate student achievement of course learning outcomes (abeysekera, 2015). teacher-centered vs. student-centered methods across academic majors across all disciplines, many studies that have attempted to answer the question of whether teacher-centered or student-centered methods are the ideal way to help students achieve learning outcomes (barrett et al., 2018; connell et al., 2016; garceau et al., 2012; prins, 2009). while significant research exists on the college professor’s utilization of particular teaching methods, the findings on what methods achieve the best results in the classroom are varied (marmah, 2014; novelli & fernandes, 2007). according to marmah (2014), amongst the variety of teaching methods, lecture is widely recognized as a traditional, teacher-centered method. however, the efficacy of the lecture method as a teaching strategy is mixed. williams and mcclure’s (2010) study of 70 students enrolled in three separate sections of a leadership and service course used lecture, experiential learning, and public pedagogy (utilization of popular media) as teaching methods. each section was taught using a singular method. the results indicated the lecture format was inadequate for imparting teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 20 leadership practices as compared to the student-centered experiential learning and public pedagogy. in contrast, deslauriers et al. (2019) in a study of 149 physics students found that students felt they learned more in a lecture-based class versus one that used active learning strategies. however, the students also had a high level of agreement that active learning was a better test of their learning. along those lines, muganga and ssenkusu (2019) found that teacher-centered methods defined as “lecture, handouts/slides/syllabus content, and teacher-directed reading/viewing” (p. 28), were more frequently used by professors than studentcentered methods. mathew and pillai (2016) found that nursing students indicated that they preferred the lecture method as compared to peer teaching, demonstrations, or discussion. other studies on teacher-centered approaches in the classroom, like the use of films, have indicated mixed results. moskovich and sharf (2012) found that the use of films “facilitated high levels of self-involvement and cooperative learning” (p. 60). derelioǧlu and sar (2010) indicate that if teachers encourage students to use films beyond just the traditional use, films can allow students to be active learners, make connections, think critically, and reflect about the content. another teacher-centered approach that has been extensively used in the classroom are quizzes (pop, weekly, or reading), primarily to get students to review, interact with the material, and enhance their learning. in a study of 55 philosophy students, tropman (2014) found that 85% of the students reported a high or moderate level of desire to read course material as a result of required reading quizzes. additionally, 89% of students indicated positive views of quizzes and “students also perceived quizzes to have a positive impact on their ability to engage in class discussion” (p. 141). across academic majors, the utilization of student-centered teaching methods is the preferred method for classroom instruction (barrett et al., 2018; bradford et al., 2016). janor et al. (2013), in a study of five finance courses taught at a business school in malaysia, focused on student-centered methods that included case studies, class discussions, group discussions, group projects, and student presentations. the researchers argued that these particular “teaching methods require students to enhance their teamwork skills, while applying knowledge to the ‘real world’ problems” (p. 114). similar results were obtained by prins (2009) who found that a “student-centered approach aided in their learning of the material” (p. 10). to further illustrate the use of student-centered methods in the professions, wehbi (2011) examined the effects on student learning using in-class experimental methods in a social work program. the results indicated that this method could expose students to a way of learning skills that could be used in a real-world setting. furthermore, student reflections on their impressions of the experiential method and their learning outcomes were positive. in a study of 58 criminal justice majors, bradford et al. (2016) found that student-centered methods like teambased learning, incentive-based learning, and the flipped classroom improved student learning and increased student engagement. teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 21 curriculums in other academic majors like sport and exercise science may incorporate many student-centered instruction strategies and often utilize a blended approach (barrett et al., 2018). garceau et al.’s (2012) study of the instructional methods utilized by 165 professors teaching a biomechanics course determined that 80.9% of professors used active learning or interactive methodologies in their classroom for both lecture (teacher-centered) and laboratory (student-centered) sessions. in a study of exercise physiology students, elmer et. al. (2016) found that 73% preferred to learn the basics before attending class to apply their understanding of class material. thus, the utilization of both active teaching methods and carefully crafted activities will foster increasing a student’s skills, allowing for improved clinical decisions as future exercise science practitioners. other approaches using student-centered methods like experiential teaching methods are widely utilized in a variety of professional disciplines that include education, nursing, criminal justice, business, exercise science, and social work (grant et al., 2016; martin, 2010; snow et al., 2019; williams & mcclure, 2010). ethical approval the current study was approved by the institutional review board at the college. all students were informed about the nature and purpose and the risks and benefits of completing the study. students completed an online informed consent prior to the administration of the surveys. complete anonymity was maintained in the study. purpose of the study and the research questions given the mixed research, this study seeks to examine preferences for teachercentered and student-centered teaching methods across different academic majors. specifically, the following research questions were investigated: 1. what are the students’ most and least preferred teaching methods across different academic majors? 2. what are some significant differences between teaching methods (teachercentered vs. student-centered) across different academic majors? methods participants the study was conducted at a small, private liberal arts college in the northeast. the survey was administered to 507 students and attempts were made to obtain data across class levels and student majors. there were 176 (34.7%) males and 331 (65.3%) females. the students ranged in age from 17 to 35 (mean age = 19.92, sd =1.62). the mean age for males was 19.78 (sd = 1.28) and for females was 19.99 (sd = 1.77). there were 141 (27.8%) freshmen, 114 (22.5%) sophomores, 142 (28.0%) juniors, and 110 (21.7%) seniors. the students teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 22 represented eight academic majors and attempts were made to obtain a sample of majors that mirrored the composition of majors at the college (table 1). table 1 breakdown of student majors academic major percent in the present study (n = 507) percent at the college (n = 1,127) business (n = 113) 22.3 18.2 education (n = 30) 5.9 7.2 social sciences (n = 52) 10.3 10.1 humanities (n = 10) 2.0 6.2 sport and exercise science (n = 72) 14.2 19.7 natural sciences (n = 41) 8.1 15.4 nursing (n = 176) 34.7 20.6 undeclared (n = 13) 2.6 2.3 measures demographic information students were administered a demographic information questionnaire that asked about their age, gender, class level, academic major, and year of graduation. student preferences for teaching methods this section asked students to indicate their preferred teaching method by responding to the statement, “this is a teaching method that i prefer be used in the classroom by the professor.” there were 27 items, and students indicated their level of agreement to the items on a 5-point likert scale (1 = strongly agree, 5 = strongly disagree). the items for this section were adapted from research by chamorro-premuzic et al. (2007); mathew and pillai (2013); novelli and fernandez (2007); and rivkin and gim (2013). the 27 items were then organized into nine clusters: lecture (5 items), films (2 items), classroom discussion (4 items), experiential activities (2 items), games/demonstrations (2 items), student presentations (3 items), case studies (3 items), quizzes (3 items), and research (3 items). based on prior research (garrett, 2008; muganga & ssenkusu; 2019; smaldino et al., 2015), the 27 items were either identified as teacher-centered (e.g., lecture where professor talks with no visuals, watching a film), or student-centered (e.g., experiential activities, free flowing classroom discussion). in addition, the investigators’ experience in different academic majors (i.e., business, sports and exercise science, and educational psychology) led to identifying some teaching teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 23 methods as a combination of both (e.g., lecture where professor talks with student interaction, demonstrations, and practice) (table 2). table 2 preferred teaching methods teaching method teachercentered studentcentered lecture lecture (professor talks) with no visuals ✔ lecture (professor talks) with handwritten notes ✔ lecture (professor talks) plus visual – powerpoint ✔ lecture (professor talks) plus visual – overhead ✔ lecture (professor talks) with student interaction ✔ ✔ films watching a short film – 20 minutes or less ✔ watching a long film – 20 minutes or more ✔ classroom discussion professor leads a classroom discussion on readings ✔ ✔ professor teaches by questioning students ✔ ✔ free flowing whole classroom discussion ✔ guest speaker (related to course topic) ✔ experiential activities all experiential activities – pairs ✔ all experiential activities – groups of three or more ✔ games/demonstrations games in the classroom ✔ demonstrations and practice ✔ ✔ student presentations individual ✔ pair of students ✔ groups of three or more ✔ case studies individual participation ✔ pair of students ✔ groups of three or more ✔ teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 24 quizzes on the readings ✔ unscheduled quizzes ✔ weekly quizzes ✔ research library research using experiential activities ✔ ✔ information search using technology ✔ ✔ course readings in the classroom ✔ results data were analyzed using ibm spss statistics for windows, version 25 (ibm corp., 2017). descriptive and inferential statistics were conducted on the data. agreement with preferred teaching methods by academic major the data in table 3 indicates the number of students who indicated the teaching method as their first, second, and third preferences. across all academic majors, students indicated the highest level of agreement for teaching methods that involved both teacher and student-centered preferences or a combination of both like lecture with student interaction (a mix of teacher-centered and studentcentered); demonstrations and practice where the teacher demonstrated skills and content (teacher-centered) while allowing students to practice those skills (studentcentered); lectures with the use of powerpoint or an overhead projector (teachercentered); professor teaches by questioning students (a mix of teacher-centered and student-centered); and free flowing classroom discussion (student-centered). these preferred teaching methods involve direct instruction from the professor while providing students opportunities for skill acquisition, promoting active learning, and allowing teacher and student interactions as well as interactions among other students. table 3 level of agreement for top three preferred teaching methods by academic major academic major first second third business (n=113) demonstrations and practice (n = 89) lecture (professor talks) with student interaction (n = 86) lecture (professor talks) plus visual – powerpoint (n = 82) education (n=30) lecture (professor talks) with student interaction (n = 29) demonstrations and practice (n = 27) games in the classroom (n = 26) teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 25 social sciences (n=52) lecture (professor talks) plus visual – powerpoint (n = 46) demonstrations and practice (n = 44) lecture (professor talks) plus student interaction/guest speaker (related to course topic) (n = 42) humanities (n=10) lecture (professor talks) with student interaction (n = 10) demonstrations and practice (n = 8) professor teaches by questioning students/free flowing classroom discussion/case studies (individual participation (n = 7) sport and exercise science (n=72) demonstrations and practice (n = 65) lecture (professor talks) plus visual – powerpoint (n = 62) lecture (professor talks) with student interaction (n = 58) natural sciences (n=41) lecture (professor talks) plus visual – powerpoint/demonstrations and practice (n = 37) lecture (professor talks) with student interaction (n = 32) lecture (professor talks) plus visual – overhead (n = 31) nursing (n=176) lecture (professor talks) plus visual – powerpoint (n = 161) demonstrations and practice (n = 154) lecture (professor talks) plus visual – overhead (n = 142) undeclared (n=13) demonstrations and practice (n = 13) free flowing classroom discussion/games in the classroom (n = 10) watching a short film – 2 minutes or less (n = 9) the data in table 4 indicates the number of students who selected their first, second, and third least preferred teaching methods. across all academic majors, students indicated that their lowest level of agreement for teacher-centered teaching methods like unscheduled quizzes, lecture with no visuals, lecture with hand-written notes, guest speaker related to the course topic, and watching a short film. these teaching methods all share common characteristics across academic majors: they tend to be predominantly teacher-centered, they do not allow teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 26 students to connect information learned in the classroom to their real life, students are passive receivers of information, and they lack any teacher-student or studentstudent interaction in the classroom. table 4 academic major and level of agreement for three least preferred teaching methods academic major first second third business (n=113) unscheduled quizzes (n = 98) lecture (professor talks) with no visuals (n = 66) library research using experiential activities (n = 57) education (n=30) unscheduled quizzes/lecture (professor talks) with handwritten notes (n = 27) weekly quizzes (n = 18) lecture (professor talks) with handwritten notes (n = 10) social sciences (n=52) unscheduled quizzes/lecture (professor talks) with no visuals (n = 36) library research using experiential activities (n = 16) student presentations – groups of three or more (n = 12) humanities (n=10) unscheduled quizzes (n = 9) library research using experiential activities/lecture (professor talks) with no visuals (n = 5) professor leads a classroom discussion on the readings/course readings in the classroom (n = 4) sport and exercise science (n=72) unscheduled quizzes (n = 53) lecture (professor talks) with no visuals (n = 49) watching a long film – 20 minutes or more (n = 31) natural sciences (n=41) lecture (professor talks) with no visuals (n = 31) unscheduled quizzes (n = 29) professor teaches by questioning students/student presentations – groups of three or more (n = 12) nursing (n=176) unscheduled quizzes (n = 146) lecture (professor talks) with no visuals (n = 144) case studies – individual participation (n = 85) teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 27 undeclared (n=13) unscheduled quizzes (n = 9) lecture (professor talks) with no visuals (n = 8) student presentations – individual (n = 4) differences in preferred teaching methods by academic major a one-way analysis of variance (anova) was conducted on the 27 items of the preferred teaching methods by academic major. the results indicated that statistically significant differences in preferences for teaching methods were obtained for 18 of the 27 items (table 5). a closer examination of the clusters/individual items of the teaching methods by academic major indicates statistically significant differences were obtained when using lecture as a teaching method. business majors (mean = 3.64) indicated a greater preference for teacher-centered methods where the professor talks with no visuals than nursing students (mean = 4.15). students in the natural sciences (mean = 2.12) had a greater preference for teacher-centered methods like lecture where professor talks with handwritten notes than students in the education major (mean = 3.03). students in the nursing major (mean = 1.70) had a greater preference for teacher-centered methods like professor talks with a visual like powerpoint over business majors (mean = 2.12). for the films cluster of items, business majors (mean = 2.76) indicated a greater preference for teacher-centered teaching methods of watching a film that was 20 minutes longer than students in the nursing major (mean = 3.35). social sciences majors (mean = 2.56) also indicated a greater preference for films 20 minutes or longer over nursing majors or sport and exercise science majors (mean = 3.24). with the classroom discussion cluster of items for teaching methods, business majors (mean = 2.34) preferred student-centered teaching methods like freeflowing whole classroom discussion over nursing majors (mean = 2.85). social science majors (mean = 2.15) also preferred free flowing classroom discussion over students in the nursing major (mean = 2.85). sport and exercise majors (mean = 1.90) preferred guest speakers (related to course topic), which is a teacher centered teaching method, more than nursing (mean = 2.55) or natural science majors (mean = 2.63). significant differences were obtained for experiential activities, between students in the sport and exercise major (mean = 2.29), who preferred (student centered) activities in groups of three or more over students in the nursing major (mean = 2.87). for student presentations, education (mean = 2.57) and social science majors (mean = 2.73) preferred individual presentations (student-centered) over students in natural sciences (mean = 2.85) or nursing majors (mean = 3.41). teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 28 with the research cluster of items, students in the natural sciences (mean = 2.54) and undeclared majors (mean = 2.54) indicated greater preferences for library research using experiential activities (a mix of teacher and student-centered teaching methods), while students in the sport and exercise major (mean = 2.46) preferred teaching methods like information search using technology (student centered) over nursing majors (mean = 2.29). also, nursing students (mean = 2.69) preferred course readings in the classroom (teacher-centered) over students in the business major (mean = 2.98). table 5 differences (means, sd’s, f values) between preferred teaching methods by academic majors clusters and items 1 . b u si n e ss 2 . e d u ca ti o n 3 . s o ci a l s ci e n ce s 4 . h u m a n it ie s 5 . s p o rt a n d e x e rc is e s ci e n ce 6 . n a tu ra l s ci e n ce s 7 . n u rs in g 8 . u n d e cl a re d f s ig n if ic a n tl y d if fe re n t p a ir s mea n (sd) mea n (sd) mea n (sd) mea n (sd) mea n (sd) mea n (sd) mea n (sd) mea n (sd) lecture lecture (professor talks) with no visuals 3.64 (1.0 7) 4.23 (.81) 3.85 (.95) 3.20 (.91) 3.83 (1.5 1) 3.85 (.93) 4.15 (.95) 3.77 (1.3 0) 3.84* * 1-7 lecture (professor talks) with handwritten notes 2.62 (1.0 0) 3.03 (1.0 6) 2.62 (.93) 2.10 (.73) 2.57 (1.0 7) 2.12 (.78) 2.49 (.98) 2.38 (.96) 2.71* * 2-6 lecture (professor talks) plus visual – powerpoint 2.12 (.96) 2.10 (.54) 1.85 (.66) 2.10 (.73) 1.81 (.81) 1.68 (.65) 1.70 (.73) 2.15 (.80) 3.92* * 1-7 lecture (professor talks) plus visual – overhead 2.33 (.97) 2.17 (.69) 2.02 (.67) 2.10 (.73) 2.24 (1.0 9) 2.02 (.82) 2.03 (.78) 2.31 (.75) 1.57 lecture (professor 2.04 1.67 (.54) 1.85 (.87) 1.50 (.52) 1.82 (.81) 1.88 (.81) 2.01 (.98) 2.38 1.65 teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 29 talks) with student interaction (1.0 6) (1.5 0) films watching a short film – 20 minutes or less 2.50 (.89) 2.20 (.92) 2.15 (.75) 2.40 (.84) 2.49 (1.0 7) 2.71 (1.0 9) 2.62 (1.0 7) 2.23 (.83) 2.13* watching a long film – 20 minutes or more 2.76 (1.0 8) 2.83 (1.0 2) 2.38 (1.0 3) 2.90 (1.1 0) 3.24 (1.1 2) 3.17 (1.0 2) 3.35 (1.1 2) 2.69 (1.1 8) 6.57* * 1-7 3-5 3-7 classroom discussion professor leads a classroom discussion on readings 3.01 (.99) 2.60 (1.0 0) 2.56 (.89) 3.00 (1.0 5) 2.67 (.98) 2.54 (.77) 2.86 (.95) 2.77 (.92) 2.26* professor teaches by questioning students 2.69 (1.1 0) 2.63 (1.0 3) 2.42 (1.1 2) 2.20 (.63) 2.69 (1.0 3) 2.78 (1.1 0) 2.95 (1.2 1) 2.69 (1.0 3) 1.94 free flowing whole classroom discussion 2.34 (1.1 0) 2.10 (.92) 2.15 (.97) 2.50 (1.4 3) 2.37 (1.1 5) 2.54 (1.0 5) 2.85 (1.1 1) 1.82 (.95) 4.94* * 1-7 3-7 guest speaker (related to course topic) 2.35 (.88) 2.33 (.84) 1.87 (.71) 2.60 (.96) 1.90 (.73) 2.63 (.91) 2.55 (.92) 2.38 (1.0 4) 7.01* * 3-6 3-7 5-6 experiential activities all experiential activities – pairs 2.47 (.95) 2.33 (.75) 2.37 (.95) 3.00 (1.1 5) 2.42 (.94) 2.56 (.89) 2.72 (1.0 4) 2.31 (1.1 0) 1.91 all experiential activities – groups of three or more 2.50 (.97) 2.17 (.74) 2.37 (.95) 3.00 (1.1 5) 2.29 (.95) 2.63 (1.1 3) 2.87 (1.0 9) 2.46 (1.1 9) 4.29* * 5-7 teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 30 games/dem onstrations games in the classroom 2.27 (.84) 2.00 (.78) 2.17 (.81) 3.10 (.73) 2.24 (.86) 2.12 (.78) 2.39 (.91) 2.15 (.55) 2.59* demonstrati ons and practice 2.03 (.72) 1.70 (.65) 1.85 (.66) 2.10 (.56) 1.88 (.71) 1.71 (.64) 1.94 (.69) 1.77 (.43) 1.71 student presentation s individual 3.00 (1.1 2) 2.57 (.81) 2.73 (.97) 2.80 (.91) 2.90 (1.2 0) 2.85 (1.0 1) 3.41 (1.0 8) 2.92 (1.5 5) 4.77* * 2-7 3-7 pair of students 2.58 (1.0 1) 2.33 (.88) 2.46 (1.0 1) 2.60 (1.1 7) 2.50 (1.0 8) 2.54 (1.0 7) 2.89 (1.0 6) 2.86 (1.2 8) 2.39* groups of three or more 2.58 (1.0 9) 2.60 (1.1 3) 2.79 (1.1 0) 2.90 (1.1 9) 2.51 (.99) 2.83 (1.0 9) 2.84 (1.1 1) 2.69 (1.1 8) 1.10 case studies individual participation 2.80 (.89) 2.37 (.76) 2.44 (.75) 2.30 (.82) 2.63 (.86) 2.56 (.70) 2.77 (.91) 2.38 (.87) 2.29* pair of students 2.53 (.90) 2.40 (.77) 2.42 (.84) 2.80 (1.0 3) 2.67 (.93) 2.46 (.84) 2.51 (.88) 2.38 (.96) .65 groups of three or more 2.64 (1.1 2) 2.60 (.81) 2.69 (.97) 2.80 (.91) 2.90 (1.2 0) 2.85 (1.0 1) 3.41 (1.0 8) 2.92 (1.5 5) .76 quizzes on the readings 2.91 (.89) 3.00 (1.0 5) 2.83 (.88) 3.10 (.99) 2.43 (1.2 0) 2.39 (.88) 2.59 (.86) 2.69 (.75) 3.43* * unschedule d quizzes 4.19 (.93) 4.40 (.77) 3.96 (.90) 4.40 (.69) 4.10 (.98) 3.88 (1.0 0) 4.29 (.88) 3.62 (1.1 2) 2.49* weekly quizzes 2.92 (1.0 0) 3.20 (1.1 2) 2.75 (.94) 2.70 (.82) 2.72 (.95) 2.37 (.88) 2.59 (1.0 5) 2.85 (.89) 2.81* * research teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 31 library research using experiential activities 3.46 (.89) 3.10 (1.0 5) 3.10 (.88) 3.30 (.99) 3.17 (1.2 0) 2.90 (.88) 3.35 (.86) 3.00 (.75) 2.50* information search using technology 2.76 (.86) 2.53 (.73) 2.54 (.77) 2.90 (.73) 2.46 (.80) 2.54 (.74) 2.99 (.90) 2.54 (.96) 4.58* * 5-7 course readings in the classroom 2.98 (.84) 2.90 (.88) 2.79 (.75) 2.40 (.84) 2.58 (.86) 2.54 (.77) 2.59 (.86) 2.69 (.75) 3.20* * 1-7 * p < .05, ** p < .01 note. 1= strongly agree, 5 = strongly disagree discussion this study set out to examine preferences for teacher-centered and studentcentered teaching methods in the classroom and academic major and has implications for teaching and learning in the classroom. the present findings indicate students’ preferred teaching methods included a mix of some teachercentered and student-centered approaches, or a combination of both. this includes lecture with student interaction, demonstrations and practice, and lecture with the use of powerpoint, free flowing classroom discussion, and games in the classroom. the least preferred teaching methods were those that were predominantly teacher-centered, including unscheduled quizzes, lectures where the professor talks with no visuals, and watching a long film. comparisons indicate significant differences for preferred teaching methods between academic majors. across all academic majors, students preferred teaching methods that involved both teacher-centered and student-centered teaching methods. these results indicate that students prefer teaching methods where the professor imparts the information in the classroom (using some visual), but also gives the students the opportunity to cover the content and apply and internalize the information by practicing the content and skills (e.g., through games, demonstrations, simulation scenarios, concept mapping, and problem-based learning) (bidabadi et al., 2016; costa, 2013; xu, 2016). across all academic majors, students indicated their least preference for teaching methods that were teacher-centered like unscheduled quizzes, lecture with no visuals, and watching a long film (20 minutes or greater). some academic majors may utilize lectures (without visuals or student interaction) as a primary teaching method; however, prior research indicates that students are passive learners and lose attention and focus, their retention of the material is lower, and they regurgitate answers (marmah, 2014). thus, it is imperative that professors not only impart knowledge, but actively work with students to reinforce and apply that knowledge in their academic major. while some research has found that teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 32 unannounced quizzes are an effective learning technique and can be beneficial and motivating in the classroom (kamuche, 2007), they may also increase panic, stress, and anxiety for students (wang, 2015), and further research is needed to determine if they work effectively across different academic disciplines. the lower preference for library research using experiential activities was surprising. typically, experiential activities (e.g., service learning, field experiences, internships) involve personal experiences that go beyond the classroom and allow students to observe and apply knowledge (york et al., 2010). students are encouraged to seek out further connections by utilizing library resources that are directly relevant to the problem at hand by examining research reports as well as primary and secondary resources by bridging the gap between the classroom and the experiential activity (spackman, 2016). when using library instruction in an ancient mediterranean civilizations class, dowling et al. (2018) found that some students had only a basic understanding of what primary and secondary sources were, and students did not have an understanding of tertiary or fact-finding sources. thus, the lower preference for this teaching method may be explained by the students’ lack of interest or commitment in seeking out additional information as well as their inability to be further involved in or reflect on the activity. differences were also observed in responses to teaching methods between academic majors, with students in the social science major indicating a greater preference for the use of films (a teacher centered method) than students in the natural sciences. while films can be a powerful educational tool that can generate interest in the subject matter being taught, very often it can also be seen by students as a time-wasting endeavor or one that a teacher uses when they are unprepared for class unless the teaching method is paired with some other assignment or activity that can assist students with successfully reconciling the material. in the classroom, teachers can also use student-centered teaching methods by pausing the movie to ask thought provoking questions to assist students in seeing connections among the movie, the theories being discussed, and real-life situations (moskovich & sharf, 2012) as well as using film clips to make the class engaging and help students learn and make connections (levey, 2015). for the classroom discussion cluster of items, students in the social science and sport and exercise science majors indicated a greater preference than students in the nursing major for a combination of both teacher-centered and student-centered approaches. classroom discussion involves various components like free flowing discussion (student-centered), guest speakers related to the course topic (teachercentered), professor leading a discussion, and teaching by questioning students (a combination of both teaching methods). for example, in the classroom, the use of a free flowing classroom discussion can appear chaotic, but if students are presented with hypothetical scenarios and asked specific questions like “why did this happen?”, “what might you do here?,” or “what would you do differently?”, these interactive discussions (in any academic major) can allow students to see multiple points of view that they may not have considered, thereby letting students to effectively reconcile and connect material in the class to their lives (kiernan & lotter, 2019). additionally, as a teaching method, classroom discussion can also teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 33 allow for higher level thinking by students to engage in collaborative discussions, examine issues logically, ask for clarifications, and reflect on assumptions. limitations, future research, and conclusion the results of the present study give rise to several issues for future research. there are a higher proportion of nursing majors in the study and a lower proportion of humanities and natural science majors. in addition, the professor’s teaching style was not examined in the study as that can impact engagement in the classroom. future studies should seek a more representative sample of academic majors as they examine teaching methods used in their classes as well as examining the professor’s teaching to determine if they can modify their teaching style to meet the needs of all students in the classroom. based on this study’s findings, several recommendations can be made for professors in the classroom. first, professors may use student-centered teaching methods in their classes, but sometimes these methods may not be successful, and students may indicate preferences for teacher-centered teaching methods where the professor can impart the knowledge and move on. thus, we recommend that professors carefully consider and evaluate the efficacy of the teaching methods they use in their classes and utilize those that promote and ensure discussion, critical thinking, student participation, and active learning (garrett, 2008). second, when using lecture as a teaching method, it is imperative that professors not only impart knowledge, but actively work with students to reinforce and apply that knowledge in their academic major. as an example, students in the sport and exercise science major should be encouraged to partner with other students to master skills taught (e.g., explaining and demonstrating an exercise). students in the nursing major can practice individual skills (e.g., inserting an iv tube) but also be encouraged to participate in group work and projects, write short reflective papers, complete peer assessments (hudson & carrasco, 2015), and complete student journals to facilitate thinking reflectively and transfer skills into nursing practice (marchigiano et al., 2010). third, it may be important to know what teaching preferences students have in the classroom. it is recommended that professors reflect on their individual styles and use this evaluation to experiment towards modification of their instruction to meet the needs of all their students. lastly, we recommend that professors examine their teaching methods across disciplines. some disciplines may be more conducive to teacher-centered methods, while others may be more appropriate for student-centered methods. it is crucial to use the most effective teaching approach that allows students to connect and internalize information while ensuring student success in the classroom. in conclusion, this study reveals that across academic majors, students prefer a mix of student and teacher-centered approaches and that some of those preferences will vary depending upon academic discipline. professors and students teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 34 could benefit from utilizing both teacher and student-centered approaches in their classrooms to ensure student engagement and success. while this study provides useful information for classroom teaching, future research examining the relationship between student’s preferences and attainment of learning goals is recommended. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references abeysekera, i. (2015). student preferences for instructional methods in an accounting curriculum. international journal of teaching and learning in higher education, 27(3), 310–319. https://ssrn.com/abstract=2765322 akdemir, e., & ozcelik, c. (2019). the investigation of the attitudes of teachers towards using student centered teaching methods and techniques. universal journal of educational research, 7(4), 1147–1153. https://doi.org/10.13189/ujer.2019.070427 alvarez-bell, r. m., wirtz, d., & bian, h. (2017). identifying keys to success in innovative teaching: student engagement and instructional practices as predictors of student learning in a course using a team-based learning approach. teaching & learning inquiry, 5(2), 128–146. http://dx.doi.org/10.20343/teachlearninqu.5.2.10 barrett, j. l., denegar, c. r., & mazerolle, s. m. (2018). challenges facing new educators: expanding teaching strategies for clinical reasoning and evidencebased medicine. athletic training education journal, 13(4), 359–366. https://doi.org/10.4085/1304359 bidabadi, n. s., isfahani, a. n., rouhollah, a., & khalili, r. (2016). effective teaching methods in higher education: requirements and barriers. journal of advances in medical education & professionalism, 4(4), 170–178. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc5065908/pdf/jamp-4170.pdf bradford, j., mowder, d., & bohte, j. (2016). you can lead students to water, but you can’t make them think: an assessment of student engagement and learning. journal of the scholarship of teaching and learning, 16(4), 33–43. https://doi.org/10.14434/josotl.v16i4.20106 carpenter, j.m. (2006) effective teaching methods for large classes. journal of family & consumer sciences education, 24(2), 13–23. https://natefacs.org/pages/v24no2/v24no2carpenter.pdf teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 35 chamorro-premuzic, t., furnham, a., & lewis, m. (2007). personality and approaches to learning predict preference for different teaching methods. learning and individual differences, 17, 241–250. https://doi.org/10.1016/j.lindif.2006.12.001 connell, g. l., donovan, d. a., & chambers, t. g. (2016). increasing the use of student-centered from moderate to high improves student learning and attitudes about biology. life sciences education, 15, 1–15. https://doi.org/10.1187/cbe.15-03-0062 costa, m. j. (2013). commentary: what does “student-centered” mean and how can it be implemented? a systematic perspective. the international union of biochemistry and molecular biology, 41(4), 267–268. https://doi.org/10.1002/bmb.20709 crookes, k., crookes, p.a., & walsh, k. (2013). meaningful and engaging teaching techniques for student nurses: a literature review. nurse education in practice, 13(4), 2239–243. https://doi.org/10.1016/j.nepr.2013.04.008 deslauriers, l., mccarty, l.s., miller, k., callaghan, k., & kestin, g. (2019). measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. proceedings of the national academy of sciences, 116(39), 19251–19257. https://doi.org/10.1073/pnas.1821936116 derelioǧlu, y., & sar, e. (2010). the use of films on history education in primary schools: problems and suggestions, 9, 2017–2020. https://doi.org/10.1016/j.sbspro.2010.12.439 dowling, a. p., wright, k., & bailey, k. (2018). academic collaboration for experiential learning: perspectives on using archival collections and information literacy in history education. college & research library news, 79(6), 323–336. https://doi.org/10.5860/crln.79.6.323 dunlosky, j., rawson, k. a., marsh, e. j., nathan, m. j., & willingham, d. t. (2013). improving students’ learning with effective learning techniques: promising directions from cognitive and educational psychology. psychological science in the public interest, 14(1), 4–58. https://doi.org/10.1177/1529100612453266 elmer, s. j., carter, k. r., armga, a. j., & carter, j. r. (2016). blended learning within an undergraduate exercise physiology laboratory. advances in physical education, 40(1), 64–69. https://doi.org/10.1152/advan.00144.2015 emaliana, i. (2017). teacher-centered or student-centered learning approach to promote learning? journal sosial humaniora, 10(2), 59–70. http://oaji.net/articles/2017/5501-1519102561.pdf teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 36 friesen, n. (2011). the lecture as a transmedial pedagogical form: a historical analysis. educational researcher, 40(3), 95–102. https://doi.org/10.3102/0013189x11404603 garceau, l. r., ebben, w. p., & knudson, d. v. (2012). teaching practices of the undergraduate introductory biomechanics faculty: a north american survey. sports biomechanics, 11(4), 542-558. https://doi.org/10.1080/14763141.2012.725764 garrett, t. (2008). student-centered and teacher-centered classroom management: a case study of three elementary teachers. journal of classroom interaction, 43(1), 34–47. https://files.eric.ed.gov/fulltext/ej829018.pdf granjeiro, e. m. (2019). research-based teaching-learning method: a strategy to motivate and engage students in human physiology classes. advances in physiological education, 43, 553–556. https://doi.org/10.1152/advan.00034.2019 grant, e., dahl, p., & bayens, g. (2016). using elvis to measure experiential learning in criminal justice internships. research in higher education journal, 30, 1–17. https://eric.ed.gov/?id=ej1100316 harder, b.n. (2010). use of simulations in health sciences: a systematic review. journal of nursing education, 4(1), 23–28. https://doi.org/10.3928/01484834-20090828-08 hudson, k., & carrasco, r. (2015). researching nursing students’ engagement: successful findings for nurses. international journal of nursing & clinical practice, 2, 150. http://dx.doi.org/10.15344/2394-4978/2015/150 ibm corporation. (2017). ibm spss statistics for windows. ibm corporation. i̇lçin, n., tomruk, m., yeşilyaprak, s. s., karadibak, d., & savcı, s. (2018). the relationship between learning styles and academic performance in turkish physiotherapy students. bmc medical education, 18, 291. https://10.1186/s12909-018-1400-2 janor, h., rahim, r. a., rahman, a. a., auzairy, n. a., hashim, n. a., & yusof, m. z. (2013). integrating student-centered learning in finance courses: the case of a malaysian research university. international education studies, 6(6), 108–123. https://doi.org/10.5539/ies.v6n6p108 kamuche, f. u. (2007). the effects of unannounced quizzes on student performance: further evidence. college teaching methods & styles journal, 3(2), 21–25. https://doi.org/10.19030/ctms.v3i2.5277 teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 37 khalil, m.k., & elkhider, i. (2016). applying learning theories and instructional design models for effective instruction. advances in physical education, 40(2), 147–156. https://doi:10.1152/advan.00138.2015 kiernan, d. a., & lotter, c. (2019). inquiry-based teaching in the college classroom: the nontraditional student. the american biology teacher, 81(7), 479–484. https://doi.org/10.1525/abt.2019.81.7.479 levey, b. r. (2015). using film clips in the classroom: something old, something new? journal of teaching and learning with technology, 4(2), 41–50. https://doi.org/10.14434/jotlt.v4n2.13140 lightweis, s. k. (2013). college success: a fresh look at differentiated instruction and other student-centered strategies. college quarterly, 6(3). http://collegequarterly.ca/2013-vol16-num03-summer/lightweis.html marmah, a. m. (2014). student’s perception about the lecture as a method of teaching in tertiary institutions. views of students from college of technology education, kumasi (coltek). international journal of education and research, 2(6), 601–612. https://www.ijern.com/journal/june2014/50.pdf marchigiano, g. m., eduljee, n., & harvey, k. (2010). developing critical thinking skills from clinical assignments: a pilot study on nursing students’ selfreported perceptions. journal of nursing management, 19, 143–152. https://doi.org/10.1111/j.1365-2834.2010.01191.x martin, h. c., (2010). the use of teaching strategies that complement learning styles in freshmen nursing students. (publication no 3428160) [doctoral dissertation, capella university]. proquest dissertations publishings. https://search.proquest.com/docview/816338078 mathew, b. a., & pillai, m. k. (2013). student’s preferences in teaching and learning methods in classroom: a cross-sectional survey. global journal for research analysis, 5(4), 200–201. https://www.worldwidejournals.com/global-journalfor-research-analysis-gjra/fileview/april_2016_1464957391__71.pdf moskovich, y., & sharf, s. (2012). using films as a tool for active learning in teaching sociology. the journal of effective teaching, 12(1), 53–63. https://files.eric.ed.gov/fulltext/ej1092141.pdf muganga, l., & ssenkusu, p. (2019). teacher-centered vs. student-centered: an examination of student teachers’ perceptions about pedagogical practice at uganda’s makerere university. cultural and pedagogical inquiry, 11(2), 16– 40. https://doi.org/10.18733/cpi29481 murphy, l., eduljee n. b., croteau, k., & parkman, s. (2020). relationship between personality type and preferred teaching methods for undergraduate college teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 38 students. international journal of research in education and science (ijres), 6(1), 100–109. https://files.eric.ed.gov/fulltext/ej1229010.pdf novelli, e. l., & fernandes, a. a. h. (2007). students’ preferred teaching techniques for biochemistry in biomedicine and medicine courses. biochemistry and molecular biology education, 35(4), 263–266. https://doi.org/10.1002/bmb.73 prins, s. c. b. (2009). student-centered instruction in a theoretical statistics course. journal of statistics education, 17(3), 1–13. http://jse.amstat.org/v17n3/batesprins.pdf rivkin, a., & gim, s. (2013). student preferences regarding teaching methods in a drug-induced diseases and clinical toxicology course. american journal of pharmaceutical education, 77(6), 123. https://www.ajpe.org/content/77/6/123 serin, h. (2018) a comparison of teacher-centered and student-centered approaches in educational settings. international journal of social sciences & educational studies, 5(1), 164–167. https://doi.org/10.23918/ijsses.v5i1p164 shreffler, m., cocco, a., & shreffler, j. (2019). an examination of the relationship between instruction type and course outcomes in sport management courses. sport management education journal, 13(2), 53–62. https://doi.org/10.1123/smej.2018-0014 smaldino, s. e., lowther, d. l., mims, c., & russell, j. d. (2015). instructional technology and media for learning. pearson. https://www.pearsonhighered.com/assets/preface/0/1/3/4/0134287487.pdf snow, k., wardley, l., carter, l., & maher, p. (2019). lived experiences of online and experiential learning programs in four undergraduate professional programs. collected essays on learning and teaching, 12, 79–93. https://doi.org/10.22329/celt.v12i0.5388 spackman, a. (2016). client-based experiential learning and the librarian: information literacy for the real world. journal of business & finance librarianship, 21(3–4), 258–273. https://doi.org/10.1080/08963568.2016.1226616 tanner, k. d. (2013). structure matters: twenty-one teaching strategies to promote student engagement and cultivate classroom equity. cbe-life sciences, 12, 322–331. https://www.lifescied.org/doi/full/10.1187/cbe.1306-0115 tews, m. j., jackson, k., ramsay, c., & michel, j. w. (2015). fun in the college classroom: examining its nature and relationship with student engagement. teacher-centered versus student-centered teaching journal of effective teaching in higher education, vol. 4, no. 1 39 college teaching, 63(1), 16–26. https://doi.org/10.1080/87567555.2014.972318 tropman, e. (2014). in defense of reading quizzes. international journal of teaching and learning in higher education, 26(1), 140–146. https://files.eric.ed.gov/fulltext/ej1043037.pdf wang, s. (2015, december 11). pop quizzes cause undue stress, should be eliminated. the campanile. https://thecampanile.org/2015/12/11/popquizzes-should-cause-undue-stress-should-be-eliminated/ wehbi, s. (2011). reflections on experiential teaching methods: linking the classroom to practice. journal of teaching in social work, 31(5), 493–504. https://doi.org/10.1080/08841233.2011.614205 williams, j. r., & mcclure, m. (2010). the effects of teaching methods in leadership knowledge retention: an experimental design of lecture, experiential, and public pedagogy. journal of leadership education, 9(2), 86–100. https://journalofleadershiped.org/wpcontent/uploads/2019/02/9_2_williams_and_mcclure.pdf xu, j. h. (2016). toolbox of teaching strategies in nursing education. chinese nursing research, 3(2), 54–57. https://doi.org/10.1016/j.cnre.2016.06.002 york, a., groves, c., & black, w. (2010). enriching the academic experience: the library and experiential learning. collaborative librarianship, 2(4), article 4. https://digitalcommons.du.edu/collaborativelibrarianship/vol2/iss4/4 microsoft word cruz_2-1.docx journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset: preparing biology students to leave the academic nest laura cruz, penn state university jennifer meadows, tennessee tech university nikki panter, tennessee tech university, (npanter@tntech.edu) abstract: as biology students prepare to complete their undergraduate degrees and continue into either a career or to another degree, the scientific skills learned in the classroom are not enough to secure their professional path. in this study, the soft skills such as the ability to work in a team and to communicate effectively were emphasized within the context of a newly designed biology course. as a required course for majors within the department of biology, students represented a wide array of experiences, skill levels, and motivations. by adopting a guided inquiry approach to teaching and learning, instructors designed a student-centered course that focused on four categories of professional skills: problem solving, communication, teamwork, and career management. data collected from student surveys were analyzed to determine the effectiveness of these interventions in enhancing student’s abilities and attitudes towards professional skills. these data suggest that students increased their proficiency in attributes valued by employers regardless of gender or major; became more likely to recognize those traits sought by employers; and gained confidence in their ability to use these skills in the workplace. keywords: mindset; communication; professional development; critical thinking; feedback the stereotypical image of the mother bird kicking her baby birds out of the nest may be a bit of a misnomer, but there is evidence that mother birds try to motivate their charges to leave the nest by incrementally increasing the distance they need to travel to get their food, eventually requiring them to start flapping their wings and taking the last steps towards full flight (cornell et al., 2017). biologists call birds in this liminal state between dependence and independence fledglings. as a metaphor, fledgling share a number of characteristics with undergraduate students who are preparing to take full flight into their chosen career paths. this begs the question of how faculty, as the figurative mother birds, effectively motivate and strengthen the abilities of our students to succeed. this article presents the results of a study of a cohort of fledglings, i.e. biology majors from a medium-sized, doctoral-granting, stem-intensive institution, and a group of faculty who endeavored to find new ways to coax them out of the nest. learning to fly is an essential skill, indeed a defining characteristic, of nearly all birds. teaching and learning in stem fields has traditionally focused on the defining aspects of science, i.e., principles, concepts, and practices necessary to solve scientific problems. this approach has a proven success record that has lasted for decades, but public attention has increasingly focused on what pundits have deemed the “stem crisis” and dire predictions of large gaps in the number of journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 48 college graduates with critical stem-related skills (cappelli, 2014; carnevale et al., 2011; drew, 2015). in an effort to close this gap, researchers have identified teaching and learning as a critical factors in both student retention and persistence (fairweather, 2008). this has translated into numerous studies of transformational teaching practices, including several that appear to be particularly compatible with teaching science, including problem-based learning, inquiry-guided learning/pogil, and flipped classrooms (borrego & henderson, 2014; felder, 2016; prince & felder, 2007). that being said, the employer surveys referenced above suggest that teaching transformation should not be limited just to the transmission of disciplinary content, but rather should also include attention to the development of professional skills. professional skills, often used synonymously with terms such as soft skills or career-readiness, are those characteristics that employers value in the workplace. they extend well beyond the science curriculum and include such traits as communication, flexibility, sociability, persistence, and professional judgement. researchers have identified strong correlations between these skills and other markers of student success, such as gpa and, perhaps most relevant to the current study, persistence (komarraju et al., 2014). growing recognition of the significance of these skills has led to nationand institution-wide efforts to find ways to articulate, integrate, and assess the degree to which universities are imbuing students with these essential skills across the curriculum (baram-tsabari & lewenstein, 2013; brownell et al., 2013; kulturel-konak et al., 2013; mohan et al., 2010; thiry et al., 2011; white et al., 2013). in the first stages of this movement, universities engaged in activities such as creating career centers; developing career readiness programming; and providing co-curricular classes. these practices continue to flourish, but the one-size-fits-all approach does have demonstrable limitations, particularly in meeting student needs in specific career paths, especially fields that are highly specialized and/or technical (feller, 2011). there are different models for providing more field-specific career development; some of the most common include the use of specialized academic advisors or targeted internship opportunities (the latter a long-standing requirement for the students in this study). another alternative is for departments or programs to take the initiative to integrate professional skills into their disciplinary curriculum, either by incorporating such skills across the curriculum or by offering designated courses devoted to professional skills in the discipline. successful examples of disciplinary-specific career development courses have been reported for chemistry, psychology, nursing, communications, and finance (lucy, 2017; peng et al., 2017). such courses also have their limitations and the question of whether or not the curricular space they fill constitutes time well spent remains open, particularly in light of the limited number of studies that evaluate such courses (wingate, 2006). our study aims to address this gap in the literature by looking closely at the development and evaluation of a three-credit, sophomorelevel communications and careers course required for all biology majors and taught by experienced biology professors. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 49 beyond the walls of the university, employers have achieved consensus on a range of skills that are increasingly highly valued. each year, the national association of colleges and employers (nace, 2016) conducts an employer survey. for 2016, 160 medium to large employers from a representative sample of economic sectors ranked leadership, teamwork, communication skills (written and verbal) and problem-solving skills as the top five most desired in new college graduates (nace, 2016). each of these characteristics ranked above those tied to knowledge of disciplinary content, including analytical/quantitative skills, technical skills, and computer skills. it is tempting to downplay these findings as applicable to a broad range of skills, not necessarily those for stem careers, but 39% of the employers who responded planned to hire math and science majors, and 23% of those planned to hire biology majors specifically. the nace survey encompasses all recent college graduates from a wide range of colleges and universities. this has the potential to do a disservice to higher performing programs, such as the institution covered in this study, a stem-focused public university. that being said, an employer study conducted by the learning assessment office in 2013 revealed that local and regional employers also identified significant gaps (ennis, 2014). as table 1 (below) reflects, employers indicated a need for improvement in the following areas: work ethic, leadership skills, and oral and written communication. each of these areas ranked among the top twelve critical skills desired by the same employers; and, this list also included ethics, critical thinking, problem-solving, adaptability, technical skills, and diversity. in response to these increasingly evident gaps between what was being taught and what employers were looking for, the biology department chose to develop an upper division required course for all majors that would integrate these competencies. the current study explores how the activities in this newly designed course affected the development of both professional skills and career efficacy for 150 students enrolled in six sections of biol 3920 at tennessee tech university (ttu). journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 50 table 1 employer ranking of career competencies (needed and actual) of recent ttu graduates note: scores are based on a 5-point scale (1=not at all proficient; 5=extremely proficient). the course: biol 3920 given the evidence from the employer surveys, the biology department chose to create a course, starting with an intensive design process incorporating evidencebased teaching strategies intended to make the course student-centered and workforce-ready. the newly designed course focused on four thematic areas (drawn from the employer surveys): career management; oral and written communication; teamwork/leadership; and critical thinking/problem-solving. each of these areas was integrated into an inquiry-guided learning framework that channeled the work of the students not only towards shared course learning outcomes, but also through tailored career pathways (lazonder & harmsen, 2016; lee, 2012). course activities included scientific writing assignments, a career portfolio, a research project, and group presentations; learning took place primarily through guided activities, instructor feedback, and peer interaction, with little to no lecture delivery. career management over the previous 10-year period covered by the employer survey, the department had enhanced its overall curriculum to include a bachelor of science in biology with four options (botany, marine biology, microbiology, and zoology) as well as biology degrees with concentrations in cellular biology, environmental biology, general biology, health sciences, and a separate bs degree in wildlife and fisheries (wfs), the latter with three concentrations (conservation, fisheries, and wildlife). the challenge for biol 3920 was not only to encompass each of these focus areas and their distinct career options, but also to support students within each of these competency employer significance recent graduates gap critical thinking/problem solving 4.58 3.59 -.99 professionalism/work ethic 4.56 3.43 -1.13 oral/written communication 4.43 3.41 -1.02 teamwork/collaborati on 4.43 3.96 -.47 leadership 3.86 3.38 -.48 journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 51 concentrations whose post-baccalaureate aspirations may take them to advanced study, directly to the workplace, or perhaps somewhere in between. at first, the instructors assumed that career development could be handled through a partnership with the campus-based career services office. what they discovered was that such offices are by nature generalists, as they serve students from all over campus, and the students did not view their input as useful. the office was simply not well equipped to offer the specific guidance that upper division students needed to be competitive for positions in laboratories or with the u.s. parks service positions, both major employers of graduate from the program. as part of the design process, the instructors chose to outsource broad capabilities (e.g., video interviewing, resume writing) to the career services office but otherwise to provide multiple structures, or tracks, within the course to allow for differentiated goals and increased motivation (wright, 2011). after consulting with the instructors on their skills levels and needs, students could choose between three tracks (graduate/professional school; applied/field position; and general/undecided), each of which had distinctive projects or activities associated with it. students who chose the graduate school track, for example, would be tasked with development a statement of purpose, while students planning to apply for field positions immediately were tasked with conducting job research. at several junctures in the course, students received feedback from the instructor “coach” including ongoing reviews of their strengths and weaknesses relative to their chosen track. oral and written communication regardless of track, students in the course were expected to complete a significant number of assignments that incorporated oral and written communication, including short papers, abstracts, and class presentations. the instructors also chose to incorporate multiple rounds of feedback and revision, a practice strongly supported in the pedagogical research literature (besley & tanner, 2011; duke, 2003; jurin et al., 2010; toit, 2012). rather than listen to lectures, the students completed most writing assignments during scheduled class time in order for them to be able to ask questions and receive immediate answers and verbal feedback from the instructor, a subject matter expert, or their peers. the instructors also provided written feedback, and students were encouraged to correct and resubmit their work, practices also well supported in the literature (hernández, 2012; reynolds et al., 2012; thompson & blankinship, 2015). in addition to these in-class assignments, the instructors chose to center the course around a semester-long project, the results of which culminated in a public poster session in which students presented to an audience consisting other students, faculty, university leaders, and community members, a practice that had been implemented with success at other institutions (tseng et al., 2013). journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 52 teamwork and leadership to develop the skills needed to complete this project, the instructors organized the students into teams starting the first weeks. in keeping with best practices in teambased learning, the instructors worked to create teams of four to five students that included a variety of degree concentrations and career aspirations, with the intent of creating a diverse social learning environment that allowed students to practice building teamwork skills in an environment that mirrors many workplaces (michaelsen et al., 2014; sibley & ostafichuk, 2015). the teams had weekly goals and assignments and expectations for sustained group communication both inside and outside of class. they held each other accountable to respective team members (through peer evaluation) and to the instructor (through their team leader). the instructors chose team leaders based on (1) displaying above-average writing skills on the first writing assignment, and (2) having a current resume (gafney & varmanelson, 2007). those selected for leadership roles were given additional responsibilities within the team-based assignments, in exchange for which team leaders had fewer individual assignments to complete. as an added incentive, the instructors introduced a level of mild competition, offering recognition to those team leaders whose team project scored most highly with the external judges (galbraith, 2012). critical thinking and problem solving team leaders bore particular responsibility to navigate their team through the steps of the major library-based research project, which began with the selection of topics (scaramozzino, 2010). in keeping with best practices of inquiry-guided learning, the students had the opportunity to provide input on possible research topics for their projects. the instructors led each class section through a brainstorming session in which students were invited to submit relevant, timely, and open-ended research questions and to suggest a minimum of two opposing viewpoints on the topic. from this list, the teams voted on which topics they wanted to tackle and identified roles for each team member, including those who would represent each of the differing perspectives on the issue. project milestones, spread out across the semester, included the identification of relevant research; evaluation of evidence from multiple perspectives; a synthesis of current understanding (displayed on the poster); and evaluation of future solutions (gormally et al., 2012). with the completion of each step, the instructors endeavored to give more control over to the students and their teams, allowing them greater freedom, flexibility, and responsibility with each step and a process of graduated self-directed learning that has proven to be effective in other stem contexts (ryan, 1993; scott et al., 2017). the study our hypothesis was that the design of the course, including the integration of the pedagogical strategies discussed above, would prove to be effective in reaching the journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 53 learning outcomes of the course as well giving students career-ready skills and influence their attitudes towards those skills. to assess the impact of these changes on the students, the researchers designed a study, consisting of a preand postassessment survey. the course sections were taught by three non-tenure track instructors each of whom used the same overall teaching strategies and assignments. consistency was maintained through a shared lms course shell and regular instructor team meetings. institutional review board (irb) approval was obtained to collect data using the surveys. the survey questions related to previous student feedback on the course and the characteristics valued by future employers. the preand post-surveys were administered using qualtrics, an online survey program. students completed the surveys during class time at the beginning and at the end of each semester. more specifically, students were asked to complete the pre-survey at the beginning of the first class meeting before the instructor discussed the course or its format. the post-survey was completed by students at the end of the last class meeting. surveys for each of the three sections were completed during that section’s scheduled class time, but within 24 hours of each other. the response rates for each survey ranged between 67% and 91%.; even at the low end, considerably higher than most student surveys at ttu. the preand post-survey results were contextualized with supplementary data sources. the researchers also reviewed assignment rubrics to develop clearly stated learning outcomes, along with measurable tasks, that students could reference as assignments were completed. student peer reviews were incorporated which involved reflection of teammate performance and self-evaluation. the university’s teaching and learning resource personnel conducted small group analyses to gather perceptions of students, individually and as a group, regarding not only the course, but about their preparation to meet prospective employers’ needs. finally, institutional data on employer needs and student knowledge was taken into consideration. comparisons between preand post-survey data, as well as those between students who had taken the course and the general student population, were made using t-tests. the participants for this study included undergraduate students from the same university. two semesters of data were collected from fall 2016 and spring 2017, which included a total of six sections and 150 students, all of whom were majoring in either biology or wfs (see table 2). “pre” columns display data from surveys at the beginning of the semester; “post” columns display data from surveys at the end of the semester, i.e., after students have completed the course. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 54 table 2 biol 3920 student survey respondent demographics pre count pre % post count post % total 137 101 gender male 57 41.6 41 40.6 female 69 50.4 47 46.5 major/concentration biology/biology 22 16.1 16 15.8 biology/environmental 3 2.2 1 1.0 biology/health science 38 27.7 24 23.8 biology/cellular & molecular 17 12.4 16 15.8 wildlife & fisheries/wildlife 28 20.4 22 21.8 wildlife & fisheries/fisheries 10 7.3 6 5.9 wildlife & fisheries/conservation 9 6.6 8 7.9 other 10 7.3 8 7.9 career track graduate/professional school 36 26.3 45 44.6 applied/field position 13 9.5 35 34.7 general/uncertain 17 12.4 20 19.8 findings taken collectively, the survey results affirmed changes in the perceived value of the four components of the course: career management, communication skills, leadership/teamwork and critical thinking (see appendix 1). that being said, the survey also showed that the students placed a relatively high value on these attributes prior to taking the course; as a result, the goal was not to change their minds, rather to find ways to help them reach these goals as effectively as possible. teamwork/leadership employers seek employees who have strong teamwork abilities. for this reason, the design of this course placed particular importance on building team skills. according to students’ responses on the survey (see appendix 1), they perceived strong teamwork skills as valuable. over the course of the semester, this perceived value increased from 91% to 98% among students. as part of an effective team, students were asked if they would be able to accept and provide constructive feedback. while both responses to these questions showed gains from the pre-test to the journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 55 post-test, students’ ability to provide constructive criticism revealed 12% growth from 83% to 95%. the steepest gains related to teamwork came when students were asked to rank their current leadership skills. in the pre-test, students 78% ranked themselves as having very strong or fairly strong leadership skills. in the post-test, 92% of students reported having very strong or fairly strong leadership skills. this 14% gain demonstrates an increase in student confidence in leading a group. female students reported having stronger teamwork abilities than males, especially at the beginning of the course, a finding which affirms other research that suggests females in stem areas place high value on teamwork skills (vaz et al., 2013). table 3 biol 3920 student survey: responses by genders (questions related to teamwork) male female survey item pre% post% pre% post% like competitive element of assignments (agree) 38 37 35 27 strong leadership skills: self (agree) 86 95 74 89 strong teamwork skills: self (agree) 89 98 94 98 provide constructive criticism: self (agree) 82 94 83 96 accept constructive criticism: self (agree) 89 95 94 96 as reflected in table 3, strong leadership skills increased by 9% for males and 15% for females during each semester, as indicated by respondents. male students experienced a greater increase in teamwork skills by 9% compared to 4% for females. females also showed a drop in their support for competitive elements, a result which affirms behavioral studies of gendered preferences (niederle & vesterlund, 2007). the ability to provide and accept constructive criticism to peers also improved by 12% and 6%, respectively, for males and 13% and 2%, respectively, for females. from these results, it would appear that efforts to enhance students experience with and appreciation for teamwork skills were successful. communication and critical thinking/problem solving the biol 3920 instructors sensed that the quality of student achievement in the areas of communication and critical thinking/problem solving had improved, but this assumption was difficult to test in the absence of a control group. one possible indicator came in the form of ongoing institutional assessment. students at ttu participate in the national study on student engagement (nsse), which provides multiple questions focused on student perceptions of communication and critical thinking. biol 3920 was selected to be part of a campus-wide course redesign program focused on inquiry-guided learning; and part of the obligations for participating in that grant-funded program were to administer a separate course journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 56 specific student survey that mirrored the nsse survey questions. when compared to nsse data for the institution as a whole, it appears that students enrolled in biol 3920 reflected significant gains in those areas associated with oral communication and critical thinking (see figure 1 below). figure 1 comparison of survey results among biol 3920 students, university freshmen (fy), and university seniors (sr) for three categories (speaking, writing, and critical thinking) note: scores are based on a 4-point scale (1: never, 2: sometimes, 3: often, 4: very often), and averaged within category. it should be noted that the nsse is administered in alternating years to first-year students (fy) and graduating seniors (sr). both sets of data are included for comparison (above) because of the wide range of students enrolled in the course. although biol 3920 is intended to be a sophomore level course, the department had struggled to get students to enroll prior to their senior year, often resulting in logistical and enrollment log jams. through increased efforts at advertising the course (including data collected in this study) and working closely with academic advisors, the instructors have succeeded in reversing this trend. there are also potential implications of these findings for student recruitment and retention. a handful of exploratory studies have noted that a lack of emphasis or integration of communication skills (or perception thereof) has acted as a barrier to enrollment in stem courses or majors, especially for students with backgrounds in the humanities and social sciences (tobias, 1990). similarly, inquiry-guided learning, and its emphasis on critical and independent thinking, has proven to be a factor in the retention of increased numbers of students from underserved 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 speaking writing thinking a ve ra ge s tu de nt r es po ns e survey results: biol 3920 vs university nsse16 fy nsse16 sr biolxxx journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 57 populations in stem fields (grandy, 1998; mau, 2003). it could be interesting to conduct a follow-up study in which we track demographics with the long-term academic and professional trajectories of students who have completed this course. career management speaking of long-term awareness, the researchers looked for indications that students not only had experience in developing professional skills, but that they understood the value of these skills in the marketplace. one survey question asked students to rank the attributes they believed were most desired by employers (see table 4), and the post-tests showed students placing higher priority on skills related to leadership, team-building, and communication (three core course components) than they had prior to the course. critical thinking, another of the core components of the course, showed no change in rank from preto post-, but remained the highest ranked of any attribute. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 58 table 4 biol 3920 student survey: perception of attributes sought by employers most important pre survey % most important post survey % change in rank pre to post critical thinking 27.0 critical thinking 26.5 0 work experience of any type 15.6 work experience of any type 13.3 0 willingness to learn 10.7 willingness to learn 10.8 0 grade point average (gpa) 9.0 professional references 7.2 +3 positive attitude 9.0 team player 7.2 +4 problem-solving ability 5.7 grade point average (gpa) 6.0 -2 professional references 4.9 written communication 4.8 +3 internships or co-ops 4.1 internships or co-ops 4.8 0 team player 3.3 leadership 4.8 +7 written communication 2.5 positive attitude 4.8 -5 relevant technical knowledge 1.6 self-motivation 3.6 +1 self-motivation 1.6 relevant technical knowledge 3.6 -1 public speaking 1.6 problem-solving ability 1.2 -7 ethics 0.8 ethics 1.2 0 data analysis & visualization 0.8 volunteer work 0.0 +2 leadership 0.8 public speaking 0.0 -3 volunteer work 0.0 computer literacy 0.0 +1 computer literacy 0.0 data analysis & visualization 0.0 -3 this value was reaffirmed by survey questions asking students to rank the usefulness of the course assignments. rankings of assignments related to professional skills, especially those that incorporated team activities, rose over the course of the semester (p=0.029), indicating a shift in perceived value from 77.4% to 88.6% of assignments categorized as very or somewhat useful. of the specific assignments (see table 5), students perceived greater value in assignments related to professional development and team activities at the end of the semester (p≤0.026) with the exception of résumé writing and interview q&a, both of which were facilitated by the university’s career center. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 59 table 5 usefulness of assignments as indicated by percentage of biol 3920 students responding “very useful” or “somewhat useful” assignment "very useful" or "somewhat useful" pre % post% resume 96.8 98.9 cover letter 89.4 100.0 interview q&a 96.0 90.8 team presentation 69.4 87.5 team poster 44.3 70.9 at the beginning of the semester, pre-survey results showed that students strongly preferred having options when completing assignments and choosing topics for team projects at a rate of 89% and 85%, respectively. this is perhaps not surprising, as other studies have affirmed that the availability of choices fosters a sense of efficacy (pintrich & schunk, 2002; van blankenstein et al., 2018). the biol 3920 students echoed that sentiment in the post-semester survey with an increased preference for choices at 94% for assignment options and 87% for topic flexibility, with noted increases for males (8–9% increase) and wfs majors (14% increase). this latter is likely a reflection of the intended career outcomes for wfs majors, which are often less oriented towards advanced study but do include multiple types of employment, including work in both the public and private sectors. these findings suggest a further inference about the students’ locus of control. the inquiry-guided learning model posits that students should become increasingly selfdirected over time, leading towards increasingly independent work and less guidance from the instructors (adedokun et al., 2013). biol 3920 was structured this way, culminating in the final project over which the student teams bore large degrees of individual and collective responsibility for identifying, analyzing, and presenting their research findings. that being said, it was not known what influence this experience would have on students’ confidence in their broader abilities to navigate the world beyond college. at first, the survey results appeared puzzling, as the remaining percentages (see table 6) largely remained stable or showed only slight increases across the different majors and concentrations. when further correlated with career track, however, the most noteworthy gains were for students who were previously undecided, an outcome that indicates these students had found an increasing sense of direction through the course structure. this conclusion that affirmed in the written responses to formal course evaluation. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 60 table 6 biol 3920 student survey: options responses by major/concentration and career track i would like to have options for assignments when fulfilling requirements for the course. i would like to have flexibility in choosing my team's project topic. major/concentration pre-survey % post-survey % pre-survey % post-survey % biology/biology 34.4 31.9 31.8 31.0 biology/ environmental 3.2 0.0 3.5 1.5 biology/health science 52.5 49.6 57.8 47.7 biology/ cellular & molecular 25.2 27.6 26.7 24.4 wildlife & fisheries/wildlife 36.8 46.3 28.8 42.9 wildlife & fisheries/fisheries 14.2 11.2 11.4 15.5 wildlife & fisheries/ conservation 11.6 14.4 15.8 18.5 other 22.1 19.1 24.4 18.5 career track graduate or professional school 49.6 46.0 31.4 29.2 applied or field position 21.2 20.4 21.9 22.6 general/uncertain 23.4 22.6 3.6 11.7 limitations and discussion this study was intended to explore how the design of this new course affected students’ abilities and attitudes towards professional skills. our findings suggest that the students increased their proficiency in four of the primary attributes valued by employers (career management, communication, critical thinking, and teamwork/leadership), regardless of gender or major. in addition, they became more likely to recognize those traits sought by employers and gained confidence in their ability to use these skills in the workplace. these results should not be overstated. this study focuses on one course offered at a single institution over a three-year period, a circumstance which presents distinct limitations, challenges, and opportunities. the impetus to developing the fledglings program started with employer perceptions of career-readiness of recent graduates. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 61 however, it will take more than three years to determine whether or not the program has made an impact outside of the classroom. the first cohort of biology majors who participated in the new course graduated in spring 2016, but ttu conducts its employer surveys every three to five years, with the next iteration expected in spring 2018. further, the survey results are reported in aggregate, and it can be difficult to pinpoint specific gains from a single program or major. nace has recently conducted several stem-specific surveys, and it might prove fruitful to follow their lead and develop a version of their work that could be administered in the primary service region of biol 3920 and/or to major employers of ttu graduates in stem fields. compelling evidence of success for the fledgling program will depend on our ability to capture the often-elusive prey of longitudinal data. the study design also faced distinct challenges. the absence of a control group, a frequent challenge for scholarship of teaching and learning studies, was compounded by the fact that the course had no clear antecedent, as students were previously expected to acquire such skills largely in either extraor co-curricular ways. we have used the institutional-specific employer survey as a proxy, but similarly to the issues with the nace data referenced above, it is not possible to specifically map its findings directly onto the students covered in this study. in keeping with best practices for research of this kind, we did choose a preand post test design to capture gains made throughout the course; but in the absence of a control, we are unable to state definitely whether or not similar gains could have been made with different instructional strategies. in the case of biol 3920, though, the primary issue at stake is not to choose between different methods of instruction per se, but whether or not such a course should be offered at all. in other words, the alternative would not be to change the instruction, but rather to change the curriculum to not include the course and to integrate the desired skills in other ways. we believe our study does make a strong case for including such a course in the curriculum. the study also has limitations in its representation. ttu is a public stem-focused institution—the only one in the state—and, as such, tends to draw students primarily from within and across the state. the majority of its graduates go on to work in the surrounding economic region, which includes nashville and oak ridge, tennessee. it is not yet known the degree to which this model may be replicable in other geographical and institutional contexts, but we hope that the promise of our preliminary results may serve to inspire others to do so. that inspiration may come from several sources. biology is not the only discipline seeking to address the challenge of integrating professional skills into its curriculum, and it may also be possible to replicate these results by adapting the model not just into other stem or stem-related fields (borzi & mills 2001; hannah et al., 2004), but perhaps many others. we encourage an open discourse across disciplines in order to prepare students for employers’ expectations. we recognize that it may take more than inspiration to replicate the fledgling model. space in the curriculum is valuable real estate and the sacrifice of content or disciplinary skill-focused curriculum to make room for a three-credit course on journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 62 professional skills may be more than some programs are able to accommodate. at several junctures, other faculty in the biology department openly questioned the continuation of the course, particularly in light of increased priorities and employer demand in targeted disciplinary areas, such as health science. the u.s. bureau of labor statistics (2015) indicates that healthcare is the fastest growing occupational field, projected to increase 19% from 2014–2024. while technical skills are fundamental, soft skills such as communication, teamwork, and time management heavily impact the satisfaction of employers, co-workers, and patients (brookes & baker, 2017; oberlin et al., 2015). part of the motivation for this study stemmed from the impetus to justify the need for time spent on these skills and provide demonstrable outcomes from the course. similar conversations are likely going on across campuses as other disciplines are wrestling with adjusting the balance between career readiness and content knowledge within and across their curriculums. this resistance from faculty was particularly pronounced after the first year, when the newly designed course garnered considerable negative feedback from the students. the instructors knew that the learning gains accrued through the design process, as reflected in the findings from this study, would be a moot point if these negative first impressions led to the course being canceled. their strategy was to support the change with evidence. researchers have posited the existence of an innovation curve, in which redesigned courses frequently undergo a dip in student evaluations followed by a subsequent rise over previous levels—a cycle which takes place over an approximately two-year period for regularly offered courses (allen et al., 2001; hodge et al., 2011; mccrickerd, 2012). several studies have speculated on reasons why this occurs, with no single factor emerging as fully convincing, but the pattern appears to be impervious to differences in instructor, instructional methods, discipline, institution, level, or student population. biol 3920 affirmed this pattern, so the instructors were able to use the existence of the innovation cycle to counter both student and faculty pressure to discontinue the course. the ability of the instructors to overcome resistance from both students and faculty is all the more remarkable when you take into consideration that none of the three instructors are tenured or even on tenure-track. the plight of adjunct faculty continues to make headlines (green, 2007; jolley et al., 2014), but there is an often-overlooked middle tier of faculty, those who teach full-time but have little or no research responsibilities (baldwin & chronister, 1996; kezar & sam, 2010; levin & shaker, 2011). all three of the faculty involved in the fledglings’ program are fulltime instructors on three-year renewable contracts; they are teaching faculty. on one hand, the contractual nature of their employment may pose a disincentive for them to lead potentially disruptive changes in teaching practice; on the other hand, teaching is their primary focus, which provides them with both the insight and motivation to lead the way in bringing evidence-based teaching and learning practices into their department and the field (christodoulou, 2017). the fledgling project suggests that these faculty can serve as significant agents of change and that perhaps more could be done to empower others in similar roles to do so. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 63 the fledgling project presents a potential model for the integration of disciplinaryfocused professional skills into the biology curriculum. this study largely affirms the efficacy of the curricular structure and strategies used to enhance not only those skills, but students’ perceptions of the efficacy of such skills. this emphasis on perception may be related to an emerging thread of education research that focuses on the significance of non-cognitive factors, such as persistence, selfefficacy, judgement, and sociability, and their connection to improved academic and professional outcomes (brinkmann et al., 2016; burtner, 2005; estrada et al., 2011; hunter et al., 2007; kreber, 2003; mendez et al., 2008). in a related vein, advanced research has been conducted to identify and assess the degree to which students in stem majors have developed an identifiable “scientific mindset” or way of thinking about science (adams et al., 2006). this articulation of a disciplinary mindset can easily be extended to other fields, replacing them word “scientific” with the appropriate adjective. in some sense, though, this mindset may be more generalizable. the experience of the biol 3920 course, now having been taught over a three-year period, with its highly responsive pedagogical approach, suggests the possibility of the existence of an integrative mindset that brings together the traits associated with a growth mindset (such as persistence, self-efficacy, judgment, and sociability) with a distinctively disciplinary, in this case, scientific, way of thinking. this fledgling mindset posits that a student can assimilate a set of discipline-based skills and then utilize those same skills in order to successfully navigate a long-term professional career for themselves. because a mindset is a belief that precedes and underlies formal learning, it is often elusive to capture or assess. our survey-based study examines different facets of the development of a fledgling mindset over the course of a one semester, discipline-specific career development course, but more work remains to be done to confirm the construct validity and replicability of our results. if the fledgling mindset can be identified through further practice and research, then perhaps it can also be learned and fostered through the guidance of many other mother birds, i.e., instructors. it may just be possible that if we teach them to believe that they can fly, they will. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references adams, w. k., perkins, k. k., podolefsky, n., dubson, m., finkelstein, n. d., & weiemann, c. e. (2006). new instrument for measuring student beliefs about physics and learning physics: the colorado learning attitudes about science survey." physical review special topics physics education research, 2(1). doi:10.1103/physrevstper.2.010101 journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 64 adedokun, o. a., bessenbacher, a. b., parker, l. c., kirkham, l. l., & burgess, w. d. (2013). research skills and stem undergraduate research students' aspirations for research careers: mediating effects of research self-efficacy. journal of research in science teaching, 50(8), 940–951. allen, g. k., wedman, j. f., & folk, l. c. (2001). looking beyond the valley: a fiveyear case study of course innovation. innovative higher education, 26(2), 103–119. baldwin, r. g., & chronister, j. l. (1996). full-time non-tenure-track faculty. nea higher education research center update, 2(5). baram-tsabari, a., & lewenstein, b. v. (2013). an instrument for assessing scientists’ written skills in public communication of science. science communication, 35(1), 56–85. besley, j. c., & tanner, a. h. (2011). what science communication scholars think about training scientists to communicate. science communication, 33(2), 239–263. borrego, m., & henderson, c. (2014). increasing the use of evidence-based teaching in stem higher education: a comparison of eight change strategies. journal of engineering education, 103(2), 220–252. borzi, m. g., & mills, t. h. (2001). communication apprehension in upper level accounting students: an assessment of skill development. journal of education for business, 76(4), 193–198. brinkmann, d., harvey, j. e., zou, c., crosthwait, r. j., elrod, c. c., & flachsbart, b. b. (2016). putting a business mindset into stem education and career planning. issues in information systems, 17(2), 242–249. brookes, g., & baker, p. (2017). what does patient feedback reveal about the nhs? a mixed methods study of comments posted to the nhs choices online service. bmj open, 7(4). doi:10.1136/bmjopen-2016-013821 brownell, s. e., price, j. v., & steinman, l. (2013). science communication to the general public: why we need to teach undergraduate and graduate students this skill as part of their formal scientific training. journal of undergraduate neuroscience education, 12(1), e6e10. burtner, j. (2005). the use of discriminant analysis to investigate the influence of non-cognitive factors on engineering school persistence. journal of engineering education, 94(3), 335–338. retrieved from http://faculty.mercer.edu/burtner_j/documents/08-ee025-04-5010.pdf cappelli, p. (2014). skill gaps, skill shortages and skill mismatches: evidence for the us. irl review, 68(2), 251–290. retrieved from http://www.nber.org/papers/w20382 carnevale, a. p., smith, n., & melton, m. (2011). stem: science technology engineering mathematics. georgetown university center on education and the workforce. retrieved from https://cew.georgetown.edu/report/stem/ journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 65 christodoulou, a. (2017). engaging in and with research to improve stem education. research papers in education, 32(4), 1–3. cornell, a., gibson, k. f., & williams, t. d. (2017). physiological maturity at a critical life-history transition and flight ability at fledging. functional ecology, 31(3), 662–670. drew, d. e. (2015). stem the tide: reforming science, technology, engineering, and math education in america. jhu press. duke, j. m. (2003). a web-based interface for student peer review, problem-based learning, and peer pressure. journal of natural resources and life sciences education, 32, 52–56. ennis, t. (2014) ttu internal employer survey. tennessee tech university. estrada, m., woodcock, a., hernandez, p. r., & schultz, p. w. (2011). toward a model of social influence that explains minority student integration into the scientific community. journal of educational psychology, 103(1), 206–222. fairweather, j. (2008). linking evidence and promising practices in science, technology, engineering, and mathematics (stem) undergraduate education. status report of the national academies national research council board of science education. retrieved from https://nsf.gov/attachments/117803/public/xc--linking_evidence-fairweather.pdf felder, r. (2016). rethink traditional teaching methods to improve learning and retention. journal of stem education, 17(1). feller, r. (2011). advancing the stem workforce through stem-centric career development. technology and engineering teacher, 71(1), 6–12. gafney, l., & varma-nelson, p. (2007). evaluating peer-led team learning: a study of long-term effects on former workshop peer leaders. journal of chemical education, 84(3), 535–539. galbraith, j. (2012). using student competition field trips to increase teaching and learning effectiveness. journal of natural resources and life sciences education, 41(1), 54–58. gormally, c., brickman, p., & lutz, m. (2012). developing a test of scientific literacy skills (tosls): measuring undergraduates’ evaluation of scientific information and arguments. cbe-life sciences education, 11(4), 364–377. grandy, j. (1998). persistence in science of high-ability minority students: results of a longitudinal study. the journal of higher education, 69(6), 589–620. green, d. w. (2007). adjunct faculty and the continuing quest for quality. new directions for community colleges, 2007(140), 29–39. hannah, a., millichamp, c. j., & ayers, k. m. (2004). a communication skills course for undergraduate dental students. journal of dental education, 68(9), 970– 977. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 66 hernández, r. (2012). does continuous assessment in higher education support student learning? higher education, 64(4), 489–502. hodge, d. c., nadler, m. k., shore, c., & taylor, b. a. p. (2011). institutionalizing large scale curricular change: the top 25 projects at miami university. change: the magazine of higher learning, 43(5), 28–35. hunter, a. b., laursen, s. l., & seymour, e. (2007). becoming a scientist: the role of undergraduate research in students’ cognitive, personal, and professional development. science education, 91(1), 6–74. jolley, m. r., cross, e., & bryant, m. (2014). a critical challenge: the engagement and assessment of contingent, part-time adjunct faculty professors in united states community colleges. community college journal of research and practice, 38(2-3), 218–230. jurin, r. r., roush, d., & danter, j. (2010). environmental communication: skills and principles for natural resource managers, scientists, and engineers. ny: springer netherlands. kezar, a., & sam, c. (2010). special issue: understanding the new majority of nontenure-track faculty in higher education–demographics, experiences, and plans of action. ashe higher education report, 36(4), 1–133. komarraju, m., swanson, j., & nadler, d. (2014). increased career self-efficacy predicts college students’ motivation, and course and major satisfaction. journal of career assessment, 22(3), 420–432. kreber, c. (2003). the relationship between students' course perception and their approaches to studying in undergraduate science courses: a canadian experience. higher education research & development, 2(1), 57–75. kulturel-konak, s., konak, a., esparragoza, i. e., & kremer, g. e. o. (2013, march). assessing professional skills in stem disciplines. integrated stem education conference (isec). symposium conducted at the meeting of institute of electrical and electronics engineers, princeton, nj. doi:10.1109/isecon.2013.6525216 lazonder, a. w., & harmsen, r. (2016). meta-analysis of inquiry-based learning: effects of guidance. review of educational research, 86(3), 681–718. lee, v. s. (2012). what is inquiry-guided learning? new directions for teaching and learning, 2012(129), 5–14. levin, j. s., & shaker, g. g. (2011). the hybrid and dualistic identity of full-time non-tenure-track faculty. american behavioral scientist, 55(11), 1461–1484. lucy, c. a. (2017). experiences and benefits of a career development course for undergraduate chemistry students. analytical and bioanalytical chemistry, 409(22), 5185–5190. mau, w. c. (2003). factors that influence persistence in science and engineering career aspirations. the career development quarterly, 51(3), 234–243. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 67 mccrickerd, j. (2012). understanding and reducing faculty reluctance to improve teaching. college teaching, 60(2), 56–64. mendez, g., buskirk, t. d., lohr, s., & haag, s. (2008). factors associated with persistence in science and engineering majors: an exploratory study using classification trees and random forests. journal of engineering education, 97(1), 57–70. michaelsen, l. k., davidson, n., & major, c. h. (2014). team-based learning practices and principles in comparison with cooperative learning and problem-based learning. journal on excellence in college teaching, 25(3-4), 57–84. retrieved from http://www.lhthompson.com/uploads/4/2/1/1/42117203/team_based_learni ng_-_group_work.pdf mohan, a., merle, d., jackson, c., lannin, j., & nair, s. s. (2010). professional skills in the engineering curriculum. ieee transactions on education, 53(4), 562–571. niederle, m., & vesterlund, l. (2007). do women shy away from competition? do men compete too much?. the quarterly journal of economics, 122(3), 1067– 1101. nace: national association of colleges and employers. (2016). job outlook 2016: the attributes employers want to see on new college graduates’ resumes. retrieved from http://www.naceweb.org/career-development/trends-andpredictions/job-outlook-2016-attributes-employers-want-to-see-on-newcollege-graduates-resumes/ oberlin, s., chapman, s., waneka, r., & spetz, j. (2015). impact of the 2010 affordable care act on the california health care labor force. report from the regents of the university of california. retrieved from https://healthforce.ucsf.edu/publications/impact-2010-affordable-care-actcalifornia-health-care-labor-force peng, h., lin, y. h., & lin, c. j. (2017). satisfaction evaluation of the effectiveness of the two models of specialty-oriented career planning course. international journal of psychological studies, 9(3), 51–63. pintrich, p. r., & schunk, d. h. (2002). motivation in education: theory, research and applications (2nd ed.). englewood cliffs, nj: prentice hall merrill. prince, m., & felder, r. (2007). the many faces of inductive teaching and learning. journal of college science teaching, 36(5), 14–20. reynolds, j. a., thaiss, c., katkin, w., & thompson, r. j., jr. (2012). writing-tolearn in undergraduate science education: a community-based, conceptually driven approach. cbe-life sciences education, 11(1), 17–25. ryan, g. (1993). student perceptions about self-directed learning in a professional course implementing problem-based learning. studies in higher education, 18(1), 53–63. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 68 scaramozzino, j. m. (2010). integrating stem information competencies into an undergraduate curriculum. journal of library administration, 50(4), 315– 333. scott, p. h., veitch, n. j., gadegaard, h., mughal, m., norman, g., & welsh, m. (2017). enhancing theoretical understanding of a practical biology course using active and self-directed learning strategies. journal of biological education, 52(2), 1–12. doi:10.1080/00219266.2017.1293557 sibley, j., & ostafichuk, p. (2015). getting started with team-based learning. sterling, va: stylus publishing, llc. thiry, h., laursen, s. l., & hunter, a. b. (2011). what experiences help students become scientists? a comparative study of research and other sources of personal and professional gains for stem undergraduates. the journal of higher education, 82(4), 357–388. thompson, l., & blankinship, l. a. (2015). teaching information literacy skills to sophomore-level biology majors. journal of microbiology & biology education, 16(1), 29–33. tobias, s. (1990). they're not dumb. they're different.: a new “tier of talent” for science. change: the magazine of higher learning, 22(4), 11–30. toit, e. (2012). constructive feedback as a learning tool to enhance students' selfregulation and performance in higher education. perspectives in education, 30(2), 32–40. tseng, k. h., chang, c. c., lou, s. j., & chen, w. p. (2013). attitudes towards science, technology, engineering and mathematics (stem) in a project-based learning (pjbl) environment. international journal of technology and design education, 23(1), 87–102. u.s. bureau of labor statistics. (2015). occupation outlook handbook. retrieved from https://www.bls.gov/ooh/ van blankenstein, f. m., saab, n., van der rijst, r. m., danel, m. s., bakker-van den berg, a. s., & van den broek, p. w. (2018). how do self-efficacy beliefs for academic writing and collaboration and intrinsic motivation for academic writing and research develop during an undergraduate research project? educational studies, 2018, 1–17. vaz, r. f., quinn, p., heinricher, a. c., & rissmiller, k. j. (2013, june). gender differences in the long-term impacts of project-based learning. american society for engineering education annual conference and exhibition. symposium conducted at the meeting of american society for engineering education, atlanta, ga. retrieved from https://www.asee.org/public/conferences/20/papers/6899/view white, h. b., benore, m. a., sumter, t. f., caldwell, b. d., & bell, e. (2013). what skills should students of undergraduate biochemistry and molecular biology programs have upon graduation? biochemistry and molecular biology education, 41(5), 297–301. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 69 wingate, u. (2006). doing away with ‘study skills’. teaching in higher education, 11(4), 457–469. wright, g. b. (2011). student-centered learning in higher education. international journal of teaching and learning in higher education, 23(1), 92–97. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 70 appendix 1 topic-related responses in agreement with statement by major/concentration pre % agree statement b/gb b/eb b/hs b/cmb wfs/w wfs/f wfs/c other career management i would prefer resume workshops provided by experienced biology professionals. 33.6 3.6 53.9 26.3 32.2 16.0 13.2 21.2 prior to taking biol 3920, did you have previous professional work experience? 30.9 20.0 44.2 22.4 38.8 10.3 12.1 21.2 communication i would like to work on writing and presentation assignments in class and getting immediate feedback. 27.6 3.2 46.5 26.3 41.9 22.4 13.6 18.6 i would have been better prepared if more feedback was received on writing and presentation assignments in other courses. 29.6 0.0 40.0 31.3 46.8 15.7 15.7 20.9 teamwork/leadership i would like to have an element of competition added to the assignments. 20.0 4.4 40.0 37.8 44.4 13.3 17.8 22.2 i would rate my current leadership skills as very/fairly strong. 26.2 4.3 61.6 23.8 42.6 11.6 12.3 17.6 i would rate my current teamwork skills as very/fairly strong. 33.5 3.1 56.7 23.1 37.1 12.2 10.7 23.5 i am able to provide constructive criticism to a peer about their work. 30.9 5.1 53.3 22.0 40.1 11.6 13.6 23.4 i am able to accept constructive criticism about my work. 30.0 3.7 55.9 21.9 39.3 14.2 14.2 20.8 journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 71 critical thinking i would like to have options for assignments when fulfilling requirements for the course. 34.4 3.2 52.5 25.2 36.8 14.2 11.6 22.1 i would like to have flexibility in choosing my team's project topic. 31.8 3.5 57.8 26.7 28.8 11.4 15.8 24.4 key: b/gb: biology/general biology; b/eb: biology/environmental biology; b/hs: biology/health science; b/cmb: biology/cellular and molecular biology; wfs/w: wildlife and fisheries science/wildlife; wfs/f: wildlife and fisheries science/fisheries; wfs/c: wildlife and fisheries science/conservation. topic-related responses in agreement with statement by major/concentration (cont’d) post % agree statement b/gb b/eb b/hs b/cmb wfs/w wfs/f wfs/c other career management i would prefer resume workshops provided by experienced biology professionals. 40.9 1.6 50.5 19.4 45.0 12.8 17.7 12.1 prior to taking biol 3920, did you have previous professional work experience? 33.7 6.7 44.5 41.0 29.1 8.2 10.7 26.1 communication i would like to work on writing and presentation assignments in class and getting immediate feedback. 31.0 2.3 44.5 31.2 46.5 11.6 17.6 15.3 i would have been better prepared if more feedback was received on writing and presentation assignments in other courses. 39.3 2.6 54.3 34.1 41.8 8.8 11.4 7.7 teamwork/leadership i would like to have an element of competition added to the assignments. 47.1 0.0 20.7 24.1 31.0 3.4 36.8 36.8 journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 72 i would rate my current leadership skills as very/fairly strong. 28.2 0.0 57.6 33.6 42.1 13.9 11.8 12.8 i would rate my current teamwork skills as very/fairly strong. 33.3 2.2 47.0 30.6 44.4 11.9 16.5 14.1 i am able to provide constructive criticism to a peer about their work. 31.0 1.9 46.1 33.7 41.6 12.5 16.6 16.6 i am able to accept constructive criticism about my work. 33.9 1.4 49.0 30.1 47.4 9.6 12.3 16.3 critical thinking i would like to have options for assignments when fulfilling requirements for the course. 31.9 0.0 49.6 27.6 46.3 11.2 14.4 19.1 i would like to have flexibility in choosing my team's project topic. 31.0 1.5 47.7 24.4 42.9 15.5 18.5 18.5 key: b/gb: biology/general biology; b/eb: biology/environmental biology; b/hs: biology/health science; b/cmb: biology/cellular and molecular biology; wfs/w: wildlife and fisheries science/wildlife; wfs/f: wildlife and fisheries science/fisheries; wfs/c: wildlife and fisheries science/conservation. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 73 appendix 2: student survey questions 1. in this course, what percentage of assignments have you heard were "busy work"? a. 0-25% b. 26-50% c. 51-75% d. 76-100% 2. pick, drop, and rank the assignments by how useful you think they would be in preparing for your career. a. online quizzes b. technical writing c. annotated bibliography d. title and abstract e. reference list f. individual presentation or facilitated discussion g. resume h. cover letter i. interview q&a j. group powerpoint k. technical communication l. peer review of writing (resumes, cover letters, abstracts) m. peer review of presentations (individual or group) n. group poster o. storyboard 3. in this course, i would like to have an element of competition added to the assignments. a. strongly agree b. mildly agree c. no opinion d. mildly disagree e. strongly disagree 4. in this course, i would have been better prepared if more feedback was received on writing and presentation assignments in other courses. a. strongly agree b. mildly agree c. no opinion d. mildly disagree e. strongly disagree 5. in this course, i would prefer resume workshops provided by experienced biology professionals. a. strongly agree b. mildly agree c. no opinion d. mildly disagree e. strongly disagree 6. in this course, i would like to have options for assignments when fulfilling requirements for the course. journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 74 a. strongly agree b. mildly agree c. no opinion d. mildly disagree e. strongly disagree 7. in this course, i would like to have flexibility in choosing my team's project topic. a. strongly agree b. mildly agree c. no opinion d. mildly disagree e. strongly disagree 8. what is your major and concentration? 9. what is your grade point average (gpa)? 10. prior to taking this course, did you have previous professional work experience? a. yes, in a wage-earning position b. yes, in an internship or co-op c. no 11. which career path best describes your intentions after graduation? a. graduate or professional school b. applied or field position c. general (considering more than one option) d. uncertain e. other (please describe) 12. to the best of your knowledge, rank the importance of the following skills sought be employers when recruiting undergraduate students (most important skill= 1). a. critical thinking b. written communication c. grade point average (gpa) d. professional references e. internships or co-ops f. work experience of any type g. volunteer work h. willingness to learn i. problem-solving ability j. leadership k. positive attitude l. team player m. self-motivation n. technical knowledge relevant to discipline o. public speaking p. ethics q. computer literacy journal of effective teaching in higher education, vol. 3, no. 2 the fledgling mindset 75 r. data analysis & visualization s. click to write item 13. i would rate my current leadership skills as: a. very strong b. fairly strong c. no experience d. fairly weak 14. i would rate my current teamwork skills as: a. very strong b. fairly strong c. no experience d. fairly weak e. very weak 15. i am able to provide constructive criticism to a peer about their work. a. strongly agree b. mildly agree c. no opinion d. mildly disagree e. strongly disagree 16. i am able to accept constructive criticism about my work. a. strongly agree b. mildly agree c. no opinion d. mildly disagree e. strongly disagree journal of effective teaching in higher education, vol. 1, no. 1 dear contributors, reviewers, and readers: we would like to thank you for your patience and support as we developed and launched our first volume of the journal of effective in teaching in higher education (jethe). we also extend gratitude for their support and counsel to the campus leaders from our institution, the university of north carolina wilmington, specifically provost marilyn sheerer, senior associate provost rich ogle, and dr. diana ashe, director of the centers for teaching excellence and faculty leadership. these individuals along with many other colleagues and graduate students who served as reviewers for this issue and undergraduate students who designed our new logo made the launch of our new publication possible. as new editors of the journal, we spent several months revising our editorial process, developing the online site for the journal (https://jethe.org), and working with reviewers and authors to prepare our first volume. our primary goal is to deliver high quality, data driven articles contributing to the scholarship of teaching and learning (sotl) in higher education. we believe the four pieces included in our first volume demonstrate this commitment and reflect the range of work we will publish in future volumes. in addition to articles that utilize quasi-experimental design to examine the effectiveness of teaching interventions, the current volume also includes articles that utilize qualitative research to explore the effectiveness of an interprofessional event and the implementation of handwriting as a learning tool in science education. all of the pieces provide meaningful implications for the practice of teaching across disciplines—an important contribution that is emphasized in jethe. we invite you to engage with the pieces in our first issue and consider making your own contributions to this work. we also invite you become a reviewer for jethe. if you are interested in reviewing for our journal, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. the deadline for submissions for our next issue of jethe will be january 15, 2019. we look forward to receiving interesting and rigorous submissions and working with some of you as authors and reviewers. -james devita and colleen reilly coeditors, jethe https://jethe.org/ mailto:jethe@uncw.edu microsoft word harrison review.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.202 range: why generalists triumph in a specialized world by david epstein, 2019. new york, ny: penguin, 339 pages, isbn 9780735214484, hardcover, $16.89. reviewed by laura m. harrison, ohio university, drlauramharrison@gmail.com if you are a person who values the broad knowledge afforded by a liberal arts education, you have probably experienced the challenge of articulating its value to the many naysayers who eschew the liberal arts in favor of a more “practical” education. this is a problem with which i wrestled, too, using vague language like “critical thinking,” “public good,” and “engaged citizenship” to try to convince people that students should be exposed to a wide range of intellectual offerings. i generally floundered in these efforts; lofty concepts are no match for convincing selling points about the material benefits of specialized fields of study leading to specific jobs. i was stuck on this conundrum until i discovered david epstein’s range: why generalists triumph in a specialized world. although range does not belong to the long line of higher education scholarship dedicated to defending the liberal arts, epstein succeeds in achieving this goal where many have faltered. epstein models the thesis of his work by providing the outsider perspective needed to make actual progress on this age-old problem of communicating the value of a broad education. he begins in the world of sports, challenging the often-touted example of tiger woods’ singularly obsessive focus on golf as the model for how to achieve great heights. woods’ approach is known as the “10,000 hour” rule: the idea that the way to develop mastery is to do the thing repeatedly for 10,000 hours. those who ascribe to this technique necessarily believe in hyper-specialization; there is no time for other interests at this level of dedication. anything outside the targeted skill is simply useless distraction from the task at hand. epstein acknowledges that this strategy worked for woods but positions him as the exception rather than the rule. he uses roger federer as a contrast with tiger woods, showing how federer became a tennis star not in spite of his lack of an early, singular commitment to the sport but because of the wide interests that led him to play creatively. epstein’s advocacy of broader interests seems counterintuitive given the dominant narratives about brilliance being driven by singular obsession. epstein exposes a hidden variable in these stories: they tend to be about skills like chess with orderly rules and predictable patterns. when we only train for immediate, short-term results in artificial contexts, we produce students who do well on tests, but cannot apply the knowledge in what psychologist robin hogarth called the “wicked environment” of the real world. as epstein explains, the world we actually live in is chaotic, demanding “conceptual reasoning skills that can connect new ideas and work across contexts” (p. 53). the implications of epstein’s central argument about the importance of range are significant for education, particularly in this current moment of life in a world review of range: why generalists triumph in a specialized world 144 journal of effective teaching in higher education, vol. 4, no. 1 shaken up by covid-19. the old arguments about the liberal arts being nice but impractical in the real world are weaker now that covid-19 has upended much of what we thought we knew about the real world. the “wicked environment” created by the pandemic demands not a singular focus on one skill but the ability to make the intellectual connections necessary to adapt to changing conditions. epstein describes this process as follows: that is, the more contexts in which something is learned, the more the learner creates abstract models, and the less they rely on any particular example. learners become better at applying their knowledge to a situation they’ve never seen before, which is the essence of creativity. (p. 77) the seemingly safe route of the hyperspecialized course of study is less stable in a world characterized by the kind of upheaval caused by pandemics (and climate change, political unrest, and other disrupters). epstein goes on to identify an equally significant disrupter that makes range necessary: the proliferation of artificial intelligence (ai), a competitor that is impossible for humans to beat on measures of short-term repetitive tasks. much has been written about computers beating chess, video game, jeopardy!, and other kinds of champions, often in the spirit of great concern about humans’ employability in an increasingly technology-driven world. epstein expands these stories by describing the arenas in which ai fails to win against humans, that is, activities requiring more long-term adaptive strategies. based on plentiful research and examples, he concludes, “the more constrained and repetitive a challenge, the more likely it will be automated, while great rewards will accrue to those who can take conceptual knowledge from one problem or domain and apply it in an entirely new one” (p. 53). epstein therefore advocates for educational experiences that inculcate this kind of conceptual knowledge. unfortunately, many schools at both the k-12 and higher education levels seem to be stuck in the mindset of narrow learning for short-term performance on tests. this is not difficult to see in k-12 schools often driven by high stakes testing but may be less obvious in higher education. the recent focus on learning outcomes, however, should give us pause in light of epstein’s argument. he refers to “an enormous and too often ignored body of work demonstrating that learning itself is best done slowly to accumulate lasting knowledge, even when that means performing poorly on tests of immediate progress” (p. 11). privileging subjects easy to measure on standardized tests is therefore a losing strategy in the wicked environment of the real world. epstein does not propose the wholesale eradication of standardized tests or learning outcomes. instead, he offers his critique in the spirit of balance in education systems that too often put the proverbial cart before the horse when it comes to student learning and assessment. muller (2019) makes a similar argument in his aptly named work, the tyranny of metrics, advocating for right-sized assessment measures that enhance rather than compete with a meaningful education. review of range: why generalists triumph in a specialized world 145 journal of effective teaching in higher education, vol. 4, no. 1 one drawback of the book for a higher education audience is that it is short on policy and practice recommendations for addressing the issues it raises. this is to be expected given that the book was written for a more general audience. the good news is that readers can extrapolate some recommendations for themselves, particularly with regard to curriculum development. epstein provides excellent talking points for why the kinds of broad knowledge instilled by the liberal arts are indeed practical for the world in which we now live. the current moment requires the ability to imagine different possibilities in a world characterized by disruption. my liberal arts professors helped me prepare for disruption in life by teaching me how to imagine different ways of being through reading and writing. invoking keats, mark edmundson explains this process as breaking free from one’s habitual self: habitual self is good for a grocery list, a laundry list, a note to the mechanic, or a note of thanks for the spotted birthday tie or the fruit-scented candle…i think sitting down to write is about getting loose from the habitual self. if you’re going to tap into what’s most creative inside you, you’ve got to find a way to outwit the pressures of the ordinary. (p. 141) having a mind capable of reimagining life beyond the habitual self will likely prove increasingly valuable in a world with more ambiguities. if students continue to think that the best recipe for success is the “safe” major leading to a similarly named career, they may be sorely mistaken. in my doctoral courses on organizational change, professors likened the situation to being a horseshoe maker right before the car was invented. a person could be the most competent and efficient horseshoe maker in the world, but buckling down and making more horseshoes would not be enough to adapt successfully to the changing conditions brought about by the invention of the car. epstein does not advocate overcorrections like abandoning vocational and/or professional courses of study; he calls for greater interdisciplinarity so that all students gain a broader knowledge base regardless of major. our students—whatever their field of study—need new vocabularies, new languages that will help them navigate the radically changed world in which we find ourselves. references edmundson, m. (2018). the heart of the humanities: reading, writing, teaching. bloomsbury, usa. muller, j. (2019). the tyranny of metrics. princeton university press. microsoft word jethe editor letter december 2020.docx journal of effective teaching in higher education, vol. 3, no. 2 dear contributors, reviewers, and readers: we are incredibly grateful for the work of our contributors and reviewers during an uncertain and difficult time for higher education around the world. the effects of covid-19 have touched nearly all aspects of life, including the review and publication process for scholarly journals. we want to reaffirm our hope that during these times you have been able to stay safe and healthy as you support your students, colleagues, communities, and families. we want to thank our reviewers and authors who overcame the obstacles presented to us in 2020 and helped us to produce a fall issue of the journal of effective teaching in higher education (jethe). although originally intended to be published in november 2020, the current issue was slightly delayed and published in mid-december 2020. although the global pandemic slowed us down, it did not prevent us from curating the five pieces included in jethe volume 3, issue 2. the current issue demonstrates our continued commitment to high quality, data driven scholarship as well as the diverse range of topics on the scholarship of teaching and learning (sotl) that we publish in jethe. each article represents a distinct perspective on teaching and learning in higher education today. one article in this issue explores perceptions of culturally responsive teaching practices at a minority-serving institution. a second article examines curricular alignment with student learning outcomes in a teacher education program. another contribution discusses transformative pedagogical practices that can be utilized to develop student voice in writing assignments, while a fourth article engages in an exploratory study of teaching personas. the fifth article included in the current issue analyzed student survey responses to examine skills developed in a biology course. as always, all articles include implications for the practice of teaching across disciplines—an important contribution that is emphasized in jethe. we also want to give a special thanks to everyone who contributed feedback about the journal during an applied learning assessment project conducted this fall semester. we received over 70 responses to the survey we shared in october, which is approximately 20% of all registered users of the journal. your responses have provided us with meaningful feedback that will help us to strengthen the journal in the future. thank you! letter from the editors ii journal of effective teaching in higher education, vol. 3, no. 1 additionally, we want to highlight the continued demand for access to articles published in jethe: as of december 2020, readers have completed nearly 5000 full-text downloads of articles published in our first three issues. on average, articles have been downloaded approximately 200 times/each. beyond downloads, we are also seeing an increase in the citation of jethe articles by other authors; notably, one article from our first issue boasts 15 additional cites with most cited by 1-2 other authors. your help and support have been invaluable as we continue to develop the journal, and we hope to continue to engage an international audience with current issues in teaching and learning in higher education. we invite you to engage with the pieces in our december 2020 issue and consider making your own contributions to this work. we also invite you to become a reviewer for jethe. if you are interested in reviewing for our journal, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. our next issue will be published in april 2021, and we look forward to working with some of you as authors and reviewers in production of that and future issues of jethe. to submit an article or download full-text copies of current and past issues please visit http://jethe.org. james devita and colleen reilly coeditors, jethe microsoft word jethe editor letter spring 2021.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.231 dear contributors, reviewers, and readers: in the past year, the effects of covid-19 have touched nearly all aspects of life, including the review and publication process for scholarly journals like ours. we are grateful for the work completed by our authors and reviewers during these difficult times and want to thank everyone for their contributions to the current issue of journal of effective teaching in higher education (jethe). we also want to reiterate our hope that you and your family have stayed safe and healthy as we continue to navigate the ongoing global pandemic. although the pandemic slowed down our review and publication process this year, it did not prevent us from curating the eight articles and our first-ever book review included in jethe volume 4, issue 1. the current issue demonstrates our continued commitment to high quality, data driven scholarship across a diverse range of topics on the scholarship of teaching and learning (sotl) that we publish in jethe. several of the articles address classroom-based practices and interventions that enhance student learning and development, including teaching art online (lai), engaging english language learners (lewis & brown), encouraging creativity in the classroom (miller, cruz, & kelley), presenting outcomes from student-centered approaches to teaching (murphy, eduljee, & croteau), and highlighting the value of reflection as pedagogy (virtue). wonder’s article focuses on student perceptions of class participation in the philippines, while ericksen and williamson-ashe examine the relationship between high-impact educational experiences and the emerging values model approach to group work. the eighth article included in this issue shares findings from a study of students’ optimism, resilience, and other health factors during covid-19 (burt & eubank)—the first article jethe has published that focuses on the effects of the ongoing pandemic. as always, all articles include implications for the practice of teaching across disciplines—an important contribution that is emphasized in jethe. as previously mentioned, we included our first-ever book review in the current issue of jethe: laura harrison’s review of david epstein’s (2019) book range: why generalists triumph in a specialized world. if interested in submitting a book review for jethe, please email the editors at jethe@uncw.edu with your proposal. despite the challenges of the pandemic, we continue to seek improvements to our review and publication processes as well as the accessibility and letter from the editors journal of effective teaching in higher education, vol. 4, no. 1 ii notoriety of jethe. in the past year we have secured dois for all past and future articles and applied to list jethe in the directory of open access journals (doaj). jethe is also included among the open-access journals supported by the university of north carolina system and listed on their new site: https://openunc.org/. additionally, we also welcomed a sotl faculty fellow to our journal team. dr. indranil ghosh, associate professor of mathematics and statistics at uncw, started as a fellow in spring 2021 and is assisting with article reviews and conducting data analysis on the article review and publication process. we also want to highlight the continued demand for access to articles published in jethe: as of april 2021, readers have completed over 6,300 full-text downloads of articles published in our first five issues. on average, articles have been downloaded approximately 210 times/each. our next issue will be published in summer 2021 and will be the first time we have published a special issue. the special issue focuses on “in the along”: curricular and pedagogical imperatives for black mattering and is edited by dr. wilson okello, uncw, and dr. tj stewart, iowa state university. the editors have curated a set of articles that contribute to a transdisciplinary understanding "of what black freedom wants [and] what black freedom requires of us" (dumas, 2018, p. 35) as we teach and advise in and around schools. marginalization and oppression from racial injustice persist in the us and throughout the world, but we hope that this special issue provides meaningful insights and encouragement for all educators to contribute to creating more inclusive and equitable classrooms, institutions, and communities. we invite you to engage with the pieces in our spring 2021 issue and consider making your own contributions to this work. to submit an article or download full-text copies of current and past issues please visit http://jethe.org. we also invite you to become a reviewer for jethe. if you are interested in reviewing for our journal, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. thank you for your continued support of jethe! james devita and colleen reilly coeditors, jethe microsoft word okello&stewart_letter.docx journal of effective teaching in higher education, vol. 4, no. 2 doi:10.36021/jethe.v4i2.255 to whom it should concern, over the past year we have labored to curate this special issue of the journal of effective teaching in higher education titled in the along: curricular and pedagogical imperatives for black mattering. this process began during a global health pandemic and the already ongoing assault on black people and lifeways, and we complete the issue in the same, if not more precarious place. while we know the elusive and insidious nature of antiblackness transcends the bounds of education and the academy, our hope is that this special issue contributes to disrupting and ultimately eradicating this violence one day. as bey (2020) reminds us, there are actions we can take that are on the way to our desire for black mattering and black freedom, that is, “there are things to be done in the interim,” which may lack complete fulfillment of our efforts but are “nonetheless in the service of that end” (p.103). it is in this spirit we deliver this issue to you. in the opening article that immediately follows this letter, we situate context to define and refine our call for black mattering in teaching, pedagogy, and education; in addition to recounting the unique offerings each of the authors provide. however, our intention for this letter is to briefly offer gratitude to all who made the special issue possible. first, we acknowledge the collective of beautiful, brilliant, and black scholars who gave of their time and capacity to serve as reviewers for in the along. during a time when so many black people were working tirelessly to embrace our lives, our work, our joy, and our pain, service in this way was an added stress on already limited energy and time: we thank you for gifting some of yours to us. we also acknowledge james devita and colleen reilly for the invitation to serve as guest editors for the issue and for allowing us the freedom to vision and shape the work without restriction or hinderance; we thank you. finally, we would like to thank the authors for contributing their work to this project and their patience as we worked our way through the process to get the issue to this place; thank you for trusting our vision for what this project might be. as we turn to the issue, we know in academe, education, teaching, and pedagogy, scholarship not only structures our work, it signifies it. with white supremacy as the pretext, yancy (2012) offered “it is white meaning making that creates the conditions under which black people are always already marked as different/deviant/dangerous” (p. 5) and which informs violence against black people in pedagogy and praxis. these same logics animate what is often stressed or repressed within research and writing. more than letter from the editors ii journal of effective teaching in higher education, vol. 4, no. 2 mere words, the ideas we champion or abandon implicate how and if black mattering might ever materialize. toni morrison (1993) warned that “oppressive language does more than represent violence; it is violence; does more than represent the limits of knowledge; it limits knowledge” (para 13). she specifically ensured to name the “proud but calcified language of the academy” (para 13) as a place this violence can readily occur, and further implicates the literature which begets knowledge production which in turn begets praxis. holding the fullness of these realities, we position in the along as a counter, a resistance, a disruption to this prevailing imposition. a moment to try and reject being defined by antiblackness; that which seeks to victimize us; to kill us. a moment to embrace ourselves as we hope to be, an attempt to call ourselves by the names we hear in our dreams. in solidarity and black love, ase. terah j. stewart & wilson k. okello references bey, m. (2020) anarcho-blackness: notes toward a black anarchism. ak press. morrison, t. 1993. nobel lecture. nobel foundation. retrieved from https://www.nobelprize.org/prizes/literature/1993/morrison/lecture/ yancy, g. 2012. look, a white!: philosophical essays on whiteness. temple university press. microsoft word morton_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi 10.36021/jethe.v4i2.174 unbroken: a critical autoethnography of a black woman’s doctoral experience during blm christina s. morton, university of michigan cspr@umich.edu abstract. in this critical autoethnography, i examine my lived experiences as a black woman doctoral student during the rise of the black lives matter movement. further, as i recount my academic journey in the wake of assaults to black life and resulting black resistance, i discuss the pedagogical interventions of black women faculty members that made me feel as if my life and work mattered in their classrooms. i revisit spoken word poems and class assignments written between 2015 and 2017 along with news articles documenting national events occurring at the time as relevant texts to help me explore and understand my experiences. i utilize critical race theory as an analytic lens, focusing on the following tenets: persistence of racism, critique of color-evasiveness, and counterstorytelling. i conclude with implications regarding how introducing graduate students to critical theory and methodologies can equip them with the tools to empirically explore and articulate their lived realities. moreover, i discuss how such explorations can be validating and healing as students navigate particularly challenging academic and sociohistorical contexts. additionally, i describe how providing students with creative outlets to express themselves in coursework can help them process their experiences and produce material that is humanizing, liberating, and life-giving. keywords: critical autoethnography, critical race theory, black lives matter, higher education, teaching i am not broken i am not silent i am not invisible… standing on a dimly lit stage, facing an audience of my peers and faculty, i felt a surge of strength course through my body as i declared these words. it was 2016, the end of the second year in my higher education doctoral program, and this spoken word performance was a key moment in my educational experience. as the black lives matter (blm) movement—a global network of organizers committed to resisting state-sanctioned violence against black people (blm, n.d.)—gained momentum nationally, i was finding the words to articulate the materiality of my existence in a doctoral program where i was once likened to an apparition. in this critical autoethnography (boylorn & orbe, 2014; holman jones, 2016; madison, 2005), i examine my lived experiences as a black woman doctoral student during the rise of the blm movement utilizing critical race theory (crt) as an analytic lens. i purposely frame this study using crt because my engagement with this theory in graduate education pivotally influenced my racialized sensemaking of my doctoral experience and the world around me. as such, it felt fitting to return to crt as i critically re-examine and recount my academic journey in the wake of assaults to black life and resulting black resistance occurring at that time. i also discuss the pedagogical interventions of black women faculty members that made unbroken: a critical autoethnography of a black woman’s doctoral experience 66 journal of effective teaching in higher education, vol. 4, no. 2 me feel as if my life and work mattered in their classrooms. i revisit spoken word poems and class assignments written between 2015 and 2017 along with news articles documenting national events occurring at the time as “relevant personal and cultural texts” to help me explore and understand my experiences (adams et al., 2015, p. 49). in the literature review that follows, i briefly discuss recent scholarship on the experiences of black women in doctoral education and provide an overview of crt to denote its history and current relevance as i embark on this autoethnographic undertaking. literature review black women in doctoral education research has demonstrated that black women in doctoral education may experience a lack of mentorship, feelings of imposterism, and differential treatment based on the intersections of their race and gender (patterson-stephens et al., 2017). additionally, black doctoral women at predominantly white institutions have reported contending with personal and vicarious racially discriminatory experiences that have made them feel unwelcome in their academic environments (shavers & moore, 2019). conditions like these can make it especially difficult for black doctoral women to feel safe being vulnerable in their academic contexts. for instance, black women attending predominantly white institutions in shavers and moore’s (2014) study described using academic masks, or professional facades, to protect themselves from stereotyping and hide their private selves in their doctoral programs. although participants adopted academic masks for protective purposes, they shared that such masking negatively affected their emotional and psychological well-being. denying aspects of themselves in professional settings left black doctoral women in the study feeling “incomplete, disconnected, and exhausted” (shavers & moore, 2014, p. 404). to navigate the various obstacles black women may confront in their doctoral programs, scholars have emphasized the importance of black women seeking and sustaining culturally responsive mentoring relationships (bertrand jones et al., 2013; gooden et al., 2020; rasheem et al., 2018). bertrand jones et al. (2013) asserted that black doctoral women benefit from relationships with mentors who are aware of the intersecting oppressions black women encounter in their lived realities. mentors can also help black doctoral women begin to understand their standpoints as insiders (e.g., doctoral students) and outsiders (e.g., black women) in the academy, which allows black women to “critique, as well as contribute to knowledge construction” (bertrand jones et al., 2013, p. 333). relatedly, in gooden et al.’s (2020) autoethnographic study, the authors expressed the importance of connecting with critically conscious mentors who can help black women navigate predominantly white institutional contexts. black doctoral women participating in rasheem et al.’s (2018) study reported benefitting from relationships with mentors who shared their identities and interests. these relationships provided black women in the study with transformational growth and personal development, as their mentors offered encouragement and promoted mentees’ success in their programs. though this critical autoethnography focuses on the pedagogical interventions of unbroken: a critical autoethnography of a black woman’s doctoral experience 67 journal of effective teaching in higher education, vol. 4, no. 2 black women faculty that helped me navigate my doctoral program during blm, literature pertaining to the mentoring experiences of black doctoral women is relevant because, as an aspiring educator and scholar, i considered my instructors to be mentors. learning from these faculty members inspired transformational growth and personal development for me as i began to apply critical theory and praxis to my own research and practice. critical race theory: an overview originating in the 1970s, crt was developed by lawyers, activists, and legal scholars committed to maintaining and advancing the gains of civil rights legislation in the 1960s, which were being attacked by oppositionists in law and politics at the time (delgado & stefancic, 2012). under the leadership of founding scholars such as derrick bell, alan freeman, richard delgado, kimberlé crenshaw, mari matsuda, angela harris, charles lawrence, and patricia williams, the crt movement quickly gathered momentum, and, by the 1980s, scholars and activists from various disciplines came together to expand its reach (zamudio et al., 2011). crt’s foundations in critical legal studies and radical feminism contributed to its critique of power and power’s relationship to the construction of social hierarchies and the domination of marginalized people (delgado & stefancic, 2012). though the tenets of crt can vary depending upon the context in which it is being applied, common propositions of crt include the following: racism is an everyday, commonplace experience for people of color; whites, as members of the dominant group, benefit psychically and materially from racial hierarchies; racism advances the interests of wealthy and working class whites; race and racism are socially constructed; the process of racialization varies depending on the evolving interests of the dominant group; people experience intersecting systems of oppression as related to their complex social identities; and the voices of people of color should be centered to contest dominant narratives (delgado & stefancic, 2012). crt’s groundbreaking theorization concerning race and racism with regard to systemic oppression has led to its adoption and evolution in fields outside of law, including education. gloria ladson-billings and william f. tate are largely credited for introducing crt to the field of education in the 1990s (zamudio et al., 2011). during an american education research association conference presentation in 1994, ladson-billings and tate specifically called for new directions in research rooted in crt to address the need for further theorization regarding race and racism in relation to educational inequity (ladson-billings, 2013; zamudio et al., 2011). since then, several scholars have responded to ladson-billings and tate’s call and have used crt as an analytic lens to better understand racial inequality throughout the education system. while crt has been taken up in legal and social science disciplines as a useful tool in both understanding and combatting racism, it has received considerable criticism from oppositionists. recently, crt garnered national attention as the trump administration issued an order for federal agencies to end racial sensitivity trainings utilizing it. in september of 2020, the office of management and budget released a unbroken: a critical autoethnography of a black woman’s doctoral experience 68 journal of effective teaching in higher education, vol. 4, no. 2 memo directing agencies to identify and terminate any contracts related to such trainings, referring to them as “divisive, anti-american propaganda” (schwartz, 2020). specifically, the memo relayed that the former president called for the cessation of these trainings on the basis that they may suggest the u.s. is a racist country or any race is “inherently racist or evil” (schwartz, 2020). despite the misunderstandings and misapplications, along with attempts to stifle its influence, crt remains a vital theoretical frame. its current contestation a clear indication of its contemporary relevance as america reckons with its longstanding history with race and racism. furthermore, i would argue that it is no coincidence that crt is receiving renewed scrutiny while the blm movement continues to gain momentum. created by three black women—alicia garza, patrisse cullors, and opal tometi— blm was initiated in response to the 2013 acquittal of george zimmerman, the man responsible for fatally shooting trayvon martin, an unarmed black teenager in 2012 (blm, n.d.). blm has since coordinated efforts worldwide to demand justice for atrocities committed against black life—including more recent demonstrations and political action in response to the murders of ahmaud arbery, breonna taylor, and george floyd. as members of the blm movement continue their forebears’ struggles for justice and black liberation, crt may once again be called upon to help maintain and advance the strides made. conceptual framework for the purpose of this project, i utilize crt as an analytic lens, focusing on the following tenets: persistence of racism, critique of colorblindness or colorevasiveness, and counterstorytelling. persistence of racism refers to the endemic nature of racism in american society (ladson-billings, 1998). i discuss how enduring legacies of racism influenced my educational experiences as well as the lives of black people such as freddie gray, sandra bland, alton sterling, and philando castile whose lives were taken by police. critique of colorblindness describes critical race theorists’ contestation of post-racial ideologies that ignore the significance of race in social stratification (zamudio et al., 2011). throughout this article, however, i use the term “color-evasiveness” advanced by dis/ability critical race theory (discrit) scholars to disrupt ableist language which conflates blindness with ignorance (annamma et al., 2017, p. 153). i offer examples of how colorevasiveness pervaded doctoral courses where race was not addressed, both in subject matter and with regard to national events happening during that period. counterstorytelling privileges the lived experiences of people of color; thereby, disrupting dominant narratives and strengthening “traditions of social, political, and cultural survival and resistance” (solórzano & yosso, 2002, p. 32). i provide examples of how i began to engage in a praxis of counterstorytelling at this time, which informed my approach to scholarship and became a means of survival and healing for me. unbroken: a critical autoethnography of a black woman’s doctoral experience 69 journal of effective teaching in higher education, vol. 4, no. 2 methodology to examine my lived experiences as a black woman doctoral student attending a predominantly white institution during the rise of the blm movement, i engage in critical autoethnography. critical autoethnography 1) blends theory and storytelling together, 2) bridges material and ethical practice by linking analysis with action, and 3) reveals ways of embodying change (holman jones, 2016). moreover, critical autoethnography addresses injustice and offers a first-person account of discrimination and difference (boylorn & orbe, 2014; madison, 2005). though traditional autoethnographies also require reflexive, systematic analyses of personal experiences as situated in cultural contexts (bhattacharya, 2017), i found critical autoethnography to be appropriate for this study in its explicit emphasis on examining one’s lived experience in the interest of advancing social justice. in this critical autoethnography, i bring together crt and stories of my educational experience, while also demonstrating how crt, the blm movement, and the pedagogical interventions of black women faculty informed my critique and contestation of anti-black racism as a black scholar. additionally, through storytelling, i embody the transformative power of critical theory for black graduate students while navigating a particularly challenging sociohistorical terrain. data utilized for this critical autoethnography included class assignments and original poetry written between 2015 and 2017, along with news articles concerning the events within that time period as relevant texts to help make meaning of my academic experiences and the sociohistorical context (adams et al., 2015). specifically, i reviewed the weekly reflective journals for dr. camille wilson’s class from fall 2016 and the final cumulative course project completed in dr. jacqueline mattis’ class in winter of 2017. for this piece, i found and referenced news articles from the proquest database. i searched for news articles either written within the time of interest or written about events that occurred within that timeframe regarding the blm movement and black lives lost as a result of police brutality. i used search terms such as “black lives matter” and “blm movement,” along with the specific names of black people who were killed by police during that period such as “tamir rice,” “freddie gray,” “sandra bland,” “alton sterling,” and “philando castille.” in total, i reviewed approximately 20 news articles, which helped me create a timeline of events leading up to and coinciding with my engagement in dr. wilson’s and dr. mattis’ respective courses. the news articles were primarily used to contextualize my experiences and assignments between 2015 and 2017, as they are artifacts of the sociohistorical moment. further, reviewing these articles deepened my reflections about the poetry i wrote and performed then, as i was able to revisit the events and reconnect with the emotions that inspired my writing of those pieces. regarding my analysis, i coded my course assignments for evidence of my interrogation and critique of the persistence of racism and color-evasiveness in my doctoral experience and higher education more broadly, along with my understanding of and engagement in the process of counterstorytelling through course activities and projects (creswell, 2009). for instance, i coded the following journal entry from dr. wilson’s class as color-evasiveness, as it demonstrated my unbroken: a critical autoethnography of a black woman’s doctoral experience 70 journal of effective teaching in higher education, vol. 4, no. 2 growing awareness and critique of education scholarship that evaded discussions of race despite it being pertinent to the topic at hand—in this case, qualitative methodology. it is interesting that race is not discussed in this piece. at first, i overlooked this omission…yet, after another look, the omission of race is glaring, and makes me think how accustomed i’ve become to not reading about race in journal articles. perhaps this is why when race is discussed it feels refreshing, and when it is not mentioned explicitly, its absence is not readily detected. this code not only captured my understanding of color-evasiveness as a concept, but also my developing thoughts about the consequences of color-evasiveness in higher education research. as i share the findings of this critical autoethnography, i utilize citation, or the quoting of pertinent texts, as a technique to “enact the intervention of theory in the writing itself” (holman jones, 2016, p. 234). in other words, i cite critical theorists throughout my writing to make meaning of my experiences and highlight the significance of theory as an intervention for me as a student. findings in the sections that follow, i offer three stories of my doctoral program experiences between 2015 and 2017 as framed by the crt tenets of color-evasiveness, persistence of racism, and counterstorytelling. within these stories, i offer valuable strategies from the blm movement and two black women faculty members regarding how black students can contest the deleterious effects of anti-black racism. such strategies include declaring the inherent worth of black life, bearing witness, and engaging in a praxis of counterstorytelling. what we can’t ignore: color-evasiveness as i entered the second year of my higher education doctoral program in the fall of 2015, i felt desperate. there was a longing inside of me to find spaces where blackness was centered rather than an afterthought, if thought of at all. my doctoral experience was unfolding against the backdrop of what would become one of the largest social movements in american history. yet, during my first year (2014–2015), it seemed that the urgent rallying cry of blm did not reach the classroom. despite the tragic deaths of members of the black community and ongoing acts of resistance happening nationally, the lesson plans for my corecurriculum courses remained unchanged, the psychological and emotional toll of these events went unaddressed by most of my department’s faculty members, and i was expected to soldier on as if what was occurring outside of the classroom was not affecting me inside of it. at the time—with the exception of courses purposefully focused on race and racism—it was uncommon to meaningfully discuss antiblackness as a department, in spite of current events that demonstrated the relevance and materiality of it on a national stage. unbroken: a critical autoethnography of a black woman’s doctoral experience 71 journal of effective teaching in higher education, vol. 4, no. 2 i distinctly recall finding it difficult to focus when tamir rice, a 12-year-old black boy was killed by a police officer while playing with a toy gun in november of 2014 (fitzsimmons, 2014). likewise, i was distracted in february of 2015 when the u.s. justice department announced that no federal civil rights charges would be brought against george zimmerman, who was found not guilty in 2013 of the murder of trayvon martin (cnn, 2020). processing the disappointment of such blatant injustice, april 2015 came and marked the death of freddie gray, a 25-year-old black man who sustained spinal cord injuries during an arrest and soon after died while in police custody (stolberg & nixon, 2015). moreover, the summer did not bring the reprieve i prayed for when that july, sandra bland—a 28-year-old black woman and activist—was found hanging in a texas jail cell after being arrested days earlier during a traffic stop (hassan, 2019). times like these, i would avoid the common throughways and gathering spaces of my higher education department—fearful that if someone asked, “how are you?” i would slip up and give them an honest reply. it was easier to keep my head down and move quickly or muster up a smile and keep the conversation light. yet, one day i crossed paths with a white faculty member who upon noticing me, exclaimed, “it’s like i’ve seen a ghost!” what this person could not have known was during that period i felt like an apparition in my doctoral program—lifelessly going through the motions while my true self was invisible to most and silent. the energy to explain why i may have been communally grieving or exhausted from harboring generational knowledge of oppression and pain was too much to give. managing the typical challenges of a doctoral program while shouldering the weight of the sociohistorical moment left me intellectually and emotionally weary. when i returned to school in the fall term of 2015, i was determined to take courses that engaged race and racism and did not reduce blackness to a variable. bell hooks (1994) wrote: i came to theory because i was hurting—the pain within me was so intense that i could not go on living. i came to theory desperate, wanting to comprehend—to grasp what was happening around and within me. most importantly, i wanted to make the hurt go away. i saw in theory then a location for healing. (p. 59) similarly, i was drawn to courses in critical theory from a hurting place with the hope that i too would find healing. from that point on, i filled my schedule with electives outside of my department in psychology, sociology, and educational studies that provided conceptual and theoretical frameworks to help me make sense of the world around me. along my academic journey, i entered the classrooms of black women faculty members who perhaps at some point in their careers were searching for healing also. as i began to learn more about crt, i experienced firsthand how theory can help one feel acknowledged and provide language to describe one’s lived realities. for example, color-evasiveness helped me unpack why the silences concerning race at my predominantly white institution made me feel invisible and as if i had lost my unbroken: a critical autoethnography of a black woman’s doctoral experience 72 journal of effective teaching in higher education, vol. 4, no. 2 voice. sue (2015) defines colorblindness, or color-evasiveness, as a “belief that race should not influence decisions or actions toward individuals or groups, which overlooks the notion that racism continues to be a significant negative factor in the lives of people of color…by intentionally or unintentionally ignoring race and racial differences, race is relegated to being an insignificant factor in the lives of people of color” (p. 77). not discussing the significance of race and racism in coursework mitigated its relevance in my lived experience. while crt helped me name the mechanisms that served to perpetuate colorevasive racism, the blm movement offered a strategy to combat it. according to its creators, the blm movement is “an ideological and political intervention in a world where black lives are systematically and intentionally targeted for demise. it is an affirmation of black folks’ humanity, our contributions to this society, and our resilience in the face of deadly oppression” (blm, n.d., para. 4). the specification that black lives matter is a contestation of color-evasive rhetoric that ignores the material consequences of anti-black racism in a society plagued by white supremacy. crt and blm empowered me critique color-evasive ideologies that ignored the salience of race and racism and assert my self-worth as a black person. in the winter of 2016, i wrote “unbroken” as a personal declaration that i would not allow racism to render me silent or invisible any longer. unbroken i am not broken i am not silent i am not invisible eyes low mouth shut neck bent that’s how you want me to be how you need me to be how you hoped to find me however, your refusal to see me does not negate my existence the space i occupy is not vacant i am not here to smile, serve, or submit my back is not your footstool so please don’t confuse my humility with timidity thinking you can humiliate or intimidate me i refuse to be disrespected, dismissed, or disregarded you don’t own me can’t define me and certainly won’t break me unbroken: a critical autoethnography of a black woman’s doctoral experience 73 journal of effective teaching in higher education, vol. 4, no. 2 i will not yield i won’t relent i will not wait for permission i will press i will push i will protest bearing witness: persistence of racism i never had the stomach for the spectacle of murder, let alone that of my people. the viral videos of alton sterling and philando castile being murdered by police officers circulating in july of 2016 were haunting for me. sterling pinned down and shot at close-range in baton rouge, louisiana, and castile shot and killed in in front of his girlfriend and her daughter while reaching for identification in falcon heights, minnesota (poniewozik, 2016). those videos reminded me of being a child watching documentaries of slavery at school. without warning, the next visual could be a black body swinging from a tree with lynch mob members and spectators looking on. i do not recall ever having a teacher say, “this might be triggering for you.” or, “this may be difficult for you to watch, if you need to avert your eyes, or leave the room, do whatever you need to take care of yourself.” why did my teachers not think to warn me? what made black death more acceptable to watch in a classroom? was the fact that it happened in the past supposed to make it less painful? why, as a child, was i expected to grapple with the complex emotions such images stirred within me alone? as these videos, and others like them, continued to spread like wildfire, i felt as overwhelmed as i did as a young girl—confused, angry, and heartbroken— struggling to process these feelings. when i returned to campus for fall courses in 2016, the horror and grief i experienced reading and watching the news that summer was still fresh. yet, i found comfort in dr. camille wilson’s critical race methodologies for qualitative research class housed in educational studies. grounded primarily in crt, the course introduced principles, philosophy, and strategies for engaging in critical methodologies, which consider “the interrelated nature of race, knowledge, power, inclusiveness, representation, and the educative value of research” (wilson, 2016, p. 1). the first session, we dove right in, collaborating as a class to develop working definitions of race and racism which we would revisit throughout the semester. as the class progressed, we also explored the tenets of crt as well as other critical theories and began to utilize such theory as lenses through which to examine our research questions, methodological approaches, and philosophical standpoints regarding qualitative research. the week we discussed the persistence of racism tenet in crt, i found an answer to the questions that troubled me as a child and an adult woman regarding the palatability of black suffering and death in america. the persistence—or permanence—of racism tenet asserts that racism in american society “is so enmeshed in the fabric of our social order, it appears both normal and natural to people in this culture” (ladson-billings, 1998, p. 11). in other words, the unbroken: a critical autoethnography of a black woman’s doctoral experience 74 journal of effective teaching in higher education, vol. 4, no. 2 commonplace nature of racism in america renders horrific and inhumane assaults to black life, such as killing a black man in front of his loved ones, “normal and natural.” yet, in naming the persistence of racism, “the strategy becomes one of unmasking and exposing racism in its various permutations” (ladson-billings, 1998, p. 11). to unmask and expose, one must bear witness. writing about his “field trip” accompanying medgar evers as he investigated the murder of a black man in mississippi with the national association for the advancement of colored people (naacp), james baldwin (2017) penned, i was to discover the line which separates a witness from an actor is a very thin line indeed; nevertheless, the line is real…this was sometimes hard on my morale, but i had to accept as time wore on, that part of my responsibility—as a witness—was to move as largely and freely as possible, to write the story and to get it out. (pp. 30–31) baldwin was writing of the weight of witnessing. though it weighed on him at times to not be directly involved with organizing in mississippi—“i did not have to sweat cold sweat after decisions involving hundreds of thousands of lives” (baldwin, 2017, p. 30)—he understood that he had a responsibility to witness and report the events there so that the stories were not lost, nor the struggle in vain. unmasking and exposing racism necessitates witnessing and naming it where it occurs. the blm movement gained momentum from the courage of individuals who decisively chose to bear witness. using cell phone footage and social media, blm mobilized activists and grassroots organizers as well as contested state violence against black bodies by gathering and disseminating evidence across digital platforms (hill, 2018). witnessing was engaging in resistance. witnessing was powerful. learning more about and discussing crt in dr. wilson’s class gave me new perspective regarding the necessity and responsibility of witnessing as a scholar. as an ongoing class assignment, dr. wilson tasked us with keeping a journal of key concepts and arguments in our course reading, along with connections we were making to our research interests and positionalities (wilson, 2016, p. 3). relating the persistence of racism tenet to the educational system, on september 20, 2016, i wrote: race and racism permeate throughout the educational system (how it’s structured, who benefits, what schools get the resources, who is in charge, what is taught, what forms of knowledge are valued). the fact that people associate racism with individual acts and irrationality strips people of color from the language to describe everyday acts of discrimination is incredibly frustrating, but it belies the importance of being bold enough to name sites of oppression (even if it isn’t polite). thinking back to my childhood upbringing in a black baptist church, witnessing described the sharing of personal stories about the goodness of god. witnessing unbroken: a critical autoethnography of a black woman’s doctoral experience 75 journal of effective teaching in higher education, vol. 4, no. 2 was testimony. being “bold enough to name sites of oppression” was a means of testifying and resisting the silencing of racism, whether it be in the educational system, criminal justice system, or elsewhere. dr. wilson’s course helped me realize that witnessing is an integral strategy in contesting the persistence of racism. further, as a scholar, i have a responsibility to witness. my understanding that bearing witness is resistance, power, and testimony, led me to use poetry as a means of witnessing. in the winter of 2017, i wrote and performed the poem below at a school event to express my feelings regarding the black lives stolen by police brutality, but also to hold others accountable for sharing the responsibility of bearing witness. i woke up and i felt it this morning, i woke up and i felt it that familiar ache spreading and shifting scorching and searing erupting in flame an all-consuming fire igniting my bones threating to burn me alive from within i can smell it, taste it even the stench of my own flesh melting away clings to the back of my throat i want to vomit, i want to scream but when i open my mouth, stretch my jaws, shut my eyes, nothing comes out i can’t breathe, i can’t breathe, i. can’t. breathe it feels like i’m dying the crushing weight of the inevitable presses down on my chest silently suffocating me as reality breaks like the dawn through my window i’m alive, but i’ve been holding my breath for what feels like forever waiting for another message, another call, another report to tell me that another one is lost another brother, sister, daughter, son, father, mother, lover…gone left broken in the street bathing the concrete in their life blood dead over skittles, cds, or maybe cigarillos draped in sheets, growing cold until someone can speak their name evoke the spirit that inhabited that body and say this one was mine you see with every death, a little piece of me dies too cause it could’ve be one of mine it could’ve been me so, i’m tired tired of stitching the tattered pieces of a heart torn open tired of that empty feeling slowly swallowing me whole tired of fearing faceless phantoms when i find myself alone but i have to get up have to face the day, have to face the questions i know will come unbroken: a critical autoethnography of a black woman’s doctoral experience 76 journal of effective teaching in higher education, vol. 4, no. 2 “did you hear about it?” of course “what do you think?” what i always do but, today let’s turn the tables i have some questions for you… when did society become so consumed with propriety that i’m expected to mourn privately? why can’t we be like children who cry without abandon, who wail and yell and kick and thrash until we are spent, and then someone attends to our pain, whispers soothing words of comfort and holds us tenderly, lovingly until we feel safe again? why do i have to tell you that my life matters? or explain that this is real? i wish it were a dream too, but what will you do when you wake up? you asked me how does it feel. i’m trying to tell you this morning, i woke up and i felt it no longer a child, through poetry, i found the words to express that was happening to black people in america was neither normal nor natural. anti-black racism must be unmasked and exposed, “even if it isn’t polite.” i will continue to bear the weight of witnessing as difficult as it continues to be. owning our stories: counterstorytelling dr. jacqueline mattis’ classroom was a sanctuary for me. in winter of 2017, a trusted mentor recommended that i enroll in dr. mattis’ class, and when i asked what exactly i should expect regarding the course content, she replied, “whatever the subject, if jacquie is teaching, you take the class.” after one session with dr. mattis, i understood what my mentor meant. when i entered that space, i immediately felt cared for, safe, and profoundly understood. i also felt excited because our lessons evolved and responded to our pressing questions, needs, and desires as emerging scholars. as hooks (1994) writes, “to enter classroom settings in colleges and universities with the will to share and the desire to encourage excitement, was to transgress” (p. 7). the course was urbanicity and positive prosocial development. building on the foundations of positive psychology, the course utilized a critical, interdisciplinary lens to examine human strengths and virtues in urban contexts (mattis, 2017, p. 1). further, we explored strategies for developing scholarship grounded in the lived realities of people residing in urban communities (mattis, 2017, p. 1). what transpired in that classroom those wednesday afternoons was a healing ministry. dr. mattis led us to interrogate raced and classed dominant narratives of urban spaces and people, and engaged us in the humanizing work of defining the urban for ourselves based on our lived experiences, empirical evidence, and theory. unbroken: a critical autoethnography of a black woman’s doctoral experience 77 journal of effective teaching in higher education, vol. 4, no. 2 though crt was not the focus of the course, upon reflection, i believe that dr. mattis’ pedagogy revealed a praxis of counterstorytelling. critical race counterstorytelling is a “method of recounting the experiences and perspectives of racially and socially marginalized people” (yosso, 2006, p. 10). further, these stories raise critical consciousness about social and racial injustice, as well as contest dominant narratives that “omit and distort the histories and realities of oppressed communities” (yosso, 2006, p. 10). counterstories can also build community among the marginalized and let them know that they are not alone (solórzano & yosso, 2002). these stories can also create new possibilities that are richer than the stories themselves or the current reality (solórzano & yosso, 2002). in dr. mattis’ class, where five of the six students were black women with research interests or upbringings in urban contexts, we engaged in the process of counterstorytelling, centering our lived experiences and those of the communities we aspired to enter as researchers. with dr. mattis as our guide, we learned how to share our stories and allow them to inform our approaches to research and life. first, we uplifted the good. we surfaced narratives of love, optimism, creativity, faith, wisdom, and altruism in urban contexts. for those of us who grew up in such environments, we told stories of how our cities and neighborhoods instilled values and taught lessons that shaped us into who we are. second, we interrogated silences in the media and research regarding urban spaces, focusing our attention on the stories that were not being told. though keenly aware of what has been said about urban contexts and people, particularly racialized, deficit-based narratives which emphasize criminality, poverty, and violence, we asked what questions were not being asked in research and how could we study the goodness we understood to exist in such spaces. third, with our new questions in mind, we searched for a heartbeat. dr. mattis would tell us that the answers to our research questions, and possibly new questions, rested with people. people are the beating heart of research and hearing others’ stories gave our questions life. we talked with people we knew about our work and research, but we primarily listened and allowed those conversations to help guide us moving forward. fourth, we imagined new possibilities for methods and theory, giving ourselves the freedom to dream of what we could create if there were no limitations. what would we need to conduct research rooted in goodness that honored the dignity of urban communities? finally, we shared our imaginings with each other. as we shared our visions and plans, we moved one step closer to actualizing them. my peers and i became strategists, collaborators, and accountability partners for one another. in our classroom, we were building a community and discovering how we could contest dominant narratives and uplift the stories that mattered to us in our scholarship. furthermore, we were given the freedom to share our ideas creatively, which gave us an opportunity to express ourselves through art. for my final project, i delivered a spoken word poetry performance dedicated to the black residents of my hometown, detroit, michigan. as i wrote my counterstory of the city and its people, i also was writing a piece of my history and imagining my future. see i love her, in the most absolute, all-encompassing, complicated sense of the word unbroken: a critical autoethnography of a black woman’s doctoral experience 78 journal of effective teaching in higher education, vol. 4, no. 2 i know her heart fiery and strong she bows to no one and bears her scars without shame at a moment when i felt afraid, angry, disheartened, and frustrated that black life was being targeted, and black people still needed to fight to assert our humanity, dr. mattis’ class gave me hope and invited me to dream. the lessons we learned with respect to counterstorytelling in research were applicable to my everyday existence. uplift the good. there is good in me and my people. interrogate silences. filter out the noise, think of what’s missing, then determine how i can use my voice to tell another story. search for the heartbeat. in times of uncertainty, look and listen for signs of life, the answer may be there. imagine. dream without limitation. share those imaginings. find others to build and create with, this work should not be done in isolation. when our class sessions ended those wednesday afternoons, i felt full. the stories we told and aspired to tell nourished me and gave me strength. i mattered. my work mattered. i was not alone. such affirmations healed me from within and reminded me how sacred teaching truly is. as hooks (1994) suggests, teaching in a manner that “respects and cares for the souls of our students is essential if we are to provide the necessary conditions where learning can most deeply and intimately begin” (p. 13). i also experienced the power of art as a means of self-expression in the classroom. having the freedom to share my ideas through poetry in a classroom setting unlocked something inside of me. i felt as if i could finally speak in my authentic voice and be received as i intended. discussion and implications when i sought i out courses in critical theory and methods, i was hurting. the antiblack racism in this nation as evidenced by the recurring assaults on black life pained me, and i understood as baldwin (2017) did that for black people, “the line which separates a witness from an actor is a very thin line indeed” (p. 30). though i was pursuing my doctorate and attaining the education my ancestors fought for, my humanity as a black person in this country was up for debate, and there was still a need to assert that my black life mattered. yet, crt offered me language to help articulate my lived reality and examine the experiences of other marginalized people. while crt offered a valuable frame of analysis, the blm movement became a case study in strategic resistance. boldly declaring the inherent worth of black life and responsibly bearing witness to injustice as a means of combatting it were clear strategies of the movement that i could apply to my academic and personal life. introducing black graduate students to critical theory and methodologies can equip them with the tools to empirically explore, make meaning of, and articulate their lived realities. such explorations can be validating and healing as students navigate challenging academic and sociohistorical contexts. critical theorization can also help black students imagine new possibilities of self, resist oppression, and dream of freedom. additionally, drs. camille wilson and jacqueline mattis, demonstrated through their teaching effective strategies to help students learn and adopt critical theory in their unbroken: a critical autoethnography of a black woman’s doctoral experience 79 journal of effective teaching in higher education, vol. 4, no. 2 practice. dr. wilson encouraged us to thoughtfully consider how critical theory could inform our epistemologies (i.e., ways of knowing), ontologies (i.e., ways of being and seeing), and methodological approaches to research through assignments such as reflective journals. in these weekly journals throughout the semester, we processed how we could apply critical frameworks to our research questions and future studies while also considering how our positionality might influence the frameworks we chose as well as our understanding and use of them. by the end of the course, we each had at least 11 weeks of journal entries that documented our reactions, reflections, and evolving ideas regarding the course content. dr. mattis offered what i refer to as a praxis of counterstorytelling that allowed our stories as researchers to guide our approach to scholarship. during the very first class meeting, dr. mattis centered our knowledge and standpoints by asking us about our communities and neighborhoods, encouraging us to uplift the good that often gets overlooked in the media and scholarship. this laid the groundwork for counterstorytelling throughout the course, as we imagined research studies that would similarly center the lived realities of our populations of interest, which were primarily communities of color—and in my case—black communities specifically. she also cultivated a community within the classroom that allowed us as students to creatively share and constructively hold one another accountable for conducting the type of research we envisioned together. giving graduate students the opportunity to reflect, map out, and share how critical theory may inform their ways of understanding and conducting research through coursework allows them to develop their ideas in a supportive environment while cultivating networks of accountability and collaboration. further, providing students with creative outlets to express themselves in coursework can help them process their experiences and produce material that is humanizing, liberating, and life-giving. exposing students to critical theory and methodologies in graduate education may also encourage them to identify, name, and contest anti-blackness in their professional careers. for instance, once aware of how color-evasiveness and racism persist in academic institutions, future educators may be more cognizant of the absenting of black scholarship from the curriculum and intentionally integrate such work in their courses. additionally, guided by counterstorytelling, researchers may begin to imagine new possibilities for studies that center black joy and love, or perhaps methodologies that honor the dignity and beauty of black life. students trained in critical theory and methods have the potential to become transformative leaders who combat anti-blackness in the spaces they occupy. finally, this critical autoethnography adds to the extant literature on black doctoral women’s experiences by discussing how transformational mentorship can occur in the classroom setting. black doctoral students who aspire toward careers in academia may benefit from witnessing faculty members employing critical theory in their pedagogy. as i was studying critical theory in dr. wilson’s and dr. mattis’ courses, i was also a student of their teaching—closely observing the ways that they uplifted counterstories in class discussions and readings or challenged us to consider how racism persists in social science research and ways to combat it. unbroken: a critical autoethnography of a black woman’s doctoral experience 80 journal of effective teaching in higher education, vol. 4, no. 2 furthermore, dr. wilson and dr. mattis were instrumental in helping me navigate the sociohistorical context and my academic environment as a black woman. the safety i felt in discarding my academic mask (shavers & moore, 2014) and being vulnerable in their classes was due in large part to our shared identities (rasheem et al., 2018) and their attention to race, racism, and social justice in their courses. conclusion much has happened since i took the stage in 2016 and declared that i was not broken, silent, or invisible. the parallels between the present moment and then are telling. during the summer of 2020, another litany of atrocities were committed against black life. yet, as evidenced by the ongoing protests and demonstrations of solidarity worldwide, the struggle to resist anti-black racism in america persists. as for me, i have once again reached for my pen to grapple with the weight of the moment. guided by the wisdom of critical theorists, the blm movement, and beloved sister scholar teachers, i will continue to contest anti-blackness in higher education by writing of my existence and the lives of others who have been rendered broken, silent, or invisible. i also think toward the future and imagine how i will help students feel whole, heard, and seen in my classroom. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references adams, t. e., holman jones, s., & ellis, c. (2015). autoethnography: understanding qualitative research. oxford university press. annamma, s. a., jackson, d. d., & morrison, d. (2017). conceptualizing colorevasiveness: using dis/ability critical race theory to expand a color-blind racial ideology in education and society, race ethnicity and education, 20(2), 147–162. https://doi.org/10.1080/13613324.2016.1248837 baldwin, j. (2017). i am not your negro. vintage books. bertrand jones, t., wilder, j., & osborne-lampkin, l. (2013). employing a black feminist approach to doctoral advising: preparing black women for the professoriate. the journal of negro education, 82(3), 326–338. https://www.jstor.org/stable/10.7709/jnegroeducation.82.3.0326 bhattacharya, k. (2017). fundamentals of qualitative research: a practical guide. routledge. black lives matter (blm). (n.d.). herstory. https://blacklivesmatter.com/herstory/ unbroken: a critical autoethnography of a black woman’s doctoral experience 81 journal of effective teaching in higher education, vol. 4, no. 2 boylorn, r. m., & orbe, m. p. (eds.). (2014). critical autoethnography: intersecting cultural identities in everyday life. left coast press. cnn. (2020, october 19). trayvon martin shooting fast facts. cnn. https://www.cnn.com/2013/06/05/us/trayvon-martin-shooting-fastfacts/index.html creswell, j. w. (2009). research design: qualitative, quantitative, and mixed methods approaches (3rd ed.). sage. delgado, r., & stefancic, j. (2012). critical race theory: an introduction (2nd ed.). new york university press. fitzsimmons, e. g. (2014, november 23). 12-year old boy dies after police in cleveland shoot him. new york times. https://www.nytimes.com/2014/11/24/us/boy-12-dies-after-being-shot-bycleveland-police-officer.html gooden, m. a., devereaux, c. a., & hulse, n. e. (2020). #blackintheivory: culturally responsive mentoring with black women doctoral students and a black male mentor. mentoring & tutoring: partnership in learning, 28(4), 392–415. https://doi.org/10.1080/13611267.2020.1793083 hassan, a. (2019, may 7). the sandra bland video: what we know. new york times. https://www.nytimes.com/2019/05/07/us/sandra-bland-brianencinia.html hill, m. l. (2018). “thank you, black twitter”: state violence, digital counterpublics, and pedagogies of resistance. urban education, 53(2), 286–302. holman jones, s. (2016). living bodies of thought: the "critical" in critical autoethnography. qualitative inquiry, 22(4), 228–237. https://doi.org/10.1177/1077800415622509 hooks, b. (1994). teaching to transgress: education as the practice of freedom. routledge. ladson-billings, g. (1998). just what is critical race theory and what’s it doing in a nice field like education? international journal of qualitative studies in education, 11(1), 7–24. https://doi.org/10.1080/095183998236863 ladson-billings, g. (2013). critical race theory: what it is not. in m. lynn and d.d. dixon (eds.), handbook of critical race theory in education (pp. 34–47). routledge. madison, d. s. (2005). introduction to critical ethnography: theory and method. in critical ethnography: method, ethics, and performance (pp. 1–16). sage publications. unbroken: a critical autoethnography of a black woman’s doctoral experience 82 journal of effective teaching in higher education, vol. 4, no. 2 mattis, j. s. (2017). urbanicity and positive prosocial development [syllabus]. department of psychology, university of michigan. patterson-stephens, s. m., lane, t. b., & vital, l. m. (2017). black doctoral women: exploring barriers and facilitators of success in graduate education. higher education politics & economics, 3(1), 157–180. poniewozik, j. (2016, july 6). a killing. a pointed gun. and two black lives, witnessing. new york times. http://www.nytimes.com/2016/07/08/us/philando-castile-facebook-policeshooting-minnesota.html?partner=bloomberg rasheem, s., alleman, a., mushonga, d., anderson, d., & ofahengaue vakalahi, h. f. (2018). mentor-shape: exploring the mentoring relationships of black women in doctoral programs. mentoring & tutoring: partnership in learning, 26(1), 50–69. https://doi.org/10.1080/13611267.2018.1445443 schwartz, m. s. (2020, september 5). trump tells agencies to end trainings on “white privilege” and “critical race theory.” npr. https://www.npr.org/2020/09/05/910053496/trump-tells-agencies-to-endtrainings-on-white-privilege-and-critical-race-theory shavers, m. c., & moore iii, j. l. (2019). the perpetual outsider: voices of black women pursuing doctoral degrees at predominately white institutions. journal of multicultural counseling and development, 47, 210–226. https://doi.org/10.1002/jmcd.12154 shavers, m. c., & moore iii, j. l. (2014). black female voices: self-presentation strategies in doctoral programs at predominantly white institutions. journal of college student development, 55(4), 391–407. https://doi.org/10.1353/csd.2014.0040 solórzano, d. g., & yosso, t. j. (2002). critical race methodology: counterstorytelling as an analytical framework for education research. qualitative inquiry, 8(1), 23–44. https://doi.org/10.1177/107780040200800103 stolberg, s. g., & nixon, r. (2015, april 21). freddie gray in baltimore: another city, another death in the public eye. new york times. http://www.nytimes.com/2015/04/22/us/another-mans-death-anotherround-of-questions-for-the-police-in-baltimore.html?partner=bloomberg sue, d. w. (2015). race talk and the conspiracy of silence: understanding and facilitating difficult dialogues on race. wiley. wilson, c. m. (2016). critical race methodologies for qualitative research [syllabus]. department of educational studies, university of michigan. unbroken: a critical autoethnography of a black woman’s doctoral experience 83 journal of effective teaching in higher education, vol. 4, no. 2 yosso, t. j. (2006). critical race counterstories along the chicana/chicano educational pipeline. routledge. zamudio, m. m., russell, c., rios, f. a., & bridgeman, j. l. (2011). critical race theory matters: education and ideology. routledge. microsoft word breeden_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi 10.36021/jethe.v4i2.259 we want to do more than survive: abolitionist teaching and the pursuit of educational freedom by bettina l. love, 2019. boston, ma: beacon press, 200 pages, isbn 9780807069158, softcover, $16.00. reviewed by roshaunda l. breeden, university of georgia, dr.rbreeden@gmail.com notable author, researcher, and educator, bettina l. love offers an unapologetic critique of the united states’ educational system in the text we want to do more than survive: abolitionist teaching and the pursuit of educational freedom. in her work, love’s text situates the idea of black mattering as a valuable conceptual framework and offers a clear vision for the future of educational justice—one rooted in the spirit of abolitionist teaching. specifically, what distinguishes love’s book from others is how she eloquently combines her lived experiences as a teacher and professor with empirical research and historical events while also paying homage to black abolitionists who paved the way for all of us. while love highlights many examples from the k-12 context, her text is a call to action for all educators— teachers, administrators, researchers, and faculty in higher education and beyond— who claim to love black children. we want to do more than survive features seven short, accessible, and thoughtprovoking chapters. in chapter 1, “we who are dark,” love sets the tone for her book centering black mattering as an integral concept. straight away, love mentions that “dark people have never truly mattered in this country except for property and labor” (p. 7). using quotes from w.e.b dubois, angela davis, toni morrison, and many others, love calls on us to (re)member the past and evoke their same urgency and passion related to black life and black mattering. while mentioning that black folx “matter to our communities, families, and ourselves” (p. 7), love asserts that america was built on racist ideals which uplift whiteness as superior and darkness as inferior. in framing the book from this lens, love helps situate how black humanity has always been questioned—even in the secondary school context. throughout each chapter, love reminds the reader of what black mattering could and should look like in education. she declares that “we who are dark want to matter and live, not just to survive but to thrive” (p. 10). she ends the chapter problematizing the current u.s. educational system and pleads for educators to move past reform to fight for systemic change in classrooms by teaching students about racial violence, oppression, and centering black joy as transformative liberation. in chapter 2, “educational survival,” love opens with a personal narrative as a new teacher to black and brown children in south florida, naming “my students and i were merely trying to survive” (p. 19). in her recollection, love highlights how her students had to navigate unfavorable environments at school (e.g., lack of resources, teacher turn-over, and high-stakes standardized testing). she names this phenomenon the educational survival complex, “in which students are left learning to merely survive, learning how schools mimic the world they live in, thus making schools a training site for a life of exhaustion” (p. 27). pairing her review of we want to do more than survive 136 journal of effective teaching in higher education, vol. 4, no. 2 experiences with empirical evidence, love provides multiple examples of how black and brown children—those who are dark—fight to survive and navigate injustice in the u.s. educational context, specifically due to white rage (e.g., indian removal act, school segregation, and no child left behind legislation). throughout this chapter, love suggests that the current u.s. educational system is built on antiblack policies and practices rooted in racism, transphobia, classism, rigid ideas of gender, and islamophobia. she ends the chapter calling for teachers to understand that the u.s. strategically built education on notions of white supremacy, antiblackness, and sexism which intentionally teaches black children to survive instead of thrive. under this current educational survival complex, children are too consumed with simply existing and staying alive to learn. in chapter 3, love revisits the concept of mattering, centering her personal experience as a black child growing up in rochester, new york. using bell hooks’ language of “homeplace,” love recalls finding spaces where black folx truly matter[ed], where souls are nurtured, comforted, and fed (p. 63). she names feeling seen, heard, and understood by her mother (patty), her first black woman teacher (mrs. johnson), and a youth empowerment program called fist—sharing these people and places were sites of resistance that taught her to value her blackness. from these interactions, love learned that teaching dark children is not an intellectual exercise or one to be taken lightly. instead, teaching black children is a deeply personal, emotional, spiritual, and communal process that requires teachers to holistically invest their time and talent. likening the work of teachers to abolitionist leaders of our day, love encouraged educators to move like civil rights activist ella baker and other organizers who used grassroots efforts, intergenerational strategies, and love for young people to move us closer toward freedom. throughout chapter 4, “grit, zest, and racism (the hunger games),” love called out character education programs (e.g., the character lab) and the educators who take part in them. she asserts that these trendy programs (e.g., grit labs) sound good in theory but are entrenched in ideas that dark and poor children lack morality and need to “work harder”—which is deep-rooted in anti-blackness. love goes on to name these programs and curriculums as distractions to civics education, which provides necessary historical context and helps equip children with the knowledge they need to fight injustices. she shares that educators must recognize that injustice requires students to become agitators who use their knowledge to fight in ways that are not always easy or comfortable. speaking truth to power, love ends by encouraging teachers to research and learn about justice, equity, and systems that impact their students so that their work is informed by truth and not trends. in “abolitionist teaching, freedom dreaming, and black joy,” the book’s fifth chapter, love, goes on to describe abolitionist teaching “as a way of life” (p. 89). again, mentioning the work of activist ella baker, love states that abolitionist teaching can take many forms. she shares that educators can create a “safe space or homeplace in their classrooms, fight standardized testing, restore justice in their curriculum, or seek justice in their own communities” (p. 89). regardless of the act, love names to be an abolitionist teacher means restoring humanity in educational review of we want to do more than survive 137 journal of effective teaching in higher education, vol. 4, no. 2 spaces. she also suggests that abolitionist teachers must be willing to put their reputation, home, and livelihoods on the line for other people's children. love ends chapter 5, acknowledging that being an abolitionist takes real work, and she reminds us that “any pedagogy that does not challenge injustice is useless because survival is not the goal” (p. 123). the goal is to matter. encouraging educators to move beyond gimmicks and best practices in education, chapter 6 asks teachers, professors, teacher preparation program faculty, and researchers to find their north star—a helpful guide that lights the way to freedom. mentioning bell hooks’ (1994) work, love asks readers to allow theories to become their north star, a location for healing that can help frame the work of educating dark children. giving readers helpful context, love asks educators to embrace theories such as critical race theory (e.g., ledesma & calderón, 2015), community cultural wealth (e.g., yosso, 2005), decolonization (e.g., patel, 2017), black feminism (e.g., hill-collins, 2002), quare studies (e.g., patrick johnson, 2001), critical white studies (e.g., matias, 2016). furthermore, love calls for the understanding of neoliberalism (e.g., martinez & garcia, 1997), economics, and the history of social movements, which helps provide full context around the experiences of dark people outside the limited scope of white supremacy. love ends the chapter by urging teachers to move away from quick fixes and tricks to turn toward theories to guide our work. chapter 7 concludes the text with, “we gon’ be alright, but that ain’t alright.” in this chapter, as with others, love provides countless examples of how whiteness and white supremacy have tried to destroy black minds and bodies. she highlights her traumatic experience with panic attacks and mentions how white doctors believe black bodies can tolerate more pain. love then declares the need for wellness, healing, and restoration when working towards freedom, particularly for black folx. in this chapter, love names that she had to “center [her]self and practice being well daily (p.158)” sharing that “in no way does being well somehow stop injustice, but it does allow you to be your best self while fighting injustice” (p. 158). aside from naming her experiences, love declares that all educators must be well. she detailed that schools can only be “sites of healing” when folx can embrace theory, interrogate systems, engage in free therapy, and work in loving and compassionate workspaces (p. 160). love ends the chapter stating that schools cannot be alright with the status quo but must make concerted efforts to ensure wellness and healing are ultimate priorities. throughout each chapter of this book, love gives suggestions for educators—k-12 and higher education faculty—to center black lives and black mattering in our work. while love contributes many practical implications for readers, i’ve provided three examples, specifically for higher education faculty. first, faculty can and should create “homeplaces” for students in their classes and offices—creating spaces where students feel seen, heard, and affirmed as scholars. while there are multiple ways to create safe spaces for students, one suggestion would be to identify a wide variety of course materials to engage students, including journal articles, blog posts, podcasts, panel discussions, webinars, and review of we want to do more than survive 138 journal of effective teaching in higher education, vol. 4, no. 2 videos that reflect multiple forms of scholarship from authors from historically and racially marginalized backgrounds. inviting minoritized folx across various identities in course materials might encourage students from all backgrounds to feel safe and visible in class discussions and assignments. additionally, providing homeplaces means creating safe spaces for black students and faculty to write, mentor, uplift, and teach one another. altogether creating a homeplace on campus means embodying a caring posture that focuses on faculty and students’ physical, mental, and emotional health—ensuring that everyone has access to health care, mental health resources, affordable housing, and food. second, faculty should consider finding their north star—using theories to guide their values and how they approach their work. for higher education teaching, using theories means thinking critically about what brings you to your work. whose scholarly traditions are you taking up? who are you reading and citing in your work? what philosophies inform your assignments, practices, and values as an educator? engaging in reflection around these questions and committing to learning new theories can help faculty find their north star. faculty interested in learning more about black mattering should consider engaging in scholarship related to black feminist, black radical, or other black knowledge traditions. third, faculty in higher education should commit to advocacy both on campus and in their personal lives. centering black mattering on-campus could include calling out problematic colleagues or troubling standardized testing in admissions. offcampus efforts include challenging oppressive systems in one’s community (e.g., interrupting homophobic family members and voting for policies that provide equitable outcomes for black families and communities). love mentions engaging in abolitionist praxis rooted in black mattering may seem difficult at first; she reminds us that the work is too important not to try. overall, love’s book is a well-argued, well-cited, well-written war cry for educators and academics everywhere. throughout the text, she seamlessly critiques the u.s. educational system and the educational survival complex, centers blackness and black mattering as a practical analysis framework, and provides strategies based on theory to guide our work for the future. using her personal narratives, combined with historical references and culturally relevant examples, love offers a space of reflection and delivers a helpful playbook for all those who claim to be educators, including those in higher education. furthermore, love’s work centers on intersectional social justice and highlights radical movements of the past as helpful tactics for our future. applying concepts from we want to do more than survive to teaching and pedagogy praxis, particularly in higher education, means focusing specifically on abolitionist teaching, black mattering, and wellness in our work as professors, preparation program faculty, and researchers. invoking an abolitionist praxis in higher education teaching requires advocacy and action in our campuses, classrooms, and beyond. from a critical race lens, abolitionist teaching includes applying methods like protesting, boycotting, and calling out other administrators, faculty, and colleagues who are racist, homophobic, transphobic, or islamophobic. review of we want to do more than survive 139 journal of effective teaching in higher education, vol. 4, no. 2 having an abolitionist ethic also means being knowledgeable, talking about, and addressing homophobia in our classrooms, research labs, offices, and communities in materials ways. regardless of one’s position or title, embodying an abolitionist ethic from the black radical traditions can provide a sense of urgency, show students they matter, and model what fighting against oppression can look like. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references hill-collins, p. (2002). black feminist thought: knowledge, consciousness, and the politics of empowerment. routledge. hooks, b. (1994). teaching to transgress: education as the practice of freedom. routledge. ledesma, m. c., & calderón, d. (2015). critical race theory in education: a review of past literature and a look to the future. qualitative inquiry, 21(3), 206– 222. https://doi.org/10.1177/1077800414557825 love, b. l. (2019). we want to do more than survive: abolitionist teaching and the pursuit of educational freedom. beacon press. martinez, e. & garcia, a. (1997, january). what is neoliberalism? corpwatch. https://www.corpwatch.org/article/what-neoliberalism matias, c. e. (2016). feeling white: whiteness, emotionality, and education. sensepublishers. patel, l. (2017, december). trump and settler colonialism. ctheory. https://journals.uvic.ca/index.php/ctheory/article/view/16713 patrick johnson, e. (2001). “quare” studies, or (almost) everything i know about queer studies, i learned from my grandmother. text and performance quarterly, 21(1), 1–25. yosso, t. j. (2005). whose culture has capital? a critical race theory discussion of community cultural wealth. race ethnicity and education, 8(1), 69–91. microsoft word kennette final version.docx journal of effective teaching in higher education, vol. 1, no. 2 universal design for learning (udl): student and faculty perceptions lynne n. kennette1, ph.d. nathan andrew wilson, abd durham college, oshawa, ontario canada abstract. universal design for learning (udl) ensures that content is accessible to the largest audience by removing learning impediments (cast, 2011a). however, few scholars have surveyed students about how much udl they encounter in their courses or how important they perceive these course modifications to be, especially in a post-secondary context. to this end, students at a canadian college were surveyed. in a follow-up survey, faculty were also asked to report on how they thought their students perceived and valued their udl usage. udl perceived usage and perceived usefulness data were compared across both students and faculty and there was much agreement across the samples. disagreements are discussed. keywords: universal design for learning, student perception, faculty perceptions, usage, usefulness universal design refers to the idea of crafting physical spaces that maximize usability for as many people as possible. it is conceptually rooted in 1980s architectural circles, specifically those established by north carolina state university’s ron mace. from architecture, educators adapted and applied the concept to the learning environment. reframed as universal design for learning (udl), teachers, instructional designers, and curriculum specialists (notably anne meyer, david h. rose, and their colleagues at the harvard graduate school of education and the center for applied special technology, cast) considered how best to make all course components barrier-free. to be most successful, advocators argued that udl guidelines should be used to frame a course from its inception, rather than retrofitting a course after the fact. properly applied to all facets of learning, udl would conceivably benefit all learners, and not solely those individuals with learning challenges or students with disabilities (cast, 2011a; courey, tappe, siker, & lepage, 2012; meyer, rose, & gordon, 2014). reducing the barriers and impediments facing all students also meant that individuals would no longer need to self-identify as requiring accommodations. while any educator should ensure that subject content is flexible and accessible to the greatest number of students, this is especially important in higher education for two reasons. first, the number of students with mental health or other learning barriers is on the rise (fichten, jorgensen, havel, & barile, 2006; ontario university & college health association, 2016; raue & lewis, 2011), which alone warrants revisiting udl principles in curriculum development. some have reported that as many as 60-80% of students with disabilities in higher education make the decision 1 corresponding author: lynne.kennette@durhamcollege.ca universal design for learning (udl): student and faculty perceptions 2 journal of effective teaching in higher education, vol. 1, no. 2 not to disclose (and therefore not obtain the required services) for fear of stigmatization by their peers, instructors, or community (black, weinberg, & brodwin, 2014; claiborne, cornforth, gibson & smith, 2010; marshak, van wieren, ferrell, swiss, and dugan, 2011; schelly, davies, & spooner, 2011; wagner, newman, cameto, garza & levine, 2005). following the principles of udl allows for the reduction of learning barriers between students with disabilities and those without, without the need for students to disclose their disability to others (alazawei, serenelli, & lundqvist, 2016). a recent meta-analysis showed almost exclusively positive student outcomes, including increasing student satisfaction and engagement (al-azawei et al., 2016). second, researchers also suggest that culturally diverse learners benefit from the implementation of udl principles (chitategmark, gravel, serpa, domings, & rose, 2012); this is important when considering the increasing number of international students on our campuses (statistics canada, 2016). as scholars rightly note: “the traditional teaching approach of ‘one-size-fits-all’ cannot meet learner diversity in contemporary learning” (al-azawei et al., 2016, p. 53). while anecdotal evidence has long suggested that individuals learn in ways that are specific to them, new knowledge produced within the past twenty-five years or so has certainly “elucidated the great variability of the human capacity to learn” (meyer, rose, & gordon, 2014, p. 49). therefore, a curriculum that is both udl based and culturally informed is key to meeting the needs of our increasingly diverse twenty-first century classrooms. the udl framework cast developed the udl framework (see appendix a for graphic organizer; cast, 2011b) to include 3 principles (divided into 9 more specific guidelines, each of which further sub-divide into multiple checkpoints): multiple means of representation, multiple means of action and expression, and multiple means of engagement (cast, 2011a). instructors are guided to craft lessons and courses with built-in scaffolds and approaches that will support and meet the needs of all students, which includes providing materials to learners in various formats, allowing flexibility in how students demonstrate their learning, and motivating students to become active agents in their own learning (courey et al., 2012). according to rose and strangman (2007), these principles mirror three key features of any learning environment and map on to three different learning networks in the brain. the what of learning (course content) is located in the posterior regions of the cortex, the how of learning (assessments) is located in the frontal regions of the cortex, and the why of learning (motivation) is in the medial regions of the nervous system. because each learner is unique as to the strength of each of these learning networks, each learner will also be unique in the way that they learn (rose & strangman, 2007; meyer, rose, & gordon, 2014). multiple means of representation each student has a preferred way to receive information. for some, it may be textual, for others visually and/or aurally, and others still critically by way of working through a given problem. by providing students with information in universal design for learning (udl): student and faculty perceptions 3 journal of effective teaching in higher education, vol. 1, no. 2 multiple ways, each student will engage more profoundly with the material, thereby making more neural connections (rose & strangman, 2007; courey et al., 2012). while students admittedly find it difficult to connect multiple representations of the same concept, it is necessary to do so in many fields (e.g., science) in order to think critically and solve problems (kozma, 2003). exposing students to multiple representations of the same concept fosters a deeper understanding, especially if they are also able to engage in dialogue about the content (kozma, 2003; courey et al., 2012). multiple means of action and expression students also vary in their preferred way to demonstrate (i.e., communicate) their mastery of course content. some students prefer writing an essay, while others prefer multiple-choice testing. some students prefer to apply their knowledge to a concrete problem, while others still prefer a more expressive option. each variance allows students to demonstrate their knowledge of the content, but in a way that benefits them. in one study, interviews revealed that giving students access to multiple means of action and expression on a discussion board (where students could respond in prose, via recording, or through illustration) allowed students to contribute their ideas in more efficient ways (goldowsky & coyne, 2016). even standardized test scores are affected by the mode used to respond: paper vs. computer. specifically, when students had greater familiarity with the responding mode, students performed significantly higher; students with greater computer literacy scored higher on the test when responding on the computer, while students with less computer familiarity performed better when completing the handwritten test (russell, 2000). it is clear from these results that providing multiple means of action and expression is beneficial to students. multiple means of engagement the affective component of learning is captured in the third udl principle: multiple means of engagement. students’ interest and motivation must be maintained throughout the learning process, and one way this can be done is through choice. the ability for control and autonomy are important for workplace engagement (pink, 2011). providing students with meaningful instruction and authentic assessments will help engage them because the significance to be derived is clear and it is concretely linked to their eventual career. frequent and personal interactions such as providing strategies for improvement as part of instructor feedback on assessments also engages students. additionally, student engagement increases when they feel safer in the learning space and limited distractions result in sharper focus. providing choice increases intrinsic motivation and the amount of effort the person puts into the task (patall, cooper, & robinson, 2008). furthermore, providing students with relevant assessments and learning experiences (i.e., those directly related to students’ goals) appear to be even more important for student motivation, at least for adolescent students (assor, kaplan, & roth, 2002). both of these (choice and relevance) are examples of this udl principle. universal design for learning (udl): student and faculty perceptions 4 journal of effective teaching in higher education, vol. 1, no. 2 experiment 1 despite a wealth of knowledge about particular aspects of udl and their benefits to student learning, gaps remain. in fact, meyer, rose, and gordon (2014) recently stressed the need for a stronger research base as udl guidelines continue to proliferate. from our perspective, we have found less research devoted to students’ perception of the udl framework in action, as it were, in the classroom. little research exists surveying students about how much udl they encounter in their courses or how important they perceive these course modifications to be to their success. a handful of studies have examined students’ perceived usage of udl in the classroom. using a pre-post design, two recent studies have found that students do perceive more udl usage by instructors after those instructors have been trained on the use of udl, especially when it came to these instructors offering more varied modes of representation (davies, shelly, & spooner, 2013; schelly et al., 2011). when compared to a control group, davies et al. (2013) actually showed that improvements in engagement occurred over time during the semester, regardless of udl training. this suggests that instructor rapport with students and the relationships which develop as the semester progresses are important for actively engaging participants’ perceived enthusiasm. however, these studies did not examine instructors’ perceived use of udl. to our knowledge, only one study has explicitly examined perceived usefulness of udl principles in the student population. black et al. (2015) showed that both students with and without a disability perceived a positive impact on their learning when instructors followed udl principles in their classroom. however, their review of udl activities within the classroom is not as comprehensive as our examination, relying on a rating scale form 1 (“not useful/important”) to 3 (“very useful/important”) and therefore could not compare usage rates with perceived usefulness. method participants participants were students enrolled in the one-year general arts and science certificate program at durham college, in ontario, canada. they were invited to complete the survey through an announcement on the program page of their learning management system. although approximately 600 students were enrolled in the program that semester, only a small subset of these students would have accessed their program page on the lms during our recruitment period and seen the announcement, so it is impossible to speculate a meaningful response rate for students. in total, only 17 students provided answers to the survey questions. however, since each of these students is providing data for all of their courses (6 per semester), we feel that the data they provide offer valuable insight into their universal design for learning (udl): student and faculty perceptions 5 journal of effective teaching in higher education, vol. 1, no. 2 experience with udl at the college. no demographic data were collected to protect student privacy, as required by our institutional research ethics board. a more detailed discussion of this is located in the concluding remarks section. materials the authors developed the survey to gather student perceptions on two key questions: “for each item, indicate how much you have experienced this in your courses.” and “for each item, indicate how useful you think these things would be in helping you learn in your courses. please answer for each item, even if you did not experience it in any of your courses.” below these prompts were 36 items which tapped into each of the three udl principles such as “include subtitles on videos,” “offer interesting and relevant major assignments”, and “provide sufficient or unlimited time for tests.” these 36 items were adapted from the list of cast (2011a) checkpoints for each of the three udl principles (appendix b for items). the questionnaire appears to be high in face validity as it lists the udl checkpoints as items. since these survey items each refer to one of the udl checkpoints, and each item is empirically supported by a list of scholarly evidence cited on the cast website in support of each of these checkpoints (cast, 2018), interested readers may refer to them for examples of best practices to use in their classrooms. the full list of the final survey items is included in the tables of the results section. procedures all students enrolled in the general arts and science program were recruited to participate in the study by way of an announcement in an informationdissemination course page for their program in the learning management system. interested students (n = 17) simply clicked on the surveymonkey url provided in the announcement and rated each of the 36 items; first for their impressions of how much that item is used in their courses, and then for how useful they perceived each of the items to be. the questionnaire took approximately 10 minutes to complete. it was live for a three-week period and students were reminded twice during that period. results and discussion usage table 1 displays how much students report encountering each of the udl elements in their classrooms. students report that faculty use these elements in their classrooms “a lot”, especially for the elements related to multiple means of representation. that is, students reported that the material was frequently presented in multiple formats, and that they felt moderately motivated to succeed in the course, and were given adequate options and flexibility for how they showed mastery of the course content. students rarely encountered field trips (“not at all” = 94%) in their courses, and were also infrequently provided with streaming video or audio of their face-to-face classes (“not at all” = 47.06%). for the most part, universal design for learning (udl): student and faculty perceptions 6 journal of effective teaching in higher education, vol. 1, no. 2 faculty decided on the course content (“not at all” = 58%) and students are not given the opportunity to resubmit course work (“not at all” = 52.94). faculty frequently post handouts (“a lot” = 81.25%) and slides (“a lot” = 70.59%) on the lms are often available outside of class by answering questions (“a lot” = 70.59%) and communicating with students (“a lot” and “a moderate amount” combined = 88.24%). faculty also provide students with rubrics for assignments (“a lot” and “a moderate amount” combined = 88.24%) and are able to monitor their progress on the lms (“a lot” = 70.59%). universal design for learning (udl): student and faculty perceptions 7 journal of effective teaching in higher education, vol. 1, no. 2 table 1 student reporting of udl usage in their classrooms. percentages for each item, indicate how much you have experienced this in your courses at durham college. how much did your teacher: not at all a little bit a moderate amount a lot unsure r epresentation present the same course content in multiple ways (graphics, video, text, graphic organizers/concept maps, etc.) 11.76 29.41 23.53 35.29 0.00 offer an electronic version of the textbook 11.76 29.41 23.53 23.53 11.76 post handouts on dc connect (or make them available digitally) 6.25 12.50 0.00 81.25 0.00 include subtitles on videos (closed captioned) 35.29 11.76 29.41 5.88 17.65 upload files can be read using text-to-speech software (e.g., word documents pdfs) 11.76 5.88 5.88 52.94 23.53 provide clear guidelines for major assignments (e.g., example/sample assignment) 5.88 17.65 35.29 35.29 5.88 include a field trip 94.12 5.88 0.00 0.00 0.00 capture class lectures and made them available to stream after class (video or podcast) 47.06 29.41 5.88 11.76 5.88 make available a glossary of terms (on dc connect, in the textbook, or other) 35.29 17.65 11.76 29.41 5.88 offer alternatives for auditory info (e.g., transcripts of videos) and visual info (e.g., description of images) 23.53 29.41 11.76 23.53 11.76 highlight patterns and relationships in the course content 25.00 12.50 31.25 18.75 12.50 e ngagem ent offer interesting and relevant major assignments 6.25 37.50 31.25 25.00 0.00 allow for some autonomy and/or control in student learning (e.g., options for assignments (topic or format); or choices on tests (choose 1 of 2 essay questions; or pick 5 of the following terms to define) 23.53 11.76 35.29 23.53 5.88 let students decide which topics are covered in the course 58.82 5.88 5.88 11.76 17.65 use hands-on activities in class 18.75 31.25 12.50 25.00 12.50 connect course content to real world experiences 5.88 29.41 17.65 47.06 0.00 communicate with students (in class, outside of class, via message board or email) 5.88 5.88 23.53 64.71 0.00 universal design for learning (udl): student and faculty perceptions 8 journal of effective teaching in higher education, vol. 1, no. 2 percentages provide clear and specific feedback on assignments 17.65 52.94 29.41 0.00 0.00 offer a choice of how students want to receive feedback on assignments (e.g., verbal or written feedback) 35.29 29.41 5.88 23.53 5.88 allow students to re-submit assignments 52.94 35.29 5.88 0.00 5.88 include peer-evaluation as part of the coursework 35.29 23.53 23.53 17.65 0.00 make powerpoint slides available to students 0.00 11.76 17.65 70.59 0.00 include group work and collaboration with other students (e.g., discussions) 0.00 23.53 17.65 58.82 0.00 provide opportunities for self-assessment/self-evaluation and reflection 5.88 29.41 11.76 52.94 0.00 answer questions about course content or assignments outside of class (e.g., discussion board, email) 5.88 5.88 17.65 70.59 0.00 use gender-neutral language and inclusive examples (race/culture, etc.) 17.65 17.65 5.88 52.94 5.88 minimize threats and distractions in the learning environment 17.65 23.53 11.76 47.06 0.00 motivate students to do their best work 11.76 23.53 17.65 47.06 0.00 e xpression flexible due dates on major assignments (e.g., allowed to turn it in late) 29.41 35.29 11.76 17.65 5.88 offer ungraded or optional assignments to practice the course content 11.76 23.53 29.41 29.41 5.88 provide sufficient (or unlimited) time for tests 29.41 11.76 35.29 23.53 0.00 provide rubrics for major assignments 5.88 5.88 17.65 70.59 0.00 guide you using increasingly difficult activities or assignments 17.65 11.76 29.41 41.18 0.00 guide goal-setting and the development of student learning strategies 11.76 5.88 35.29 47.06 0.00 provide opportunities for students to monitor progress (e.g., grades posted on dc connect) 5.88 11.76 11.76 70.59 0.00 note: dc connect refers to our learning management system. universal design for learning (udl): student and faculty perceptions 9 journal of effective teaching in higher education, vol. 1, no. 2 usefulness table 2 shows how useful students perceived each udl element to be. with the exception of a few elements − field trips, peer evaluations, allowing students to resubmit assignments, giving students the choice of selecting which content is covered in the course, and capturing the lecture to make them available for later streaming − the udl elements were rated as being helpful to student learning at least a moderate amount. this was especially true for having faculty available outside of class time to answer questions (“a lot” = 100%), providing rubrics for assignments (“a lot” = 90.91%), sharing lecture slides (“a lot” = 91.67%), providing clear feedback on assignments (“a lot” = 91.67%), regular communication with students (“a lot” = 91.67%), motivating students to do their best work (“a lot” = 91.67%), and posting handouts on the lms (“a lot” = 91.67%). it seems that a key element for students is faculty communication. universal design for learning (udl): student and faculty perceptions 10 journal of effective teaching in higher education, vol. 1, no. 2 table 2 students’ usefulness ratings of udl principles. percentages for each item, indicate how useful you think these things would be in helping you learn in your courses. please answer for each item, even if you did not experience it in any of your courses. not at all a little bit a moderate amount a lot unsure r epresentation present the same course content in multiple ways (graphics, video, text, graphic organizers/concept maps, etc.) 0.00 0.00 16.67 83.33 0.00 offer an electronic version of the textbook 16.67 0.00 25.00 33.33 25.00 post handouts on dc connect (or make them available digitally) 0.00 0.00 0.00 91.67 8.33 include subtitles on videos (closed captioned) 8.33 8.33 8.33 66.67 8.33 upload files can be read using text-to-speech software (e.g., word documents pdfs) 16.67 0.00 8.33 66.67 8.33 provide clear guidelines for major assignments (e.g., example/sample assignment) 0.00 0.00 8.33 83.33 8.33 include a field trip 25.00 25.00 16.67 8.33 25.00 capture class lectures and made them available to stream after class (video or podcast) 25.00 8.33 16.67 50.00 0.00 make available a glossary of terms (on dc connect, in the textbook, or other) 0.00 8.33 8.33 83.33 0.00 offer alternatives for auditory info (e.g., transcripts of videos) and visual info (e.g., description of images) 8.33 16.67 16.67 58.33 0.00 highlight patterns and relationships in the course content 8.33 8.33 16.67 50.00 16.67 e ngagem ent offer interesting and relevant major assignments 0.00 8.33 8.33 75.00 8.33 allow for some autonomy and/or control in student learning (e.g., options for assignments (topic or format); or choices on tests (choose 1 of 2 essay questions; or pick 5 of the following terms to define) 0.00 0.00 16.67 83.33 0.00 let students decide which topics are covered in the course 25.00 8.33 8.33 41.67 16.67 use hands-on activities in class 0.00 0.00 16.67 75.00 8.33 connect course content to real world experiences 0.00 0.00 8.33 83.33 8.33 universal design for learning (udl): student and faculty perceptions 11 journal of effective teaching in higher education, vol. 1, no. 2 percentages communicate with students (in class, outside of class, via message board or email) 0.00 0.00 8.33 91.67 0.00 provide clear and specific feedback on assignments 0.00 8.33 0.00 91.67 0.00 offer a choice of how students want to receive feedback on assignments (e.g., verbal or written feedback) 0.00 16.67 33.33 41.67 8.33 allow students to re-submit assignments 16.67 8.33 16.67 41.67 16.67 include peer-evaluation as part of the coursework 8.33 16.67 8.33 58.33 8.33 make powerpoint slides available to students 0.00 0.00 0.00 91.67 8.33 include group work and collaboration with other students (e.g., discussions) 0.00 16.67 16.67 58.33 8.33 provide opportunities for self-assessment/self-evaluation and reflection 8.33 8.33 33.33 50.00 0.00 answer questions about course content or assignments outside of class (e.g., discussion board, email) 0.00 0.00 0.00 100.00 0.00 use gender-neutral language and inclusive examples (race/culture, etc.) 0.00 0.00 16.67 66.67 16.67 minimize threats and distractions in the learning environment 8.33 0.00 8.33 83.33 0.00 motivate students to do their best work 0.00 0.00 8.33 91.67 0.00 e xpression flexible due dates on major assignments (e.g., allowed to turn it in late) 8.33 8.33 8.33 58.33 16.67 offer ungraded or optional assignments to practice the course content 8.33 8.33 8.33 75.00 0.00 provide sufficient (or unlimited) time for tests 8.33 16.67 8.33 66.67 0.00 provide rubrics for major assignments 0.00 0.00 0.00 90.91 9.09 guide you using increasingly difficult activities or assignments 8.33 0.00 8.33 66.67 16.67 guide goal-setting and the development of student learning strategies 0.00 0.00 16.67 83.33 0.00 provide opportunities for students to monitor progress (e.g., grades posted on dc connect) 0.00 15.38 0.00 84.62 0.00 note: dc connect refers to our learning management system. universal design for learning (udl): student and faculty perceptions 12 journal of effective teaching in higher education, vol. 1, no. 2 summary taken together, the usage and usefulness data paint a consistent picture: students are exposed to many elements of udl in their classrooms and they find most of these elements to be useful to their individual learning. they perceive faculty communication as a key component of their success (i.e., feedback on assignments, communication outside of class, and responding to emails). fortunately, the elements they encounter most (e.g., rubrics, communication, and sharing lecture slides) are also the ones they perceive as most helpful for their learning, while many of the elements they did not frequently encounter (e.g., field trips, streaming lectures, choosing course content) were also perceived as not particularly valuable. one limitation should be pointed out however, and that is the small sample of students who responded to the survey: only 17 out of several hundred students in the program provided us with data, but their responses are still valuable as a semirepresentative sample of our student population. see the concluding remarks section for a more elaborate discussion. experiment 2 student perceptions are important as they provide a glimpse into their realities. however, faculty perceptions are equally valuable and there are limited publications surveying faculty on their use and perceived usefulness of udl principles. assessing faculty perceptions of udl in the classroom with give us a different perspective on the same reality and allow us to better understand what is happening in classrooms, and how useful specific elements of udl are perceived to be by faculty. method participants participants were faculty teaching in the one-year general arts and science certificate program at an ontario college. approximately 40 faculty members were invited to complete the survey through email with an equal proportion being fulltime employees and contract instructors. in total, 11 faculty members responded to the survey, yielding a response rate of approximately 28%. no demographic data were collected to maintain our colleagues’ privacy and reduce the risk of identification. for a more detailed discussion, see the concluding remarks section. materials the authors adapted the student survey from experiment 1 as necessary to reflect the faculty’s perspective. for example, in the student survey (experiment 1), the first question asked “for each item, indicate how much you have experienced this in your courses,” while in the survey for faculty, the question was reworded to “for each item, indicate how much you use it in your courses.” the second question was similarly reworded to reflect a faculty’s perspective. faculty responded to the same 36 items as had the students in experiment 1. universal design for learning (udl): student and faculty perceptions 13 journal of effective teaching in higher education, vol. 1, no. 2 procedures faculty teaching courses in the general arts and science program were recruited via email to participate in the study. interested faculty clicked on the surveymonkey url provided in the email and completed the adapted survey from experiment 1, first rating their usage of each of the 36 items, and then their perceived usefulness. the questionnaire took approximately 10 minutes to complete. it was live for a two-week period and faculty were sent one reminder email one week after the initial recruitment email. results and discussion usage table 3 shows how frequently faculty report using each of the udl in their classrooms. the most used elements were posting content on the lms (handouts “a lot” = 81.82; slides “a lot” = 100%), providing clear guidelines for assignments (“a lot” = 81.82%), answer questions and communicate with students outside of class (“a lot” = 90.91% and 81.82%, respectively), provide feedback on assignments (“a lot” = 90.91), and allow students to monitor their progress in the course by posting grades to the lms (“a lot” = 100%). the udl elements that faculty reported not using very frequently include field trips (“not at all” = 72.73%), capturing and streaming lectures (“not at all” and “a little bit” combined = 100%), allowing students to decide course content (“not at all” and “a little bit” combined = 100%), offering a choice of how students receive feedback on assignments (“not at all” and “a little bit” combined =90.91%), and allowing students to resubmit assignments (“not at all” and “a little bit” combined = 72.73%). universal design for learning (udl): student and faculty perceptions 14 journal of effective teaching in higher education, vol. 1, no. 2 table 3 faculty reporting of udl usage in their classrooms. percentages for each item, indicate how much you use it in your courses at durham college. how much did you: not at all a little bit a moderate amount a lot unsure r epresentation present the same course content in multiple ways (graphics, video, text, graphic organizers/concept maps, etc.) 0.00 0.00 45.45 54.55 0.00 offer an electronic version of the textbook 27.27 0.00 9.09 63.64 0.00 post handouts on dc connect (or make them available digitally) 0.00 0.00 18.18 81.82 0.00 include subtitles on videos (closed captioned) 18.18 9.09 18.18 54.55 0.00 upload files can be read using text-to-speech software (e.g., word documents pdfs) 0.00 9.09 9.09 54.55 27.28 provide clear guidelines for major assignments (e.g., example/sample assignment) 0.00 9.09 9.09 81.82 0.00 include a field trip 72.73 9.09 18.18 0.00 0.00 capture class lectures and made them available to stream after class (video or podcast) 63.64 36.36 0.00 0.00 0.00 make available a glossary of terms (on dc connect, in the textbook, or other) 36.36 9.09 0.00 54.55 0.00 offer alternatives for auditory info (e.g., transcripts of videos) and visual info (e.g., description of images) 45.45 9.09 36.36 9.09 0.00 highlight patterns and relationships in the course content 9.09 18.18 9.09 54.55 9.09 e ngagem ent offer interesting and relevant major assignments 9.09 18.18 27.27 45.45 0.00 allow for some autonomy and/or control in student learning (e.g., options for assignments (topic or format); or choices on tests (choose 1 of 2 essay questions; or pick 5 of the following terms to define) 27.27 18.18 27.27 27.27 0.00 let students decide which topics are covered in the course 63.64 36.36 0.00 0.00 0.00 use hands-on activities in class 0.00 9.09 36.36 54.55 0.00 connect course content to real world experiences 0.00 0.00 27.27 72.73 0.00 communicate with students (in class, outside of class, via message board or email) 0.00 0.00 18.18 81.82 0.00 universal design for learning (udl): student and faculty perceptions 15 journal of effective teaching in higher education, vol. 1, no. 2 percentages provide clear and specific feedback on assignments 0.00 0.00 9.09 90.91 0.00 offer a choice of how students want to receive feedback on assignments (e.g., verbal or written feedback) 63.64 27.27 0.00 9.09 0.00 allow students to re-submit assignments 54.55 18.18 18.18 9.09 0.00 include peer-evaluation as part of the coursework 54.55 0.00 45.45 0.00 0.00 make powerpoint slides available to students 0.00 0.00 0.00 100.00 0.00 include group work and collaboration with other students (e.g., discussions) 18.18 9.09 18.18 54.55 0.00 provide opportunities for self-assessment/self-evaluation and reflection 27.27 0.00 45.45 27.27 0.00 answer questions about course content or assignments outside of class (e.g., discussion board, email) 0.00 0.00 9.09 90.91 0.00 use gender-neutral language and inclusive examples (race/culture, etc.) 0.00 9.09 27.27 63.64 0.00 minimize threats and distractions in the learning environment 0.00 0.00 27.27 72.73 0.00 motivate students to do their best work 0.00 9.09 27.27 63.64 0.00 e xpression flexible due dates on major assignments (e.g., allowed to turn it in late) 18.18 36.36 36.36 9.09 0.00 offer ungraded or optional assignments to practice the course content 9.09 27.27 45.45 18.18 0.00 provide sufficient (or unlimited) time for tests 9.09 9.09 27.27 54.55 0.00 provide rubrics for major assignments 9.09 9.09 18.18 63.64 0.00 guide students using increasingly difficult activities or assignments 0.00 9.09 27.27 54.55 9.09 guide goal-setting and the development of student learning strategies 9.09 27.27 18.18 36.36 9.09 provide opportunities for students to monitor progress (e.g., grades posted on dc connect) 0.00 0.00 0.00 100.00 0.00 note: dc connect refers to our learning management system. universal design for learning (udl): student and faculty perceptions 16 journal of effective teaching in higher education, vol. 1, no. 2 usefulness table 4 shows how useful faculty perceived each udl element to be for students. many of the elements that faculty did not include in their courses were elements that faculty also did not perceive as helpful to student learning. for example, including a field trip (“not at all” and “a little bit” combined = 55.55%), allowing students to decide course content (“not at all” and “a little bit” combined = 77.77%), allowing students to re-submit assignments (“not at all” and “a little bit” combined = 44.44%), and including peer-evaluation as part of the course (“not at all” and “a little bit” combined = 44.44%). where faculty felt that students benefitted most from udl principles were by presenting material in multiple ways (“a lot” = 77.78%), posting handouts and slides on the lms (“a lot” = 77.78% for each), providing clear guidelines on assignments (“a lot” = 88.89%), providing interesting major assignments (“a lot” = 77.78%) and hands-on learning activities (“a lot” = 77.78%), answering questions outside of class time (“a lot” = 100%), motivating students to do their best work (“a lot” = 100%), minimizing threats (“a lot” = 88.89%), and posting grades on the lms to allow students to monitor their progress (“a lot” = 100%). universal design for learning (udl): student and faculty perceptions 17 journal of effective teaching in higher education, vol. 1, no. 2 table 4 faculty’s rating of the perceived usefulness of udl principles for students. percentages for each item, indicate how useful you think these things are in helping your students learn in your courses. please answer for each item, even if you do not use it in any of your courses. not at all a little bit a moderate amount a lot unsure r e p re s e n ta tio n present the same course content in multiple ways (graphics, video, text, graphic organizers/concept maps, etc.) 0.00 11.11 11.11 77.78 0.00 offer an electronic version of the textbook 22.22 22.22 33.33 11.11 11.11 post handouts on dc connect (or make them available digitally) 0.00 11.11 0.00 77.78 11.11 include subtitles on videos (closed captioned) 0.00 0.00 44.44 33.33 22.22 upload files can be read using text-to-speech software (e.g., word documents pdfs) 0.00 22.22 22.22 22.22 33.33 provide clear guidelines for major assignments (e.g., example/sample assignment) 0.00 0.00 11.11 88.89 0.00 include a field trip 11.11 44.44 22.22 11.11 11.11 capture class lectures and made them available to stream after class (video or podcast) 22.22 11.11 22.22 22.22 22.22 make available a glossary of terms (on dc connect, in the textbook, or other) 22.22 0.00 22.22 55.56 0.00 offer alternatives for auditory info (e.g., transcripts of videos) and visual info (e.g., description of images) 11.11 33.33 33.33 22.22 0.00 highlight patterns and relationships in the course content 11.11 11.11 11.11 55.56 11.11 e n g a g e m e n t offer interesting and relevant major assignments 0.00 0.00 22.22 77.78 0.00 allow for some autonomy and/or control in student learning (e.g., options for assignments (topic or format); or choices on tests (choose 1 of 2 essay questions; or pick 5 of the following terms to define) 11.11 22.22 22.22 44.44 0.00 let students decide which topics are covered in the course 33.33 44.44 11.11 11.11 0.00 use hands-on activities in class 0.00 11.11 11.11 77.78 0.00 connect course content to real world experiences 0.00 11.11 88.89 0.00 0.00 universal design for learning (udl): student and faculty perceptions 18 journal of effective teaching in higher education, vol. 1, no. 2 percentages communicate with students (in class, outside of class, via message board or email) 0.00 11.11 88.89 0.00 0.00 provide clear and specific feedback on assignments 11.11 0.00 88.89 0.00 0.00 offer a choice of how students want to receive feedback on assignments (e.g., verbal or written feedback) 22.22 33.33 22.22 11.11 11.11 allow students to re-submit assignments 33.33 11.11 22.22 33.33 0.00 include peer-evaluation as part of the coursework 33.33 11.11 33.33 22.22 0.00 make powerpoint slides available to students 0.00 11.11 11.11 77.78 0.00 include group work and collaboration with other students (e.g., discussions) 11.11 11.11 22.22 55.56 0.00 provide opportunities for self-assessment/self-evaluation and reflection 11.11 11.11 33.33 44.44 0.00 answer questions about course content or assignments outside of class (e.g., discussion board, email) 0.00 0.00 0.00 100.00 0.00 use gender-neutral language and inclusive examples (race/culture, etc.) 0.00 22.22 22.22 55.56 0.00 minimize threats and distractions in the learning environment 0.00 0.00 0.00 100.00 0.00 motivate students to do their best work 0.00 0.00 0.00 88.89 11.11 e x p re s s io n flexible due dates on major assignments (e.g., allowed to turn it in late) 33.33 0.00 44.44 22.22 0.00 offer ungraded or optional assignments to practice the course content 11.11 0.00 33.33 55.56 0.00 provide sufficient (or unlimited) time for tests 11.11 11.11 33.33 44.44 0.00 provide rubrics for major assignments 0.00 11.11 33.33 55.56 0.00 guide you using increasingly difficult activities or assignments 0.00 0.00 33.33 44.44 22.22 guide goal-setting and the development of student learning strategies 11.11 0.00 44.44 33.33 11.11 provide opportunities for students to monitor progress (e.g., grades posted on dc connect) 0.00 0.00 0.00 100.00 0.00 note: dc connect refers to our learning management system universal design for learning (udl): student and faculty perceptions 19 journal of effective teaching in higher education, vol. 1, no. 2 summary taken together, the usage and usefulness data provided by faculty show that elements of udl are consistently included in their curriculum, perhaps because they perceive these elements as valuable to student learning. the elements that faculty did perceive as valuable to students’ learning were typically integrated into their course curriculum. although the response rate for faculty was also small, it is likely more representative of all the courses in the general arts and science program as full-time faculty teach 8-10 courses per year, while contract employees typically teach 2-4; full-time faculty appeared to be more likely to complete the survey based on the open-ended comments that were made by respondents and participants’ self-identification to the researchers. general discussion comparing the data from students (experiment 1) and faculty (experiment 2) can show us where discrepancies may exist for usage and perceived usefulness for elements of udl. usage importantly, there is much agreement on which udl principles are present in the classroom, but there were also some interesting perceived differences. one of these elements was offering an electronic version of the textbook. perhaps students were not aware that e-books existed for their courses, because two-thirds of faculty reported that an e-text was available, but less than one-quarter of students did. so, although the electronic format may be provided as an option to students, they may not actually be aware of this alternate means of representation. in terms of using hands-on activities in the classroom, faculty believe they are providing students with far more hands-on activities than students report experiencing them. this difference is likely due to how this question was operationalized by each group of respondents, (i.e., which activities would be considered hands-on) and expectations for the classrooms (e.g., instructors may have higher expectations based on their knowledge of best practices in the classroom). and, although many faculty respondents report that they do not offer many alternatives for auditory and visual information, students reported a greater amount of alternatives in their classrooms. here again, the difference may have more to do with a mismatch between student and faculty expectations for alternative formats. one key element where students and faculty disagreed had to do with faculty highlighting patterns and relationships to students in the classroom. students perceived far less highlighting of these relationships than faculty report using. this discrepancy may be due to the level at which faculty are highlighting these patterns; if they are beyond the cognitive abilities of the students, these patterns may not be fully understood by students. research by kennette and frank (2013) universal design for learning (udl): student and faculty perceptions 20 journal of effective teaching in higher education, vol. 1, no. 2 and bowman, frame, and kennette (2013) have shown that peers are in a unique position to bridge the knowledge gap between expert instructors and novice students as their cognitive sets are more similar and can therefore explain concepts or relationships between them at the students’ level. related to this, and possibly caused by the same underlying issue, faculty believe they are almost always providing clear feedback on assignments, but students are more divided on this. the final element that we wish to highlight is related to the principle of engagement. faculty believe they are providing a safe and distraction-free environment, but may not be aware of what can be distracting to students (in both their personal and academic lives), as student perceptions differed from faculty’s perceptions. we may think our students are not distracted in the classroom, but they report that they are. we may not notice, of course, but there may be students nearby whispering or students on facebook on their laptops near the front of the class. there are policies we may be able to put in place to reduce in-class distractions (such as asking students with laptops to sit near the back of the class), but it is unlikely that instructors can eliminate all distractions in the learning environment. usefulness for the elements that students and faculty disagreed on in terms of usefulness, students typically perceived the udl elements as more useful than faculty did. for example, there was disagreement on the value of including peer-evaluation as part of the coursework. students felt that evaluating their peers was a more critical feature of their success than faculty did. this is perhaps because students have experienced dysfunctional groups more than faculty have been made aware and they would like to have a mechanism to declare these transgressions; alternately, students may feel that peer-evaluations would serve as a deterrent to nonparticipatory group members. this is an interesting finding as, anecdotally, faculty report students’ dislike for group work (kennette & hanzuk, 2017). an accurate explanation is beyond the scope of this paper, and, although we can speculate, future research should address this paradox empirically. capturing lectures and posting online for later viewing or listening was another area of disagreement as to its perceived helpfulness for understanding course material. two-thirds of student respondents felt that capturing lectures for later streaming was moderately or very helpful, while less than half of faculty respondents felt this way. of course, it is possible that students’ perceived value for this item reflects, in part, a desire to be able to miss class without missing content. however, since students also reported moderate usefulness for text-to-speech compatible files, perhaps it is simply that students prefer to have access to auditory course content, perhaps to listen to while riding the bus to campus. finally, far more students saw autonomy or control (e.g., assignment or test format options) as being helpful to their learning than did faculty. we know from the literature on motivation (see pink, 2011) that autonomy and control are important components of motivation and this is likely what is driving this difference in perception as faculty already have most of the control in the typical classroom. universal design for learning (udl): student and faculty perceptions 21 journal of effective teaching in higher education, vol. 1, no. 2 concluding remarks the good news is that there is much agreement in which udl principles are present in the classrooms, but there were also some differences in the student and faculty perspectives. some of these differences may have to do with the frame of reference each is bringing to the classroom or by the lack of explicit transparency by faculty (for example, having a reflection as part of an assignment, but not labelling it as such). one important limitation which warrants further discussion is the small sample size and the lack of demographic data collected to describe our sample. especially in experiment 1 (students), respondents were a very small portion of the total population and may not have been a representative sample. as this study was descriptive in nature and we had not hypothesized tied to any particular demographic criteria, our school’s irb discourages the collection of these data as they significantly increase the risk of identification. however, the lack of sample demographics and the low response rate should not necessarily be interpreted as our sample coming from an unrepresentative subset of the program’s student population. it is also important to remember that each student is enrolled in 6 courses per semester, so each respondent is providing data on 12 courses in the program. in ontario colleges, which would be similar to 2-year schools in the us, there is very little research conducted by faculty, and so students are not used to seeing these opportunities. furthermore, there are typically no incentives for participation in research by students. in our experience, participation rates to in-person classroom recruitment are typically only around 10% and offering an incentive negligibly increases this rate (by 1-2%). research is simply not part of the expectations for students at our school. what can we learn from these data? in general, students find udl principles to be useful for their learning and faculty are overall pretty good at including these elements in their curriculum. when designing curriculum, particular consideration may be given to the elements that students found especially beneficial, however faculty should also ensure that they include elements from all three principles of udl. it is important to note that the data reported here are student and faculty perceptions. student perceptions of usefulness are not backed up by performance data. future research should examine whether the perceived usefulness of these udl principles by students correlates with student performance data such as course grades. references al-azawei, a., serenelli, f., & lundqvist, k. (2016) universal design for learning (udl): a content analysis of peer-reviewed journal papers from 2012-2015. universal design for learning (udl): student and faculty perceptions 22 journal of effective teaching in higher education, vol. 1, no. 2 journal of the scholarship of teaching and learning, 16(3), 39-56. retrieved from https://files.eric.ed.gov/fulltext/ej1104867.pdf assor, a. kaplan, h. & roth, g. (2002). choice is good, but relevance is excellent: autonomy-enhancing and suppression teacher behaviours predicting students’ engagement in schoolwork. british journal of educational psychology, 72, 261-278. doi: http://dx.doi.org/10.1348/000709902158883 black, r. d., weinberg, l. a., & brodwin, m. g. (2015). universal design for learning and instruction: perspectives of students with disabilities in higher education. exceptionality education international, 25(2), 1-26. retrieved from https://ir.lib.uwo.ca/cgi/viewcontent.cgi?article=1131&context=eei bowman, m., frame, d. l., & kennette, l. n. (2013). enhancing teaching and learning: how cognitive research can help. journal on excellence in college teaching: brain-based learning (special issue), 24(3), 7-28. cast (2011a). universal design for learning guidelines version 2.0, wakefield, ma. retrieved from http://udlguidelines.cast.org/binaries/content/assets/udlguidelines/udlg-v20/udlg_fulltext_v2-0.doc cast (2011b). universal design for learning guidelines version 2.0 [graphic organizer], wakefield, ma. retrieved from http://udlguidelines.cast.org/binaries/content/assets/udlguidelines/udlg-v20/udlg_graphicorganizer_v2-0.pdf cast (2018). universal design for learning guidelines. center for applied special technology. retrieved from http://udlguidelines.cast.org chita-tegmark, m., gravel, j. w., serpa, m. l. b., domings, y., & rose, d. h. (2012). using the universal design for learning framework to support culturally diverse learners. journal of education, 192(1), 17-22. doi: 10.1177/002205741219200104 claiborne, l. b., cornforth, s., gibson, a., & smith, a. (2010). supporting students with impairments in higher education: social inclusion or cold comfort? international journal of inclusive education, 15(5), 513–527. doi: 10.1080/13603110903131747 davies, p. l., schelly, c. l., & spooner, c. l. (2013). measuring the effectiveness of universal design for learning intervention in postsecondary education. journal of postsecondary education and disability, 26(3), 195-220. retrieved from https://eric.ed.gov/?id=ej1026883 fichten, c. s., jorgensen, s., havel, a., & barile, m. (2006). college students with disabilities: their future and success. retrieved from https://eric.ed.gov/?id=ed491585 goldowsky, b. n. & coyne, m. (2016, april). supporting engagement and comprehension online through multiple means of expression. in h. tokagi (chair), accessible multimedia. symposium conducted at the meeting of the web for all conference, montreal, canada. retrieved from https://dl.acm.org/citation.cfm?doid=2899475.2899488 kennette. l. n. & hanzuk, w. (2017, may 1). group work and collaborative activities for all. faculty focus. https://www.facultyfocus.com/articles/instructional-design/group-workcollaborative-activities/ universal design for learning (udl): student and faculty perceptions 23 journal of effective teaching in higher education, vol. 1, no. 2 kennette, l. n., & frank, n. m. (2013). the value of peer-review opportunities for students in writing-intensive classes. psychology teaching review, 19(2), 106-111. kozma, r. (2003). the material features of multiple representations and their cognitive and social affordances for science understanding. learning and instruction, 13(2), 205-226. doi: http://dx.doi.org/10.1016/s09594752(02)00021-x marshak, l., van wieren, t., ferrell, d. r., swiss, l., & dugan, c. (2011). exploring barriers to college student use of disability services and accommodations. journal of postsecondary education and disability, 22(3), 151–165. retrieved from https://eric.ed.gov/?id=ej906688 ontario university & college health association (2016). national college health assessment. retrieved from http://oucha.ca/ncha.php patall, e. k., cooper, h., & robinson, j. c. (2008). the effects of choice on intrinsic motivation and related outcomes: a meta-analysis of research findings. psychological bulletin, 134(2), 270-300. doi: 10.1037/0033-2909.134.2.270 pink, d. (2011). drive: the surprising truth about what motivates us, new york, ny: riverhead books. raue, k., & lewis, l. (2011). students with disabilities at degree-granting postsecondary institutions (nces 2011–018). u.s. department of education, national center for education statistics. washington, dc: u.s. government printing office. retrieved from https://nces.ed.gov/pubs2011/2011018.pdf rose, d. h., & strangman, n. (2007). universal design for learning: meeting the challenge of individual learning differences through a neurocognitive perspective. universal access in the information society, 5(4), 381-391. doi: https://doi.org/10.1007/s10209-006-0062-8 russell, m. (2000). it's time to upgrade: tests and administration procedures for the new millennium. the secretary's conference on education technology 2000, u.s., department of education. retrieved from https://files.eric.ed.gov/fulltext/ed452833.pdf schelly, c. l., davies, p. l., & spooner, c. l. (2011). student perceptions of faculty implementation of universal design for learning. journal of postsecondary education and disability, 24(1), 17-30. retrieved from https://accessproject.colostate.edu/assets/pdfs/schelly,%20davies,%20spo oner%202011.pdf statistics canada (2016). international students in canadian universities 20052005 to 2013-2014. retrieved from http://www.statcan.gc.ca/pub/81-599x/81-599-x2016011-eng.htm wagner, m., newman, l., cameto, r., garza, n., and levine, p. (2005). after high school: a first look at the postschool experiences of youth with disabilities. a report from the national longitudinal transition study-2 (nlts2) menlo park, ca: sri international. retrieved from www.nlts2.org/reports/2005_04/nlts2_report_2005_04_complete.pdf universal design for learning (udl): student and faculty perceptions 24 journal of effective teaching in higher education, vol. 1, no. 2 appendix a graphic organizer of udl framework (cast, 2011b). universal design for learning (udl): student and faculty perceptions 25 journal of effective teaching in higher education, vol. 1, no. 2 appendix b student items (experiment 1). dc connect is our college-branded learning management site (lms). present the same course content in multiple ways (graphics, video, text, graphic organizers/concept maps, etc.) offer an electronic version of the textbook post handouts on dc connect (or make them available digitally) include subtitles on videos (closed captioned) upload files can be read using text-to-speech software (e.g., word documents pdfs) provide clear guidelines for major assignments (e.g., example/sample assignment) include a field trip capture class lectures and made them available to stream after class (video or podcast) make available a glossary of terms (on dc connect, in the textbook, or other) offer alternatives for auditory info (e.g., transcripts of videos) and visual info (e.g., description of images) highlight patterns and relationships in the course content offer interesting and relevant major assignments allow for some autonomy and/or control in student learning (e.g., options for assignments (topic or format); or choices on tests (choose 1 of 2 essay questions; or pick 5 of the following terms to define) let students decide which topics are covered in the course use hands-on activities in class connect course content to real world experiences communicate with students (in class, outside of class, via message board or email) provide clear and specific feedback on assignments universal design for learning (udl): student and faculty perceptions 26 journal of effective teaching in higher education, vol. 1, no. 2 offer a choice of how students want to receive feedback on assignments (e.g., verbal or written feedback) allow students to re-submit assignments include peer-evaluation as part of the coursework make powerpoint slides available to students include group work and collaboration with other students (e.g., discussions) provide opportunities for self-assessment/self-evaluation and reflection answer questions about course content or assignments outside of class (e.g., discussion board, email) use gender-neutral language and inclusive examples (race/culture, etc.) minimize threats and distractions in the learning environment motivate students to do their best work flexible due dates on major assignments (e.g., allowed to turn it in late) offer ungraded or optional assignments to practice the course content provide sufficient (or unlimited) time for tests provide rubrics for major assignments guide you using increasingly difficult activities or assignments guide goal-setting and the development of student learning strategies provide opportunities for students to monitor progress (e.g., grades posted on dc connect) 1 corresponding author’s email: sheri.stover@wright.edu journal of effective teaching in higher education, vol. 1, no. 1 learner-centered design: is sage on the stage obsolete? sheri stover, ph.d. wright state university sharon g. heilmann, ph.d., wright state university amelia hubbard, ph.d., wright state university abstract. this quantitative research study examined one instructor’s redesign of her introductory anthropology course (n = 265) from teacher-centered (tc) to learning-centered (lc) and the resulting impact on her students’ perceptions of teaching presence (tp), social presence-interaction (sp-i), social presence participation (sp-p), cognitive presence (cp), and satisfaction (sat). using the community of inquiry (coi) survey (swan et al., 2008) in a face-to-face classroom environment; results indicated that implementing a lc classroom compared to a tc classroom was found to have a significantly positive impact on students’ perceptions of tp (p = .021), sp-i (p < .001), sp-p (p < .001), cp (p = 002), and sat (p = .022). multiple regression results indicated that tp, sp-i-, and sp-p were able to predict 42% of students’ level of satisfaction score with tp having the highest level of prediction (β=.37). preliminary evidence suggests that instructors who implement lc teaching methodologies can have a positive impact on tp, sp-i, sp-p, cp, and sat. keywords: learner-centered, community of inquiry, teaching presence, social presence, cognitive presence faculty in higher education are educated to be content experts upon graduation, yet few college professors take pedagogically-oriented classes or get experience on the best methods to convey their content expertise to their students (fertig, 2012). as a result, teacher-centered (tc) instruction is the most dominant form of instruction with more than half of faculty reporting a heavy reliance on lecture in “all” or “most” of the courses (eagan et al., 2014). tc instruction can be defined as a classroom teaching methodology that is designed so that the instructor is the person primarily in charge of providing direct instruction or lectures to a passive student audience with little input from the students (weimer, 2013a). scholarship on learning-centered (lc) instruction suggests that such approaches have many academic benefits including increased student retention and better preparation post-graduation (matlin, 2002; sternberg & grigorenko, 2002). lc instruction can be defined as learning that requires students to become active learners where they become more responsible for their own knowledge creation (weimer, 2013a). while there are benefits to a lc course design, faculty may feel nervous about moving away from a tc methodology, which can change the dynamics of classroom structure and create fear among faculty that such shifts may lead to poor student course evaluations (carrell & west, 2010; braga, paccagnella, & pellizzari, 2014). therefore, before faculty change teaching models, it is important to determine whether such a change has any effect. mailto:sheri.stover@wright.edu learner-centered design: is sage on the stage obsolete? 2 journal of effective teaching in higher education, vol. 1, no. 1 using the framework of the community of inquiry (coi), this study explores the effects of switching from a tc to lc model on students’ perceptions of teacher presence (tp), social presence (sp), cognitive presence (cp), and satisfaction (sat) in a large introductory anthropology course. the instructor for this course subscribed to the belief that, “education’s role is to challenge inequality and dominant myths rather than socialize students into the status quo. learning is directed toward social change and transforming the world, and ‘true’ learning empowers students to challenge oppression in their lives” (stage, muller, kinzie, & simmons, 1998, p. 57). to meet this educational goal the instructor, like others (eagen et al., 2014), opted to redesign her course from teacher-centered (tc) to learning-centered (lc) in an effort to allow students to take more control of their own academic development with the expectation that such a change would improve students’ perceptions of tp, sp, cp, and sat. literature review the foundation for lc teaching is built upon the social constructivism theory, which was developed by lev vygotsky (1978). vygotsky was a cognitivist who felt that students needed to be more responsible for constructing their own knowledge, and therefore could not passively listen to an allknowing instructor’s lecture. vygotsky also argued that learning cannot be separated from a social context so that students must be part of a learning community. he felt that learning is a collaborative process where students can have higher levels of development (zone of proximal development) with the help and support of teachers and peers. lc classrooms need to be designed with collaborative learning methodologies where students have opportunities to work together. further, lc classrooms need to be designed so that activities are complex and relevant, authentic, require social negotiation, encourage multiple perspectives, require students’ ownership of learning, and encourage self-awareness of knowledge construction (reynolds & kearns, 2017). in a lc classroom, students take more responsibility for their own knowledge creation. lc learning environments allow students to work collaboratively in a community of inquiry with the other participants in the class to help negotiate meaning of the course content, diagnose existing misconceptions, and challenge any currently held biases to move toward deeper and more meaningful learning outcomes (garrison, 2011; vega & tayler, 2005). in a tc classroom, the teacher is the sole leader and the person taking the active role in presenting lectures. the students ‘primary job in the classroom is to listen and to take notes (rogers & frieberg, 1994). however, in lc classrooms, learning is the responsibility of the group who form a community of inquiry. garrison (2011) defines a community of inquiry as “a group of individuals who collaboratively engage in purposeful critical discourse and reflection to construct personal meaning and confirm mutual understanding” (p. 15). the community of inquiry (coi) is a theoretical framework that emphasize the need for effective educational experiences to include high levels of the three interdependent elements of teaching presence (tp), social presence (sp), and cognitive presence (cp). tp can be defined as “the design, facilitation and direction of cognitive and social processes learner-centered design: is sage on the stage obsolete? 3 journal of effective teaching in higher education, vol. 1, no. 1 for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes” (garrison, 2011, p. 24). sp can be defined as “the ability of participants to identify with a group, communicate purposefully in a trusting environment, and develop personal and affective relationships progressively by way of projecting their individual personalities” (p. 23). the final coi element, cp can be defined as “the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical community of inquiry” (p. 24). students’ perception of teaching presence may change when a class moves toward a lc design. students’ attention is focused on the instructor in the tc classroom because the instructor is the “centerpiece of instruction” (mccarthy & anderson, 2000, p. 279). the instructor’s role in a tc classroom is to be the expert who disseminates information to students usually using a lecture teaching methodology. students have had years of experiences with tc learning environments and have come to believe it is the responsibility of the instructor to take control of the teaching (howard & baird, 2000). the advantage of teachers maintaining the expert role is the respect given by students for sharing their expertise in class lectures. the disadvantage of teacher as expert is that students rely on the instructor to disseminate information and can be intimidated to take ownership of their own knowledge creation (roberson, 2014). the instructor’s role changes in a lc classroom. knowledge creation is no longer the sole responsibility of the instructor, and students become partners in their own knowledge creation. wood (1994) found that students go through all the steps psychologists associate with trauma when they are forced to assume more control of their learning. they will go through shock, denial, strong emotion, resistance and withdrawal, surrender and acceptance, struggle and exploration, return of confidence, and integration and success. students’ perception of social presence may also be impacted when moving toward a lc design. students have few opportunities to interact with other students in tc classrooms as most instruction is direct lecture transmitted from the instructor. in a lc classroom, the class works collaboratively as a community of inquiry (garrison, 2011) to help each other make sense of what they are learning (brophy, 1999). lipman (2003) discusses the power of having students work as a community of inquiry because students can build on each other’s ideas and can help each other to draw inferences. lipman identifies some of the class characteristics when students work as a community of inquiry to include inclusiveness, participation, shared cognition, relationships, quest for meaning, feelings of social solidarity, deliberation, challenging, reasonableness, questioning, and deciding. there have been many research studies to show the benefits of students working collaboratively to include social benefits (social support, building diversity, developing learning communities), psychological benefits (increased self-esteem, reduced anxiety, positive attitude toward instructors), and academic benefits (promotes critical thinking skills, actively involves students in the learning process, classroom results improved, models problem solving, personalizes large lectures, increased student motivation) (laal & ghodsi, 2011). learner-centered design: is sage on the stage obsolete? 4 journal of effective teaching in higher education, vol. 1, no. 1 implementing a lc classroom can have an impact on students’ cognitive presence. lc classrooms are designed so students are more actively involved in their own learning and have opportunities to reflect and regulate their learning, so this results in more effective learning (lambert & mccombs, 2000) research has shown that students who have higher personal involvement in their own knowledge creation have higher levels of personal commitment and confidence which leads to higher levels of student achievement (alexander & murphy, 2000). bandura (1963) developed the social learning theory, where he posits that learning can only take place in a social context where students have an opportunity to observe and then model behaviors. therefore, lc teaching has a positive impact on students’ cognitive development. the adoption of lc teaching methodologies may also have an impact on students’ level of satisfaction. lc education can increase students’ levels of satisfaction due to having more control over their own learning (alexander & murphy, 2000). however, not all students react well to lc teaching methodologies. by the time students come to higher education, many have a fixed mindset toward tc classrooms where they view the students’ role as passively taking notes, listening to lectures, reading the course material, and taking tests. in a tc classroom students mostly work along and only seek the instructor if they have questions or issues (doyle, 2008). howard and baird (2000) found that some students resist becoming active participants in a lc classroom because they feel they are the paying customer and the instructor is the one being paid to provide the information. weimer (2014) suggested that students resist group work because they do not value the input of other students, and only value the faculty member because he/she is considered the expert knowledge provider. instructors may be hesitant to incorporate lc teaching strategies such as group work into their classes due to issues such as free riding where only a few members do most of the work (brooks & ammons, 2003), social loafing where the group exerts less work than they would individually since they feel a reduced accountability (kao, 2013), and conflict when group members cannot work together due to personality issues (taylor, 2011). students may report disliking group work because it requires students to do the hard work instead of passively listening to the instructor. students may resent being required to struggle through the material instead of having it spoon-fed to them by the instructor. they may also dislike being accountable to their group members which means they will need to complete group prep work and cannot sleep during class (taylor, 2011). research hypothesis this research study examines the impact of one instructor’s redesign of her introductory anthropology course from tc to lc. this study will examine the impact that the redesign from tc instruction to lc instruction had on her students’ perceptions of teacher presence (tp), social presence-interaction (sp-i), social presence-participation (sp-p), cognitive presence (cp), and students’ level of satisfaction (sat). h1: students perception of teaching presence (tp) will be positively impacted by a learner centered (lc) course design. learner-centered design: is sage on the stage obsolete? 5 journal of effective teaching in higher education, vol. 1, no. 1 h2: students perception of social presence-interaction (sp-i) will be positively impacted by a learner-centered (lc) course design. h3: students perception of social presence-participation (sp-p) will be positively impacted by a learner-centered (lc) course design. h4: students perception of cognitive presence (cp) will be positively impacted by a learner centered (lc) course design. h5: students level of satisfaction (sat) will be positively impacted by a learner centered (lc) course design. h6: students’ perceptions of teaching presence (tp), social presenceinteraction (sp-i), social presence-participation (sp-p), and cognitive presence (cp) will positively impact students’ perceptions of satisfaction (sat) such that increased tp, sp-i, sp-p, and cp will have a corresponding, positive increase in perceptions of students’ sat correlation in predicting students’ sat. methodology participants the participants in this study were enrolled in an introductory anthropology class at a medium sized midwestern university. while there were 265 students enrolled in these classes; absenteeism, opting out, or incomplete surveys resulted in 143 completed surveys, for a 54% response rate. students identified as female (n = 76), male (n = 62) and other (n = 1). the majority of students were in the 18-24 range (n = 117), with other students in the age ranges of 25-30 (n = 7), 31-40 (n = 5), 41-50 (n = 2), and 50+ (n = 3). most participants described themselves as caucasian (64.3%), other (15.7%), black/african american (10.7%), asian (5.7%), and hispanic/latino (3.6%). students identified themselves as freshman (33.3%), sophomore (31.1%), junior (17%), and senior (18.5%). there were 11.8% of the students who reported being an international or foreign national student. class structure the professor teaching these classes participated in a year-long learning community designed to help her learn how to redesign her classes from passive tc format to a lc active-learning format. a researcher, other than the course instructor, observed a class during the 2015 spring term, the 2015 fall term, and the 2016 spring term. the outside researcher took detailed notes by scripting class activities and documenting times for activities. the scripted class activities were then coded by the researcher into tc or lc activities. tc activities were those activities where the instructor was almost entirely in charge and usually conducting a lecture. the lc activities were those activities where the student was primarily in charge by completing activities such as a quiz or group project. activities where there was a group discussion where all were participating were equally divided between tc and lc. learner-centered design: is sage on the stage obsolete? 6 journal of effective teaching in higher education, vol. 1, no. 1 table 1: class descriptive data class term n teacher-centered learner-centered class #1 2015 spring 99 82% 18% class #2 2015 fall 72 32.5% 67.5% class #3 2016 spring 94 38% 62% class #1 spring 2015: this class was taught in a fixed-seat auditorium style classroom. the class was designed with primarily a tc design where the instructor conducted her classes showing powerpoint slides accompanied by an instructor lecture. students were passive learners for the majority of the class as their primary role was to listen to the instructor lecture and take notes. most of the class was instructor lecture, and there was some opportunity for interaction with instructor questions and clicker polls. class #2 fall 2015 and class #3 spring 2016: these classes were taught in a classroom specifically designed to support active learning activities. these classes were primarily taught in a lc design where the responsibility of learning was shared with the students. students sat in groups of six at a round table where each student had his or her own laptop. the classroom was equipped with whiteboards throughout the classroom to allow group work; software on the laptops to permit active learning activities such as polling, quizzing, etc.; software on the computers that permitted the instructor to display individual and group work; software on the computers that permitted the instructor to manage and monitor all classroom computers; and a swipe system to monitor class attendance. the instructor completely redesigned her class toward more lc instruction where students were actively engaged throughout the class. the instructor conducted short min-lectures, and students would apply their knowledge many times during class by completing individual quizzes, group quizzes, group problem-solving at the whiteboards, group work, individual assignments, polling questions, class questions, and other activities. instrument students in this research study were given a survey that included the community of inquiry (coi) survey (swan et al., 2008) then coi framework was originally developed to guide research in online learning environments (garrison, anderson, & archer, 2000). however, since the coi attempts to measure learning environments that “support discourse and reflection in a community of inquiry” (arbaugh et al., 2008), the survey was consistent with the goals of this research study and was chosen to utilize for this research study. the coi survey was slightly modified to be administered to students so it was appropriate for a face-to-face classroom environment (see appendix a). swan et al. (2008) validated that the coi survey appropriately measured for the three factors of teaching presence (tp), social presence (sp), and cognitive presence (cp) when delivered in an online learning environment. stover and ziswiler (2017) used the coi survey in a research study conducted in a face-to-face learning environment that was conducted with students in 7 different classes with 417 students. stover and ziswiler conducted an learner-centered design: is sage on the stage obsolete? 7 journal of effective teaching in higher education, vol. 1, no. 1 exploratory factory analysis (efa) with principal axis factoring and varimax rotation on the data gathered from results of the coi survey to reduce the number of questions and determine the number of factors (norris & lecavalier, 2010). items with a primary factor load of .4 or below and items with higher than a .32 cross-loading were removed (tabachnick & fidell, 2012). the stover and ziswiler (2017) efa resulted with two different factors for the sp. one sp group (q14, q15, q16) pertained to questions about students interactions in the class, so this factor was renamed social presence-interaction (sp-i). the other sp group (q17, q18) pertained to questions about students participation in the class, so this factor was renamed social presence-participation (sp-p). the efa also resulted in a reduction of questions. therefore, the revised four factor groupings and the reduced number of questions in each factor were used to analyze the data for this research study as depicted in table 2. table 2: coi survey item groupings after factor analysis teaching presence tp social presence interaction sp-i social presence participation sp-p cognitive presence cp q1 q2 q3 q4 q5 q6 q8 q9 q13 q14 q15 q16 q17 q18 q32 q33 q34 note. adapted from stover, s. e., & ziswiler, k. (2017, in press). impact of active learning environments on community of inquiry. international journal of teaching and learning in higher education (29)3. the survey administered to the students in this research study also included a student satisfaction questionnaire. the semantic differential technique (osgood, suci, & tannenbaum, 1957) was used so that the students were asked to select between 15 sets of bipolar adjectives (for example, anxiety or security) to express their feelings while participating in the class. eight students outside the class enrollees were given a mixed up list of adjectives and asked to select the bipolar opposites. results indicated 100% agreement on 7 terms, 87.5% agreement on 5 terms, 75% agreement on 1 term, and 62.5% agreement on 2 terms. exploratory factor analysis (efa) with principal axis factoring and varimax rotation was used to identify the underlying relationships between the student satisfaction adjective items (norris & lecavalier, 2010). in an effort to identify one group of adjectives to represent a student satisfaction, items were removed until all remaining items had a primary factor load of .4 or above and did not have any cross-loading of higher than .32 to ensure adequate item communalities (costello & osborne, 2005). after removing those items that did not meet the specified criteria, eight items bi-polar adjective remained to represent the factor of student satisfaction as shown in table 3. learner-centered design: is sage on the stage obsolete? 8 journal of effective teaching in higher education, vol. 1, no. 1 the survey was given to students during the final week of the academic semester and was administered by a researcher other than the instructor to ensure students’ anonymity. students completed the scantron survey and results were tabulated and saved to an excel spreadsheet that was exported to spss for analysis. table 3: student survey items after factor analysis items factor load 1. dissatisfaction – satisfaction .90 2. defeat – success .85 3. lack of confidence – confidence .84 4. anxiety – security .81 5. confusion – clarity .80 6. frustration – well being .77 7. disconnected connected .75 8. bored excited .67 bi-polar adjectives scored by students using a seven-point semantic differential technique results the revised coi survey factors and revised student satisfaction (sat) survey items were used to compare classes for tp, sp-i, sp-p, cp, and sat. the cronbach alpha scores indicated good to excellent levels of consistency (devellis, 2012) for tp (α = .90), sp-i (α = .80), sp-p (α = .83), cp (α = .84), sat (α = .93). shapiro-wilk’s test (p < .05) (razali & wah, 2011) and a visual inspection of their histograms, normal q-q plots and box plots indicated the tp, sp-i, sp-p, cp, and sat scores did not appear to be normally distributed. therefore, the kruskal-wallis non-parametric method was used for analysis to compare tp, sp-i, sp-p, cp, and sat scores between classes. h1 teaching presence (tp) a kruskal-wallis h test showed that there was a statistically significant difference in tp scores between the three classes, χ2 (2) = 9.180, p = 0.01, with a mean rank tp score of 68.35 for class #1, 62.66 for class #2, and 87.82 for class #3. post hoc tests were conducted to evaluate pairwise differences among the three groups. the results of these tests indicated a significant difference between class #1 and class #3 (p = .021) and class #2 and class #3 (p = .004). therefore, the lc classroom appeared to have a positive impact on students’ perception of tp for class #3. see table 4 for all hypotheses results. h2 social presence: interaction (sp-i) a kruskal-wallis h test showed that there was a statistically significant difference in sp-i scores between the three classes, χ2 (2) = 26.742, p = .000, with a mean rank sp-i score of 49.28 for class #1, 76.30 for class #2, and 93.19 for class #3. post hoc tests were conducted to evaluate pairwise differences among the three groups. the results of these tests indicated a significant difference between class #1 and learner-centered design: is sage on the stage obsolete? 9 journal of effective teaching in higher education, vol. 1, no. 1 class #2 (p = .001), a significant difference between class #1 and class #3 (p = .001p cannot be 0), and a significant difference between class #2 and class #3 (p = .028). therefore, the lc classroom had a positive impact on students’ perception of sp-i for class #2 and class #3. table 4: class impact on tp, sp-i, sp-p, and cp class n mean rank class n mean rank χ2 eta2 p teaching presence class #1 49 53.27 class #2 52 48.87 .570 .006 .45 class #1 49 40.08 class #3 42 52.90 5.352 .059 .02* class #2 52 40.30 class #3 42 56.42 8.155 .088 .004* social presence-i class #1 49 40.62 class #2 52 60.78 12.163 .122 .001*** class #1 49 33.65 class #3 42 60.40 23.593 .262 .001*** class #2 52 42.02 class #3 42 54.29 4.812 .052 .021* social presence-p class #1 49 45.11 class #2 52 56.55 4.003 .040 .05* class #1 49 35.94 class #3 42 57.74 15.966 .177 .000*** class #2 52 41.00 class #3 42 55.55 6.917 .074 .01** cognitive presence class #1 49 49.54 class #2 52 52.38 .239 .002 .63 class #1 49 38.30 class #3 42 54.99 9.148 .102 .01** class #2 52 40.91 class #3 42 55.65 6.882 .074 .01** satisfaction class #1 38 43.72 class #2 52 46.80 .305 .003 .58 class #1 38 33.84 class #3 41 45.71 5.281 .068 .02* class #2 52 42.07 class #3 41 53.26 3.948 .043 .05* *significant at < 05 level; **significant at < .01; ***significant at <.001 h3 social presence: participation (sp-p) a kruskal-wallis h test showed that there was a statistically significant difference in sp-p scores between the three classes, χ2 (2) = 17.473, p = .001, with a mean rank sp-p score of 56.05 for class #1, 71.05 for class #2, and 91.79 for class #3. post hoc tests were conducted to evaluate pairwise differences among the three groups. the results of these tests indicated a significant difference between class #1 and class #2 (p = .045), a significant difference between class #1 and class #3 (p = .001), and a significant difference between class #2 and class #3 (p = .009). therefore, the lc classroom appeared to have a positive impact on students’ perception of sp-p for class #2 and class #3. h4 cognitive presence (cp) a kruskal-wallis h test showed that there was a statistically significant difference in cp scores between the three classes, χ2 (2) = 10.536, p = .005, with a mean rank cp score of 62.84 for class #1, 66.79 for class #2, and 89.14 for class #3. post hoc tests were conducted to evaluate pairwise differences among the three groups. learner-centered design: is sage on the stage obsolete? 10 journal of effective teaching in higher education, vol. 1, no. 1 the results of these tests indicated a significant difference between class #1 and class #3 (p = .002), and a significant difference between class #2 and class #3 (p = .009). therefore, the lc classroom appeared to have a positive impact on students’ perception of cp for class #3. h5 student satisfaction a kruskal-wallis h test showed that there was a statistically significant difference in student satisfaction scores between the three classes, χ2 (2) = 6.219 p = .045, with a mean rank student satisfaction score of 58.07 for class #1, 62.37 for class #2, and 77.96 for class #3. post hoc tests were conducted to evaluate pairwise differences among the three groups. the test did indicate a significant difference between class #2 and class #3 (p = .047) and then also between class #1 and class #3 (p = .022). therefore, the lc classroom had a positive impact on student satisfaction for class #3. h6 prediction of students satisfaction level a multiple stepwise regression was computed in order to evaluate students’ level of satisfaction based on their perception of tp, sp-i, sp-p, and cp. results indicated an adjusted r2 of 42%, a moderate to large size effect according to cohen (1988). the multiple regression model statistically significantly predicted satisfaction level, f(4, 129) = 32.430, p < .0001, adj. r2 = .420. three of the four variables (tp, spi, sp-p) added statistically significantly to the prediction, p < .001. regression coefficients and standard errors can be found in table 5. as depicted in table 5, tp (β=.37) was the most significant contributor in the model. tp, sp_p, and sp_i accounted for 42% of the explained variance, and cp was excluded from the stepwise regression computation. all betas for the remaining variables were statistically significant at the p < 05 level.. further, collinearity indices, variance inflation factor, and tolerance indicators all met rule-of-thumb requirements, indicating the model was not significantly impacted by collinearity (best & wolf, 2015). table 5: summary of linear regression with stepwise entry predictor ∆r2 β step 1 .29 tp .54** step 2 .39 tp .42** sp-p .35** step 3 .42 tp .37** sp-p .24* sp-i .22* total adj r2 .42** n 131 *p < .01; **p < .001 learner-centered design: is sage on the stage obsolete? 11 journal of effective teaching in higher education, vol. 1, no. 1 discussion the goal of this research paper was to examine the impact of a redesign from tc to lc instruction on students’ perceptions of tp, sp-i, sp-p, cp, and sat. implementing learner-centered instructional strategies requires many instructors to change their teaching paradigms and to redesign their classes to shift more of the responsibility of learning to the students. students may resist moving to a lc design due to the requirement of increased cognitive efforts, student dependency on instructors doing most of the work, and resistance to working with other students (tolman & kremling, 2017). instructors worry switching to a lc class format may result in lower student evaluations due to student resistance to the lc design. in order for instructors to begin to adopt lc teaching philosophies, they need to have training and support while redesigning their classes. the instructor for these introductory anthropology courses joined a year-long learning community facilitated by the university learning teaching center that helped teach her strategies about redesigning her class for more lc active-learning methodologies. the training helped her redesign her class from the tc design in class #1 to the lc design in class #2. the redesign of her courses resulted in an increased in students’ perception of cp and sat and also increases in sp-i and sp-p at significant levels. while the instructor was teaching class #2, she asked a teaching assistant (ta) to take notes for any issues or concerns with each class that were brought up by class participants and observed by the ta. these ta observations led to the instructor making about 100 tweaks to the class before she taught class #3. none of these ta observations resulted in changes to entire activities but more minor modifications to instructions and directions to add clarity or to clear up confusion that was noted during the previous class. the instructor again taught the anthropology class using the lc design for class #3 during the 2016 spring term. while the activities and class structure remained the same as class #2, the suggestions by the ta during the previous term resulted in clearer instructions and directions for class activities. these updates were noticeable by students and resulted in less confusion by the students. class #3 resulted in improved scores at significant levels from class #1 in tp (p = .021), spi (p =<.001), sp-p (p < .001), cp (p = .002), and sat (p = .022), as indicated in table 4. while the class activities remained almost exactly the same from class #2 to class #3, the instructor tweaks and clarifications resulted in improved scores from class #2 to class #3 at significant levels in tp (p = .04), sp-i (p = .028), spp (p = .009), cp (p = .009) and sat (p = .047), depicted in table 4. since the course activities, course assignments, and course assessments remained the same from class #2 and for class #3, it can be argued that the significant increases in tp, sp-i, sp-p, cp, and sat may be impacted by the instructor’s course redesign. it also shows that implementing the class designed with learner-centered principles may take a semester of teaching the same course before the instructor has mastered the new teaching philosophies and make improvements to course activities. the changes implemented by the instructor for her third time teaching learner-centered design: is sage on the stage obsolete? 12 journal of effective teaching in higher education, vol. 1, no. 1 resulted in improvements in students’ perceptions of tp, sp-i, sp-p, cp, and sat at significant levels, as depicted in table 4. instructors that wish to move from a tc to a lc design will reformat their courses so that students are required to take more responsibility for their own learning. this means students will be the ones to do their own knowledge creation instead of coming to class and passively listening to instructor lectures. instructors can hold students accountable for completing course assignments by requiring students to complete assessments at the start of each class with activities such as clicker questions or learning management system (lms) quizzes. students can use their cell phones to respond to clicker questions or complete lms quizzes if they are in large auditorium classrooms without access to laptop computers. incorporating educational technologies such a clickers and lms quizzes allow assessments to be implemented in large classes due to the immediate automated grading capabilities that can provide immediate feedback to students. the inclusion of frequent assessment activities allows students to take more ownership of their own learning because it gives them frequent feedback about their learning progress so they can learn from their mistakes (blumberg, 2015). instructors can include lc activities during class by conducting short mini-lectures and then include case studies where students are required to work together to apply the course concepts to solve problems. the case studies should not have one correct answer, but be open-ended responses that empower students to construct their own meaning to solve the problem (blumberg, 2015). instructors that do not have access to robust course redesign workshops may need to develop other support structures. faculty can connect with other like-minded colleagues on campus to create their own learning community. the advantage of creating a faculty learning community is to share best practices, share support resources, and brainstorm ideas to overcome obstacles. faculty can also attend an academic conference focused on teaching and learning to come away with strategies for lc class design. while lc instruction requires students to become more actively involved in their own knowledge construction, it does not diminish the role of the instructor or make them obsolete. the instructor is working just as hard (and probably harder) before each class to design class activities, build quizzes, structure group projects, and other lc activities. the instructor also continues to conduct short min-lectures during class to provide meaning, add context, provide instruction, and clarify any misconceptions. the instructor also needs to continue to update the lc activities to make modifications when students are confused, make the activities more productive and effective, and keep them updated. the instructor’s role changes in lc classrooms, but the instructor’s presence continues to have a substantial influence on students’ perceptions of sp-i, sp-p, and cp. therefore, when designing lc classrooms, the design should always start with the instructor, and the importance of keeping the instructor involved in the learning process should not be forgotten. the lc model allows faculty and students to work together more interactively instead of limited interaction in a lecture environment. learner-centered design: is sage on the stage obsolete? 13 journal of effective teaching in higher education, vol. 1, no. 1 weimer (2013b) points out that academics tend to see teachers as either having a philosophy of tc with lectures (currently considered bad) or a lc teacher who actively engages students (which is now considered good). instead of believing that tc instruction is juxtaposed against lc, perhaps a combination of both teaching philosophies can have the biggest impact on students’ learning. instructors can include short mini-lectures when they need to intervene and set boundaries to allow students to make choices and assume more responsibility of their own learning. cox and yearwood (2014) make the case that teachers are multifaceted individuals and feel that students need to know instructors not only as course designers, but also as sages who push students to develop deeper and more critical thinking skills. perhaps the best mode of instruction is to have a combination of tc and lc instruction. faculty have terminal degrees in their discipline area and often do not have many opportunities to take courses in effective course design. faculty may be nervous about receiving poor student evaluations with lc course designs (carrell & west, 2010; braga et al., 2014) due to negative perceptions caused by changes in instructor roles, increased requirements for more group work, impact on students’ beliefs on knowledge creation, and decreased levels of satisfaction because of increased requirements to take control of their own learning. these results support the expectations that, in this course, the instructor’s redesign to lc teaching methodologies had a positive impact on students’ perceptions of tp, sp-i, sp-p, cp, and sat. the results also show that we cannot forget the importance of the instructor on students in these areas as the multiple regression analysis showed that the presence of a teacher had the largest impact on students’ level of satisfaction. while students assume more responsibility for their knowledge creation in lc classrooms, the presence of the instructor continues to have a substantial impact on students’ level of satisfaction. study limitations and further areas of study it is impossible to have perfectly controlled conditions in a live classroom. conducting research in an authentic educational classroom opens up the research study to commonly occurring fluctuations that cannot be controlled such as technology issues, lack of control of student enrollment, term influences (fall, spring), external influences (politics, current events), and a plethora of other issues that instructors are faced with every day while teaching their classes. these fluctuations certainly have an impact on students’ perceptions of tp, sp-i, sp-p, cp, and sat. however, it is critical that research studies are conducted live classrooms to measure students’ perceptions of design change in authentic environments. the data were gathered at one mid-western, mid-sized institution that has 96% acceptance rates. therefore, it is unknown if the educational preparation or willingness to change for the students in this research study had an impact on their perceptions; therefore, the results cannot be generalized to other locations. further research involving control and experimental groups is warranted. the coi survey was originally designed for use in online classroom settings. stover and ziswiler (2017) conducted a factor analysis on data from a study in a face-tolearner-centered design: is sage on the stage obsolete? 14 journal of effective teaching in higher education, vol. 1, no. 1 face classroom using the community of inquiry (coi) survey (swan et al., 2008). the factor analysis resulted in only 17 of the 34 questions meeting the guidelines for an exploratory factor analysis. instead of the three original factors identified in the coi for delivery in online settings, the factor analysis identified one additional factor when implemented in face-to-face classroom settings. the questions on the coi needed to be slightly modified to be appropriate for a class in a face-to-face teaching environment. therefore, there need to be additional studies on the modified version of the coi survey to evaluate its validity in face-to-face classroom environments. while the results of this research study cannot be generalizable, these data provide preliminary evidence that faculty wishing to migrate from tc to lc teaching methodologies can have a positive impact on their students’ perceptions of tp, spp, sp-i, cp, and sat. an additional area of research could be evaluating the impact of the lc classroom on students’ learning, drop rates, and impact on minority students. conflicts of interest the author(s) declare(s) that there is no conflict of interest regarding the publication of this article. learner-centered design: is sage on the stage obsolete? 15 journal of effective teaching in higher education, vol. 1, no. 1 references alexander, p., & murphy, p. (2000). the research base for apa's leaner-centered psychological principles. in n. lambert, & b. mccombs (eds.), how students learn (pp. 25-60). washington, d.d.: american psychological association. arbaugh, j., cleveland-innes, m., diaz, s., garrison, d. r., ice, p., richardson, j., & swan, k. (2008). developing a community of inquiry instrument: testing a measure of the community of inquiry framework using a multi-institutional sample. the internet and higher education, 11(3–4), 133–136. bandura, a. (1963). social learning and personality development. new york: holt, rinehart, and winston. best, h. & wolff, c. (2015). the sage handbook of regression analysis and causal inference. thousand oaks: sage publications blumberg, p. (2015). how critical reflection benefits faculty as they implement learner-centered teaching. new directions for teaching and learning, 2015(144), 87-97. doi:10.1002/tl.20165 braga m., paccagnella m., pellizzari m. (2014). evaluating students’ evaluations of professors. economics of education review, 41, 71–88. https://doi.org/10.1016/j.econedurev.2014.04.002 brooks, c. m., & ammons, j. l. (2003). free riding in group projects and the effects of timing, frequency, and specificity of criteria in peer assessments. the journal of education for business, 78, 268–272. brophy, j. (1999). perspectives of classroom management: yesterday, today and tomorrow. in h. freiberg (ed.), beyond behaviorism: changing the classroom management paradigm, 43-56. boston: allyn and bacon. carrell s. e., & west j. e. (2010). does professor quality matter? evidence from random assignment of students to professors. journal of political economy, 118(3), 409–432. doi: 10.1086/653808 cohen, j. (1988). statistical power analysis for the behavioral sciences ( 2nd edition). hillsdale, nj: lawrence erlbaum associates. costello, a. b., & osborne, j. w. (2005). best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. practical assessment, research & evaluation, 10(7), 1-9. cox, j. r., & yearwood, d. (2014, march 21). in defense of teaching. faculty focus. retrieved from http://www.facultyfocus.com/articles/philosophy-ofteaching/defense-teaching/ devellis, r.f. (2012). scale development: theory and applications. los angeles: sage. doyle, t. (2008). helping students learn in a learner-centered environment. sterling, va: stylus. eagan, m. k., stolzenberg, e. b., berdan lozano, j., aragon, m. c., suchard, m. r. & hurtado, s. (2014). undergraduate teaching faculty: the 2013–2014 heri faculty survey. los angeles: higher education research institute, ucla. retrieved from http://www.heri.ucla.edu/monographs/heri-fac2014monograph.pdf fertig, j. (2012, march 29). how do professors learn to teach (or do they)? the john william pope center for higher education policy. retrieved from http://www.facultyfocus.com/articles/philosophy-of-teaching/defense-teaching/ http://www.facultyfocus.com/articles/philosophy-of-teaching/defense-teaching/ http://www.heri.ucla.edu/monographs/heri-fac2014-monograph.pdf http://www.heri.ucla.edu/monographs/heri-fac2014-monograph.pdf learner-centered design: is sage on the stage obsolete? 16 journal of effective teaching in higher education, vol. 1, no. 1 http://www.popecenter.org/2012/03/how-do-professors-learn-to-teach-ordo-they/ garrison, d. r., anderson, t., & archer, w. (2000). critical inquiry in a text-based environment: computer conferencing in higher education. the internet and higher education, 2(2-3), 87-105. garrison, d. r. (2011). e-learning in the 21st century: a framework for research and practice (second edition). new york, ny: routledge. howard, j. r., & baird, r. (2000). the consolidation of responsibility and students’ definitions of situation in the mixed-age college classroom. the journal of higher education, 71(6), 700-721. lambert, n., & mccombs, b. (2000). introduction: learner-centered schools and classrooms as a direction for school reform. in n. lambert, & b. mccombs (eds.), how students learn (pp. 1-15). washington, d.c.: american psychological association. laal, m., & ghodsi, s. m. (2011). benefits of collaborative learning. social and behavioral sciences , 31, 486-490. lipman, m. (2003). thinking in education (2nd edition). new york, ny: cambridge university press. matlin, m. w. (2002). cognitive psychology and college-level pedagogy: two siblings that rarely communicate. in d. f. halpern, & m. d. hakel (eds.), applying the science of learning to university teaching and beyond. (pp. 87103). san francisco: jossey-bass. mccarthy, j.p., & anderson, l. (2000). active learning techniques versus traditional teaching styles: two experiments from history and political science. innovative higher education, 24(4), 279-294. norris, m., & lecavalier, l. (2010). evaluating the use of exploratory factor analysis in developmental disability psychological research. journal of autism and developmental disorders, 40(1), 8-20. doi:10.1007/s10803-009-0816-2 osgood, c. e., suci, g. j., & tannenbaum, p. h. (1957). the measurement of meaning. urbana, il: university of illinois press. razali, n. m., & wah, y. b. (2011). power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. journal of statistical modeling and analytics, 2(1), 21-33. reynolds, h. l, & kearns, k. d. (2017). a planning tool for incorporating backward design, active learning, and authentic assessment in the college classroom. college teaching, 65(1), 17-27. https://doi.org/10.1080/87567555.2016.1222575 roberson, s. (2014). improving teaching and learning: three models to reshape educational practice. education, 134(3), 340-358. rogers, c., & freiberg, j. (1994). freedom to learn (3rd ed.). upper saddle river, nj: merrill publishing. stage, f. k., muller, p. a., kinzie, j., and simmons, a. (1998). creating learner centered classrooms: what does learning theory have to say? washington, d.c.: eric clearinghouse on higher education and the association for the study of higher education. sternberg, r. j., & grigorenko, e. l. (2002). the theory of successful intelligence as a basis for instruction and assessment in higher education. in d. f. halpern, http://www.popecenter.org/2012/03/how-do-professors-learn-to-teach-or-do-they/ http://www.popecenter.org/2012/03/how-do-professors-learn-to-teach-or-do-they/ learner-centered design: is sage on the stage obsolete? 17 journal of effective teaching in higher education, vol. 1, no. 1 & m. d. hakel (eds.), applying the science of learning to university teaching and beyond (pp. 45-54). san francisco: jossey-bass stover, s. e., & ziswiler, k. (2017, in press). impact of active learning environments on community of inquiry. international journal of teaching and learning in higher education, 29(3). swan, k. p., richardson, j. c., ice, p., garrison, r., cleveland-innes, m., & arbaugh, j. b. (2008). validating a measurement tool of presence in online communities of inquiry. e-mentor, 2(24), 1-12. tabachnick, b. g., & fidell, l. s. (2012). using multivariate statistics (sixth edition). upper saddle river, nj: pearson education, inc. taylor, a. (2011). top 10 reasons students dislike working in groups…and why i do it anyway. biochemistry and molecular biology education, 39(2), 219-220. tolman, a. o., & kremling, j. (2017). why students resist learning: a practical model for understanding and helping students. sterling, va: stylus. vega, q. c. & tayler, m. r. (2005). incorporating course content while fostering a more learner-centered environment. college teaching, 53(2), 83-86. vygotsky, lev (1978). mind in society. london: harvard university press. weimer, m. (2013a). learner-centered teaching: five key changes to practice. san francisco, ca: jossey-bass. weimer, m. (2013b, january 16). teacher-centered, learner-centered or all of the above. faculty focus. retrieved from http://www.facultyfocus.com/articles/teaching-professor-blog/teachercentered-learner-centered-or-all-of-the-above/ weimer, m. (2014, april 23). class discussion challenge: getting students to listen and respond to each other’s comments. faculty focus. magna publications. wood, d.r. (1994). problem-based learning: how to gain the most from pbl. waterdown, ontario: donald r. woods. http://www.facultyfocus.com/articles/teaching-professor-blog/teacher-centered-learner-centered-or-all-of-the-above/ http://www.facultyfocus.com/articles/teaching-professor-blog/teacher-centered-learner-centered-or-all-of-the-above/ learner-centered design: is sage on the stage obsolete? 18 journal of effective teaching in higher education, vol. 1, no. 1 appendix a table a1: coi survey with revised questions teaching presence 1. *the instructor clearly communicated important course topics. 2. *the instructor clearly communicated important course goals. 3. *the instructor provided clear instructions on how to participate in course learning activities. 4. *the instructor clearly communicated important due dates/time frames for learning activities. 5. *the instructor was helpful in identifying areas of agreement and disagreement on course topics that helped me to learn. 6. *the instructor was helpful in guiding the class towards understanding course topics in a way that helped me clarify my thinking. 7. the instructor helped to keep course participants engaged and participating in productive dialogue. 8. *the instructor helped keep the course participants on task in a way that helped me to learn. 9. *the instructor encouraged course participants to explore new concepts in this course. 10.instructor actions reinforced the development of a sense of community among course participants. 11.the instructor helped to focus discussion on relevant issues in a way that helped me to learn. 12.the instructor provided feedback that helped me understand my strengths and weaknesses. 13.*the instructor provided feedback in a timely fashion. social presence 14.*getting to know other course participants gave me a sense of belonging in the course. 15.*i was able to form distinct impressions of some course participants. revised: i was able to form distinct impressions (ideas, feelings, or opinions) of some course participants. 16.*online or web-based communication is an excellent medium for social interaction. revised: class discussions are an excellent tool for social interaction. 17.*i felt comfortable conversing through the online medium. revised: i felt comfortable talking during class. 18.*i felt comfortable participating in the course discussions. 19.i felt comfortable interacting with other course participants. 20.i felt comfortable disagreeing with other course participants while still maintaining a sense of trust. 21.i felt that my point of view was acknowledged by other course participants. 22.online discussions help me to develop a sense of collaboration. revised: class discussions help me to develop a sense of collaboration. learner-centered design: is sage on the stage obsolete? 19 journal of effective teaching in higher education, vol. 1, no. 1 cognitive presence 23.problems posed increased my interest in course issues. revised: course problems and activities increased my interest in course issues. 24.course activities piqued my curiosity. 25.i felt motivated to explore content related questions. 26.i utilized a variety of information sources to explore problems posed in this course. 27.brainstorming and finding relevant information helped me resolve content related questions. 28.online discussions were valuable in helping me appreciate different perspectives. revised: class discussions were valuable in helping me appreciate different perspectives. 29.combining new information helped me answer questions raised in course activities. revised: applying new information helped me answer questions raised in course activities. 30.learning activities helped me construct explanations/solutions. 31.reflection on course content and discussions helped me understand fundamental concepts in this class. 32.*i can describe ways to test and apply the knowledge created in this course. 33.*i have developed solutions to course problems that can be applied in practice. 34.*i can apply the knowledge created in this course to my work or other non-class related activities. 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree *questions remaining after factor analysis (swan, richardson, ice, garrison, cleveland-innes, & arbaugh, 2008) microsoft word jethe+editor+letter_final.docx journal of effective teaching in higher education, vol. 4, no. 3 dear contributors, reviewers, and readers: as you will note from the delayed publication of the fall 2021 issue of the journal of effective teaching in higher education (jethe), the ongoing effects of covid-19 continue to delay the review and publication process for scholarly journals like ours. despite the delays, we are grateful for the work completed by our authors and reviewers during these difficult times and believe that the nine articles included in this issue reflect the resiliency of higher education faculty and scholars throughout the world. volume 4, issue 3 of jethe includes nine articles on critical and emerging issues in teaching and learning from scholars in the us and canada. the first three articles included in this issue focus on topics related to inclusive teaching in higher education. the first article, “creating an inclusive learning community to better serve minority students” by lin and kennette, is the first literature review we have published in jethe. in addition to serving as an example for other authors seeking to publish in this format, lin and kennette provide a useful overview of approaches to creating more inclusive classrooms in higher education. in the second article, calkins and rivnay examine outcomes from using an inclusive learning approach (i.e., jigsaw design) in a nursing class to promote cooperative problem-solving. lopez, penner-williams, and carpenter de cortina research the use of culturally responsive teaching in an online course in the third article included in this issue. the fourth and fifth articles included jethe 4.3 focus on interdisciplinary approaches to learning. rima and rodriguez focus on the assessment of an interdisciplinary project in psychology and criminal justice, while gastreich and milakovic explored how new knowledge was generated in interdisciplinary work in science, poetics and immersive field ecology. the next two articles focus on the use of technology in teaching, including litton’s article on the relationship between using videos in online courses and student performance as well as iluzada, wakefield, and alford’s evaluation of student learning, attention, and satisfaction from using personal technology. the final two articles demonstrate the diverse topics and approaches we value and publish in jethe. yeo and lafave’s article titled “a play in three acts: confused, conflicted and convinced (learning to reflect in athletic therapy)” highlights the benefits of creativity and reflection in teaching in higher education. the final article by kosal focuses on mindfulness in the college classroom and examines the effects of mindful practices on student performance. as always, all articles include implications for the practice of teaching across disciplines—an important contribution that is emphasized in jethe. despite the challenges of the pandemic, we were able to publish three quality issues of jethe for the first-time in one calendar year. in addition to reviewing the nine articles included in volume 4, issue 3 published in fall 2021, we also want to remind readers of our special issue (4.2) titled “in the along”: curricular and pedagogical imperatives for black mattering that was co-edited by dr. wilson letter from the editors ii journal of effective teaching in higher education, vol. 4, no. 3 okello, uncw, and dr. tj stewart, iowa state university. in just over two months since its publication in mid-fall 2021, articles from the special issue have been downloaded over 1,500 times at an average rate of 175+ times per article. demand for articles from the spring 2021 issue (4.1) of jethe further represents the growing reach of the journal with downloads exceeding 3,100 at an average rate of 350 full-text downloads per article. we want to thank you all for your interest in and support of our efforts at jethe and invite you to continue to contribute your own work to the journal. to submit an article or download full-text copies of current and past issues please visit http://jethe.org. finally, we would like to invite you to become a reviewer for jethe. if you are interested in reviewing for our journal, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. thank you for your continued support of jethe! james devita and colleen reilly coeditors, jethe microsoft word parra final version.docx journal of effective teaching in higher education, vol. 1, no. 2 promoting significant learning: a case study in computational chemistry ruben d. parra, depaul university, rparra1@depaul.edu abstract. this paper describes pedagogical efforts implemented to promote significant learning in a computational chemistry course. the taxonomy of significant learning advanced by fink (2013) is used as a framework to discuss the results. fink identifies six categories of learning: foundational, application, integration, human dimension, caring, and learning how to learn. significant learning occurs when all six kinds of learning are promoted. in the computational chemistry course, the learning outcomes are aligned with these kinds of learning, and students are provided with learning activities to get them engaged with the course content. these activities purposefully activate students’ prior knowledge, increase their motivation, develop basic skills, apply and integrate what they learn in realistic situations, develop an appreciation for computational chemistry as an ally in science, and further their ability to work independently and with others, and to continue learning about the subject matter beyond the course. keywords: effective teaching, computational chemistry, course design, significant learning the concept of learning is a complex one as seen by the many proposed definitions and learning theories. learning theories serve as frameworks for research in education (lederman & lederman, 2015). some definitions of learning refer to gaining knowledge and expertise (knowles, 2012), while others emphasize changes in attitudes and dispositions. in particular, reif (2008) considers that upon learning the learner can do things he/she could not do before. defining learning is important in the context of teaching as intentional support to facilitate learning. these notions about teaching and learning appear supported by research on how learning works (ambrose et al., 2010) and on how people learn (bransford et al., 1999). davis & arend (2013) correlate teaching success with learning, implying the need to identify and facilitate ways of learning that best matches the intended learning. biggs (2003) underscores the need for developing instructional strategies that increase the level of students’ engagement in deep learning. according to biggs, deep learning is revealed in the ability to build new knowledge from previous knowledge. in contrast, surface learning relies on accumulating ideas as isolated and unconnected items. fink (2013) also understands learning in terms of change, and submits that the change experienced by the learner has to be worthwhile. therefore, quality learning should correlate with the amount of effort that goes into teaching and the effort put by the learner during the learning process. when the change experienced by the learner as a result of instruction is lasting and important in the learner’s life, fink deemed the change as significant learning. thus, it seems appropriate to infer that “effective teaching leads to significant and lasting learning”. this definition entails the challenge of designing and implementing significant learning experiences. to help meet this challenge, fink developed a promoting significant learning: a case study in computational chemistry 94 journal of effective teaching in higher education, vol. 1, no. 2 taxonomy of learning and called it “taxonomy of significant learning”. this taxonomy was proposed as a successor of the well-known bloom’s taxonomy of the cognitive domain (bloom, 1956). in its original form, bloom’s taxonomy included six levels: knowledge, comprehension, application, analysis, synthesis, and evaluation. the taxonomy is hierarchical in nature with the first three categories listed representing lower levels of cognition, while the others representing higherorder levels. since the publication of bloom’s taxonomy, other learning taxonomies have been proposed. anderson et al. (2001) proposed a version of bloom’s taxonomy wherein the hierarchical nature of the original version remains but three categories are renamed and all categories are expressed as verbs. one important aspect of this revised version is that the authors explicitly consider how the taxonomy interconnects with and acts on different levels of knowledge: factual, conceptual, procedural, and metacognitive. an alternative to bloom’s taxonomy is the solo taxonomy (structure of observed learning outcomes) that serves as a framework to examine the extent and quality of learning at the surface, deep, and conceptual levels on a scale of five levels of increasing sophistication (biggs & collis, 1982). in contrast to bloom’s taxonomy, the solo taxonomy provides a useful guide for establishing criteria for both writing and evaluating learning outcomes based on the desired levels of understanding (biggs, 2003). marzano’s taxonomy is yet another alternative to bloom’s taxonomy that provides a framework for distinguishing higher from low-order thinking (marzano & kendall, 2007). in addition to the cognitive domain, marzano’s taxonomy emphasizes both the metacognitive and the self-system. the metacognitive system involves monitoring the learner’s own process, while the self-system addresses the emotional response to learning. in contrast to the hierarchical nature of the learning taxonomies described previously, fink’s taxonomy is relational and synergistic. here, six categories of learning are identified: foundational knowledge, application, integration, human dimension, caring, and learning how to learn. the nature of these categories ensures that promoting one type of learning increases the likelihood of success in achieving the other kinds of learning. fink’s taxonomy extends beyond the cognitive domain to include aspects of the affective domain like human dimension and caring. a brief description, taken from fink (2013), of the six categories is presented below. foundational knowledge: acquiring basic understanding of basic data, concepts, relationship, and perspectives as well as the ability to recall this knowledge in the future. application: using foundational knowledge. it includes developing particular skills, learning how to manage complex projects, and developing the ability to engage in various kinds of projects. integration: connecting and relating various things to each other. human dimension: learning about oneself and others. it involves addressing the important relationships and interactions we have with ourselves and others. promoting significant learning: a case study in computational chemistry 95 journal of effective teaching in higher education, vol. 1, no. 2 caring: developing new feelings, interests, and values. it involves caring more deeply about something. learning how to learn: becoming a better learner. it implies developing the ability to inquire about a subject to construct and expand knowledge, and the ability to monitor and regulate own learning. according to fink, “truly” significant learning is expected to take place when all six learning categories are promoted. the taxonomy of significant learning appears most adequate for the design and the promotion of the kind of learnings that occurs in an upper level chemistry course, for example. in particular, this paper describes various pedagogical approaches implemented to promote significant learning in an upper level computational chemistry course. computational chemistry in the author’s institution, computational chemistry is an elective course for undergraduate chemistry seniors. students enrolled in the ms program or the combined bs/ms program are eligible to enroll provided they meet the prerequisites which include having completed both organic chemistry and quantum chemistry at the undergraduate level. the course has been offered in the spring quarter of every other year since 2002, although intentional design of the course based on the taxonomy of significant learning started in 2006. the average course enrollment has been twelve students ranging from eight to twenty students. moreover, the course has consistently enrolled an almost equal number of female and male students. most of the students have little to no research experience and no previous exposure to any kind of computational chemistry. one goal of the course is for students to be able to determine the kind of research questions that may be effectively addressed using computational quantum chemistry as a tool. through hands-on activities, students develop the necessary skills to apply what they learn in the course to tackle chemistry-related research questions. to accomplish these goals, students are introduced to the fundamental principles and practical applications of computational quantum chemistry. course significant learning one challenge in the course is to strike a balance between coverage and depth. given the many computational chemistry approaches and methodologies, it is impractical to cover them all in just one course. moreover, these computational approaches are not necessarily based on a single theoretical framework. some are based on classical physics, others on quantum theory, and yet some others are based on a hybrid of these fundamental theories. the level of readiness for the students to apply and build upon these various theories constitutes another challenge. the fact that the course meets for three hours a week for 10 weeks (quarter system) needs to be factored in the design of the course. given the aforementioned challenges, the following questions are considered: what is it that promoting significant learning: a case study in computational chemistry 96 journal of effective teaching in higher education, vol. 1, no. 2 students need to learn and be able to do as a result of this course? what could realistically be achieved given the time constraints of the course, and the background knowledge of the students? how to design the course so as to help facilitate the kinds of learning students need to demonstrate in the course? how to determine that the intended learnings have occurred? as it turns out, the taxonomy of significant learning provides an important framework to help answer these questions. accordingly, one component in the design of the computational chemistry course is the identification of the learning outcomes of the course, and their alignment with fink’s taxonomy of learning as detailed below. each of the learning outcomes below is to be read as “the student will be able to” with the sentence being completed in the specification of the outcome. foundational knowledge – … define computational chemistry. – … identify and explain the governing principles of computational quantum chemistry. – … identify and describe key concepts, terminologies, approximations, and conventions. – … describe similarities and differences between various computational quantum chemistry methods. application – … solve a chemistry research question effectively and efficiently by • determining appropriate computational quantum chemistry methods to use. • using an appropriate computer program to carry out quantum chemistry calculations. • working effectively in a linux and windows operating system environment. • managing complex projects. – … present computational chemistry results both orally and in writing. – … analyze and evaluate the quality of computational chemistry results, and propose alternate ways of improving the quality of results. integration – … integrate computational chemistry with other branches of chemistry, and relevant disciplines such as biochemistry, organic chemistry, etc. human dimension – … work independently and with others in a respectful and productive way to solve problems or address issues guided by scientific and ethical principles. caring – … value computational chemistry as an essential ally in modern research in chemistry and in related disciplines. promoting significant learning: a case study in computational chemistry 97 journal of effective teaching in higher education, vol. 1, no. 2 learning how to learn – … build on what has been learned to determine what else is needed or wanted to learn about computational chemistry, and then developing concrete actions for achieving the desired new learning. to help students meet the learning outcomes of the course, several instructional strategies and activities were used as outlined next. essay accounts one pedagogical approach adopted in the course is an essay assignment that reads: “your assignment is to use your current academic background to ponder a research question in chemistry that you are interested in, and then provide your approach in the form of a systematic methodology to address the question in a way that either completely answers it or at least provides you with significant insight about the subject matter.” (see appendix for description of the assignment). writing as a learning tool has been shown to be an effective approach to help students deepen their learning and understanding of a subject matter (bean, 2011). in particular, essay assignments may be designed to incorporate higher levels of bloom's taxonomy as exemplified in the literature on project-based learning (halpern, 2014; krauss & boss, 2013). the assignment requires students to use their current knowledge in chemistry. it also promotes critical thinking (application dimension). consequently, students are given an opportunity to demonstrate sufficient depth in their thinking to formulate a chemistry question and then to make appropriate judgments about how to tackle it in an effective and efficient manner. the assignment promotes integration as students need to formulate a question based first on their current academic background, and then refine the question and the ways to solve it by integrating some of the computational chemistry tools they learn in the course. as for caring, students demonstrate that they already care about the content of the course by virtue of choosing to enroll in this elective course. nonetheless, students are seen to develop a much deeper appreciation of what computational chemistry can do for them upon completion of the course in general and the essay assignment in particular; the human dimension is factored in as students are formulate a question they feel passionate about but still they are required to persuade others of the importance of solving the question. although only 10% of the overall grade, the essay assignment has been instrumental in the realization of a significant learning experience for the students in the course. at first, most students feel at odds with the assignment because they are not used to formulating research questions, let alone developing a plan to tackle them. one key to the success of the assignment, however, is the explicit connection of the assignment to what students already know and to the things they care about. through these connections, students interact with the course content in a more meaningful way, and become more motivated to learn. through frequent meetings with individual students, i help them develop their initial take on the assignment, and after that, students take ownership of the assignment. as students learn more about computational chemistry, they move the assignment successfully promoting significant learning: a case study in computational chemistry 98 journal of effective teaching in higher education, vol. 1, no. 2 forward in all cases. regular contact with students is maintained to provide formative feedback prior to the submission of the essay accounts. the continuous support from the university writing center is also vital to help students address writing issues. mastering fundamental skills every class meeting is divided into two periods each of 90 minutes. the first period is devoted to the acquisition of foundational knowledge and to the development of deep learning through group discussions (about assigned readings or issues in assigned projects/activities). the second period is devoted to hands-on activities to master fundamental skills. students are taken to the computer laboratory where they work in their assigned computer station. for students to be able to apply what they learn in the course, they need to master the following operational skills: work in and navigate between both linux and windows computer environments; work with the computer software used to set up and carry out computational chemistry calculations, and the software used to help visualize, collect and analyze the results of the calculations. to facilitate learning and mastering of these skills, i developed a series of activities for students to engage in during class. there are additional activities for homework assignments. each activity is given with specific learning outcomes, so that students know what it is that they will be able to do as a result of the activity. the first few hands-on activities focus on mastering numerous linux commands that students need to use regularly such as remote login, create directories, create and edit files, and so on. at first, students feel visibly overwhelmed with the myriad commands they need to learn. in just a couple of weeks, however, students demonstrate a level of competence that is appropriate for them to work independently in a linux system. building upon the skills mastered, the pedagogical efforts are directed towards developing the skills necessary to work with the computer software used to run the calculations and visualize the results. such skills include setting up and running calculations, troubleshooting, creating/editing input files, collecting and visualizing data, etc. several in-class activities and related homework assignments are developed for students to master the mentioned skills to the point they can work independently. students are also directed to video clips (on the web or that i prepared myself) that touch base on issues of practical relevance like setting up calculations and visualization and analysis of results. half way through the quarter, students have acquired the skills necessary to begin working on the research project. one reason for the success of the hands-on activities is their practical nature and the on-thespot feedback. the activities are designed in a scaffolding manner to build continuously on “small wins”, that is, students develops confidence and a sense of accomplishment via the achievement of very specific targets or milestones. students need immediate feedback since they would not be able to move on to the next step unless they succeed in the step they are working on at any given time. one feature of the hands-on activities is that it promotes peer-led teaching and learning. students who grasp the skills more quickly than others tend to help their peers on site, or even after class. this social aspect adds to the human dimension of learning, and helps narrow the gap in the performance among students with varying abilities. promoting significant learning: a case study in computational chemistry 99 journal of effective teaching in higher education, vol. 1, no. 2 evaluating quality of results upon mastering the basic operational skills, students start reading selected computational chemistry publications. we discussed the publications during lecture with regard to research questions addressed, methodology chosen, and results quality. these readings help provide a framework to discuss and examine the foundational knowledge being imparted in the course, i.e. the basic principles and practical applications of computational quantum chemistry methods. as part of the discussion, students work in small groups and are required to suggest ways in which the quality of the results could be improved. during the hands-on portion of the class, students are asked to reproduce themselves portions of one or more of the research publications discussed. then, students are asked to examine whether the quality of the results could indeed be improved by implementing some of the suggestions made during the lecture discussions. students find the activity of reproducing portions of a real research paper very reassuring and stimulating. it is also instrumental in helping students build capacity to undertake their own research project. developing and managing a research project at week five, each student is assigned a project so they can apply what they are learning in the course and integrate what was learned in other courses. the project is designed to shed light into a research question in chemistry using known methodologies in chemistry and including some of the computational chemistry tools presented in the course. each student is given a unique project, although most projects are designed to be complementary projects. in some instances, the project is assigned based on the research question proposed in the essay assignment. most often, students work on projects that i know can be completed in five weeks or less. the project resembles an authentic performance task (hansen, 2011) as students are challenged to integrate and apply what they have learned to scenarios that mirror closely a real-life professional setting. students are encouraged to discuss with one another about their individual projects. as a result, students are seen to develop and maintain constructive working relationships. through peer interactions, students develop a learning community that enables them to learn from one another. during the first three weeks of the project, students demand constant support, but for the final weeks students are largely independent and able to bring the project effectively to completion. throughout the project, students experience changing and sometimes frustrating situations. students are challenged to manage what is in essence a complex project. students need to make decisions, organize information, plan and implement different tasks that need to work in concert to achieve the goal of the project. students are challenged to exercise creative thinking as there are usually different yet valid approaches that can be used to develop the project, with some more efficient or effective than others. students need to make practical decisions given the constraints of time and computer resources at hand. moreover, the research project helps promote students’ ability to become independent selfpromoting significant learning: a case study in computational chemistry 100 journal of effective teaching in higher education, vol. 1, no. 2 directed learners as they reflect on, and adjust as needed, the approach or strategies used to tackle the project. i provide timely feedback and support (via email, face-to-face, over the phone, etc.) to help students learn to monitor progress, restructure priorities and tasks on the fly, and redefine some of the proposed tasks or even consider alternatives to ensure success in the long run (meaning completion of the project). lastly, successful completion of the project promotes students’ communication skills because the students are required to write a report in the format of a scientific peer-reviewed journal and to give an oral powerpoint presentation to the entire class by the end of the quarter. i help students with the preparation of the powerpoint presentation both in person and by directing them to appropriate youtube videos available on the web on how to present scientific information using powerpoint. the acs (american chemical society) writing style is used in the course. given that this is an upper level chemistry course, students are usually familiar with the acs style. by examining peer-reviewed articles published in journals sponsored by the american chemical society, students develop a more practical sense of the acs writing style. students also learn that different scientific journals have different citation styles. for example, students are asked to compare the writing styles in two different journals: the journal of the american chemical society, and the computational and theoretical chemistry journal. the former is an acs journal, the latter is not. additionally, students are required to make use of the university writing center for support in their writing of the final report. since the writing center has tutors in all disciplines, including chemistry, it can assist students not only with specific issues of grammar but also with writing styles. results the success of the course is revealed by the extent to which students achieve the intended learning outcomes established for the course. in aggregate terms, most students have achieved a better than satisfactory level of competence (80% or better) every time the course has been offered (a total of 9 times since 2002). a typical grading scheme is given below 1. 30% hands-on activities and homework 2. 10% essay 3. 10% midterm 4. 40% paper based on assigned project (written in the format of a peerreviewed journal). 5. 10% oral defense of paper on assigned project through a powerpoint presentation given to the entire class. the grading scale follows a standard grade range of a (90%-100%), b (80%-90%), c (70%-80%), d (60%-70%), and f < 60%. more evidence comes from the observation that by the end of the course students can actually do things they were not able to do prior to taking the course. that is, students are comfortable working in the computational chemistry environment, handling the pertinent software, setting up and running calculations, formulating research questions, designing and promoting significant learning: a case study in computational chemistry 101 journal of effective teaching in higher education, vol. 1, no. 2 implementing research plans, collecting, analyzing and communicating information, evaluating the quality of results, and proposing creative ways to improve upon the results. students’ perceptions of learning, as revealed in the course evaluations, provide indirect evidence of the success of the course. a few representative comments (sampled from the different times the course has been offered) referring to aspects of the course students found most beneficial are given below: “learning how to use the computer programs to run calculations” “the final project allowed me the opportunity to see the vast applications of computational chemistry while also learning the challenges associated with working in this field” “the in-class work, it gave me the hands on experience that i would need instead of just having pure reading material” “i enjoyed replicating data from a previous publication” “just understanding the basic tools and the information that can be gleaned using computational chemistry is useful. it will be a useful perspective to have in the following chemistry courses” “it helped in showing how all previous courses can bring help to computational chemistry; it made the understanding of it more helpful” one impressive outcome is the fact that some students continue working in their assigned project well after the course is over. these students want to finish their projects at a level acceptable for publication in a peer-reviewed journal. as a result, there have been eight publications resulting largely by the students’ motivation, dedication and commitment to continue learning: (parra, knewstub, kusion, & moreno, 2019; falk & moses, 2015; parra & streu, 2011; parra & hill, 2010; parra & ohlssen, 2008; parra, arena, & sankissa, 2007; gharbonpour, wemhoff, kofoed, & parra, 2007; parra, yoo, & wemhoff, 2006). some students have incorporated what they learned in the course to other academic aspects of their lives, such as for their honors program thesis, or to continue into ms/bs option where they use computational chemistry as an ally. discussion and conclusions students’ feedback and my own professional development have been instrumental in changes made to the course over the years. for example, in the first two offerings of the course, i placed emphasized breadth over depth. i was following the model i experienced as a student in my computational chemistry courses. most of the textbooks i considered for the course also favors breadth over depth. accordingly, i spent much time on the theoretical foundations of computational chemistry sacrificing valuable hands-on practice. as one student put it, the course was “interesting but overwhelming”. suggestions from students included focusing on one computational chemistry approach, and adding more depth to the assigned promoting significant learning: a case study in computational chemistry 102 journal of effective teaching in higher education, vol. 1, no. 2 projects. i searched around for best teaching practices that could be adapted to improve my teaching and hence the learning experiences of my students. when i became aware of fink’s taxonomy, i found it to be an adequate framework to plan and design the course. i started implementing the significant learning taxonomy in 2006. since then, the course focuses on the approach to computational chemistry based on molecular orbital theory. hands-on activities in the course has increased, and scaffolding for the assigned projects are more intentional. one striking feature of the course that started with the 2006 offering is the peerreviewed publications by the students based on their projects. no such publications came out from the 2002 and 2004 offerings. overall, the taxonomy of significant learning has proven to be an important reference framework in the design, implementation, and assessment of the course since 2006. the synergistic aspect of the various learning experiences that students are exposed to in the course is noteworthy. in planning the course, i consider building upon the strengths and prior knowledge of the students. the importance of time limitations cannot be overestimated. for example, the possibility of publishing the research with students from a course like this is perhaps unrealistic. therefore, publishing is not part of the course grading. moreover, publication quality requires what is known as “high level” calculations. in practical terms, this translates into very time and computerresource demanding calculations that students would not be able to meet. this is especially true because the computer resources used are for teaching rather than for research purposes. the publications originated in this course were possible because of a combination of several factors: students’ motivation to continue working on their projects after the course ended; students using my research lab computers to carry out publicationquality calculations; students still having my support. the number of students in the course is also a consideration. the supervision and support of individual research projects become increasingly more difficult and perhaps impractical as the number of students increases. with twenty students enrolled in 2018, for example, the research projects were designed for students to work in pairs. in other disciplines, it may even be more difficult to publish within the time window of a course. for instance, a course where students engage in research that involves human subjects will need approval from the institutional review board or irb approval. the process of getting irb approval itself takes time making publication unlikely during the course lifetime. some additional considerations made when planning and teaching the course include: • make explicit the learning outcomes for the course. • provide guidance for students to work with content in meaningful ways. • align learning outcomes and the instructional strategies selected for the course. in particular, provide students with ample opportunities to engage in learning activities that are explicitly designed to help them develop and apply key skills in relevant tasks. • use assessment methods that align with the course learning outcomes. promoting significant learning: a case study in computational chemistry 103 journal of effective teaching in higher education, vol. 1, no. 2 • develop rubrics and descriptive criteria for the desired outcomes, so that students know what they need to do to demonstrate the level of achievement expected of them. • rely on various forms of evidence to determine the quality and extent of learning. for example, evidence includes many performance tasks that challenge students to use judgement and innovation while demonstrating their ability to use knowledge and skills attained in the course. • inform students of relevant learning support resources within and outside the university. for example, the university center for writing based learning, the university library, and a variety of online resources. • provide opportunities for frequent and varied assessment activities to facilitate timely and targeted feedback to students. especially to help students learn how to monitor their own progress. • create a course climate that fosters good interactions among students and with the instructor. • use a scaffolding approach so that students can move gradually from a beginning to a mastery level through a continual sequence of “small” wins that encourage the heart, and of “small” failures that prompt reflection and ability to learn from mistakes. promoting significant learning is worthwhile, but it does require a great deal of commitment from the part of both the instructor and the students. achieving significant learning demands purposeful planning and consideration of such things as the nature of the course (undergraduate vs graduate, and upper vs lower levels), the interconnectedness of all learning activities, the number of students, and the time available for instructor and students to engage with the course. the intentional design and assessment of activities aligned with the learning domains in fink’s taxonomy is especially tricky, but there are helpful books (angelo & cross, 1993; barkley & major, 2016). examples of application in various disciplines exist including religious studies (jones & hilaire, 2012), as well as special education, spanish, biology, economics and others (fink & fink, 2009; levine-fallahi, 2008). these examples demonstrate the diverse ways in which the taxonomy of significant learning has been used. interestingly, examples of fink’s taxonomy in chemistry education research (cooper & stowe, 2018) appears to be missing illustrating the paucity of information in this area and the need for publications like the work presented here. in contrast, examples of application in chemistry of bloom’s taxonomy (mcguire, 2015), solo taxonomy (hodges & harvey, 2003), and marzano’s taxonomy (toledo & dubas, 2016) can be found. the need for further research and applications of the significant learning taxonomy is apparent. given the known high attrition rates in general chemistry (ye, shuniak, oueini, robert, & lewis, 2016), this course is an excellent candidate to explore ways of improving students learning experiences and performance using the taxonomy of significant learning, and to compare the results with those from other learning taxonomies. i am exploring this possibility in my general chemistry course, in particular by incorporating learning activities designed to support the learning how to learn dimension in synergy with the foundational and application dimensions. as for the computational chemistry course, i plan to promoting significant learning: a case study in computational chemistry 104 journal of effective teaching in higher education, vol. 1, no. 2 strengthen the integration dimension further by having students consider and tackle issues in computer-aided drug design and atmospheric chemistry. i also plan to challenge students to consider how computational chemistry could be integrated into and applied to chemical education. conflicts of interest the author declare that there is no conflict of interest regarding the publication of this article. references ambrose, s. a., bridges m. w., dipietro, m., lovett, m. c., & norman, m. k. (2010). how learning works: seven research-based principles for smart teaching. san francisco, ca: josey-bass. anderson, l., & krathwohl, d. (eds.). (2001). a taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives— complete edition. new york, ny: addison wesley longman. angelo, t. a., & cross, p. k. (1993). classroom assessment techniques: a handbook for college teachers. san francisco, calif: josey-bass. barkley, e. f., & major, c. h. (2016). learning assessment techniques. san francisco, ca: josey-bass. bean, j.c. (2011). engaging ideas: the professor’s guide to integrating writing, critical thinking, and active learning in the classroom (2nd ed.). san francisco, ca: josey-bass. biggs, j. b., & collis, k. f. (1982). evaluating the quality of learning: the solo taxonomy. new york: academic press. biggs, j. (2003). teaching for quality learning at university (2nd ed.). maidenhead, berkshire, united kingdom. srhe, and open university press. bloom, b. s. (ed.). (1956). taxonomy of educational objectives, book 1: the cognitive domain. new york: longman. bransford, j., brown, a. l., cocking, r. r., & national research council (u.s.). (1999). how people learn: brain, mind, experience, and school. washington, d.c: national academy press. cooper, m. m. & stowe, r. l. (2018). chemistry education research: from personal empiricism to evidence, theory, and informed practice. chem. rev., 118: 6053-6087. davis, j. r., & arend, b. d. (2013). facilitating seven ways of learning: a resource for more purposeful, effective, and enjoyable college teaching. sterling, va: stylus. falk, n. & moses, a. (2015). diol-mediated versus water-mediated proton transfer reactions. depaul discoveries, 4(1), 1-7. fink, l. d. & fink, a. k. (2009). designing significant learning experiences: voices of experience. issue #119 in the "new directions for teaching and learning" series. san francisco: jossey-bass. fink, l. d. (2013). creating significant learning experiences: an integrated approach to designing college courses, revised and updated. hoboken, nj: wiley. promoting significant learning: a case study in computational chemistry 105 journal of effective teaching in higher education, vol. 1, no. 2 ghorbanpour, m., wemhoff, m. p., kofoed, p. & parra, r. d. (2007). intramolecular hydrogen bonding effects on chelate binding of ions by aromatic amides: a dft study. journal of undergraduate chemistry research, 6, 135-139. halpern, d. f. (2014). thought and knowledge: an introduction to critical thinking (5th ed.). new york, ny: psychology press. hansen, e. j. (2011). idea-based learning: a course design process to promote conceptual understanding. sterling, va: stylus. hodges, c. l. & harvey c. l. (2003). evaluation of student learning in organic chemistry using the solo taxonomy. j. chem. ed., 80(7), 785-787. jones, j. l., hilaire, r. s. (2012). creating significant learning experiences: a case study in the college religion classroom. the journal of effective teaching, 12(3), 34-43. knowles, m. s., holton, e. f., swanson, r. a. (2015). the adult learner: the definitive classic in adult education and human resource development (8th ed.) new york, ny: routledge. krauss, j. & boss, s. (2013). thinking through project-based learning: guiding deeper inquiry. corwin press. lederman, n. g., & lederman, j. s. (2015). what is a theoretical framework? a practical answer. j sci teacher educ, 26, 593-597. levine, l. e., fallahi, c. r., nicoll-senft, j. m., tessier, j. t., watson, c. l., & wood, r. m. (2008). creating significant learning experiences across disciplines. college teaching, 56(4), 247-254. mcguire, s. y. (2015). teach students how to learn: strategies you can incorporate into any course to improve student metacognition, study skills, and motivation. sterling, va: stylus. marzano, r., & kendall, j. (2007). the new taxonomy of educational objectives (2nd ed.). thousand oaks, ca: corwin press. parra, r. d., yoo, b. & wemhoff, m. (2006). conformational stability of a model macrocycle tetraamide: an ab initio study. journal of physical chemistry a., 110, 4487-4494. parra, r. d., arena, a., & sankisa, s. (2007). conformational preferences of carbonic acid and its sulfur derivatives, h2c(=x)o2-nsn ( x = o/s; n = 0-2). journal of molecular structure, 815, 31-40. parra, r. d. & ohlssen, j. (2008). cooperativity in intramolecular bifurcated hydrogen bonds: an ab initio study. journal of physical chemistry a, 112, 3492-3498. parra. r. d. & hill, b. (2010). conformational analysis of n-fluorine, 2,2’-x (n = 3 or 3’; x = bifuran, bithiophene, or thienylfuran): ab initio, density functional, and multi-coefficient correlation results. journal of molecular structure, 940, 61-69. parra, r. d. & streu, k. (2011). cooperative effects in regular and bifurcated intramolecular oh…o=c interactions: a computational study. computational and theoretical chemistry, 977, 181-187. parra, r. d., knewstub, s., kusion, p., & moreno, g. (2019).cyclic dimers of formamidine with its n halogenated formamidine analogues: structure, energetics, and proton-halonium transfer. computational and theoretical chemistry, 1153, 1-11. promoting significant learning: a case study in computational chemistry 106 journal of effective teaching in higher education, vol. 1, no. 2 toledo, s. & dubas, j. (2016). encouraging higher-order thinking in general chemistry by scaffolding student learning using marzano’s taxonomy. j. chem. ed., 93, 64-69. ye, l., shuniak, c., oueini r., robert, j. & lewis, s. (2016). can they succeed? exploring at-risk students’ study habits in college general chemistry. chem. educ. res. pract., 17, 878-892. promoting significant learning: a case study in computational chemistry 107 journal of effective teaching in higher education, vol. 1, no. 2 appendix essay this assignment represents 10% of your overall grade. you will complete this assignment in two different accounts called account 1 and account 2 respectively. each one of these accounts deals with essentially the same assignment. however, the accounts are expected to be written at increasingly deeper levels of critical thinking. accordingly, the 10% is distributed to mirror the increased expectations as follows: account 1 (3%) and account 2 (7%). purpose: the main purpose of this activity is to provide you with an opportunity to demonstrate your ability to make appropriate judgments about how to tackle a research question in chemistry in an effective and efficient manner. thus, what matters the most here is the quality of your critical thinking represented in this essay. assignment: your assignment is to use your current academic background to ponder a research question in chemistry that you are interested in, and then provide your approach in the form of a systematic methodology to address the question in a way that either completely answers it or at least provides you with significant insight about the subject matter. format: your essay will not be composed of separate sections, but it should include the following components: • claim: there should be a clearly identifiable research question in your first paragraph. the research question should be suitable for investigation using the tools provided by the field of chemistry, especially including computational chemistry. please note that the research question that you formulate in essay account 1 most likely will change or be refined as you developed a better understanding of what computational chemistry can do to assist you in your research. the expected gradual change in your formulation of the research question and the way you will tackle it should appear in your essay account 2. • analysis: identify and describe the research strategy you have deemed appropriate to tackle your research question. it is critical that you are not simply creating a bulleted list of steps to follow in your quest to answer your research question. rather, you should be writing fully developed paragraphs to support your chosen methodology. • persuasive elements: think about what would persuade your specific audience and use that to make your points more effective. it will be important to provide the reason or motivation that you have in pursuing this research. promoting significant learning: a case study in computational chemistry 108 journal of effective teaching in higher education, vol. 1, no. 2 audience: imagine that you are writing to a group of people who may be willing to finance your research so long as you convince them that the question you are addressing is an important one, that it can be addressed satisfactorily with the methodology you propose, and that you understand the requirements for solving the problem. requirements: 4-5 pages, double spaced, 12 pt. times new roman font, 1-inch margins due dates: account 1: thursday, april 14th 2016 account 2: thursday, may 25th 2016 microsoft word jenkins_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi: 10.36021/jethe.v4i2.184 showing out: africanisms and hip-hop mindfulness as black cultural praxis of excellence, resistance, joy, and love toby s. jenkins, phd, university of south carolina, jenki279@mailbox.sc.edu gloria boutte, phd, university of south carolina kamania wynter-hoyte, phd, university of south carolina abstract. in this essay, we center hip-hop culture and black cultural legacies. we envision and offer a two-fold framework which illuminates the intersection between the two. we explore ways that the black cultural experience (or better yet black cultural praxis) has always brilliantly and organically demonstrated the shape and form of a scholarship of consequence. black cultural praxis, or reflective action with a black emancipatory influence, has always allowed freedom of movement, freedom of body, freedom of tongue, and freedom of voice. we translate what this cultural praxis teaches and urges regarding the transformation, unbinding, and freeing of both educators and educational spaces. we demonstrate how the intersection of hip-hop culture and black cultural legacies can be instructive and transformative to educators. we invite educators to reimagine their classroom spaces by not only focusing on learning about hip hop but from it as well. keywords: hip-hop, black culture, africanisms, african cultural legacies this article illuminates how black cultural practices have long served as both evidence and example of what black mattering looks like in daily life. we suggest that it is the audacity in the everydayness of black culture that college educators must come to not just appreciate and accommodate but rather to adopt and incorporate in their own professional philosophies and practices. we therefore argue that higher education cannot simply include black cultural experiences and activities into the college experience and call that mattering. there must be a mindset change among those who teach and lead that is informed by black cultural values, beliefs, habits, and histories. in this essay, we center hip-hop culture and black cultural legacies. we envision and offer a two-fold framework which illuminates the intersection between the two. we explore ways that the black cultural experience (or better yet black cultural praxis) has always brilliantly and organically demonstrated the shape and form of a scholarship of consequence. black cultural praxis, or reflective action with a black emancipatory influence, has always allowed freedom of movement, freedom of body, freedom of tongue, and freedom of voice. we translate what this cultural praxis teaches and urges regarding the transformation, unbinding, and freeing of both educators and educational spaces. we demonstrate how the intersection of hip-hop culture and black cultural legacies can be instructive and transformative to educators. we invite educators to reimagine their classroom spaces by not only focusing on learning about hip hop but from it as well. showing out 44 journal of effective teaching in higher education, vol. 4, no. 2 overview of african cultural legacies tete ka asom ene kakyere a proverb of the akan people (“ancient things remain in the ear.”) (vansina, 1971) in his song dna (2017), kendrick lamar, 12-time grammy artist, boldly proclaims that he has loyalty and royalty inside his dna. lamar brilliantly explains the powerful impact of epigenetics and cultural legacies by highlighting that power, poison, pain, and joy are all a part of his dna. in other words, our ancestors’ experiences and cultural legacies are encoded in our dna and includes triumphs and joys. from a black emancipatory framework, king and swartz (2014) refer to lamar’s understanding of the depth of black cultural legacies and the impact on black people’s contemporary lives as (re)membering. (re)membering can be understood as an act of reassembling cultural legacies that europeans sought to dismember. these cultural legacies include africans’ identities, essence, and beings including languages, names, families, drums, and gods that were stripped from africans during the enslavement and colonization of black people around the globe. culture is the essential nucleus of what makes us human and “(t)he mind through memory carries culture from generation to generation” (vansina, 1985). when we fail to situate hip-hop within african diasporic culture or truncate black history by starting our narratives with enslavement, it essentially reduces black people to acultural beings—and, therefore, non-human. against all odds, black people do and have done what people all over the globe do—pass down cultural legacies—often in tacit, unseen, and powerful ways. worldwide, we see evidence of black cultural legacies among african descendant people. we will unpack and demonstrate the seamless connection to hip-hop and describe hip-hop as agentive, potent, and lifesustaining for black scholars—whether they are educators or students. notwithstanding the caveat regarding overgeneralizations about cultural groups, a significant body of research has demonstrated general cultural strengths and legacies among people of the african diaspora (boykin, 1994; hale, 2001; halebenson, 1986; hilliard, 1992; king, 2005; shade, 1997). instead of adopting the common deficit-based beliefs (e.g., “what’s wrong with black students?”, “they don’t know anything,” “they don’t want to learn,” or “we have to fix them”), educators would benefit from using students’ cultural strengths as starting points for instruction (boutte, 2016). a summary of 11 of the major dimensions of black culture (or cultural legacies) that have been cited in the literature as deep culture can be seen in table 1. these dimensions are sustained by black people across the globe—often in unconscious ways. they are stable though they morph over time and contexts; thus, educators should be intentional about observing how they show up. ultimately, the vision is for educational institutions to integrate these (or other emerging or existing) cultural strengths as part of the pro-black nucleus of curriculum and instruction, instead of merely using them as a bridge to learning eurocentric content. table 1 showing out 45 journal of effective teaching in higher education, vol. 4, no. 2 dimensions of african american culture african american cultural dimensions 1. spirituality—an approach to life as being essentially vitalistic rather than mechanistic, with the conviction that nonmaterial forces influence people’s everyday lives. 2. harmony—the notion that one’s fare is interrelated with other elements in the scheme of things, so that humankind and nature are harmonically conjoined. 3. movement—an emphasis on the interweaving of movement, rhythm, percussiveness, music, and dance, all of which are taken as central to psychological health. 4. verve—a propensity for relatively high levels of stimulation and for action that is energetic and lively. 5. affect—an emphasis on emotions and feelings, together with a specific sensitivity to emotional cues and a tendency to be emotionally expressive. 6. communalism/collectivity—a commitment to social connectedness, which includes an awareness that social bonds and responsibilities transcend individual privilege. 7. expressive individualism—the cultivation of a distinctive personality and proclivity for spontaneous, genuine personal expression. 8. oral tradition—strengths in oral/aural modes of communication, in which both speaking and listening are treated as performances, and cultivation of oral virtuosity. the ability to use alliterative, metaphorically colorful, graphic forms of spoken language. 9. social time perspective—an orientation in which time is treated as passing through a social space rather than a material one, and in which time can be recurring, personal, and phenomenological. time is acknowledged as a social construct. 10. perseverance—ability to maintain a sense of agency and strength in the face of adversities. 11. improvisation—substitution of alternatives that are more sensitive to black culture. the hip-hop mindset overview in two years, hip-hop will turn 50. for the past five decades, hip-hop has fully and unapologetically embraced black youth and young adults, situating their ways of thinking, being, and doing in the center of what is now a global culture. hip-hop has showing out 46 journal of effective teaching in higher education, vol. 4, no. 2 served as an important cultural system of inclusion for black intellectual thought and growth (chang, 2005; hamilton, 2004; george, 2005; jenkins, 2013b, 2020). at the same time that black and brown students were being silenced and pushed out of traditional p-12 classrooms, they began to create a cultural alternative. hiphop became a cultural space that welcomed their social criticisms; affirmed their lived experiences; and cheered on their expressions of rage against the power structures of america. hip-hop culture has always brought marginalized communities into the center and allowed them to testify. this happens not just within the music, but more often, in the varied spaces created within the larger culture. in the article, “hip hop: a culture of vision and voice,” sean mccollom (n.d.) stresses that, hip hop embraces these artistic elements, most definitely. but it also has blended and transcended them to become a means for seeing, celebrating, experiencing, understanding, confronting, and commenting on life and the world. hip-hop, in other words, is a way of living—a culture. for many young adults, hip-hop has been such a critical cultural life experience that they have carried it with them onto their college campus and into their college experience. in his book, hip-hop culture in the lives of college students, petchauer (2012) shares the cultural value of resistant or underground spaces (as they are known in hip-hop). according to petchauer (2012), the notion of the underground is a contemporary hip-hop lexicon that communicates ideas of nonconformity, self-creation, personal freedom, and alternative space. within hip-hop culture, the underground is the space where folks “keep it real.” the power structures of large commercial institutions cannot intercede and control the thoughts and work production of underground artists. campus hip-hop spaces often create this same sort of grass-roots, culturally free space. not only is there a sense of true intellectual and personal freedom, but in many ways, there is an intentional commitment to move in uncontrolled, uncensored, and unoppressed ways. hip-hop emcee and educator, krs-one offers the idea that hip-hop is essentially about conscious movement. this means that it goes beyond being a culture that is simply inherited and taken for granted. rather, hip-hop cultural participation demands intentionality: when you look at the terms, hip and hop-just what the terms mean—not getting philosophical yet. just the term hip-hop. “hip” means to know and “hop” means to move. awareness-movement. consciousness-movement, action, activity. intelligence-movement, action activity. so hip-hop means conscious movement. springing, leaping. “i’m hip to my hop.” “i know why i move.” “my movements-i’m aware of what i’m doing.” “i’m deliberate” (i am rap, https://fb.watch/2453sf5_uv/). this concept of “conscious movement” is the foundation of the hip-hop mindset. how do we, as higher education instructors, move with a cultural intentionality as professionals? how do we incorporate a cultural purposefulness in our practice? in the context of the hip-hop mindset, conscious movement is not about learning how showing out 47 journal of effective teaching in higher education, vol. 4, no. 2 to include hip-hop in the classroom so that your students can feel more included. rather, this is about wrestling with how black culture and black cultural production matters to you as a person. adopting a hip-hop mindset does not mean learning how to use hip-hop activities to teach other subjects. rather, this framework pushes educators to sit as cultural learners and to allow hip-hop culture to teach you. black culture is the subject, not the tool. the hip-hop mindset makes central black habits of mind—ways of thinking, being and doing—and situates these mindsets as meaningful forms of knowledge that can teach us new approaches to leadership and professional practice (jenkins, 2021). for colleges and universities to become authentic spaces of black mattering, hip-hop culture specifically and black culture more broadly must move from the campus underground (specialized courses, co-curricular programs, themed professional trainings) and become one of the major knowledge sources that informs how college educators and administrators lead, teach, research, advise, and engage with students and colleagues. the most critical takeaway is understanding hip-hop as a culture that embodies an ethic of excellence, freedom, and love. the hip-hop mindset is a theoretical framework informed by multiple research studies of hip-hop culture including a phenomenological study of culture in the lives of college students (jenkins, 2013a); an ethnographic study of community-based hip-hop spaces (jenkins, 2013b); a lyrical analysis of the knowledge production found in hip-hop music (jenkins, 2011); a narrative study of the impact of campus open mics (jenkins et al., 2017); and a phenomenological study of hip-hop habits of mind among elite educators and scholars (jenkins, 2021). this collective work establishes hip-hop as a cultural way of being and doing that directly influences how one shows up in the world—whether that means how they show up as a student or how they show up as a professional. when we say “show up,” we mean the values, beliefs, attitudes, and behaviors that a person brings into any space that they enter—their mindset. there are nine practices that form the hip-hop mindset. a practice is defined as “a way of doing something” or “the application of an idea or belief” (https://www.dictionary.com/browse/practice). it is important to stress that in hip-hop cultural spaces, people are free to not just have their beliefs, values and attitudes, but they are also free to put their beliefs into action and to display those attitudes fully. these practices, which form the hip-hop mindset framework, are organized below in table 2. table 2 the hip-hop mindset frameworki mindset drive: having large ambitions, dreams and goals practice 1 hunger: dreaming, grinding, and working to not just meet one’s goals but to “slay” them. practice 2 competitiveness: having high expectations and ambition to be the best. black culture cheers and applauds those who “show up” and “show out.” further, it was two black men (muhammad ali showing out 48 journal of effective teaching in higher education, vol. 4, no. 2 and l.l. cool j) who made popular the term, “g.o.a.t.” (greatest of all time) practice 3 honor & kinship: kinship is about shining independently while maintaining community connectedness. honor concerns having gratitude for the ways that others have “put you on” and learning to do the same. mindset approach: the way one performs or “shows up” in their work practice 4 creativity/originality: being original and/or unique; a value for innovating—being able to transform or remix something old into something new. hip-hop artists are “seers” or visionaries who can see utility in what others may view as worthless. practice 5 authenticity/integrity: representing oneself and/or community in a way that is real, clear, and true. hip-hop is about accountability and showing up as your full, whole self (not a watered-down version to please others). practice 6 ingenuity/cultural efficacy: ingenuity concerns being clever and inventive. but before you invent anything, you must first believe that you can. cultural efficacy is an optimistic belief in, appreciation for, and respect of one’s culture and the people in it. you cannot represent the culture if you do not love it. mindset posture: one’s presence within their work space practice 7 confidence: a strong belief in one’s ability. confidence concerns having the guts to be a non-conformist; to tell the truth; and to try something new. practice 8 claiming space: “owning” any space that you occupy. to own it means to confidently enter, move through it, and transform it knowing that you have the right (the talent, the skills, the ability) to do so. practice 9 commanding attention: having a dynamic and engaging presence and knowing how to move the community. speaking boldly and bravely (speaking loudly, in your language, and about the things that matter to your people). in hip-hop, microphones project your voice, they do not silence voices. being heard is important. as illustrated in table 2, cultural authenticity, cultural confidence, and cultural efficacy are central aspects of the hip-hop mindset. this is particularly salient for educators. cultural efficacy concerns a sense of cultural agency-having positive beliefs about a culture, understanding the strong contributions of a culture, authentically loving a culture (jenkins, 2013a). before educators can adopt a culturally inclusive pedagogical practice, they must first simply believe that black culture matters—that it is positive and worthy of study in the first place. for nonblack educators, this might require developing a personal relationship with the showing out 49 journal of effective teaching in higher education, vol. 4, no. 2 culture so that they can experience the deep professional benefits that occur when led by black culture. while many black professionals love black culture, they may not identify with hip-hop culture or may not view it as valuable or beneficial. several scholars have documented their love/hate relationship with hip-hop due to the misogynistic and homophobic messages found in some hip-hop music (bradley, 2015; durham et al., 2013; hurt et al., 2006; love, 2016a; morgan, 1999). therefore, when educators engage in hip-hop based education, it is imperative to create a space where students can value and critique hip-hop as a larger culture that both includes and moves beyond its musical element. this might mean pushing oneself to be re-educated to critically understand hip-hop culture while integrating it into professional practice. for many educators, regardless of race, if we allow ourselves to be led by hip-hop culture, we will come to see that black youth have been organically demonstrating what black mattering looks like for decades. hip-hop artists do not hesitate to proclaim black culture and black people as dope, fly, fresh, and the best. they will write a whole song and not feel compelled to talk about anything but the black experience. they use quintessentially african beats and rhythms, and they dance in ways that allow their body to be free. the youth that created hip-hop, created their own freedom using their culture as the blueprint. they simply followed what they felt. we need more of this “freedom feeling” in education as opposed to the conventional colonized and controlling pedagogies often used in higher education. because the reality is students are not the only ones suffering and miserable in educational institutions, a hip-hop mindset can influence educators’ drive, approach, and posture as a professional. educators also need to “get free.” the pressure to conform is strong in educational environments for black students, teachers, administrators, and scholars. what hip-hop culture, and the broader african cultural experience from which it is rooted, gifts black people is the permission to be our authentic selves. cultural intersections of the hip hop and black cultural dimensions in this section, we explicate the intersection of the hip-hop mindset framework with the african cultural legacies framework in order to demonstrate the consistency, pervasiveness, and continuity of black culture. we seek to particularly analyze and demonstrate how a contemporary extended culture like hip-hop aligns perfectly with the longer history of african cultural legacies. the values, practices, moods, beliefs, and overall vibes are all congruent. what this ultimately means is that in order to center blackness as an educator, foundational truths about black culture cannot be escaped. we also pose several critical questions that push educators to wrestle with the implications that these habits of mind and cultural legacies have for their professional practices. as presented in table 3, the intersections between hip-hop mindsets (drive, approach, and posture) and black culture (oral tradition; verve, affect, communalism/collectivity; expressive individualism; social time perspective; movement; perseverance; spirituality; harmony; and improvisation) are apparent and powerful. showing out 50 journal of effective teaching in higher education, vol. 4, no. 2 table 3 intersections between hip-hop and black cultural dimensions black cultural dimensions hip-hop cultural practices critical questions oral tradition • the spoken word: lyrics, rhymes, verbal freestyles, poetry • vocal creativity: flow, style, and creative content • microphone: projects voice and symbolizes your words are important • how much or how little is the oral tradition embraced in our instructional practices? • as instructors, how do we flow? • how comfortable are we with our own voice? • to whom am i passing the mic? in other words, what voices are included as scholars and experts? (quaye & harper, 2007; wichelns, 2021; millardball et al., 2021; saunders & kardia, 1997). • what language is allowed in the spaces that we curate? • can students actually say it “slick” or must they conform to particular ways of speaking and expressing? • can we, as the educator, relax our own tongue and engage students more creatively? verve • booming bass, aggressive emcees • audacious acts • over-the-top antics • showiness • loudness • “in your face” attitudes • as educators, how loud can we turn up the volume? • what kind of energy are we willing to bring to faculty meetings? • what is the level of change we are comfortable creating in our curriculum? • how will we reconcile “keeping it real” with showing out 51 journal of effective teaching in higher education, vol. 4, no. 2 students and the pressures to “maintain professionalism” that come from the institution? • who gets to decide what professionalism looks like? affect • freedom to express emotions • speaking truth to power • rawness and realness • as educators, how do we feel? • are we allowing ourselves to feel? • can our classrooms become a space where everyone is allowed to be human rather than expert machines with all of the answers? communalism/collectively • community of practice (squad) • shared interests and experiences with community • expectation to “represent” the community • who is our crew? • how do they make us better? • what larger community experiences do we draw from in our work? • how diverse is that experience? • what have we experienced in life? • are we comfortable claiming our experience and telling the truth about who we are? expressive individualism • a value for being unique and different • a desire to stand out • independence and interdependence: each member of the squad has their own success and works collaboratively to make the larger brand successful • what is our contribution? • what is our talent? • what magic do we bring to your work? showing out 52 journal of effective teaching in higher education, vol. 4, no. 2 social time perspective • the party doesn’t start until the place is packed • freedom to wait until everything is ready • how can we work to ensure that students are ready to engage rather than diving into a preplanned lesson? • how might educators slow down and make time for community building before instruction begins? movement • breakdancing • body language (posture/swag) • how can we recognize movement as critical to the psychological and physical welfare of students and educators? • how do we positively or negatively/stereotypically read body language or everyday body movement? perseverance • telling (and hearing) the story of endurance is valued • hanging in there when on the “come up” (trying to make it) • what have students and educators endured in order to be where we are? • who are we currently and how did we get there? spirituality • drawing from and embracing natural, non-physical, forms of energy that flow through the human experience • what is our purpose, calling, and passion? • how does teaching fit with our purpose, calling, and passion? harmony • cyphers as circles of energy in which performers share the mic and commune in a shared space • hip hop artists recognize and embrace the inherent beauty in their geologic • how can we create spaces not only where people are in harmony, but places where we are also in harmony with the rhythms of nature? showing out 53 journal of effective teaching in higher education, vol. 4, no. 2 spaces (e.g., urban, rural, suburban) regardless of pervasive deficit narratives about these spaces. improvisation • the dj’s creative talent for mixing and ear for rhythm • freestyling in dancing and emceeing • creativity in dressing, language, etc. • as instructors, how are you fostering space for teachable moments or discussions that were not on the course agenda? • how are you allowing for creative expression in your instruction or in your classroom? • what languages are welcomed in the classroom? oral tradition is one of the most obvious ways that hip-hop culture carries forward african cultural legacies. when most people think of hip-hop, they envision an emcee. the oral tradition in hip-hop concerns lyrics, rhymes, and verbal freestyles that tell stories or share opinions. the spoken word. the microphone, as a visual symbol, communicates that one’s voice matters. passing the mic or grabbing the mic is an important act of creating or taking space to allow new voices to speak. dropping the mic is a swag-filled act of confidence that indicates there is nothing left to be said—i said it all. but voice is central in hip-hop. sharing the experience, shouting the truth, saying what needs to be said, passing down street knowledge or family wisdom are all elements of hip-hop culture inherited from this african cultural legacy. an educator that is not comfortable speaking freely, openly, and truthfully may not be comfortable creating a space for students to do the same (riddle, 2018; boutte et al., 2011; rademacher, 2018). verve: booming bass, aggressive emcees, audacious acts, over-the-top antics, showiness, boastfulness—hip-hop is loud, energetic and in your face. those that love it, value the way that hip-hop hypes us and moves us. the energy is always maxed to the limit. it makes us feel alive. verve concerns both the cultures of quiet that drive many p-12 classroom environments and college campuses, and it also concerns our level of comfort with turning up our energy as professionals (rademacher, 2018). affect: “one of the most important things hip-hop gives its community is the permission to feel and to express those feelings. from anger to rage; from love to lust; from top dog confidence to underdog hunger; being centered in a hip-hop mindset is to be centered in authenticity and truth and to express that truth showing out 54 journal of effective teaching in higher education, vol. 4, no. 2 regardless of the consequences” (jenkins, p. 2021). hip-hop is probably one of the few spaces in society where young adults are not only rewarded for speaking truth to power, but they are actually expected to be as raw and real as possible. one of the critical characteristics of effective teaching in p-20 educational settings, particularly in times of social unrest, is the ability to allow students to react to the world around them (linsenmeyer & lucas, 2017; sathy et al., 2020). flexibility to amend the syllabus, throw out the lesson plan and create a space where students can just speak and heal is essential for educators who are comfortable facilitating such discussions. for others, acknowledging the need to thoughtfully (but quickly) plan a guided engagement around the issues is also okay. but educators should let students know that we will talk, and that silence will not be the response to social unrest in our classrooms (landis, 2008). as citizen scholars, we can reflect on what we are feeling as well. communalism/collectivity: the crew or squad is a major aspect of hip-hop culture. association with a highly skilled community of artists is critical to the reputation of a hip-hop artist. the members of a crew also push each other to be better. they are essentially a community of practice who share a deep interest in something (hip-hop) and work collaboratively to get better at it (jenkins, 2021). but hip-hop culture is not simply about a cliquish association to a small crew or squad. artists speak quite often to their community experience—larger social realities that might involve a shared since of place, circumstance, joy, resilience, oppression, and hardship (love, 2016a, 2016b). it is also important to note that hip-hop began with young cultural conveners planning parties in rec rooms and gatherings at local parks (chang, 2005). people gathering, dancing, socializing, and having fun has always been essential to hip-hop culture. expressive individualism: hip-hop culture is a particularly vivid example of this african cultural legacy, which is markedly different from western concepts of individualism (boutte et al., 2017). as mentioned earlier, within hip-hop crews, each member shines and stands out individually, but they are still very much a part of a crew or community whose resources, association, power, expertise, talents, and affirmation are very important to the individual artist. each person’s success makes the crew look good. but largely hip-hop artists are always striving to stand out as unique and different-to make their own unique contribution to the field. we need a lot more of this in the field of education. the ability to cultivate one’s distinct professional personality or, in other words to just “be ourselves,” is vital. too often, educators are being molded and “trained” to be the same rather than distinctly different. a hip-hop crew wants that difference to shine rather than be shut down. many educators need to start the process of becoming more of ourselves by first thinking deeply about who we are professionally. social time perspective: the african diasporic community is notoriously known for viewing time as a flexible concept (streamas, 2021). hip-hop is no different. there is actually brilliance in this perspective when we view it through the lens of something like a hip-hop concert. the concert promoter might be concerned with the schedule and timeline (they are paying people by the hour, and they want to avoid riots of angry fans). but the artist only cares about taking the stage when showing out 55 journal of effective teaching in higher education, vol. 4, no. 2 everything is right—they aren’t concerned with being on time. this means the concert doesn’t begin with a bunch of empty seats. critical mass is important. regardless of what is advertised on the poster, parties don’t start until the place is packed, or it will look like a failure. in many ways, this is about cultivating a culture of care and mattering. ensuring that the community is present is important. within the education arena, this might mean more than checking to ensure that all bodies are in the classroom. a student being physically present and a student being mentally present are two different things. this is about not taking the stage until you have an audience. ensuring there is transitional time to get into the mood of learning is necessary. this gives the students time to address issues or release emotions that may have occurred on the way to class, to munch a snack if their bellies are still empty, or to greet and say what’s up to friends and comrades. there is much to do within the time constraints of any course, whether it is a high school or college classroom. but no learning will occur (whether you are talking or not) until the crowd is ready and present. movement: hip-hop, of course, has an explicit focus on movement through breakdance. the term breakdance comes from the practice of creating a break in the music so that dancers could perform. that is how essential movement was to the culture. at the old-school hip-hop parties, space was created both in the music by the dj and on the dance floor by the crowd for dancers to show off and show out. ultimately, this was the culture creating space for bodies to do what they need to do: move. this is important insight for p-20 educational environments because it seems the higher one climbs in education the less mobile the learning becomes. kindergartners are always moving while participants in doctoral seminars might often sit in a conversational circle. even in that very literal example, and accounting for differing levels of physical mobility among classroom participants, more can probably be done to mobilize the higher education classroom. this might include embracing place-based learning and going off campus or project-based learning that requires the agility and flexibility for students to engage and communicate with each other (braxton et al., 2004; kuh, 2008). it might also involve adopting instructional strategies like flipped classrooms that situate the lecture at home allowing for more action in the classroom. or it may be as simple as just allowing more freedom for students to experience content in both mental and physical ways. it is a critical insight for educators to recognize that sitting still isn’t natural at any age. perseverance: so many hip-hop songs talk about the hardships an artist has endured and the ways they pushed to come out on top. the act of remembering as a source of affirmation and motivation is a cultural legacy present throughout the african diaspora and hip-hop culture. it makes sense that a community of people that have been the target of american disenfranchisement, oppression, and discrimination would speak on how they survived this experience if given a microphone to talk to the world. hip-hop rhymes and lyrics are a form of testimony to the tenacity of a community—to not just make it and stay alive but to thrive and come out winning. more opportunities for all of us in education to share our stories and tell our truths are needed. offering opportunities for cultural reflection, selfauthorship, and testimony is very important in the education of black college showing out 56 journal of effective teaching in higher education, vol. 4, no. 2 students (harper, 2010; patton et al., 2017; rendon et al., 2004) and the educators who teach and advise them. understanding ourselves as cultural beings has been affirmed as the critical first step in anti-racist work (gorski & dalton, 2019; helms, 1995, 1997; kendi, 2019). every person on a college campus needs an opportunity to explore deeply where they have been, what they have been, and who they currently are (boyd, 1998). spirituality: musical genres often create clear lines and distinctions between certain types of music. for example, black spiritual music might be classified as gospel with hip-hop music only being included if the content explicitly discusses a religious faith. but many artists and cultural producers argue that their work is a mix of the social, the political, and the spiritual (giorgis, 2015; robinson, 2020). hip-hop performers and artists often view their craft as part of their life purpose to speak truth to power on behalf of humanity—particularly for and with people who have been or are minoritized. when the “spiritual” is understood as that which transcends religious boundaries, it becomes a space that includes any form of cultural production that expresses the natural, non-physical form of energy that flows through the human experience. black cultural music production has always tapped this sense of spirituality through drums, banjo, guitar, organ, dance, voice, or beatbox. the power of the music and the audience’s soul-filled response to it affirms the supernaturalness of life. spirituality in this sense and particularly in relation to the field of education has no concern with religious teaching. it concerns what drives us as human beings. often those who are brave enough to unapologetically and publicly love blackness do so because they feel that love in the core of their being (rodriguez, 2001). central to the very concept of spirituality is connectedness. it is not simply that an incredible energy exists, it is about your connection to that energy. likewise, when it comes to anti-racism in education, it is not enough that there now exists an incredible amount of energy for this work. educators must wrestle with their connection to it, and, more specifically, their connection to black people. in his work examining the “class-inflected nature of activist identifies,” stephen valocchi (2012) suggests that among activists, the closer a person’s life is (or has been) to the struggle for which they fight, the more their sense of activism shifts from being a career commitment to being a calling or way of life. harmony: in many ways, hip-hop culture has served as an example of humankind’s hardheaded determination not to give up on the idea that harmony is possible. the incredible forms of economic, political, social oppression, and racism faced by the communities of youth and young adults that both created and carried forth hip-hop culture (chang, 2005; hill, 2016; jenkins, 2006; lomax, 2011) would make anyone doubt that living in harmony is the natural course of life. the building of the united states required the exact opposite of harmony—it called for physical division of people and severe differences in the lived experiences of those people (kendi, 2016; wilkerson, 2020)—one community living in peace and the other living in terror. the land that surrounded african americans from slave labor camps to housing projects were social jails that limited movement and freedom (alexander, 2010). but yet, young people saw community as more than the physical land or showing out 57 journal of effective teaching in higher education, vol. 4, no. 2 buildings that surrounded them. they clung to the people close to them and to the culture that those people created. they found ways to stay conjoined and develop a sense of community and placemaking through the culture. this is why hip-hop artists constantly reference where they are from, even when they are from economically oppressed neighborhoods. using the lyrics of their songs, hip-hop artists culturally transformed housing projects like queensbridge and marcy houses into badges of honor and historical landmarks (jenkins, 2021). they choose to show their community love, not judgement or contempt. living in harmony has long been an african value. and black people, black activists, black educators, and black students continue to do whatever is necessary (protest, call-out, call-in) to transform our institutions into spaces where black people can truly live in peace. improvisation is also central to hip-hop culture. the most obvious form is through the techniques of djing, dancing, and emceeing. these performative aspects of the culture require an ability to freestyle and improvise. but beyond the musical elements, the larger culture itself is a story of improvisation—the creation of an alternative experience that was more sensitive to black and latinx youth culture. if schools today are still actively struggling to better include and welcome black student cultures and voices, imagine what schools were culturally like in the 1980s. young people like dj kool herc (a student at an alternative vocational school credited with co-hosting the first hip-hop party that launched the culture) might not have been a part of creating hip-hop if their schools were actually spaces where they could engage and work out their cultural ideas (jenkins, 2021). those type of cultural incubator spaces did not exist at the time, so young people took to basement rec rooms and neighborhood parks and created them. ultimately, improvisation is about invention, being able to create something more relevant, more accessible, more interesting than what exists. this is every educator’s call to action in the current moment. when what we have does not work, it is time to create something new—to flip and expand the script. conclusion: black cultural praxis as strategies for black mattering black cultural frameworks, like the intersection of the hip-hop mindset and african cultural legacies, provide strategies for activating black mattering university courses in actionable ways. in this case, black mattering is not simply about caring for black students or even having fundamental values and beliefs that racism, inequity, and discrimination are wrong. these frameworks pull us into a wider conversation about black people broadly—black communities, black histories, black ancestors, black youth, black artists, black teachers, black educational leaders, black neighbors, and black activists. educators at all levels can begin to reckon with how they interact and engage (or limit interaction and engagement) with the beauty and depth of blackness that surrounds them every day. is it possible to effectively teach black college students in a classroom if black culture is detested when heard on the radio, observed in meetings, or walked past it on the streets? if high levels of verve, affect, movement, and vocal expression are in direct opposition to how classrooms are managed, then black culture might actually not matter in respective educational showing out 58 journal of effective teaching in higher education, vol. 4, no. 2 spaces. this is why we argue that simply creating a lesson plan incorporating hiphop or black culture is inadequate. our classrooms and professional philosophies at large are in need of a cultural revolution. educators must build our capacity to not just understand cultural ways of being and doing when we see it in students, but more importantly, we must come to understand how we can embrace those same mindsets and cultural legacies in how we act, move, think, create, and facilitate as professionals. how can we move, flow, and approach teaching differently? there is much to take away from the intersections of these frameworks. for educators, freeing ourselves will entail being more flexible, honest, human, vocal, expressive, creative, unique, and authentic. in her powerful talk, “the masters tools will never dismantle the master’s house,” audre lord (1984) states: those of us who stand outside the circle of this society's definition of acceptable women; those of us who have been forged in the crucibles of difference—those of us who are poor, who are lesbians, who are black, who are older—know that survival is not an academic skill. it is learning how to take our differences and make them strengths. for the master's tools will never dismantle the master's house. they may allow us temporarily to beat him at his own game, but they will never enable us to bring about genuine change. and this fact is only threatening to those women who still define the master's house as their only source of support. (p. 2) in this article, we argue for educators of all races to do a better job embracing and embodying black culture in their professional ways of life. whether you are a white college professor with little exposure to and knowledge about black cultural experiences and histories or a black college professor who has been taught to discard black culture in order to succeed, we offer two cultural frameworks that can be used to redirect professional mindsets and reorder the professional movement of educators in ways that are informed by black cultural ways of knowing, being, and doing. we often hope to have comrades in the struggle—university presidents who get it, faculty who honestly admit they need to do better, and administrators who are radically pushing, challenging, and changing the system. but, the reality is that some institutions have these people and some institutions just do not. some educators in higher education like the institution just the way it is—it works for them. we cannot define the master’s house as our only knowledge base or source for professional development. instead, we must revolutionize the way that we conceive of concepts like “knowledge,” “professionalism,” and “scholar.” this means challenging ourselves to imagine various types of life experiences as spaces of significant learning and professional development. ultimately, we must begin to view culture itself as a source of knowledge that can and should inform the ways we conceptualize, develop, and engage our professional approach and practice. showing out 59 journal of effective teaching in higher education, vol. 4, no. 2 conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references alexander, m. (2010). the new jim crow: mass incarceration in the age of colorblindness. new press. boutte, g. s. (2016). educating african american students. and how are the children? routledge. boutte, g., george, j., bryan, n., winter-hoyte, k., uyoata, u.e. (2017). using african diaspora literacy to heal and restore the souls of young black children. international critical childhood policy studies journal, 6(1), 66–79. boutte, g., lopez-robertson, j., powers-costello, e. (2011). moving beyond color blindness in early childhood education. early childhood education journal, 39, 335–342. doi:10.1007/s10643-011-0457-x boyd, h. (1998). in memoriam dr. john henrik clarke (1915–1998). the black scholar, 28(3/4), 50–52. http://www.jstor.org/stable/41068808 boykin, a. w. (1994). afrocultural expression and its implications for schooling. in e. hollins, j. king, & w. hayman (eds.), teaching diverse populations: formulating a knowledge base (pp. 243–273). state university of new york press. bradley, r. n. (2015). barbz and kings: explorations of gender and sexuality in hiphop. in j. a. williams (ed.), cambridge companion to hip-hop (181–191). cambridge university press. braxton, j., hirschy, a., mcclendon, s. (2004). understanding and reducing college student departure. jossey-bass. chang, j. (2005). can’t stop won’t stop: a history of the hip-hop generation. st. martin’s press. durham, a., cooper, b., & morris. s. (2013). the stage hip-hop feminism built: a new directions essay. signs, 38(3), 721–37. george, n. (2005). hip-hop america. penguin. giorgis, h. (2015). black art is dangerous, because it marries the personal and political. the guardian. showing out 60 journal of effective teaching in higher education, vol. 4, no. 2 https://www.theguardian.com/commentisfree/2015/feb/22/black-artdangerous-personal-political gorski, p., & dalton, k. (2019). striving for critical reflection in multicultural and social justice teacher education: introducing a typology of reflection approaches. journal of teacher education, 71(3), 357–368. hale, j. (2001). learning while black: creating educational excellence for african american children. johns hopkins university press. hale-benson, j. e. (1986). black children: their roots, culture, and learning styles (rev. ed.). johns hopkins university press. hamilton, k. (2004). making some noise: the academy’s hip-hop generation— scholarship on the genre moves beyond a project of legitimization into a more self-critical, challenging realm. black issues in higher education, 21(5), 34. harper, s. (2010). an anti-deficit achievement framework for research on students of color in stem. in s. r. harper and c. b. newman (eds.), students of color in stem: engineering a new research agenda (pp. 63–74). jossey-bass. helms, j. e. (1995). an update of helm's white and people of color racial identity models. in j. g. ponterotto, j. m. casas, l. a. suzuki, & c. m. alexander (eds.), handbook of multicultural counseling (pp. 181–198). sage publications. helms, j. e. (1997). toward a model of white racial identity development. in p. g. altbach, k. arnold, & i. c. king (eds.), college student development and academic life: psychological, intellectual, social and moral issues (pp. 49– 66). routledge. hill, m. l. (2016). nobody: casualties of america’s war on the vulnerable, from ferguson to flint and beyond. atria books. hilliard, a. g. (1992). behavioral style, culture, and teaching, and learning. journal of negro education, 61(3), 370–377. hurt, b., gordon, s. s., & winters, b. (2006). hip-hop: beyond beats and rhymes [film]. god bless the child productions, independent television service; national black programming consortium; corporation for public broadcasting. media education foundation. i am rap (2016, september 11). krs-one hablando de hip-hop. extracted from the internet on june 20, 2021 from https://www.facebook.com/iamrap.es/videos/1114926478587268/ showing out 61 journal of effective teaching in higher education, vol. 4, no. 2 jenkins, t. (2006), mr. nigger: the challenges of educating african american males in american society. journal of black studies, 37(1), 127–155. https://doi.org/10.1177/0021934704273931 jenkins, t. (2011). a beautiful mind: black male intellectual identity in hip-hop. journal of black studies, 42(8), 1231–1251. https://doi.org/10.1177/0021934711405050 jenkins, t. (2013a). my culture, my color, my self: heritage and resilience in the lives of young adults of color. temple university press. jenkins, t. (2013b). de (re) constructing ideas of genius: hip hop, knowledge, and intelligence. the international journal of critical pedagogy, 4(3), 11–23. jenkins, t. (2020). imagination, power & brilliance: hip-hop as a politic of educational survival. in e. adjapong & i. levy (eds)., hiphoped: the compilation on hip-hop education (volume 2, pp. 87–97). peter lang. jenkins, t. (2021). the hip-hop mindset as a professional practice: air-walking and trashtalking. routledge. jenkins, t., endsley, c., & jaksch, m. (2017). the open mic night: campus programs that champion student voice and engagement. stylus publishing. kendi, d. i. x. (2016). stamped from the beginning. avalon publishing group. kendi, i. (2019). how to be an anti-racist. one world. king, j. e. (2005). black education: a transformative research and action agenda for the 21st century. lawrence erlbaum. king, j. e., & swartz, e. e. (2014). “re-membering” history in student and teacher learning: an afrocentric culturally informed praxis. routledge. king, j. e., & swartz, e. e. (2016). the afrocentric praxis of teaching for freedom: connecting culture to learning. routledge. krs-one (2008). the gospel of hip hop: the first instrument. powerhouse books. kuh, g. (2008). high impact education practices. association of american colleges & universities. lamar, kendrick. (2017). dna. on damn. top dawg. aftermath. interscope. landis, k. (2008). start talking: a handbook for engaging difficult dialogues in higher education. university of alaska anchorage and alaska pacific university. showing out 62 journal of effective teaching in higher education, vol. 4, no. 2 linsenmeyer, w., & lucas, t. (2017). student perceptions of the faculty response during the civil unrest in ferguson, missouri. international journal of teaching and learning in higher education, 29(3), 524–533. lomax, t. (2011). in search of our daughters’ gardens: hip hop as womanist prose. bulletin for the study of religion, 40(3), 15–20. https://doi.org/10.1558/bsor.v40i3.004 lorde, a. (1984a). the master’s tools will never dismantle the master’s house (comments at the “the personal and the political panel,” second sex conference, new york, september 29, 1979). in sister outsider (pp. 110– 113). sister visions press. (original work published 1979) love, b. (2016a). complex personhood of hip hop & the sensibilities of the culture that fosters knowledge of self & self-determination. equity & excellence in education, 49(4), 414–427. love, b. (2016b). good kids, mad cities: kendrick lamar and finding inner resistance in response to ferguson usa. cultural studies—critical methodologies, 16(3), 320–323. https://doi.org/10.1177/1532708616634837 mccollom, s. (n.d.). hip hop: a culture of vision and voice. kennedy center website. accessed october 15, 2020. https://www.kennedycenter.org/education/resources-for-educators/classroom-resources/mediaand-interactives/media/hip-hop/hip-hop-a-culture-of-vision-and-voice/ millard-ball, a., desai, g., & fahrney, j. (2021, march). diversifying planning education through course readings. journal of planning education and research. https://doi.org/10.1177/0739456x211001936 morgan, j. (1999). when chickenheads come home to roost: a hip-hop feminist breaks it down. simon & schuster. patton, l., haynes, c., & croom, n. (2017). centering the diverse experiences of black women undergraduates. naspa journal about women in higher education, 10(2), 141–143. https://doi.org/10.1080/19407882.2017.1331627 pell, h. (2020). incorporating the stories of women, people of color, and lgbtq+ scientists into the physics classroom. the science teacher, 87(7). https://www.nsta.org/science-teacher/science-teacher-march2020/incorporating-stories-women-people-color-and-lgbtq petchauer, e. (2012). hip-hop culture in college students’ lives: elements, embodiment and higher edutainment. routledge. showing out 63 journal of effective teaching in higher education, vol. 4, no. 2 quaye, s. j., & harper, s. r. (2007). shifting the onus from racial/ethnic minority students to faculty: accountability for culturally inclusive pedagogy and curricula. liberal education, 92(3), 19–24. rademacher, t. (2018, january 31). maybe we should stop telling kids to be quiet. educationpost. https://educationpost.org/maybe-we-should-stop-telling-kidsto-be-quiet/ rendon, l., amaury, n., and pepe, j. (2004). theoretical considerations in the study of minority student retention in higher education. in j. m. braxton (ed.), reworking the student departure puzzle (pp. 5–25). vanderbilt university press. riddle, a. (2018, february 26). when white teachers avoid conversations about race, we marginalize students of color. education week. https://www.edweek.org/teaching-learning/opinion-when-white-teachersavoid-conversations-about-race-we-marginalize-students-of-color/2018/02 robinson, s. (2020). the aesthetics of the black arts movement. black art in america, 1(1). https://www.blackartinamerica.com/index.php/2020/08/24/the-aestheticsof-the-black-arts-movement/ rodriguez, s. (2001). giants among us: first-generation college graduates who lead activist lives. vanderbilt university press. doi:10.2307/j.ctv16h2nct sathy, v., hogan, k. sims, c. (2020). a dozen-plus ways you can foster educational equity. inside higher education. https://www.insidehighered.com/advice/2020/07/01/list-practical-ways-nonblack-faculty-members-can-help-dismantle-educational#.yginqriqkaw.link saunders, s., kardia, d. (1997). creating inclusive college classrooms. university of michigan center for research on learning & teaching. https://crlt.umich.edu/gsis/p3_1 shade, b. j. (1997). culture, style, and the educative process: making schools work for racially diverse students (2nd ed.). charles t. thomas. streamas, j. (2020) the war between “school time” and “colored people’s time.” teaching in higher education, 25(6), 709–721. https://doi.org/10.1080/13562517.2020.1782882 vansina, j. (1971). once upon a time: oral traditions as history in africa. daedalus, 100, 442–468. vansina, j. (1985). oral tradition as history. university of wisconsin press. showing out 64 journal of effective teaching in higher education, vol. 4, no. 2 valochi, s. (2012). activism as a career, calling and way of life. journal of contemporary ethnography. https://doi.org/10.1177/0891241612467202 washington college (n.d.). diversifying our syllabi: suggestions to incorporate diversity, equity & inclusion in our classes, https://www.washcoll.edu/_resources/documents/diversifying%20our%20s yllabi.pdf wichelns, k. (2021). black realism: or, a syllabus is still a terrible thing to waste. american literary realism, 53(2), 100–105. wilkerson, i. (2020). caste: the origins of our discontents. random house. microsoft word phelps_ward_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi: 10.36021/jethe.v4i2.186 researching, learning, and healing within the master’s house: a collaborative autoethnography robin phelps-ward, edd, ball state university, rjphelpsward@bsu.edu keneisha harrington, mal, clemson university lashia bowers, phd, coastal carolina university dion harry, med, north carolina state university maurice williams, msed, clemson university cherese fine, phd, southern illinois university-edwardsville travis smith, phd, university of florida abstract. this paper tells the story of resistance and efforts to work and mend within an anti-black institution: higher education. through a collaborative autoethnographic approach, seven black academics connected to the action research collective team (a group focused on supporting graduate students and cultivating equitable campus climates), explored how doing research as a team served as a mechanism for healing from the trauma of anti-black racism. this paper illustrates how researching, learning, and healing can manifest within research teams by emphasizing visibility, shared experience, authenticity, and community. keywords: blackness, collaborative autoethnography, research team “for the master's tools will never dismantle the master's house.” (lorde, 1984, p. 112) the backdrop: introduction whether the master’s tools manifest as strategic diversity plans, officers, committees, or book clubs, black people within the academy know all too well that such devices will not spur the revolution necessary to fully value and protect black lives. black people know such performative expressions of institutionally espoused values—which ultimately amount to aesthetically appealing documents, videos, and websites—will not lead to structural change, and these actions are not enough. they are not enough to prevent a student from being called the n-word when going to buy groceries in their college town, and they are certainly not enough to ensure students’ voices will be heard when they march in protest against racism. listening sessions, town halls, and diversity plans do not undo or adequately address the pain, isolation, and dehumanization of existing as a black person at a historically and predominantly white institution that communicates in obvious and understated ways their continued disregard for black lives. paradigmatic shifts in every arena (from research and teaching to advising and accounting) are necessary to truly transform institutions of higher education into more just, equitable, and anti-racist spaces. given the current sociopolitical environment that reinforces neoliberalism and maintains the cis-hetero white researching, learning, and healing within the master’s house 12 journal of effective teaching in higher education, vol. 4, no. 2 supremacist imperialist patriarchal regime, black folks must find pockets of reformation (as we always have) within predominantly white institutions to creatively resist, subvert, heal from, and undo oppressive systems. this paper tells the story of such resistance and efforts to work and attempt to mend within. in particular, we (a team of seven black academics) who worked collaboratively as part of the broader action research collective (arc), reflect on our collective research experiences as black scholars within arc to answer the research question: how can the act of doing research serve as a healing space for black people? gp are you with me?: about the action research collective (arc) during the wake of local and national anti-racist activism in the fall semester of 2016, arc formed at clemson university to engage in conversations about access and equity specific to the experiences of graduate students of color (gsoc). what started as a space for authentic discussion and a mechanism to support graduate students’ scholarly identities and research capacities, grew into a community of research partners (faculty, students, administrators, etc.) who engage in critical scholarly inquiry for positive and sustainable change at clemson and other predominantly white institutions (pwis). the goal of the arc team’s initial research study was to explore the experiences of gsoc at clemson university and understand their needs, obstacles, support, and ability to thrive at a pwi using a critical participatory action research design rooted in photovoice (bowers et al., 2020; dillard, 2020; latz, 2017). participating in critical participatory action research allows members to reclaim their agency “as they seek to promote greater self and social change” (dillard, 2020, p. 48). a major objective of the arc team was to serve as a counter-space for gsoc. counter-spaces are sites where “deficit notions of people of color can be challenged and where a positive collegiate racial climate can be established and maintained” (solórzano et al., 2000, p. 70). situated within an emancipatory research counter-space model “where students experience socialization processes of interaction, integration, and learning within an environment dedicated to inquiry that embraces emancipatory pedagogy, community, intersectionality, and critical praxis,” arc members organized the space to resist hegemonic ideologies of researching and collaborating (phelps-ward, 2020, p. 260). given this manuscript’s space confines, we refer readers to readings by bowers et al. (2020) and phelps-ward (2020) to learn more about the contours of the arc team’s history, organization, and maintenance. for us, by us, about us we want to be up front about our positionality as the authors of this manuscript. we are a group of seven black scholars brought together through our work within the clemson arc team. this work centers our experiences to encourage more black scholars, educators, and graduate students to create and find spaces that honor blackness. this paper is quite simply, for us, by us, and about us. as a way to resist traditional writing norms that tend to place value in an objective third person, we use “we” throughout to refer to ourselves as a group of authors, employ songs from black culture as manuscript headings, and provide a playlist for readers to heighten their somatic engagement with our narratives. further, we use black researching, learning, and healing within the master’s house 13 journal of effective teaching in higher education, vol. 4, no. 2 sitcom character pseudonyms in place of our names as we remain acutely aware of the potential negative consequences that often befall black academics when they speak their truths. the following paragraphs provide brief introductions for each of us. maxine shaw i am a black woman educator. i was familiar with arc, but thought it was only for students. during one of my conversations with gina, i mentioned i wanted to be engaged in more research, so she invited me to an arc meeting. after that meeting, i joined. fresh prince i am a black man, a christian, husband, father, son, and brother. arc was one of my saving graces during my transition to clemson. i was invited to this space by upper-level graduate students who became like guardian angels to me while at clemson. arc was family, community, and my village. khadijah winslow i am a black, womyn, mother, scholar, and doctoral candidate. my first semester at clemson i struggled adjusting to the subtle culture of anti-blackness, and i strongly considered transferring. during one of my lowest moments as a student i was introduced to arc, and i had a meeting with gina payne to learn more about it. after attending my first meeting, i knew i had found a home within the walls of the pwi. this community became my saving grace. dwayne wayne i am a black man and full-time educational leadership doctoral student at clemson university. fresh prince introduced me to the arc organization and invited me to attend a meeting. after attending the meeting, i immediately wanted to be involved as a teammate and help with projects while serving as a research partner. my involvement with arc has enhanced my team leadership skills and love for research. moesha mitchell i am a black womyn, educator, counselor, and doctoral candidate at clemson university in p-12 educational leadership. fresh prince invited me to an arc meeting. i quickly realized the benefits of being on a research team with people who looked like me. tj henderson i am a black man and full-time doctoral student in educational leadership. gina payne was the person who invited me to this space after knowing i wanted to researching, learning, and healing within the master’s house 14 journal of effective teaching in higher education, vol. 4, no. 2 explore more work that involved students of color as well as explore more areas of research. arc appealed to me as an avenue to make meaning of the concepts i was learning and the conversations i was having in class, as well as a way to put into practice what i began to learn from my assessment class in the previous semester. gina payne i am a black feminist, woman, mother, and assistant professor of higher education. i arrived at my work with arc at the nexus of feeling isolated as a black woman in a predominantly white department, experiencing the cultural taxation of supporting several graduate students with minoritized identities at clemson university who (like me) finally had enough of the institution’s racism, and interested in taking advantage of a graduate school internal grant to support graduate students’ professional development. i created arc alongside graduate students for “us.” back stabbers: anti-blackness in the ivory tower in academia, a system built on white supremacy (wilder, 2013), anti-blackness is pervasive. in today’s society, black scholars experience racism and oppression through the policing of black bodies on university campuses (west, 1993), the african american education debt (ladson-billings, 2006), and the commodification of black bodies as a source of university income (dancy et al., 2018). for example, during the resurgence of the black lives matter movement (in the summer of 2020), after police officers murdered breonna taylor, george floyd, and a slew of additional black people, college students protested across the country insisting on systemic change within the justice system and on their campuses, but in some cases black students and athletes had to decide not to voice their opinions or experiences and risk losing university-funded scholarships and/or other benefits (see frederick et al., 2017). this is not new. historically, the ivory tower has been a place of silencing, trauma, assault, unpaid labor, and multi-level racism for black faculty, staff, and students (dancy et al., 2018; kelly et al., 2017; volpe et al., 2021). as dancy et al. (2018) put it, “universities were created to educate the offspring of colonizers” (p. 182). as black people continue to create coalitions and forge communities in a system that was not designed for us, we partake in a social exchange that reifies anti-blackness. we make conscious decisions about how we will engage in and with this oppressive system. we can decide to give in and or we can decide to resist. arc is a space we created through conscious resistance efforts of organizing and meeting outside of the confines of a formal classroom setting, working collaboratively on a research project to address campus racism, and centering our daily experiences of anti-blackness through storytelling, reflective writing during meetings, and fellowship through social gatherings. who can i run to?: black healing black people in academia contend with more than racial trauma, we also face an assault on our humanity stemming from trauma rooted in anti-blackness (fairfax, 2020). while many scholars write about healing from racial trauma, a paucity of researching, learning, and healing within the master’s house 15 journal of effective teaching in higher education, vol. 4, no. 2 research addresses healing from the trauma of anti-black racism. to address black healing, we need to situate how scholars have analyzed racial trauma and antiblack racism. only then can we begin to conceptualize how healing from the trauma of anti-blackness manifests in academic spaces. racial trauma impacts people on a psychological and physiological level through symptoms similar to post-traumatic stress disorder such as flashbacks, hypervigilance, avoidance, headaches, and heart palpitations (comasdíaz et al., 2019). ways to cope with racial trauma include addressing racism, avoiding internalizing racist ideologies, and seeking support (chioneso, 2020 et al.; truong & museus, 2012). the field of mental health and psychology highlights the necessity to move beyond just coping and on to healing. healing from racial trauma is the unlearning of internalized racial stereotype messages about one’s own socially constructed race and race of others through actively working to stop participating in the system of racism (singh, 2019). the trauma of anti-blackness goes deeper than racial conflict; it is “an irreconcilability between the black and any sense of social or cultural regard” (dumas, 2016, p. 13). black people may collectively heal from antiblackness trauma by maintaining memories of african spiritual traditions, practicing sacred rituals, and engaging with social organizations (fairfax, 2020). in the spirit of sankofa, black people may return to african culture to understand what it means to heal. defined through an african-centered lens, healing is collectively shifting the mind, body, and spirit from suffering to repairing by maintaining a harmonious balance between the physical and spiritual realm (myers, 2013). scholars in p-12 education highlight the importance of critically altering their classroom pedagogical practices to promote spirit healing (hines & wilmot, 2018; love, 2016). professors in higher education can use critical pedagogical approaches such as intentional praxis, interdisciplinarity content, and courageous transparency (see tuitt et al., 2018) to help students begin the collective healing process by creating spaces for students to unlearn internalized blackness and restore harmony within. started from the bottom: research teams although research remains a mainstay of life in the academy, fueling the publish or perish mentality, hunt for grant dollars, and competition within the network of colleges and universities with very-high research institution goals and aspirations, few who lead large research projects gain instrumental education experiences to lead such research teams. in this paper, we define a research team as any group of three or more people working collaboratively on an empirical study. this definition includes all methodologies across disciplines, fields, and foci (e.g., humans and non-human related research) and the various configurations of parties who comprise the research team (e.g., faculty, students, staff, community members, etc.). in addition to minimal literature focused on the socialization and formal education about how to organize and manage research teams (kim, 2018; krockover et al., 2001; schmidt et al., 2012), a relatively small body of scholarly literature focuses on how such teams may serve as sites of support, communitybuilding, and activism, particularly against anti-black racism. researching, learning, and healing within the master’s house 16 journal of effective teaching in higher education, vol. 4, no. 2 we explore what might it look like for more graduate faculty to lead initiatives in which they teach about the work of building emancipatory research counter-spaces (phelps-ward, 2020). such research teams socialize graduate students, faculty, and staff to engage in critical scholarly inquiry and serve as spaces of healing and respite. further, we aim to discuss implications for ways to shift narratives of research for productivity to research for community-building, belonging, and healing. ultimately, we inquire about the possibilities and benefits for everybody black when community-building is centered and prioritized within the research process. this is how we do it: methods in an effort to carve out counter-spaces (solórzano et al., 2000) within the racist academy, we (a group of seven black people holding varying identities and roles as faculty and doctoral students) helped develop a research team for graduate students of color (gsoc) and, ultimately through this collaborative autoethnographic (cae) study (chang et al., 2013), began examining our own experiences of healing from racial trauma and our experiences as arc members. we decided to engage a cae methodological approach given its self-reflection orientation and opportunity for us to engage in solo and collective work. we engaged in a multi-step process of individual journaling, group discussions, and coding to answer the following research question: how can the act of doing research serve as a healing space for black people? data collection and analysis participants for this collaborative autoethnographic study included seven black researchers who also served as members of the arc. three of us are faculty members and four are doctoral students (at the time of authoring this manuscript). cae refers to a collaborative, autobiographical, and ethnographic research method that places researchers in the dual role of researchers and participants (chang et al., 2013). cae emphasizes situating the autobiographical data in a sociocultural context to understand the meaning of societal and cultural experiences and perspectives. for this study, we engaged in two rounds of self-reflection through written narratives in response to a set of prompts designed to elicit narratives about our experiences in arc (i.e., how did you enter the arc team? what was it like to participate? what did you learn? what did you gain? how did arc serve as a healing space?). such prompts led us to reflect about how and why we became involved in arc and what arc meant to us. we used a multi-step data collection and analysis process to analyze our written reflections. first, we met to discuss our responses to an initial set of reflective prompts about our overall experiences; we each talked about what we wrote and why. we video-recorded our discussion to assist us with triangulating data in the analysis phase. our discussion led us to identify additional prompts centered on healing and anti-black trauma given that our conversation gravitated toward the healing nature of the space. we returned to solo-writing for another round of selfreflection before engaging in a second group discussion. after reviewing each researching, learning, and healing within the master’s house 17 journal of effective teaching in higher education, vol. 4, no. 2 person’s written reflections (a total of 27 written pages) and the first videorecording of our group discussion (a total of two hours), we gathered for a third virtual meeting to share initial insights and build a plan for data analysis. we decided to pair nascent qualitative researchers with a more experienced researcher to code each participants’ written reflections. each pair was responsible for analyzing two participants’ written reflections (neither of which were their own) using charmaz’s (2006) process of initial line-by-line coding noting specific words, phrases, incidences, and actions in participants’ narratives. after reviewing all of our initial codes within a collaborative, cloud-based, electronic spreadsheet, we each created researcher memos to support the development of larger axial codes across the data. finally, we reconvened as a research team to discuss our axial codes to collectively synthesize, categorize, and build themes. never would have made it: findings four major themes emerged from the analysis in regard to our research question: how can the act of doing research serve as a healing space for black people? through our analysis, we found arc served as a healing space for us as black people by providing opportunities for visibility, recognizing shared realities, being authentic, and being in community. we discuss each of these themes next with excerpts from our written reflections. visibility conducting research collaboratively created a space for us to feel seen, heard, valued, and appreciated. outside of arc, navigating the research environment at an institution where not many people shared our identities or cultural values was difficult. during our engagement with arc’s research and each other, we developed a psychological sense of connection to our campus community as we worked to transform it. visibility showed up in two ways during the study: through shared blackness (i.e., apparent cultural similarities) and having a seat at the research table as a valued contributing member of the arc team. arc members expressed being able to walk into research meetings and events excited about seeing other black people and being “unapologetically black.” the fresh prince stated, “i felt like people heard me and cared about what i had to say. people in that space honored my blackness and didn’t try to tell me that i wasn’t professional, or i should act like this or that.” in essence, members of the team appreciated being seen for who they were. moreover, arc provided a space where social cues are recognized and valued. for example, khadijah described the feeling of belonging by stating, it’s like how black people walk past each other and give each other “the nod” no matter what, but we’re doing that in a physical space. it’s like saying i see you; i feel you; it’s going to be alright. researching, learning, and healing within the master’s house 18 journal of effective teaching in higher education, vol. 4, no. 2 these shared cultural experiences in arc provided enthusiasm toward conducting research collectively in a campus environment where black people are typically invisible. participating in collaborative research also helped us feel like we belonged as a contributing member of the team. this level of comfort helped us develop appreciation for various perspectives, opinions, and voices of other teammates. maxine and the fresh prince (respectively) described the feeling of visibility as, arc allowed me to show up as a collaborator and a resource, not only for research but for the other projects within arc. i got over me “just being an administrator,” and i feel that my knowledge and contributions to the space are valued and respected. (maxine) as time went on, arc became a safe space both for me personally and academically. i was able to ask questions during the meeting and not feel ashamed or that my questions were dumb. (fresh prince) team members viewed themselves as research partners whose ideas and suggestions were important and heard. ultimately, we felt we had a proverbial seat at the table. as moesha described, “i was not hoping for a seat at the table. instead, i was the table. we were the table.” these conditions of visibility shaped how arc members viewed themselves and each other. moesha explained, “arc is a sacred place that encompasses many wonders.” in this sacred space, black people were seen, heard, and appreciated. shared realities the theme of shared realities refers to the ways doing research collaboratively helped us witness and experience shared emotions, stories, and experiences. through the process of recognizing our shared realities, we were able to reflect on similar experiences of trauma from anti-black racism. together we received validation and understanding of our experience. we expressed viewing arc as a space to connect with others about the realities of living through traumatic anti-black racial situations in academia. dwayne articulated, “storytelling manifests as a vehicle for healing by allowing me to reflect on my emotions about my current environment.” sharing personal stories allowed us to reflect and connect with others who may have felt similar emotions. fresh prince shared, “being able to share my experiences with other students was a freeing experience for me. i felt as if i was able to unload burdens and negative energy that i had been carrying for so long.” although we each held different roles within the university, we were able to support each other and be transparent about our personal experiences. gina expressed: researching, learning, and healing within the master’s house 19 journal of effective teaching in higher education, vol. 4, no. 2 on days when i would come from department meetings or interactions exhausted, worn out, overwhelmed, overlooked, invalidated, and undervalued, i would go to arc and connect with people who understood what i was going through (‘cus they experienced it too, just in another role, maybe as a student, maybe as a staff member), cared about my presence and well-being, and were willing to offer advice and support. gaining support from others in different roles helped foster opportunities for healing. members also expressed the notion that black people cannot rely on the oppressors for their healing. for example, while the university may create associations, trainings, and forums, these opportunities seldom provide counterspaces where we can genuinely and openly express ourselves with minimal surveillance, filtering, or code-switching. khadijah shared, we heal each other by seeing each other where we are and relating to each other. we tell stories, share our experiences and connect on levels that are more than academic…and i think that’s how we heal as a race. we must heal ourselves because the oppressors aren’t going to do it for us. collectively, we also shared stories of triumphs as a reminder of our abilities to conquer anti-blackness. we were able to “see” ourselves and each other. khadijah reported growth when she said, “i honestly feel like i’m a boss…i can look back on those comments from my professor and my early feelings of inadequacy and imposter syndrome and laugh because i know how capable i am.” authenticity “coming into a space where i could be my authentic self and be loved and valued is healing.” – khadijah as black people at a pwi, we often have to “put up a front,” use code-switching, and employ other means to survive the campus environment. within an oppressive space, black people often express feelings of discomfort and invalidation. arc allowed space for various forms of black expression to collectively engage, share experiences, and conduct research with a spirit of authenticity. gina explained this occurrence: what kept me going is knowing i had a space to be myself, to laugh, to experience and to not worry about the pressures of being uncomfortable, exploited, or misnamed that i typically had in other spaces on campus. authenticity refers to a representation of a preferred state of being or set of qualities that people in a particular time and place have come to agree represent an ideal (vannini & williams, 2009). in the context of arc, authenticity reveals the ways researching collaboratively allowed us to have a space to exist and engage in research as our full selves. arc made us feel comfortable enough to remove our walls and be ourselves. tj stated, “in this space, arc breaks down those barriers researching, learning, and healing within the master’s house 20 journal of effective teaching in higher education, vol. 4, no. 2 and fears and provides students with the opportunity to express their full selves and receive validation.” arc is a space where the survival mechanisms rooted in anti-blackness were less needed in the arc environment given a sense of comfort and the promotion of authenticity. as researchers and scholars, arc made it easier to embrace our blackness. moesha shared, in arc, my language is scholar-appropriate, but i do not have to exert extra energy to code-switch into a way i would speak in a research group that does not aim to hear all voices. i could reference something from black culture such as the phrase “black girl magic” and the arc members would know exactly what i am referring to. the ability to be authentic, be in community, feel supported, and experience validation promoted individual and collective healing for arc members. less energy was directed towards “putting up a front” and more toward being one’s authentic self and appreciating blackness. community forging communities of resistance in anti-black spaces has historically been a way black people strive toward healing and thriving despite oppression (johnson & lubin, 2017; lorde, 198; spencer, 2016). through our research, we found that arc served as a community of both physical and emotional support for black members. the experience of “doing it together” and engaging in collaborative research that had the potential to subvert anti-blackness within the academy allowed us to feel a sense of collective healing when engaging in our work. gina said, “i gained a community of support outside of a department and college that felt isolating, racist, and invalidating.” these comments showcase the necessity of community for black folks who are actively trying to heal within racist systems. tj said it this way: as a collective, arc allows us to build community out of common interests and “misery” we experience as gsocs…arc has served as a vehicle that has connected graduate students who may have never met otherwise, building mentorship opportunities, strengthening our network, and validating the experiences we have at clemson, along with developing ways to disrupt and transform the negative experiences. within our written reflections, the majority of us discussed how arc became a place of emotional support, noting the level of vulnerability we were able to have within the arc community. for example, maxine said, “[healing from racial trauma comes from] having safe spaces to be vulnerable, where there’s an opportunity to share stories, feelings, and other thoughts.” moesha echoed this sentiment and said, “this was my first experience discussing my goals with my classmates in a family/community atmosphere…this was healing for me because many of my close friends and family members do not understand what i am trying to do.” these researching, learning, and healing within the master’s house 21 journal of effective teaching in higher education, vol. 4, no. 2 excerpts illustrate the profound impact arc had on black team members through the strong emotional support the space provided. several of us within the cae discussed the importance of having the physical support of other black people in predominantly white spaces for events and milestones such as dissertation proposals and conference presentations. the fresh prince said, “for me, collective healing showed up by way of supporting each other outside of the space.” for some of us, this type of support served as a reminder that we are not alone on our journeys. ultimately, we each shared how being a part of the arc community provided multiple layers of support that helped us navigate within a pwi and resist racist messages about our value, capability, and power as scholars. such layers of support provided space to acknowledge and overt and covert racism we experienced and offered an opportunity for us to find ways to address and combat such racism through research. thinking of a master plan: discussion and implications our research illustrates the possibilities of research teams designed to center and emphasize collective care and social action, particularly in a racist campus environment. we focus our discussion on resistance to anti-blackness in the academy and recommendations for and by black people working within colleges and universities as students, staff, faculty, and administrators. ultimately, we advocate institutional leaders (e.g., graduate school deans) allocate additional resources to support the development of emancipatory research counter-spaces (phelps-ward, 2020) that bring black people together to collectively dream, heal, and resist within the structures of higher education. the white supremacist structure of the academy seldom allows for spaces where we as black folks (and all the ways we present, live, speak, love, believe, and know) can simply be ourselves, confidently engaging in research that seeks to dismantle oppression. when such counter-spaces (solórzano et al., 2000) do exist, they typically manifest within black and multicultural centers and programs and organizations that center minoritized populations (e.g., black student governments, fraternities and sororities, and identity-based living-learning communities). such counter-spaces rarely exist within research teams and are not emphasized within the scholarly literature. our findings support the notion that community-based research teams can lessen fears of engaging in research while providing opportunities for collective criticalconsciousness raising and community. research woes (i.e., feelings of impostor syndrome in the area of research and writing skills, fears of pursuing a research topic rooted in blackness, and epistemological limitations born from objectivist and eurocentric socialization) stem from anti-blackness. as a team of researchers on the arc team, we experienced research woes and anti-blackness as we navigated uninviting spaces full of microaggressions (e.g., predominantly white classes with few opportunities to engage with texts by people of color), invalidation (e.g., questions about why [we] decided to focus our interests or research on blackness), and exploitation (e.g., being asked to work in spaces without being valued for our researching, learning, and healing within the master’s house 22 journal of effective teaching in higher education, vol. 4, no. 2 intellectual contributions) as aligned with much of the literature on black student and faculty experiences (dancy et al., 2018; kelly et al., 2017; volpe et al., 2021). we resisted anti-blackness through storytelling, naming the racism we experienced, celebrating with each other, and seeing value in what each person brought to the team’s work. the act of reflecting on our trauma stemming from anti-blackness— and sharing those stories with each other—afforded a space to discuss and process our thoughts, feelings, and desires for a better campus. although the full arc team is not all black (i.e., the team includes white and non-black team members all committed to the project of centering members with minoritized identities and working to improve the campus racial climate), arc exists as a space where black members feel comfortable to talk about racism and our collective plans to dismantle it. the concepts of visibility, authenticity, shared experiences, and community contribute to the how of starting to heal from the trauma of anti-blackness through research teams. arc built a community of scholars through faculty teaching how to conduct critical participatory action research, role modeling a process that emphasized care, and supporting members through the process, regardless of level of experience with research. the team’s work even evolved to a point where students felt empowered to teach new members how to conduct research and support each other through the research process. black educators can facilitate similar spaces, but not without first unlearning and relinquishing white notions of leading, researching, and teaching. such work involves seeing students as knowledge holders, resisting the temptation to do fast research, and leaning into healing practices (e.g., reflection, storytelling, dialogue, and fellowship). further, graduate programs must teach graduate students (i.e., future faculty and administrators) how to establish research teams rooted in community-building and liberation. while such action would require an ideological shift and intentional curriculum, the change presents opportunities for a generation of faculty and administrators who cultivate research teams and labs that build students up rather than break them down—an emancipatory perspective. to actualize such change, university leaders must reward those who initiate and maintain research teams where emancipatory research counter-spaces can flourish within the promotion and tenure process through funding and allocations of campus space and financial resources. can we talk for a minute?: conclusion the research team experiences of black people working within pwis warrant more research attention. the act of resisting, healing, and loving all while engaging in research should not go overlooked as a liberatory practice. when factoring in a sociopolitical climate where racism, economic suffering, and health disparities persist, black folks still find a way to make it and find healing in the process. researching, learning, and healing within the master’s house 23 journal of effective teaching in higher education, vol. 4, no. 2 figure 1 researching, learning, and healing the master’s house playlist note: readers can listen to the song’s referenced throughout the manuscript via the playlist curated by the authors: https://apple.co/2oe2ttd conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. researching, learning, and healing within the master’s house 24 journal of effective teaching in higher education, vol. 4, no. 2 references bowers, l., harrington, k., phelps-ward, r., allen, c., smith, t., & dsouza, n. (2020). #mytruthmatters: narratives of graduate students of color at a predominately white institution. journal of ethnographic and qualitative research, 14(4), 233–248. https://www.academia.edu/49949135/bowers_et_al_2020_my_truth_matter s_jeqr_1 chang, h., ngunjiri, f. w., & hernandez, k. c. (2013). collaborative autoethnography. left coast. charmaz, k. (2006). constructing grounded theory: a practical guide through qualitative analysis. sage. chioneso, n., hunter, c., gobin, r., mcneil smith, s., mendenhall, r., & neville, h. (2020). community healing and resistance through storytelling: a framework to address racial trauma in africana communities. journal of black psychology, 46(2-3), 95–121. https://doi.org/10.1177/0095798420929468 comas-díaz, l., hall, g., & neville, h. (2019). racial trauma: theory, research, and healing: introduction to the special issue. the american psychologist, 74(1), 1–5. https://psycnet.apa.org/fulltext/2019-01033-001.pdf dancy, t. e., edwards, k. t., & davis, j. e. (2018). historically white universities and plantation politics: anti-blackness and higher education in the black lives matter era. urban education, 53(2), 176–195. https://doi.org/10.1177/0042085918754328 dillard, n. (2020). designing research to dismantle oppression: utilizing critical narrative analysis and critical participatory action research in research on mothering and work and beyond. journal of international women's studies, 21(7), 47–60. https://vc.bridgew.edu/jiws/vol21/iss7/5/ dumas, m. (2016). against the dark: anti-blackness in education policy and discourse. theory into practice, 55(1), 11–19. https://doi.org/10.1080/00405841.2016.1116852 fairfax, g. n. (2020). the need to be: since 1619, trauma and anti-blackness. phylon, 57(1), 56–75. https://www.jstor.org/stable/26924987 frederick, e., sanderson, j., & schlereth, n. (2017). kick these kids off the team and take away their scholarships: facebook and perceptions of athlete activism at the university of missouri. journal of issues in intercollegiate athletics, 10, 17–34. https://csri-jiia.org/kick-these-kids-off-the-team-andtake-away-their-scholarships-facebook-and-perceptions-of-athlete-activismat-the-university-of-missouri/ researching, learning, and healing within the master’s house 25 journal of effective teaching in higher education, vol. 4, no. 2 hines, d. e., & wilmot, j. m. (2018). from spirit-murdering to spirit-healing: addressing anti-black aggressions and the inhumane discipline of black children. multicultural perspectives, 20(2), 62–69. https://doi.org/10.1080/15210960.2018.1447064 johnson, g. t., & lubin, a. (eds.). (2017). futures of black radicalism. verso books. kelly, b. t., gayles, j. g., & williams, c. d. (2017). recruitment without retention: a critical case of black faculty unrest. the journal of negro education, 86(3), 305–317. kim, c. (2018). building faculty-student research teams via a vertically integrated project (vip) framework. scholarship and practice of undergraduate research, 1(4), 48. krockover, g., adams, p., eichinger, d., nakhleh, m., & shepardson, d. (2001). action-based research teams: collaborating to improve science instruction: injecting energy into science education. journal of college science teaching, 30(5), 313–317. ladson-billings, g. (2006). from the achievement gap to the education debt: understanding achievement in u.s. schools. educational researcher, 35(7), 3–12. https://www.jstor.org/stable/3876731 latz, a. o. (2017). photovoice research in education and beyond: a practical guide from theory to exhibition. routledge. lorde, a. (1984). the master’s tools will never dismantle the master’s house. sister outsider: essays and speeches. crossing press. love, b. (2016). anti-black state violence, classroom edition: the spirit murdering of black children. journal of curriculum and pedagogy, 13(1), 22–25. http://dx.doi.org/10.1080/15505170.2016.1138258 myers, l. (2013). restoration of spirit: an african-centered communal health model. journal of black psychology, 39(3), 257–260. https://doi.org/10.1177/0095798413478080 phelps-ward, r. (2020). emancipatory research counter-spaces: theorizing about black doctoral student socialization. in j. c. weidman & l. deangelo (eds.), socialization in higher education and the early career: theory, research and application (pp. 241–268). springer. schmidt, a. h., robbins, a. s. t., combs, j. k., & freeburg, a. (2012). a new model for training graduate students to conduct interdisciplinary, interorganizational, and international research. bioscience, 62(3), 296–304. https://doi.org/10.1525/bio.2012.62.3.11 researching, learning, and healing within the master’s house 26 journal of effective teaching in higher education, vol. 4, no. 2 singh, a. (2019). the racial healing handbook: practical activities to help you challenge privilege, confront systemic racism, & engage in collective healing. new harbinger publications. solórzano, d., ceja, m., & yosso, t. (2000). critical race theory, racial microaggressions and campus racial climate: the experiences of african american college students. journal of negro education, 69(1/2), 60– 73. https://www.jstor.org/stable/2696265 spencer, r. c. (2016). the revolution has come. duke university. tuitt, f., haynes, c., & stewart, s. (2018). transforming the classroom at traditionally white institutions to make black lives matter. to improve the academy: a journal of educational development, 37(1), 63–76. https://doi.org/10.1002/tia2.20071 truong, k., & museus, s. (2012). responding to racism and racial trauma in doctoral study: an inventory for coping and mediating relationships. harvard educational review, 82(2), 226–254. https://scholar.harvard.edu/truong/publications/responding-racism-andracial-trauma-doctoral-study-inventory-coping-and vannini, p., & williams, j. p. (eds.). (2009). authenticity in culture, self, and society. ashgate. volpe, v. v., katsiaficas, d., & neal, a. j. (2021). “easier said than done”: a qualitative investigation of black emerging adults coping with multilevel racism. cultural diversity and ethnic minority psychology, 27(3), 495–504. http://dx.doi.org/10.1037/cdp0000446 west, c. (1993). race matters. beacon. wilder, c. s. (2013). ebony & ivy: race, slavery, and the troubled history of america’s universities. bloomsbury press. microsoft word okello&stewart_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi: 10.36021/jethe.v4i2.255 because we know: toward a pedagogical insistence on black mattering wilson k. okello, university of north carolina at wilmington, okellow@uncw.edu terah j. stewart, iowa state university abstract. black mattering is contested terrain. as we write this, more than 25 states and municipalities have proposed or passed legislation banning critical race theory (crt) and the incorporation of material(s) that upset the normative curricular and pedagogical conditions of whiteness. against this backdrop, what is mattering for black people? this essay interrogates the formations and utilizations of educational pedagogies and curriculum. it raises questions about the implicit intent of these mechanisms on and for the lives of black people, with specific attention to the notion of mattering. plainly, pedagogies and curriculum that have failed to center or theorize the ways anti-blackness facilitates projects of unmattering have engendered a set of conditions that reproduce anti-black racism in and beyond the educational context. to address these conditions, we conceptually trouble the notion of mattering by meditating on a set of priorities urgently embodied by black passage into and through anti-black spaces and places. keywords: black mattering; anti-blackness; pedagogy; curriculum doors can be real, imaginary, and metaphorical. doors leading into the middle passage did more than transport black people; it transformed black bodies into black flesh. entering through its gates meant that one would never return (the same). for brand (2012), "one enters a room and history follows; one enters a room and history precedes. history is already seated in the chair in the empty room when one arrives" (p. 6). thus, "our inheritance in the diaspora is to live in this inexplicable space" (p. 3), in the along, to consider what this door, or doors, instantiated in and through black bodies. these opening pages theorize the door as a portal; a reckoning; an entry; a refusal, for we know: perhaps, we have always have call it borrowed, brokered over bones and flesh, broken and spilled libation, running beneath our seas beat and bless us forward, though boats be tossed to and fro, be still our rest, and peace and because we know, this essay interrogates the formations and utilizations of educational pedagogies and curriculum. it raises questions about the implicit intent of these mechanisms on and for the lives of black people, with specific attention to the notion of mattering. plainly, pedagogies and curriculum that have failed to center or theorize the ways anti-blackness facilitates projects of unmattering have engendered a set of conditions that reproduce anti-black racism in and beyond the educational context. to address these conditions, we conceptually trouble the notion of mattering by meditating on a set of priorities urgently embodied by black passage into and through anti-black spaces and places. because we know 2 journal of effective teaching in higher education, vol. 4, no. 2 mattering as it is understood in the student development canon, mattering is in conversation with marginality. schlossberg (1989) examines this relationship to discuss the importance of mattering for students navigating higher education, particularly those in transition. according to schlossberg (1989), transitions represent those events or non-events that affect how one understands themselves concerning others. the premise presumes that when students experience marginalization, they tend to feel like they do not matter to anyone. mattering coheres on the core belief that [we] matter to someone, that our living means something to those with whom we are in a relationship. mattering operates on two primary levels, interpersonal and social (rosenberg, 1985; rosenberg & mccullough, 1981). the interpersonal dimension accounts for the ways one matters to others in their network while the social dimension considers one's indispensability in the social structure and institutions that encapsulate living. explicating this point, rosenberg and mccullough (1981) highlighted three fundamental dimensions for mattering: attention, ego-extension, and dependence. regarding attention, individuals feel like they command recognition and notice by others, such that their presence is intensely felt and their absence missed. egoextension denotes the belief that others are genuinely interested in what one has to say, think, or desire. relatedly, dependence posited that others need or rely on an individual in some way. schlossberg (1989) located a fourth dimension that she proposed as appreciation, or the sense that one is esteemed and deemed worthy for their efforts by those in authority. insignificance or the feeling that one does not matter can prompt an existential crisis. following schlossberg (1989), the classroom should be read as a site of transitionpassage, entrance through doors, both physical and metaphorical—and therefore, must grapple with the potentiality of mattering. mattering, in so far as it stresses connectivity, is built on optimism, or "the force that moves you out of yourself and into the world in order to bring closer that satisfying something that you cannot generate on your own" (berlant, 2011, p. 2) but feel drawn to as a pathway into some form of belonging. constructing a sense of worth on optimism, however, can have its drawbacks. when optimism fails or becomes "cruel," the object that demands one's attachment "actively impedes the aim that brought you to it initially" (p. 2). stated differently, the object/scene/attachment becomes cruel when it "ignites a sense of possibility that makes it impossible to attain the expansive transformation for which a person or a people risks striving" (p. 2). against this backdrop, what is mattering for black people? the language of mattering, as written, gives way to the subjectivity of perception, where, all things being equal, one may feel as if they matter; in the same way, another may suggest that they do not matter. however, understated in these claims to mattering is the function of anti-blackness as a structuring apparatus that functions to enclose— surveil, patrol, monitor, and place limitations on black livability. because we know 3 journal of effective teaching in higher education, vol. 4, no. 2 mattering and ordinary crisis importantly, anti-blackness puts forth the irreconcilable truth that black people exist in a structurally antagonistic relationship with society. wilderson (2010) considers blackness "a paradigmatic impossibility" because to be black is to be "the very antithesis of a human subject" (p. 9). anti-black scholarship is necessarily motivated by the question of black suffering, as it interrogates the psychic and material assault on black flesh. taken together, the body becomes the site for the manifestation of anti-blackness—the constant humiliation, murder, disgust, and disregard for black life and humanity. anti-blackness, more specific than discussions on the effects of racism toward black people, insists on "grappling with the position of the black person as socially dead—denied humanity and thus ineligible for full citizenship and regard within the polity" (dumas, 2016, p. 12). the presumption of life and citizenship that this stance argues against is verified by mappings of historical terror in the western, united states context, involving black flesh (okello, 2020). an honest theorization on anti-blackness must continuously wrestle with what it means to have one's very existence as black constructed as a problem—for white people, for the public (good), for the nation-state even as a problem to be resolved in the educational context. here, the very technologies and imaginations that allow a social recognition of the humanness of others systematically exclude this possibility for black people. in this sense, black people cannot be human, and, for all intents and purposes, they are something other than human. this baseline assumption explicitly critiques the social dimensions of mattering discussed by rosenberg and mccullough (1981) and advanced by schlossberg (1989) and raises questions about the interpersonal tenets. critique of the interpersonal rests on the belief that sociopolitical systems are inextricably bound to the personal and, as such, inform how individuals engage and make meaning of interpersonal relationships. if anti-blackness is permanent and enduring, then blackness is perpetually at odds with the notions of appreciation, attention, importance (ego-extension), and dependence, tenets germane to mattering. in other words, passage through the door of no return as a heuristic for reading the value of black people, and further, their capacity to belong, interrupts possibilities threaded in and assumed by schlossberg's (1989) dimensions of mattering. accordingly, the question for mattering, theorized through anti-blackness, asks, what does it mean to matter when the organization of human sociality and politics do not, and cannot, comprehend the black as legible and worthy of mattering? this inherent conflict outlines the ordinariness (shange, 2019) of anti-blackness, where blackness in educational learning spaces is understood as a type of "impasse shaped by a crisis in which people find themselves developing skills for adjusting to newly proliferating pressures" (berlant, 2011, p. 8). in the impasse, black people, specifically black pedagogues, have embodied what it means to curate a liberatory agenda that comes to terms with the cruel optimism of (not) mattering. relatedly, the impasse calls forth an acute recognition of power relations and enclosures surrounding them toward ways of living and sustaining the self. because we know 4 journal of effective teaching in higher education, vol. 4, no. 2 black pedagogies the realities of anti-blackness leave little room for pretense in what is required of teaching and learning in higher education. speaking at uc berkeley, baldwin (1974) would say it this way: for a black person to get an education in this country, [they have] to have a lot of guts first of all, and this institution is like many institutions, which means that it is a racist institution, there is no way around that. all the american institutions are racist. and to get an education under those circumstances is a tremendous act of the will. black people very largely educate themselves. baldwin posits that white institutions and the pedagogies and curriculum that direct teaching and learning, at best, fail black students, and at worst, mis-educate them. it would seem then that black students, by way of traditional schooling, are gaining a political education about what is expected of them if they are to fit into the united states’ social structure. even in the most progressive models, one may still experience ideologies of absence (stewart, 2019) that enclose black ways of knowing and being in support of respectable comportments that privilege whiteness. of note, schooling achieved through a liberal discursive tradition is markedly unoriginal in the deployment of reform-based practices that stress acquiescence and unthreatening expressions of black being, such that theorizing blackness will not consequently undo repressive regimes’ forms of governance. progressive projects, generated by white institutions, equally, if not more discreetly, depend on obedient black subjects (cox, 2015; shange, 2019) against all likelihood, the transformation of black bodies into black flesh, humans into things (shange, 2019), gave birth to a different kind of creativity. this ontoepistemological metamorphosis would forever define what black was, is, or was capable of becoming (alexander, 2004). through the door and into the hold (wilderson, 2010), where death waited in stow, survival meant imagining and curating life-giving practices that would intervene on denials of citizenship. encased in these practices were inheritances that m. jacqui alexander (2005) would call pedagogies. the language of pedagogies invokes curriculum or the production and reification of knowledge, both formally and informally, in and across various sites. at the heart of pedagogies is the imperative "of making the world in which we live intelligible to ourselves and each other—in other words, teaching ourselves" (alexander, 2005, p. 6). relatedly, the language of pedagogies emphasizes interdependence, established by a mutual investment in teaching and learning. pedagogies, in this way, are epistemic and ontological projects that summon "subordinated knowledges that are produced in the context of the practices of marginalization" (p. 7). through the door of no return, the middle passage represents one such context of marginalization that produced pedagogies bent on mattering. black students, through the gates of higher education, and classrooms within them, point to another such context. because we know 5 journal of effective teaching in higher education, vol. 4, no. 2 black mattering in an open letter to angela davis in 1970, james baldwin abides in the protracted diminutive history of blackness in the united states. we begin here and record baldwin at length as one point of entry into black mattering. we lift a portion of this letter to demonstrate the ways black mattering can flow beyond the formal educational context and be enacted as social-political praxis. additionally, the scene and circumstances that compelled baldwin to write this letter offer a picture of carceral technologies that work to enclose the lives, spirits, and freedom of black people. davis' imprisonment is an egregious manifestation of anti-black sentiment. yet, black people broadly, and black girls and women in particular, in schools daily, must devise social and cultural choreographies to disrupt the pedagogical rhythms of captivity, progressive (read: anti-racist) or otherwise. baldwin (1970) wrote: we must do what we can do, and fortify and save each other—we are not drowning in an apathetic self-contempt, we do feel ourselves sufficiently worthwhile to contend even with inexorable forces in order to change our fate and the fate of our children and the condition of the world! we know that a man is not a thing and is not to be placed at the mercy of things. we know that air and water belong to all mankind and not merely to industrialists. we know that a baby does not come into the world merely to be the instrument of someone else's profit. we know that democracy does not mean the coercion of all into a deadly—and, finally, wicked—mediocrity but the liberty for all to aspire to the best that is in him, or that has ever been. we know that we, the blacks, and not only we, the blacks, have been, and are, the victims of a system whose only fuel is greed, whose only god is profit. we know that the fruits of this system have been ignorance, despair, and death, and we know that the system is doomed because the world can no longer afford it—if, indeed, it ever could have. and we know that, for the perpetuation of this system, we have all been mercilessly brutalized, and have been told nothing but lies, lies about ourselves and our kinsmen and our past, and about love, life, and death so that both soul and body have been bound in hell... if we know, and do nothing, we are worse than the murderers hired in our name. if we know, then we must fight for your life as though it were our own—which it is—and render impassable with our bodies the corridor to the gas chamber. for, if they take you in the morning, they will be coming for us that night. (para. 14) there is no going back. baldwin knows this. black mattering assumes passage into the longer wakes of history for black people, and once there, a sitting with realities of that passage. if to move through these doors is to take up a posture of history's burden and willfully inject that consciousness into one's teaching philosophy, mattering cannot remain idle-static, immobile, stagnant, and must be complicated in ways that resist objectivity. mattering invested in living black people must think beyond the conventional tropes that define acceptance and appreciation and wrestle with the cruelty of anti-black racism as a process of necropolitical organizing, where black people are already condemned to death and dying. because we know 6 journal of effective teaching in higher education, vol. 4, no. 2 pedagogies of the flesh as black mattering at the very least, we know that freedom and therefore mattering, "is a practice...to want freedom is to welcome struggle" (love, 2019, p. 9), but what are the pathways, innovations, decisions, and nondecisions, that facilitate this flight and resist generic solutions to the problem of racism? offering insight on a praxis of black mattering, gwendolyn brooks (1991), in her poem speech to the young: speech to the progress—toward (among them nora and henry iii), wrote "live not for battles won/live not for the-end-of-the-song/ live in the along" (p. 23). as a poetics of survival, to live in the long "proposes an injunction, a proposition…a declaration of war" (kelley, 2002, p. 192) that commits to building possibilities and new conceptions of black liberation. though nestled in the hold of the ship (wilderson, 2010), to live in the along is to enact an imagination bent on black people's material and epistemic desires. to live in the along is to "produce beautifully, visceral, and eloquent literature, photography, visual art, and films that explain and endure our suffering, soundscapes…body movements that express pain and joy simultaneously" (love, 2019, p. 8). moreover, following baldwin (1970), black mattering, in part, is actualized at the convergence of radical divestment in anti-blackness and loving [black] flesh. regarding the former, the possessive investment in anti-blackness cannot be rectified by learning "how to be anti-racist." it necessitates a radical divestment in the project of anti-blackness and a refocusing of energies and resources. it requires an abolitionist ethic that surrenders negotiations with anti-black structures and refuses the project altogether. divestment is clear in that whiteness is irredeemable, and black mattering begins and ends at [our] belief in its merits. this commitment gives way to loving flesh. although the surveillance of black people, materialized by anti-blackness and systems of terror means that black people must wrestle with external and internal disciplining that reads their bodies as unlovable things, loving flesh is personal, discursive, and collective, engendering a willingness to look, see, and account for the embodied lives of black people (okello, 2020). consequently, to take up black mattering pedagogically is to dedicate oneself to enlivening flesh. pedagogies of the flesh note the constraints of carecerality as more than a physical occurrence, but the psychic, spiritual, and emotional states that black people are trained into and take on as ways of being. pedagogies of the flesh fully engage the black body-mind as a living, moving entity deserving of humanity. baldwin's letter to davis projects this mattering. affirming the collective at the onset, baldwin gets clear about who and what is at stake. crossing over and beyond the tendency to lift individuality in the quest for living, baldwin employs and affirms the collective we as a site of struggle. one gets the sense here that mattering could not just be about ego-extension, as in tying one's sense of value to the works that they might otherwise perform, but he clarifies that davis requires protection because the collective we are deserving of protection. to "fortify" and "save" each other re-presents the best of the black radical tradition, whereas because we know 7 journal of effective teaching in higher education, vol. 4, no. 2 mutuality does not presume sameness; instead, mutuality is an invitation to materialize freedom that says, i cannot be free until we are all free. in this special issue, robin phelps-ward, keneisha harrington, lashia bowers, travis smith, cherese fine, dion harry, and maurice williams consider how to structure research teams such that they serve as a community building space and, one, where black scholars can experience belonging as they negotiate anti-black educational places. similarly, ginny jones boss, tiffany davis, and christa porter underscore the potential hazards that lay in waiting should instructors continue to abandon framing their courses deliberately—through syllabus, citations, and course designs—with black mattering in mind. clarification there exists a will to social, political, and cultural amnesia in a western, united states context that derides the need for close examinations. this derision fears the interruption of progress. forgetting seems to be more desirable, less threatening. forgetting, at times, "stands in for never having known or never having learned something, the difference between staying in tune with the source of our wisdom and relying on borrowed substitutes" (alexander, 2005, p. 276). baldwin (1970) interrupts this training by confronting miseducation levied against black people, "we have all been mercilessly brutalized, and have been told nothing but lies, lies about ourselves and our kinsmen and our past, and about love, life, and death." the recognition seems to be that a corrective is necessary, and thus, memory intervenes on individual and collective amnesia, nostalgia, and the logic that promotes fragmentation of the mind, body, and spirit. taking up this project, toby jenkins, gloria boutte, and kamania wynter-hoyte posit that higher education cannot simply include black cultural experiences and activities into the college experience and call that mattering. there must be a mindset change among those who teach and lead that is informed by black cultural values, beliefs, habits, and histories. relatedly, in the spirit of leading with black cultural habits and beliefs, christina morton’s offerings describes how providing students with creative outlets to express themselves in and coursework to meaningfully recognize global antiblack solidarity at work can help them process their experiences and produce material that is humanizing, liberating, and life-giving. emphasizing agency baldwin emphasizes agency (okello & white, 2019) in the absence and, perhaps, the impossibility of full regard and appreciation. critically engaging the concept, he moves to consider the body's movement against methods of enclosure, writing, "we are not drowning in an apathetic self-contempt, we do feel ourselves sufficiently worthwhile to contend even with inexorable forces in order to change our fate and the fate of our children." notably, he does not implore an agency devoid of history or hard memory. so, too, it should not be read as a variant of rational western sensibilities that situate public and private appreciation in the realm of possibility; it is an embodied agentic response to anti-black exploitation woven into the tapestry of western social life that insists on the black people's capacity to curate their healing. in this issue, led by afrofuturism, stephanie toliver and elizabeth gilliam because we know 8 journal of effective teaching in higher education, vol. 4, no. 2 put forth black feminist wondaland (bfw) as framework to account for how black women reckon with the misogynoir enacted against them, how they celebrate themselves as an act of radical resistance, and how they reclaim their joy in a society bent on keeping us in a state of sorrow. accountability/intercession intercession proposes a deliberate doing that is willing to put one's own life on the line to advance the life chances of another: "if we know, then we must fight for your life as though it were our own—which it is—and render impassable with our bodies the corridor." this appeal, different from dependence, which proffers that a relationship exists because one is needed or relied on, the willingness to intercede, already renders value and works at the edge of possibility and imaginatively beyond to create life. intercession bends on the richness of black onto-epistemologies and cosmologies, and, in doing so, pedagogically asks, what does the collective need to become to survive? here, the classroom/assignment/text/letter becomes a convergent space of otherwise possibilities, the site where we discard the unuseful to pick up that which is necessary. theorizing collective survival, robert robinson meditates on the potentialities of black educational spaces, particularly, stealin’ the meeting, a black antebellum practice of creating otherwise literacy practices under repressive circumstances. beyond the classroom site, ti'era worsley and reanna roby work to address a void in maker education literature by specifically considering how black joy can foster spaces of pride, creativity, and ownership for and by black youth. the issue concludes with a book review where, in addition to reviewing the text, roshaunda breeden uses love’s (2019) we want to do more than survive: abolitionist teaching and the pursuit of educational freedom as an analytic to suggest possibilities for a praxis of black mattering and abolitionist teaching in higher education, specifically. endless beginnings black mattering is contested terrain. as we write this, more than 25 states and municipalities have proposed or passed legislation banning critical race theory (crt) and the incorporation of material(s) that upset the normative curricular and pedagogical conditions of whiteness. we understand this current contestation against crt, and larger critiques of oppression, as an historical script that has functioned to control what counts as legitimate knowledge and how it is reproduced through white institutions. to be clear, black mattering decenters whiteness, and, purposefully, centers blackness. taken together, these contributions offer important beginnings, with the understanding that anti-blackness is enduring and adaptable. and so, let us begin again. conflicts of interest the authors declare that there are no conflict of interest regarding the publication of this article. because we know 9 journal of effective teaching in higher education, vol. 4, no. 2 references alexander, e. (2004). the black interior: essays. graywolf press. alexander, m. j. (2005). pedagogies of crossing: meditations on feminism, sexual politics, memory, and the sacred. duke university press. baldwin, j. (1970, november 7). an open letter to my sister, angela y. davis. https://www.nybooks.com/articles/1971/01/07/an-open-letter-to-my-sistermiss-angela-davis/ baldwin, j. (1974). james baldwin speaks at uc berkeley in 1974[video]. youtube. https://www.youtube.com/watch?v=8klsr2tb5pa berlant, l. (2011). cruel optimism. duke university press. brand, d. (2012). a map to the door of no return: notes to belonging. vintage canada. brooks, g. (1991). blacks. third world press. cox, m. a. (2015). shapeshifters: black girls and the choreography of citizenship. durham, nc: duke university press. dumas, m. j. (2016). against the dark: antiblackness in education policy and discourse. theory into practice, 55(1), 11–19. kelley, r. d. (2002). freedom dreams: the black radical imagination. beacon press. love, b. (2019). we want to do more than survive: abolitionist teaching and the pursuit of education freedom. beacon press. okello, w.k. (2020). “loving flesh”: self-love, student development theory, and the coloniality of being. journal of college student development, 61(6), 717– 732. doi: 10.1353/csd.2020.0071 okello, w.k., & white, k. (2019). a black feminist reconstruction of agency. in e. s. abes, s. r. jones, & d-l. stewart (eds.), rethinking college student development theory using critical frameworks (pp. 142–s157). stylus publishing. rosenberg, m. (1985). self-concept and psychological well-being in adolescence. in r.l. leahy (ed.), the development of the self (pp. 205–246). academic press. rosenberg, m., & mccullough, b. c. (1981). mattering: inferred significance and mental health among adolescents. research in community & mental health, 2, 163–182. because we know 10 journal of effective teaching in higher education, vol. 4, no. 2 schlossberg, n. k. (1989). marginality and mattering: key issues in building community. new directions for student services, 48, 5–15. https://doi.org/10.1002/ss.37119894803 schlossberg, n. k., lasalle, a., & golec, r. (1990). mattering scales for adult students in postsecondary education. washington, dc: american council on education, center for adult learning and educational credentials. (eric document reproduction service no. ed341772). shange, s. (2019). black girl ordinary: flesh, carcerality, and the refusal of ethnography. transforming anthropology, 27(1), 3–21. https://doi.org/10.1111/traa.12143 stewart, d-l. (2019). ideologies of absence. anti-blackness and inclusion rhetoric in student affairs practice. student affairs in education & the school of higher education, colorado state university. https://sahe.colostate.edu/ideologies-of-absence/ wilderson, f. b., iii. (2010). red, white, and black: cinema and the structure of u.s. antagonisms. duke university press. microsoft word torres et al_final.docx journal of effective teaching in higher education, vol. 5, no. 1 https://doi.org/10.36021/jethe.v5i1.244 crossword puzzles and knowledge retention elisa r. torres, university of mississippi medical center, etorres@umc.edu p. renée williams, university of mississippi medical center wondwosen kassahun-yimer, university of mississippi medical center xiaoshan zhu gordy, university of mississippi medical center abstract. students perceive crossword puzzles as enjoyable. in addition to students’ perceptions, crossword puzzles actually improve knowledge retention. however, crossword puzzles increased exam scores for some students but not others. recommendations have been made for students to create puzzles for their classmates to complete with the rationale that students are encouraged to research and understand the material in order to write meaningful clues for the puzzle. while students enjoy creating their own crossword puzzles, the association between students creating crossword puzzles and knowledge retention is unknown. the purpose of this project was to determine if creating crossword puzzles and completing peers’ crossword puzzles were associated with improved knowledge retention indicated by higher scores on quizzes. students in a research course from two institutions across three semesters had the option each week to upload a blank puzzle they created prior to completing each other’s puzzles and taking a quiz. quiz scores were compared between those who did versus did not create their own puzzles and complete their peers’ puzzles. results varied by institution and programs as well as the same program within the same institution but different semesters. results highlight the importance of moving beyond student perceptions and towards assessing knowledge retention while taking into consideration institution, program, and semester. keywords: crossword puzzles; knowledge retention; active learning; educational activities; educational measurement active learning is defined as “anything that involves students doing things and thinking about the things they are doing” (bonwell & eison, 1991, p. 19). in contrast to traditional teacher-centered learning that often positions students as passive recipients of instructional content (gregory, 2002), active learning requires students to become active participants who must make decisions and solve problems (franklin et al., 2003). crossword puzzles are an example of active learning. crossword puzzles require the ability to remember factual knowledge and provide foundational knowledge, the basic understanding that is necessary for other kinds of learning (fink, 2013). thus, crossword puzzles would meet a course objective of demonstrating foundational knowledge, which is at the remembering level of anderson’s revision of bloom’s crossword puzzles and knowledge retention 19 journal of effective teaching in higher education, vol. 5, no. 1 taxonomy of educational objectives (anderson & krathwohl, 2001; krathwohl, 2002). crossword puzzles improve knowledge retention (abuelo et al., 2016; gaikwad & tankhiwale, 2012; murphy et al., 2016; nirmal et al., 2020; orawiwatnakul, 2013; patrick et al., 2018; singh matreja et al., 2021). however, crossword puzzles increased exam scores for some students but not others (davis et al., 2009). recommendations have been made for students to create puzzles for their classmates to complete with the rationale being that students are encouraged to research and understand the material in order to write meaningful clues for the puzzle (davis et al., 2009). while students enjoy creating their own crossword puzzles (coticone, 2013), the association between students creating crossword puzzles and knowledge retention is unknown. crossword puzzles can be cumulative or noncumulative. cumulative learning is an assignment that requires students to draw on information they learned earlier in the semester (lang, 2016). for example, a weekly cumulative crossword puzzle assesses content in the course up to that point as opposed to a noncumulative weekly crossword puzzle that only assesses content for that week. students who have cumulative assessments retain more information than students who have noncumulative assessments (beagley & capaldi, 2016; khanna et al., 2013; lawrence et al., 2013). the association between students creating and completing cumulative crossword puzzles with knowledge retention is unknown. creating and completing weekly cumulative crossword puzzles may be a method to address the issue that some students can struggle with a semester-long scaffolding project (vandiver & walsh, 2010) because by the time they get towards the end of the semester, they have forgotten what they learned early in the semester. one way to address this issue may be to have weekly cumulative assignments throughout the semester. the purpose of this quality improvement project was to determine if creating and completing weekly cumulative crossword puzzles were associated with improved knowledge retention as indicated by higher scores on corresponding weekly cumulative quizzes. method sample participants comprised students from two state institutions across three semesters. the first semester included students in a doctorate of nursing practice (dnp) or doctor of philosophy (phd) program at a university in the midwest of the united states. students were enrolled in nursing research, a required course in the dnp (optional in the phd) program that fulfilled the american association of colleges of nursing (2006) essentials of doctoral education for advanced nursing practice: clinical scholarship and analytical methods for evidence-based practice. this was a 16-week hybrid course where students met three hours face-to-face once a month for four months, with the rest of the course online. prior to the start of the course, crossword puzzles and knowledge retention 20 journal of effective teaching in higher education, vol. 5, no. 1 the university’s education institutional review board (irb) reviewed and exempted this quality improvement project. the second and third semesters included students enrolled in a master of science in nursing (msn) program at a university medical center in the southeast united states. students were enrolled in research design and methods for advanced nursing practice, a required course that fulfilled the american association of colleges of nursing (2011) essentials of master’s education in nursing: essential iv: translating and integrating scholarship into practice. this was a 16-week online course. prior to the start of the course, it was determined by the irb that this quality improvement project was not considered research and therefore would not be reviewed by the irb according to institutional guidelines. there were no conflicts of interest in either course. context both courses examined quantitative and qualitative research methods; the interrelationships among theory, research, and practice; the research process; critical evaluation of research findings; and ethical criteria for the protection of human subjects in research. the prerequisites were graduate standing or consent of the instructor. the courses were taught by a phd-prepared nurse with an active program of research, who designed the current quality improvement project as a participant in the wisconsin teaching fellows and scholars (wtfs) program. the wtfs program offered university of wisconsin faculty and teaching academic staff a unique opportunity to collaborate with other teachers from various disciplines across the university of wisconsin system. in addition to discussing the pedagogical literature, participants were guided through the process of completing a scholarship of teaching and learning (sotl) project with input from fellow participants and the program co-directors. intervention students were assigned two chapters most weeks from the textbook nursing research: generating and assessing evidence for nursing practice (polit & beck, 2012). online quizzes that corresponded with the chapter readings were assigned. all questions came from the test bank of the course textbook available at no cost. completing weekly quizzes was associated with improved knowledge retention; higher scores on the weekly cumulative quizzes were associated with higher scores on the cumulative final exam (torres, 2019), providing evidence of predictive validity, a form of criterion validity (polit & beck, 2012). evidence for reliability in the quizzes was found with a cronbach alpha of .69. for extra credit and to assist with preparing for the weekly cumulative quizzes, weekly cumulative crossword puzzles that corresponded with the chapter readings were due four days prior to the quizzes. uploading a blank puzzle and corresponding answer key to the learning management site four days prior to the quiz due date allowed students sufficient time to complete each other’s puzzles prior to taking the weekly quiz. students were provided a link to a site where they could complete the puzzles: crossword labs (https://crosswordlabs.com/). every crossword puzzles and knowledge retention 21 journal of effective teaching in higher education, vol. 5, no. 1 puzzle was to have at least one term from each chapter, with the length of the puzzles increasing each week from at least two terms on the puzzle in the first week to at least 26 terms on the final puzzle at the end of the semester. students scored 100% if they submitted their puzzles on time and the answers were correct and 0% if their puzzles were not submitted on time or any answer was not correct. two phd-prepared faculty checked each submission to ensure adherence to guidelines and entered assigned puzzle grades into a spreadsheet; one faculty member also checked the completeness and accuracy of the data. all of the extracredit crossword puzzles combined were worth an additional 1% of the final grade in the course. the crossword puzzle assignment was created in response to students having difficulty understanding the content in other assignments at the understanding, applying, analyzing, evaluating, and creating levels. statistical analysis reliability was assessed with the parallel-forms procedure (waltz et al., 2005) to determine if uploading a blank puzzle and corresponding answer key to the learning management site four days prior to the quiz due date was comparable to completing each other’s puzzles prior to taking the weekly quiz. po is the proportion of observed agreements in classifications in both puzzles (100% if submitted on time and correctly, and 0% if not submitted, not submitted on time, or not submitted correctly). k is the proportion of persons consistently classified in the same category with both puzzles beyond expected by chance. kruskal-willis was performed for the entire sample comparing weekly quiz scores between those who did versus did not complete the corresponding weekly puzzle correctly to determine the predictive validity of the puzzle scores on quiz scores or whether the observed quiz outcomes were associated with the crossword puzzle intervention. the same analyses were performed stratified by semester using mann-whitney u test with a bonferonni correction (p < .0167) to account for multiple analyses and reduce the risk of type i error since context matters (ogrinc et al., 2019), such as internal elements (modality: hybrid vs. 100% online), external elements (institution), and characteristics of the individuals (masters vs. doctoral students). results the first semester comprised 14 students in a dnp program and 2 in a phd program with all students completing the course. the second semester was a year after the first semester and comprised 20 students in a msn program at a separate institution from the first semester with all students completing the course. the third semester immediately followed the second semester and comprised 53 students in the same msn program as the previous semester, with 51 students completing the course. the total sample size was 87. the parallel-form reliability found po ranged from 0.75 to 1.00, while k ranged from -0.56 to -0.00 (see table 1). crossword puzzles and knowledge retention 22 journal of effective teaching in higher education, vol. 5, no. 1 table 1 parallel form reliability of creating versus completing crossword puzzles puzzle semester 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1st po 1.00 0.94 0.88 0.88 0.94 0.88 0.94 0.88 1.00 1.00 1.00 0.94 1.00 0.94 k -0.56 -0.53 -0.50 -0.36 -0.34 -0.36 -0.25 -0.36 -0.02 -0.06 -0.02 -0.16 -0.06 -0.16 2nd po 0.95 1.00 1.00 0.95 0.95 0.75 0.85 0.90 0.75 0.90 0.85 0.80 0.90 0.80 k -0.11 -0.49 -0.81 -0.47 -0.47 -0.35 -0.43 -0.34 -0.31 -0.34 -0.25 -0.20 -0.18 -0.17 3rd po 0.88 0.94 0.96 0.86 0.90 0.90 0.94 0.88 0.96 0.98 1.00 0.96 0.90 0.94 k -0.23 -0.24 -0.24 -0.14 -0.11 -0.09 -0.06 -0.13 -0.05 -0.03 -0.00 -0.04 -0.10 -0.06 as can be seen in table 2, when all the semesters were combined, results from the kruskal-willis test found those who completed puzzles 5, 6, 7 and 13 correctly had statistically significantly higher scores on the corresponding quizzes compared to those who did not complete those puzzles. the same analyses were performed stratified by semester using the mann-whitney u test. in the first semester (n = 16), those who completed puzzle 13 correctly had significantly higher scores on the corresponding quiz compared to those who did not (97.6 vs. 90.0, p = .022), although this was not statistically significant with a bonferonni correction. in the second semester (n = 20), those who completed puzzles 6 (93.3. vs. 80.3, p = .039) and 11 (92.5 vs. 82.0, p = .002) had significantly higher scores on the corresponding quiz, although only week 11 was statistically significant after bonferroni corrections. in the third semester (n = 53), those who completed puzzle 7 had significantly higher scores on the corresponding quiz (92.9 vs. 89.0, p = .019), although this was not statistically significant with a bonferonni correction. table 2 mean quiz scores based on puzzle completion total (n = 87) 1st semester (n = 16) 2nd semester (n = 20) 3rd semester (n = 51) completed assignment did not complete assignment completed assignment did not complete assignment completed assignment did not complete assignment completed assignment did not complete assignment quiz 1 93.1 n = 51 91.7 n = 36 96.4 n = 14 100 n = 2 91.7 n = 6 89.3 n = 14 91.9 n = 31 92.5 n = 20 crossword puzzles and knowledge retention 23 journal of effective teaching in higher education, vol. 5, no. 1 p-value .77 .93 .90 .78 quiz 2 98.5 n = 65 97.7 n = 22 100 n = 13 100 n = 3 97.1 n = 17 83.3 n = 3 98.6 n = 35 100 n = 16 p-value .64 1.00 .15 .33 quiz 3 93.4 n = 66 97.6 n = 21 91.7 n = 12 95.8 n = 4 95.5 n = 19 100 n = 1 92.8 n = 35 97.9 n = 16 p-value .06 .60 .80 .07 quiz 4 95.7 n = 49 96.4 n = 38 95.5 n = 11 95.0 n = 5 92.4 n = 16 90.8 n = 4 98.3 n = 22 97.4 n = 29 p-value .99 .91 .82 .52 quiz 5 88.1 n = 53 82.6 n = 34 94.2 n = 12 85.0 n = 4 86.0 n = 15 82.0 n = 5 86.5 n = 26 82.4 n = 25 p-value .016 .08 .45 .16 quiz 6 94.1 n = 46 89.9 n = 41 97.0 n = 11 90.0 n = 5 93.3 n = 12 80.3 n = 8 93.2 n = 23 92.6 n = 28 p-value .04 .12 .04 .83 quiz 7 92.7 n = 50 88.7 n = 37 94.7 n = 11 95.0 n = 5 91.0 n = 14 82.2 n = 6 92.9 n = 25 89.0 n = 26 p-value .005 .74 .13 .019 quiz 8 93.7 n = 44 91.7 n = 43 94.9 n = 11 90.0 n = 5 93.0 n = 14 82.3 n = 6 93.4 n = 19 93.8 n = 32 p-value .24 .18 .08 .90 quiz 9 93.7 n = 46 91.8 n = 41 94.1 n = 9 95.8 n = 7 92.3 n = 11 88.7 n = 9 94.1 n = 26 91.8 n = 25 p-value .26 .40 .30 .25 quiz 10 86.8 n = 51 87.4 n = 36 85.3 n = 10 87.7 n = 6 84.9 n = 14 77.0 n = 6 88.3 n = 27 89.9 n = 24 p-value .60 .79 .35 .60 quiz 11 91.9 n = 44 89.9 n = 43 87.8 n = 7 91.0 n = 9 92.5 n = 11 82.0 n = 9 92.9 n = 26 92.4 n = 25 p-value .40 .41 .002 .72 quiz 12 91.6 n = 44 90.5 n = 43 89.6 n = 10 92.4 n = 6 88.1 n = 10 84.7 n = 10 94.0 n = 24 92.3 n = 27 p-value .87 .49 .35 .91 crossword puzzles and knowledge retention 24 journal of effective teaching in higher education, vol. 5, no. 1 quiz 13 94.5 n = 44 89.9 n = 43 97.6 n = 10 90.0 n = 6 91.3 n = 12 88.0 n = 8 94.9 n = 22 90.3 n = 29 p-value .019 .02 .31 .22 quiz 14 95.9 n = 43 91.2 n = 44 97.1 n = 11 95.7 n = 5 96.0 n = 9 92.0 n = 11 95.3 n = 23 90.1 n = 28 p-value .20 .44 .20 .79 no changes were made in the puzzle assignment between semesters due to small sample sizes. there was no missing data. an unintended consequence was a dramatic increase in faculty workload when the number of students increased from 20 in the fall to 53 in the spring without an increase in the number of faculty in the course. discussion creating cumulative crossword puzzles and completing peers’ cumulative puzzles were associated with higher scores on most corresponding quizzes. these results are consistent with previous literature that found students who have cumulative assessments retain more information than students who have noncumulative assessments (beagley & capaldi, 2016; khanna et al., 2013; lawrence et al., 2013). a strength of the current quality improvement project is that it incorporated recommendations for students to create puzzles for their classmates to complete, encouraging students to research and understand the material in order to write meaningful clues for the puzzle (davis et al., 2009). the current results that found crossword puzzles were associated with higher scores on most corresponding quizzes coincide with previous studies that found crossword puzzles improve knowledge retention (abuelo et al., 2016; murphy et al., 2016; nirmal et al., 2020; orawiwatnakul, 2013; patrick et al., 2018; shawahna & jaber, 2020; singh matreja et al., 2021; zamani et al., 2021). however, there was not a statistically significant association between most puzzles and their corresponding quizzes. similar results have been found where completing crossword puzzles was not always associated with improved knowledge retention (sumanasekera et al., 2020), and student question generation was not associated with statistically significant improvement in achievements (aflalo, 2018). in the current analysis, regardless of semester, most of the quiz scores were in the 80s and 90s, resulting in a ceiling effect or a reduction in the amount of upward change that is detectable (polit & beck, 2012). our recommendation is to only incorporate crossword puzzles for those quizzes with habitually low scores, such as below 80%, for at least two semesters in the same course at the same institution. the current results move beyond students’ subjective perceptions of crossword puzzles and towards objectively assessing knowledge retention. crossword puzzles and knowledge retention 25 journal of effective teaching in higher education, vol. 5, no. 1 reliability of the crossword puzzle assignment results from the po indicated a good level of agreement in classifying learners as 100% where both puzzles were submitted on time and were correct, versus 0% if not submitted, not submitted on time, or not correct. however, results from the k indicated uploading a blank puzzle and corresponding answer key to the learning management site was not consistent with completing each other’s puzzles prior to taking the weekly quiz. there were several reasons for this discrepancy. some created a puzzle but did not complete their peer’s puzzle even though both were required in order to earn full credit. similarly, some completed their peer’s puzzle but did not create their own. some did not upload a corresponding answer key. some uploaded a puzzle from a previous week by mistake, and others did not upload their puzzles by the due date. some provided a nonfunctional link to their puzzles. thus, results of the criterion-referenced parallel-forms procedure indicate agreement in correctly completing both puzzles. however, not surprisingly, creating one’s own puzzles is not consistent with completing peers’ puzzles. the effectiveness of creating versus completing crossword puzzles should be examined further. strengths and limitations of the crossword puzzle assignment a major strength of the current work is its addition to a body of literature objectively assessing knowledge retention resulting from crossword puzzles in a variety of disciplines such as english (orawiwatnakul, 2013) and health professions (abdulmajed et al., 2015), including pharmacology (patrick et al., 2018), nursing (shawahna & jaber, 2020), dentistry (nirmal et al., 2020), and sociology (davis et al., 2009). to our knowledge, the current project is the first to examine crossword puzzles in graduate education and in a hybrid course. there are several limitations. the results are limited to the students in the current quality improvement project. efforts were made to minimize this limitation by including students across three semesters, across three programs (msn, dnp and phd), across two instructional modalities (online and hybrid), and across two institutions. in addition, this quality improvement project had a quasi-experimental design as there was an intervention without randomization, thereby threatening internal validity (polit & beck, 2012). efforts were made to minimize this limitation by including a control group, i.e. those who did not complete the puzzle assignment. finally, the sample size was small with deviations from normality necessitating non-parametric statistics, which are not as powerful as parametric statistics (polit & beck, 2012). efforts were made to minimize this limitation by including a bonferroni correction to adjust for multiple testing. conclusion students perceive crossword puzzles as an enjoyable, creative, and innovative way to learn (abuelo et al., 2016; franklin et al., 2003; gaikwad & tankhiwale, 2012; singh matreja et al., 2021; sumanasekera et al., 2020; zamani et al., 2021). based on the current results, our recommendation is to only incorporate crossword crossword puzzles and knowledge retention 26 journal of effective teaching in higher education, vol. 5, no. 1 puzzles for those quizzes with habitually low scores for at least two semesters in the same course at the same institution. an important implication is the importance of moving beyond students’ subjective perceptions and towards objectively assessing knowledge retention. the current work should be replicated in additional disciplines beyond nursing; undergraduate and graduate education; and face-toface, hybrid, and online courses. acknowledgements i would like to gratefully acknowledge the wisconsin teaching fellows and scholars program, especially drs. cyndi kernahan and david voelker for their guidance and feedback. conflicts of interests the authors declare that there are no conflicts of interest regarding the publication of this article. funding the authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: this work was supported by the clinical and translational science award program through the nih national center for advancing translational sciences [grant numbers ul1tr000427, kl2tr000428] and the mississippi center for clinical and translational research [grant number 5u54gm115428]. the funding sources had no role in the project design, collection, analysis, interpretation of data, writing of the report, or the decision to submit for publication. the content is solely the responsibility of the author and does not necessarily represent the official views of the nih. references abdulmajed, h., park, y. s., & tekian, a. (2015). assessment of educational games for health professions: a systematic review of trends and outcomes. medical teacher, 37(supplement 1), s27-32. https://www.tandfonline.com/doi/full/10.3109/0142159x.2015.1006609 abuelo, a., castillo, c., & may, s. a. (2016). usefulness of crossword puzzles in helping first–year bvsc students learn veterinary terminology. journal of veterinary medical education, 43, 255–262. https://pubmed.ncbi.nlm.nih.gov/27111003/ aflalo, e. (2021). students generating questions as a way of learning. active learning in higher education, 22, 63–75. https://journals.sagepub.com/doi/10.1177/1469787418769120 american association of colleges of nursing. (2006). essentials of doctoral crossword puzzles and knowledge retention 27 journal of effective teaching in higher education, vol. 5, no. 1 education for advanced nursing practice: clinical scholarship and analytical methods for evidence-based practice. author. https://www.aacnnursing.org/portals/42/publications/dnpessentials.pdf american association of colleges of nursing. (2011). the essentials of master’s education in nursing. author. https://www.aacnnursing.org/portals/42/publications/mastersessentials11.pd f anderson, l. w., & krathwohl, d. r. (2001). a taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives. addison wesley longman. beagley, j. e., & capaldi, m. (2016). the effect of cumulative tests on the final exam. problems, resources, and issues in mathematics undergraduate studies, 26, 878–888. https://www.tandfonline.com/doi/abs/10.1080/10511970.2016.1194343 bonwell, c. c., & eison, j. a. (1991). active learning: creating excitement in the classroom. 1991 ashe-eric higher education reports. association for the study of higher education—eric clearinghouse on higher education. https://eric.ed.gov/?id=ed336049 coticone, s. r. (2013). utility of self-made crossword puzzles as an active learning method to study biochemistry in undergraduate education. journal of college science teaching, 42(4), 33–39. https://eric.ed.gov/?id=ej1011749 davis, t. m., shepherd, b., & zwiefelhofer, t. (2009). reviewing for exams: do crossword puzzles help in the success of student learning? journal of effective teaching, 9(3), 4–10. https://uncw.edu/jet/articles/vol9_3/davis.pdf fink, l. d. (2013). creating significant learning experiences: an integrated approach to designing college courses. jossey–bass. franklin, s., peat, m., & lewis, a. (2003). non–traditional interventions to stimulate discussion: the use of games and puzzles. journal of biological education, 37, 79–84. https://www.tandfonline.com/doi/abs/10.1080/00219266.2003.9655856 gaikwad, n., & tankhiwale, s. (2012). crossword puzzles: self-learning tool in pharmacology. perspectives on medical education, 1(5–6), 237–48. https://pubmed.ncbi.nlm.nih.gov/23240102/ gregory, m. r. (2002). constructivism, standards, and the classroom community of inquiry. educational theory, 52, 397–408. https://eric.ed.gov/?id=ej673909 khanna, m. m., brack, a. s., & finken, l. l. (2013). shortand long-term effects of crossword puzzles and knowledge retention 28 journal of effective teaching in higher education, vol. 5, no. 1 cumulative finals on student learning. teaching of psychology, 40, 175–182. https://journals.sagepub.com/doi/abs/10.1177/0098628313487458 krathwohl, d. r. (2002). a revision of bloom's taxonomy: an overview. theory into practice, 41, 212–218. lang, j. m. (2016). small teaching. jossey-bass. lawrence, n. k. (2013). cumulative exams in the introductory psychology course. teaching of psychology, 40, 15–19. https://journals.sagepub.com/doi/abs/10.1177/0098628312465858 murphy, m., spillane, k., cully, j., navarro-pardo, e., & moret-tatay, c. (2016). can word puzzles be tailored to improve different dimensions of verbal fluency? a report of an intervention study. journal of psychology, 150(6), 743–54. https://pubmed.ncbi.nlm.nih.gov/27224052/ nirmal, l., muthu, m. s., & prasad, m. (2020). use of puzzles as an effective teaching-learning method for dental undergraduates. international journal of clinical pediatric dentistry, 13(6), 606–610. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc8060935/pdf/ijcpd-13606.pdf ogrinc, g., armstrong, g. e., dolansky, m. a., singh, m. k., & davies, l. (2019). squire-edu (standards for quality improvement reporting excellence in education): publication guidelines for educational improvement. academic medicine, 94, 1461–1470. https://pubmed.ncbi.nlm.nih.gov/30998575/ orawiwatnakul, w. (2013) crossword puzzles as a learning tool for vocabulary development. electronic journal of research in educational psychology, 11, 413–428. https://www.redalyc.org/pdf/2931/293128257006.pdf patrick, s., vishwakarma, k., giri, v. p., datta, d., kumawat, p., singh, p., & matreja, p. s. (2018). the usefulness of crossword puzzle as a self-learning tool in pharmacology. journal of advances in medical education & professionalism, 6(4), 181–185. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc6191832/pdf/jamp-6181.pdf polit, d. f., & beck, c. t. (2012). nursing research: generating and assessing evidence for nursing practice. wolters kluwer health. shawahna, r., & jaber, m. (2020). crossword puzzles improve learning of palestinian nursing students about pharmacology of epilepsy: results of a randomized controlled study. epilepsy & behavior, 106, 107024. https://www.epilepsybehavior.com/article/s1525-5050(20)30203-1/fulltext singh matreja, p., kaur, j., & yadav, l. (2021). acceptability of the use of crossword puzzles and knowledge retention 29 journal of effective teaching in higher education, vol. 5, no. 1 crossword puzzles as an assessment method in pharmacology. journal of advances in medical education & professionalism, 9(3), 154–159. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc8273532/ sumanasekera, w., turner, c., ly, k., hoang, p., jent, t., & sumanasekera, t. (2020). evaluation of multiple active learning strategies in a pharmacology course. currents in pharmacy teaching & learning, 12(1), 88–94. https://www.sciencedirect.com/science/article/abs/pii/s1877129718300698 torres, e. r. (2019). cumulative quizzes in a nursing research course for nursing doctoral students. journal of nursing education, 58, 243–246. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc6639015/pdf/nihms1031999.pdf vandiver, d. m., & walsh, j. a. (2010) assessing autonomous learning in research methods courses: implementing the student-driven research project. active learning in higher education, 11(1), 31–42. https://journals.sagepub.com/doi/10.1177/1469787409355877 waltz, c. f., strickland, o. l., & lenz, e. r. (2005). measurement in nursing and health research. springer publishing company. https://www.thefreelibrary.com/measurement+in+nursing+and+health+res earch%2c+3rd+ed-a0143215378 zamani, p., biparva haghighi, s., & ravanbakhsh, m. (2021). the use of crossword puzzles as an educational tool. journal of advances in medical education & professionalism, 9(2), 102–108. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc8106739/pdf/jamp-9102.pdf microsoft word high impact practices emerging values modelericksen williamson-ashe.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.219 high-impact educational practices’ influence on the emerging values model: group work impact kirsten s. ericksen, norfolk state university, ksericksen@nsu.edu sandra williamson-ashe, norfolk state university abstract. this research examines the impact of various high-impact educational practices integrated in different group elements (groups course, in-class group activities, and a learning community) on student perceived group work experience related to the emerging values model. the emerging values model found academic group work to be beneficial for students, specifically associated with peer value and group work values. different types of high-impact educational practices related to groups are examined using the emerging values model. implications for teaching group work and personal and professional growth in group work are examined. the integration of group work in the higher education classroom and application to professional environments is discussed extensively. keywords: group work, high-impact practices (hips), values model teaching faculty are persistent in their efforts to determine the most effective methods to help students gain knowledge and skills towards their profession. many have found high-impact educational practices to benefit students learning (finley & mcnair, 2013; kinzie, 2010; kuh, 2008). research has also found that students report benefits to participation in group work, resulting in the emerging values model (williamson-ashe & ericksen, 2017). this research considers various group high-impact practices using the emerging values model as a lens. do certain types of group high-impact practices result in more values associated with the ev model? do different hips create different outcomes related to the ev model? what is the impact on students’ personal and professional development? it is apparent that various educational methods are used to teach social work including experiential activities, journals, small-group activities, and lectures (strozier, 2008). group work is often used in the classroom as a teaching tool to enhance a students’ academic achievement and experience in group socialization (kolb, 1984; humphrey, 2014). group work is an essential skill for the practicing social worker and noted in four of the nine competencies required for social work students to master (council on social work education, 2015). educators are responsible for guiding students in the knowledge and understanding of group application, so they become effective professionals. social work students need to obtain the necessary group work skills to become proficient group facilitators. high-impact practices high-impact practices (hips) have been found to be beneficial for student success (carini et al., 2006; kinzie, 2010). educational practices with high-impact include engaging environments and practices such as internships, learning communities, high-impact emerging values model 61 journal of effective teaching in higher education, vol. 4, no. 1 undergraduate research, collaborative projects, service-learning, and capstone projects (kuh, 2008). finley and mcnair (2013) found high-impact practices greatly benefit underserved populations and cohen (2010) reported the use of experiential learning in the classroom to be an effective method to teach group process and development. the high-impact practices in learning communities enhance social justice education for social work students (ericksen & williamson-ashe, 2019). incorporating these and other high-impact practices in the classroom can benefit student learning. in-class group activities have been demonstrated to enhance comprehension about the group process (humphrey, 2014). poort et al. (2020) assert trust between group members is essential for group functioning and performance. incorporating group activities within the class setting has been found to be effective in cultivating impartiality and supporting work with diverse group members (ericksen, 2019). results with stem students demonstrate the inclusion of structured team related course assignments lead to students’ confidence in various non-academic professional skill development (cruz et al., 2020). likewise, cooperative learning and problem-based learning contribute to a team approach that is transferable to many professions (michaelsen et al., 2014). emerging values model the original baylor college of medicine value teams research found the instructional method of team learning as highly appreciated by students within the psychiatry discipline (levine et al., 2004). specifically, levine et al. (2004) determined that team learning is beneficial for communication, problem solving, and student engagement. levine et al. (2004) suggested an examination of different disciplines would be beneficial, leading to the subsequent research with social work students (williamson-ashe & ericksen, 2017; williamson-ashe & ericksen 2019). williamson-ashe and ericksen (2017) distributed the amended values team survey (vts) to social work students which demonstrated emerging themes related to students’ perceived value from participation in group work. these findings resulted in the development of the emerging values model (ev) (see figure 1) (williamson-ashe and ericksen, 2017). the emerging values model (ev) represents how academic group work was beneficial to students, specifically asserting merit associated with peer value and group work value (williamson-ashe & ericksen, 2017). the ev model was developed based on graduate student participation in group assignments (williamson-ashe & ericksen, 2017) and was recently validated for the peer and group work values reducing the categories from four to three, specifically peer values related to skills category (sc) and collaboration category (cc) and group work values related to decision-making and problem-solving category (dp) (williamson-ashe & ericksen, 2019). the ev model asserts group participants report several values from their group experiences. high-impact emerging values model 62 journal of effective teaching in higher education, vol. 4, no. 1 figure 1 the emerging values model (ev model) as the emerging values model (ev) suggests, student participants reported peer values and group work values through their involvement in course group assignments. students included specific skills they gained and the value of collaboration with their peers as valuable in their success as students. additionally, student participants found group work to be valuable in decision-making and problem solving (williamson-ashe & ericksen, 2019). context: group work course elements application to the ev model this research was conducted to examine the impact of various types of high-impact education group work practices (ranging from an entire group course to integrating group activities within a course) on the emerging values model in the classroom. various group work high-impact educational practices were implemented including, specific group work content, in-class group activities, group assignments, and class group experiential activities outside the classroom. these results can inform educators about any connections between the implementation of hips and specific group work values related to the ev model. methods sample the participants in this research included students attending a minority-serving four year institution in the southeastern part of the united states with a student value themes peer value promotes student success skills collaboration group work value rewarding process productive decision-making & problem solving high-impact emerging values model 63 journal of effective teaching in higher education, vol. 4, no. 1 population of approximately 5,500 at the time of this research. the convenience sample included students enrolled in three different social work courses: applied social work skills with groups course (a graduate level advanced groups course), and two different sections from the undergraduate sophomore level core course human behavior and the social environment i (one regular class and one learning community class in which students with common interests participated in activities together, outside the classroom, related to the course content). the selection of these courses was based on the intention to examine the outcomes across curricular programs (graduate and undergraduate courses) and different highimpact educational practices (specifically different group elements present) in each class and application to the emerging values model. each of these courses offered varying degrees of exposure to group work concepts/elements and application. all students enrolled in these sections of the courses were invited to participate voluntarily in the research (56 students invited) and provided with a consent form approved by the institutional review board for the campus. students were informed their participation was voluntary without repercussions. amended value teams survey the amended value teams survey (vts) was administered as a preand posttest to those students willing to participate during the second week of classes (after the course drop date) and during the final week for the course. the amended vts included the original 14 statements with 5 point likert scale responses ranging from strongly disagree to strongly agree and incorporated demographic information and three descriptive questions (including group knowledge level and one benefit and one challenge related to group participation based on this course). data collection data were gathered from the three separate cohorts, representing different integrations of high-impact educational group work practices (group course cohort, in-class group activities cohort, and a learning community group cohort). together the total sample, from 56 invited potential individuals, included 43 (77%) participants whose self-reported demographics were as follows: 4 males (9%) and 39 females (91%) and 36 african american/black (84%), 3 caucasian/white (7%), and 4 individuals (9%) who declined to share their race/ethnicity. there were 19 graduate students (44%) and 24 undergraduate students (56%). the applied social work skills with groups course (group course cohort) contained master’s level students in social work, and 19 of the 20 students participated in the research. undergraduate course attendance was a challenge which impacted the ability to gather data from these courses. several students notified the professor they were ill and not in attendance the date the posttest was administered in class. for the inclass activities course cohort, 20 students completed the pre-test while only 16 students participated in the posttest (out of a potential 22 students). eight students in the learning community group cohort completed the posttest compared to the 10 who completed the pretest (out of a potential 14 student participants). this resulted in the 43 total student participants. high-impact emerging values model 64 journal of effective teaching in higher education, vol. 4, no. 1 treatment: high-impact practices each cohort was exposed to different high-impact group work practices. in-class group activities cohort the human behavior and the social environment i (hbse) course incorporated inclass group activities. the sophomore undergraduate students in this core required course discussed the dynamics of systems and impact on the development and well-being of humans from preconception through childhood. throughout the semester, these students participated in three specific in-class group activities. these activities required small group discussion and application of the course material during class time. the first in-class activity included the review of a case study with group discussion and analysis of the possible interventions using the concept approaches/models examined in class. the second course lecture specifically focused on the impact of the community and the importance of group work including the development/facilitation of class rules as a demonstration of group development. the final in-class activity examined the impact of trauma on childhood development, including a small group discussion (4-5 people) to determine potential trauma-informed care interventions for a specific case. learning community cohort the human behavior and the social environment i learning community course (learning community cohort) included all of the in-class group work engagement mentioned above in the regular non-lc section of the hbse course with the inclusion of two additional co-curricular activities outside the class to enhance the classroom content. cocurricular activities consisted of a daylong visit to the general assembly (the state capital) and meeting with elected representatives to advocate for a particular area of need and participation in a de-stress on-campus bowling activity. students prepared for the visit to the general assembly in small groups, developing a political issue to research and practicing a 30-second elevator speech that was delivered during the group meeting with the state delegate. the learning community faculty participated in the casual de-stress bowling event located at the campus bowling alley the week before final exams. groups course cohort the applied social work skills with groups course (groups course cohort) was a graduate level advanced groups elective course. students participating in this graduate course had taken a general introduction to group work course as a prerequisite. the content for this elective course focused on the assessment, synthesis, and application of group work with a special emphasis on at-risk populations. all graduate students completed a course assignment that entailed participation in a task group for planning and delivery of a student determined event to celebrate social work month. the overall course emphasis was group work, student engagement in group assignments, and a group simulation experience each high-impact emerging values model 65 journal of effective teaching in higher education, vol. 4, no. 1 week with both facilitation and participation in a group. although students were not expected to pretend to be a part of the presented population (e.g., alcoholics, children of divorce), they engaged in the group activities however they related to the content with processing following their participation (both during class and through journaling). for example, if the group activity was designed for the population of grieving parents, students were expected to participate in the experiential group and relate to the content the group facilitator shared such as the emerging theme “loss.” data analysis the data were analyzed to compare qualitative (narrative data responses) and quantitative results (pre-post test results for the amended values team survey) to determine differences within group participants scores (amended values team survey pre-post scores) and between the different groups (groups course, in-class group activities, and learning community). a paired-samples t-test was conducted to compare all three cohorts pre-post self-ratings following participation in the respective course. the participants’ responses to the survey/inventory were compared with the narrative data responses to determine any similarities and differences. as rubin and babbie (2017) suggest, comparative analysis of the qualitative data included coding of the data and identification of emerging themes with patterns. initially, open coding was used to analyze the data while axial coding followed to examine any reoccurring patterns and determine the emerging themes from the data (creswell & poth, 2018). findings/results the amended values team survey likert scale items and open-ended responses indicated an impact from participation in the various high-impact practices and courses (group course cohort, group activities cohort, and learning community cohort). the ev model constructs and the matching prominent high-impact practice results are discussed below. quantitative findings ev model construct, peer value: collaboration both qualitative and quantitative data support the ev model construct, peer value: collaboration. the related likert scale survey question on the amended values team survey was “the ability to collaborate with my peers will be necessary if i am to be successful as a student” (question #1, amended values team survey). this survey question was found to be the only question significant for all three cohorts (experiential learning, learning community, and in-class activities). specifically, a paired-samples t-test was conducted to compare all cohort group pre-collaboration rating and post-collaboration rating after attending the respective course. there was a statistically significant difference in the total combined cohort scores for pre-collaboration rating (m = 4, sd = .857) and post-collaboration rating high-impact emerging values model 66 journal of effective teaching in higher education, vol. 4, no. 1 (m = 4.512, sd = .436) following participation in their respective course t(43) = 3.078, p = .01. the total combined cohort results from the survey question indicated there is a statistical significance in the preand post-collaboration rating scores for the self-reported ratings. in addition, a paired-samples t-test was conducted to compare each of the cohorts separately for pre-collaboration rating and post-collaboration rating after course attendance. there was not a significant difference in the learning community cohort scores; however, there was a significant difference in the group course cohort scores for pre-collaboration rating (m = 4.05, sd = .576) and post-collaboration rating (m = 4.47, sd = .46) following participation in the graduate level course t(19) = -1.80, p = .09. for the in-class activities cohort, there was also a significant difference in the scores for precollaboration rating (m = 3.85, sd = 1.128) and post-collaboration rating (m = 4.44, sd = .59) following participation in the in-class activities cohort t(16) = 1.99, p = .068. table 1 represents the mean difference for self-rating of the importance of collaboration for student success. table 1 mean difference within groups for self-rating of ability to collaborate with peers to be successful as a student as reported by the amended values team survey (question #1) subjects pretest mean posttest mean mean difference t df p group course cohort 4.05 variance 0.576 4.47 variance 0.46 .42 -1.80 18 .09 learning community cohort 4.2 v0.76 4.75 v-0.188 .55 -1.744 7 .13 in-class activities cohort 3.85 v-1.128 4.44 v-0.496 .59 -1.987 15 .068 all cohorts 4 v-0.857 4.512 v-0.436 .51 -3.078 39 .01 note. p<.10 statistical significance was found for survey item 1, “the ability to collaborate with my peers will be necessary if i am to be successful as a student,” suggesting students recognize the necessity of collaborating with peers in order to be successful in their academic endeavors. this finding suggests that all cohort student participants viewed their collaboration with peers as an important part of being high-impact emerging values model 67 journal of effective teaching in higher education, vol. 4, no. 1 successful as a student. the peer value and, specifically, collaboration is found to be a significant prominent value expressed between all three high-impact practice cohorts, supporting the emerging values model (ev). ev model construct, peer value: skills the survey item #3, “ability to work with my peers is a valuable skill,” was found to be statistically significantly different for pre-post scores, as found in table 2. the amended values team survey question 10 was related to the ev model peer value: skills. for the group course cohort posttest, 12 (63%) of the respondents noted they agreed or strongly agreed with the statement “it is important to volunteer to lead groups” (survey # 10) with 7 neither agreeing or disagreeing on the likert scale. similarly, out of the in-class activities cohort, 8 (50%) of the respondents selected agree or strongly agree while 8 respondents (50%) remained neutral selecting “neither agree or disagree” with the statement “it is important to volunteer to lead groups.” these findings are consistent with williamson-ashe and ericksen’s (2017) original analysis (11 [55%] of 20 participants selected agree or strongly agree), demonstrating supportive evidence and consistency in the ev model. ev model construct, group work value: decision making and problem solving all three high-impact practice cohorts indicated decision making and problem solving as an emerging group work value. the amended value survey item number five (5), “solving problems in a group is an effective practice i have learned,” was directly related to the group work value: problem solving ev model construct. the significant difference in pre-post scores for the group work course cohort demonstrated this prominence (see table 1). for the learning community cohort, all respondents (100%) indicated support for the survey statement 5, “solving problems in a group is an effective practice i have learned,” with 4 responding “strongly agree” and 4 responding “agree.” similar findings emerged for the inclass activities cohort with 94% of respondents indicating “strongly agree” (6 respondents) or agree” (9 respondents) to survey item #5. for survey question 9, “solving problems in groups leads to better decisions than solving problems alone,” 75% of learning community cohort participants were in agreement, 63% of in-class activities cohort participants were in agreement, and 72% of groups course cohort respondents were in agreement. these results exceed the 50% of student participants in agreement found during the original research (williamson-ashe & ericksen, 2017), further supporting the ev model constructs. qualitative findings using systematic comparison several themes emerged in the data. emerging themes included collaboration comprising subcategories of camaraderie and extra help/assistance (ev modelpeer value: collaboration), new perspectives and ability to work with peers (ev modelpeer value: skills), and knowledge gain related to the subcategories of groups, general knowledge and problem solving (ev model group work values: decision making & problem solving). these emerging themes high-impact emerging values model 68 journal of effective teaching in higher education, vol. 4, no. 1 within the group high-impact practice cohorts match several ev model constructs (see table 2). table 2 emerging value model (ev model) constructs related to emerging themes within group high-impact practice cohorts emerging values model (ev) construct: emerging themes: high-impact practices cohort: data/quotes peer value collaboration peer collaboration: extra help /assistance camaraderie experiential learning group course cohort learning community cohort one benefit was “camaraderie” “having extra help with work” “getting to know classmates” in-class activities cohort “you get more work done in time” “furthering knowledge on what others had to say about assignment” “you are able to have help when completing work” peer value: skills peer learning: learning from peers new perspectives group course cohort learning community cohort “i personally learned more about myself/others which allowed me to be comfortable” “one benefit was “creative icebreakers and activities that my classmates provided” “you can understand the material better from other people perspective” “allowed me to see things on different perspectives” “learning different perspectives” “things don’t always go as you planned, and when you get lemons make lemonade” in-class activities cohort “getting different perspectives on a subject” “being able to work with others and see other peoples understanding” “being able to get a better understanding of something from a peer” “learning to be open minded of other people ideas” group work value: decision making and problem solving knowledge gain: groups group course cohort “understanding group dynamics in group facilitation” “i am now able to navigate through the phases of the group” “learning skills to effectively facilitate groups” high-impact emerging values model 69 journal of effective teaching in higher education, vol. 4, no. 1 general in-class activities cohort one benefit of group work “get more knowledge” “working in a group brings in more information” ev model construct, peer value: collaboration the most prominent emerging theme was collaboration, which directly matches the peer value: collaboration construct related to the emerging value model (ev model). additionally, all three cohorts reported the emerging subcategory “camaraderie.” specifically, one learning community cohort participant reported the benefit “getting to know classmates” (respondent #2) while “camaraderie” (respondent # 8) was noted by a groups course cohort participant. another respondent highlighted the benefits of collaboration with comments such as “you get more work done in time” (respondent # 8, in-class activities cohort participant). ev model construct, peer value: skills emerging themes supported the ev model construct peer value: skill including the theme new perspectives. some benefits noted by student participants from the inclass activities cohort were “engaging with other people and being innovative learning new things” (respondent #2, in-class activities cohort) and “getting different perspectives on a subject” (respondent #14, in-class activities cohort). further support of the peer value: skills construct emerged in data from the learning community cohort; when asked about the benefit of group participation, “learning new perspectives” (respondent #7, learning community cohort) was noted. a participant in the group course cohort indicated that after participation in the course they were “getter better with speaking in front of groups and learning to work with different types of personalities” (respondent #17, group course cohort). similarly, an in-class activities participant reported “learning to be open minded of other people ideas” (respondent #13, in-class activities cohort). ev model construct, decision making and problem solving one participant noted gains in understanding problem solving in groups through the general knowledge gain theme “get more knowledge” (respondent # 10, in-class activities cohort). respondents in the group course cohort shared an increase in their knowledge specifically related to group work, commenting that “i am now able to navigate well through the phases of the groups life-cycle” (respondent #18, group work course cohort), and shared that they are “learning skills to effectively facilitate groups” (respondent #9, group work course cohort). group course cohort prominence additionally, several emerging values model constructs were highly supported by the specific survey item findings for the groups course high-impact practice cohort. a paired-samples t-test was conducted to compare each of the amended values high-impact emerging values model 70 journal of effective teaching in higher education, vol. 4, no. 1 team survey items pre-post ratings after attending the respective courses. there was a prominent statistically significant difference found in three (3) group course cohort survey items related to the emerging values model. specifically, the following emerging themes match with the emerging values model constructs: waste of time (ev modelgroup work value: productive), ability to work with peers (ev modelpeer value-skills), and solving problems (ev modelgroup work value: decision-making & problem solving). the pre-post reverse scoring results for the survey item “it is a waste of time to work in groups” indicated there is a statistical significance for the self-reported ratings for pre-waste of time rating (m = 2, sd = .737) and post-waste of time rating (m = 1.368, sd = .443) following participation in the graduate level course t(19) = 2.534, p = .03. for the item “ability to work with peers,” there was a significant difference in the scores for pre-peers rating (m = 4.421, sd = .349) and post-peers rating (m = 4.895, sd = .094) following participation in the in-class activities cohort t(19) = -3.101, p = .01. the results for survey item (#5), addressing the ability to effectively solve problems in a group, revealed a significant difference in the scores for pre-solving problems rating (m = 4.211, sd = .271) and post-solving problems rating (m = 4.737, sd = .194) following participation in the groups course cohort t(19) = -3.363, p = .01. table 3 represents the prominent mean difference for self-rating for the group course cohort and specific item analysis for the value of group work, the ability to work with peers, and learning to solve problems in a group. table 3 mean difference for self-rating of specific survey items for the group course high-impact practice cohort. as reported by the amended values team survey survey item pretest mean posttest mean mean difference t df p *it is a waste of my time to work in groups (#2) 2 v-0.737 1.368 v-0.443 0.632 2.534 18 .03 the ability to work with my peers is a valuable skill. (#3) 4.421 v-0.349 4.895 v-0.094 .474 -3.101 18 .01 solving problems in a group is an effective practice that i have learned. (#5) 4.211 v-0.271 4.737 v-0.194 .526 -3.363 18 .01 note. p< 0.05 *reverse scoring for item high-impact emerging values model 71 journal of effective teaching in higher education, vol. 4, no. 1 within the group course cohort, the most significant pre-post differences were found for three (3) items: waste of time (reverse scoring), the ability to work with peers, and learning to solve problems in a group. these survey items support the emerging value model constructs for the groups course cohort. limitations the difference in class sizes may have contributed to the variations reported by participants. a smaller class size means more individual instruction from the professor, which could result in a favorable response to their experience. the small sample size makes it difficult to generalize the results. graduate students’ participation in an elective course could suggest their preference for group work might also influence the outcomes. discussion and implications these findings suggest participation in various high-impact practice group elements (three cohorts; group course, learning community, and group activity integration) promotes a gain in knowledge and value for group work. all three cohorts reported some benefit from their participation in the group experience. different types of benefits were expressed for the three different cohorts. these results can be used to support educators’ use of group work in the class and challenge students to work with group members towards effective outcomes, which prepares them for employment. finding ways to enhance students’ connections between course material and their careers is optimal. collaboration is the ability to effectively work with others and is one of the primary skills employers are seeking (monster.com, topresume.com). others use a similar term, such as “teamwork skills” (indeed.com, thebalancecareers.com), to recognize the value of this characteristic. students’ high-impact practice experiences demonstrate their ability to connect the value of participation in groups (in various formats), and teamwork is becoming an essential expectation from hiring professionals. educators can assist in the development of these professional skills in students to best prepare them for future employment. the high-impact practice of experiential group work in the class can contribute to students’ knowledge about collaborating with other group members while increasing the value of group work for participants/future professionals as indicated by the ev model. this research further validates the ev model and its relevance to the high-impact practice of engaging group work. it is not surprising that since the courses have varying engagements in group activities, there are different outcomes. the largest impact related to the emerging values model was from the groups course cohort. this suggests more frequent engagement and direct group work interaction, leads to a greater appreciation of groups, and supports the ev model emerging values: peer value (skills, collaboration) and group work value (productive, decision-making and problem solving). high-impact emerging values model 72 journal of effective teaching in higher education, vol. 4, no. 1 the largest pre-post statistical difference was reported by those participants in the high-impact practice group course cohort. perhaps not surprisingly, an elective graduate course teaching group content had the most significant support for the ev model. these graduate students are in fact studying the group process and learning about specific group benefits, which translates to an understanding about groups. additionally, since this is an elective course, students have a personal interest and therefore may see more value in group work and have more commitment to the success of groups. these graduate students have taken an initial graduate level group course and likely already taken an undergraduate course including group content; therefore, this group course content builds on their previous knowledge and experience. the prominence for the peer value: collaboration across all three cohorts demonstrates the value of offering the high-impact practice of group work engagement in various formats during a course. exposing students to group work and preparing them for effective professional development in groups and teams can be promoted in the classroom. educators should continue to promote group work education within their courses as they are able. the results for the peer collaboration construct in the emerging values model (williamson-ashe & ericksen, 2017) indicate a statistical difference in pre-post results, suggesting participants gained a new perspective about peer collaboration following participation in the course. this high-impact practice format indicates a gain for all participants. collaboration is an essential skill for effective professional development and is highly valued by potential employers. the peer value: collaboration (ev model) is helpful to real-world application for students. it is necessary for individuals to be able to work in various types of groups/teams in their profession. learning communities have been found to be high-impact practices that enhance the learning experiences for students (kuh, 2008); therefore, the lower perceived benefits are surprising for this group. students’ self-selection to participate in the learning community also suggests that they prefer working with like-minded individuals in a group setting. perhaps participation in this learning community (lc) was impacted by the optional nature of the activities outside of classroom time and/or students not viewing the lc as a group. the last-minute unavoidable change to the itinerary for the trip to the general assembly (state capital) may have been upsetting to some of the student participants and may be reflected in their responses. a lack of group cohesiveness can also impact the group experience as noted by the learning community cohort with responses like “not everyone completes their part of the assignment” (respondent #8) and echoed by the group course cohort frustration involved in “getting everyone to stay engaged and participate” (respondent # 9). these research findings support the benefit of in-class group activities, which are similar to the approach of small group discussions that enhance student classroom learning (knight, 2000). both the group course cohort and the in-class activities cohort courses required active engagement in group activities to accomplish school high-impact emerging values model 73 journal of effective teaching in higher education, vol. 4, no. 1 assignments while learning communities participated in additional voluntary group activities outside the classroom. it is important for students to learn the skills associated with group work as noted by potential employers (cruz et al., 2020). this research suggests students learn the value of group work through the high-impact practices of experiential small group participation, as supported in the literature (cruz et al., 2020). although some students complain about the challenges of working in groups with inferences such as “some group members do not participate” (respondent #3, learning community cohort), the data overwhelming supports the group work decisionmaking & problem-solving found in the emerging values model. supportive data is also shared by respondents about the benefit of group knowledge: “understanding the process of groups was a valuable benefit for me” (respondent #19, group course cohort) and “working in group brings in more information” (respondent # 9, in-class activities cohort). these findings suggest that incorporating group activities and engaging students in groups within course content can influence their perceptions about the value of group work. this is significant to note as it supports the social work curriculum and cswe competencies, which assert the importance of group work in social work. social work students will facilitate and practice psychoeducational, therapeutic, and support groups as professionals, and, therefore, educators need to adequately prepare them through group work in the class (cohen, 2010). engaging in classroom group work develops an appreciation for the values associated with group interactions. conclusion high-impact educational practice group elements (experiential learning-group course content, in-class activities, learning communities) were found to promote a gain in knowledge and value for group work, supporting the ev model. participants in the group course (highest group engagement and content) were found to report the highest impact of group work; however, all students shared benefits from the group experience. exposure to group work was positive for all students, developed their appreciation for group work, and prepared them for professional group facilitation. the inclusion of high-impact group activities in various formats improves students’ knowledge and value for group work. this better prepares students for the workforce where teamwork and collaboration are prominent. this research demonstrates the ev model offers valuable considerations in the implementation of group work in the class to help perpetuate both personal and professional growth and development. group work develops valuable transferable skill for all professions. additionally, the ev model offers a good reminder to educators and students that experiential group work assists in the best understanding of the process to become effective group professionals and that they need to overtly discuss the positive outcomes of group work. high-impact emerging values model 74 journal of effective teaching in higher education, vol. 4, no. 1 conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references carini, r. m., kuh, g. d., & klein, s. (2006). student engagement and student learning: testing the linkages. research in higher education, 47(1), 1–32. doi:10.1007/s11162-005-8150-9. cohen, m. b. (2010). using student task groups to teach group process and development. social work with groups, 34(1), 51–60. https://www.tandfonline.com/doi/full/10.1080/01609513.2010.503384 council on social work education. (2015). educational policy and accreditation standards. https://cswe.org/getattachment/accreditation/accreditationprocess/2015-epas/2015epas_web_final.pdf.aspx creswell, j. w., & poth, c. n. (2018). qualitative inquiry and research design: choosing among five approaches (4th ed.). sage publications. cruz, l., meadows, j., & panter, n. (2020). the fledgling mindset: preparing biology students to leave the academic nest. journal of effective teaching in higher education, 3(2), 47–75. https://jethe.org/index.php/jethe/article/view/55/36 ericksen, k. s. (2019). educational instruction for group work with diverse members: innovative student classroom engagement using a game-based learning activity. currents in teaching and learning, 11(1), 157–182. https://www.worcester.edu/currents-archives/ ericksen, k. s., & williamson-ashe, s. (2019). holistic justice instruction through high-impact educational practices at historically black colleges and universities. urban social work, 3(1), s100–s114. doi: 10.1891/24748684.3.s1.s100 finley, a., & mcnair, t. (2013). assessing underserved students’ engagement in high-impact practices. association of american colleges and universities. https://aacu.org/sites/default/files/files/assessinghips/assessinghips_tggra ntreport.pdf humphrey, k. (2014). lessons learned from experiential group work learning. social work with groups, 37(1), 61–72. https://www.researchgate.net/publication/273480976_lessons_learned_fro m_experiential_group_work_learning high-impact emerging values model 75 journal of effective teaching in higher education, vol. 4, no. 1 kinzie, j. (2010). student engagement and learning: experiences that matter. in j. christensen hughes & j. mighty (eds.), taking stock: research on teaching and learning in higher education (pp.139–153). school of policy studies, queens university at kingston. knight, c. (2000). critical content of group work for the undergraduate social work practice curriculum. the journal of baccalaureate social work, 5, 93–111. kolb, d. a. (1984). experiential learning: experience as the source of learning and development. prentice hall. kuh, g. (2008). high-impact educational practices: what they are, who has access to them, and why they matter. association of american colleges and universities. https://www.aacu.org/leap/hips levine, r., o'boyle, m., haidet, p., lynn, d., stone, m., wolf, d., & paniagua, f. (2004). transforming a clinical clerkship with team learning. teaching and learning in medicine, 16(3), 270–275. http://brando.med.uiuc.edu/facultydev/classroom/interactivemethods/levine .pdf michaelsen, l. k., davidson, n. & major, c.h. (2014). team-based learning practices and principles in comparison with cooperative learning and problem–based learning. journal of excellence in college teaching, 25(3–4), 57–84. https://eric.ed.gov/?id=ej1041362 poort, i., jansen, e., hofman, a. (2020). does the group matter? effects of trust, cultural diversity, and group information on engagement in group work in higher education. higher education research and development. https://doi.org/10.1080/07294360.2020.1839024 rubin, a., & babbie, e. (2017). research methods for social work (9th ed.). cengage learning. strozier, a. l. (2008). group work in social work education: what is being taught? social work with groups, 20(1), 65–77. williamson-ashe, s., & ericksen, k. s. (2017). social work student perceptions of group work and the presence of value themes that correspond to group work success. journal of social work values and ethics, 14(2), 43–53. https://jswve.org/download/fall_2017_vol._14_no._2/43-social-workstudent-perceptions-of-group-work-2017-14-2.pdf williamson-ashe, s., & ericksen, k.s. (2019). validating social work student value theme perceptions using the emerging values model. journal of social work values and ethics, 16(1), 53–62. https://jswve.org/download/spring2019/full-issue-16-1-jswve-spring2019.pdf microsoft word jethe_editor_final.docx journal of effective teaching in higher education, vol. 5, no. 1 dear contributors, reviewers, and readers: as you will note from the delayed publication of the spring 2022 issue of the journal of effective teaching in higher education (jethe), the ongoing effects of covid-19 continue to delay the review and publication process for scholarly journals like ours. despite the delays, we are grateful for the work completed by our authors and reviewers and believe that the six articles included in this issue reflect our commitment to high quality scholarship of teaching and learning (sotl) across a range of disciplines. volume 5, issue 1 of jethe includes six articles on various topics related to teaching and learning in higher education from scholars in the us and canada. multiple articles explore the use of specific teaching techniques and their respective effects on students’ experiences and learning outcomes. in the first two articles, dandotkar and colleagues discuss the influence of reflective writing on students’ epistemic beliefs in psychology courses while torres et al. examine the relationship between knowledge retention and the use of crossword puzzles as a teaching tool in a graduate-level nursing course. next, dorland discusses her examination of the use of design thinking in an interdisciplinary undergraduate course in the article “that’s a good question: using design thinking to foster question formulation skill development.” the current issue also includes an article by anderson and colleagues that explored the use of a jigsaw teaching technique across multiple semesters of a diversity course for human development and family science majors. these four articles provide insights on the benefits and challenges of utilizing these innovative teaching techniques across disciplines. the two additional articles in volume 5.1 focus on cyberbullying in online education and the importance of reading in the college classroom, respectively. cox and raditch utilized document and policy analysis in their article titled “teaching online and cyberbullying: exploring cyberbullying policies.” in the final article, hollander and colleagues engaged in a mixed methods study to examine “the importance of college reading: professors’ and students’ views.” as always, all articles include implications for the practice of teaching across disciplines—an important contribution that is emphasized in jethe. we are honored to present the work of the authors included in the current issue. we hope to see our audience engage with these articles and want to thank you all for your continued interest in and support of our efforts at letter from the editors ii journal of effective teaching in higher education, vol. 5, no. 1 jethe. we are currently accepting submissions for our fall 2022 and future issues and invite you to contribute your own work to the journal. to submit an article and/or download full-text copies of current and past issues please visit http://jethe.org. finally, we would like to invite you to become a reviewer for jethe. if you are interested in reviewing for our journal, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. thank you for your continued support of jethe! james devita and colleen reilly coeditors, jethe microsoft word 213-article text-820-2-15-20210331.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.213 pedagogical reflection: demonstrating the value of introspection emily e. virtue, western carolina university, evirtue@wcu.edu abstract. instructors in higher education are often asked to reflect on their pedagogical choices in formulaic, detached, rote ways such as end of the year faculty evaluations or in response to peer review of teaching. yet, because of the parameters for these reflections, they often lack depth or much consideration. particularly because higher education institutions, especially in the united states, are focused on assessment, outcomes, student performance, and retention, little time is focused on particular pedagogical choices or interaction with students. numerous studies demonstrate that faculty-student interaction has a remarkable impact on student success. this paper, a scholarly personal narrative (spn), explores the value of sustained pedagogical reflection and how such reflection can benefit instructors and their students. keywords: pedagogy, reflection, higher education, scholarly personal narrative (spn) conversations about teaching in higher education often center on assessment, outcomes, and student performance (fook & sidhu, 2010). each of these factors can be linked to teacher preparation and pedagogical practices (henard & roseveare, 2012). numerous studies demonstrate that faculty interaction with students can have a major impact on student success (maestas et al., 2007; mayhew et al., 2016; schreiner et al., 2011; strayhorn, 2012). yet, those who teach undergraduate students often do not share their experiences beyond personal conversations. further, many may not take the time to reflect on their own mindsets and dispositions related to teaching. instructors and researchers have been conditioned to address our teaching in a detached, analytical manner devoid of emotion, even though many instructors consider teaching an incredibly personal and emotional process. it is time to reimagine the value that personal reflection and writing have on teacher preparation and pedagogical practice. in this scholarly personal narrative (spn), i explore the role of sustained reflection on teaching and how such reflection was a benefit to both me and my students. scholarly personal narrative seventeen years ago, robert nash (2004) published his first book on scholarly personal narrative (spn), a form of qualitative inquiry that is similar to, yet distinct from, traditional forms of educational inquiry such as autoethnography and phenomenology. spn allows for creativity and voice to shine through in its composition. in contrast to the rigid nature of academic writing, in spn the author’s voice is considered vital to knowledge delivery. such writing is highly accessible and readable. the narrative form invites readers in to experience the life of the author/narrator. bruner (1987) suggested that narratives, particularly those that pedagogical reflection: demonstrating the value of introspection 129 journal of effective teaching in higher education, vol. 4, no. 1 are autobiographical, “reflect prevailing theories” (p. 15) about one’s culture thus making such a form useful to narrator and reader alike, similar to nash’s conception of universalizability. spn calls for universalizability in the findings: a way to take one’s specific experience and appropriately broaden the ideas to other related contexts (nash, 2004; nash & bradley, 2011). through sustained critical reflection, i share my experiences demonstrating both the unique nature of being in a classroom with students while at the same time pointing to the commonalities many teachers share. narrative writing, more broadly speaking, allows authors to “character[ize] culture” by using “narrative models” (bruner, 1987, p. 15). this spn highlights my own experiences engaging in reflection guided by phronesis and utilizes my voice as essential to the argument. researchers, teachers, and stakeholders should reimagine what it means to engage in the hard work of preparing students for undergraduate study—particularly what it means for the teachers themselves. as teachers, our stories have merit and can serve to encourage and prepare new teachers and shed light on the realities of undergraduate teaching. reflection and teaching while the literature on pedagogy and reflection is vast and dates back as early as dewey (1933), scholars disagree on the best approach to take for reflection. brookfield (2017) argued that teachers must be critically reflective of their teaching experience because one can never be sure of the effect one has on a student’s learning experience. critical reflection “is the sustained and intentional process of identifying and checking the accuracy and validity of our teaching assumptions” (p. 3). when teachers critically reflect, they can ascertain what and how the student is learning and if any content has affected students in an unexpected or unintentional way. moreover, reflection forces teachers to identify any underlying assumptions they might have about teaching methods or students (brookfield, 2017). like other scholars (birmingham, 2004; brookfield, 2017), mankey (2014) suggested that reflection must occur continually to be effective. the benefits of reflection are difficult to summarize as outcomes are dependent on goals and context. fook and gardner (2007) noted, however, that critical reflection can lead to informed choice, “more enlivened teaching” (p. 130), emotional grounding, and improved professional practice. mankey (2014) argued that “reflection serves us best when we look inward to our own intentions, emotions, and actions in order to compare them with what we espouse our beliefs to be” (p. 86). palmer (2007) noted the importance of being “inward bound” so that teachers might eventually be “outward bound” and able to influence others. in recent years, teacher preparation programs have maintained a strong emphasis on self-reflection (blanchard et al., 2018; civitillo et al., 2019; hozebin, 2018; mccoy & lynam, 2020; sahli & benaissi, 2019; snead & frieberg, 2019), yet that expectation is not as common or studied as acutely in instructors in higher education. pedagogical reflection: demonstrating the value of introspection 130 journal of effective teaching in higher education, vol. 4, no. 1 phronesis as reflection my approach to reflection builds on birmingham’s (2004) theory of pedagogical reflection based on phronesis—a virtue presented by aristotle that is a “unifying and essential habit of the mind” (p. 314). i was drawn to this model of reflection because the concept of phronesis suggests a practice that becomes regular and holistic, rather than a one-off attempt at reflection. birmingham (2004) argued that reflective instructors must critically reflect on pedagogical choices so that they might best serve students. reflection, therefore, must be sustained, critical, and address specific questions or areas of concern for the instructor. further, what an instructor finds as a result of reflection must be shared with teaching peers and institutional stakeholders. traditional forms of scholarly work (for example a fivechapter dissertation or five section article), however, do not align with knowledge dissemination of personal experience. i argue that personal experiences and knowledge should appear more consistently in academic work and that the formatting of knowledge sharing should allow for creativity. as humans, we are drawn to stories; it is how we first make meaning of our world and as sarte (bruner, 1987) points out, we are surrounded by stories. therefore, storytelling as a means of knowledge production is the very essence of who we are. bruner (1987) extends this to mean that we expect to learn of lives through storytelling. in my previous years of teaching, the english department would require instructors to write a yearly reflection of their teaching. with very little direction, we were asked to explain what it is we do in the classroom and how effective we think we are. i despised this practice and, in all honestly, largely copied and pasted what i wrote the previous year in an attempt to simply complete the task and call it a day. because of my inconsiderate approach to the task, i did not benefit from the practice. it merely seemed like a hoop i had to jump through. while the intent of the exercise might have been for instructors to reflect on their work, i found it difficult to summarize in one page all the ways i interacted with and taught my students over the course of a year. i knew what i did in the class, but i did not often think about the big picture—what it meant for my overall teaching practice or for my students. phronesis, on the other hand, invites the teacher to be personal, consider the community, to avoid reducing the work into a few simple phrases, and provide new opportunities to construct knowledge (birmingham, 2004). clandinin (1992) explained how practical knowledge (identified by reflection) is “carved out of and shaped by situations; knowledge that is constructed and reconstructed as we live out our stories and retell and relive them through processes of reflection” (p. 125). the iterative process of reflection that i engaged in served to acknowledge the context of my classroom and the situations therein. moreover, as clandinin (1992) and birmingham (2004) suggested, my reflection of these experiences serve to create new knowledge. pedagogical reflection: demonstrating the value of introspection 131 journal of effective teaching in higher education, vol. 4, no. 1 “i write entirely to find out what i'm thinking, what i'm looking at, what i see and what it means.” ― joan didion the challenge with bridges the basis of my reflective practice was teaching in a summer bridge (sb) program. many institutions of higher education are committed to creating better access to a college degree (garcia & paz, 2009; nemelka et al., 2017). bridge programs exist as one way institutional leaders increase access and are so named because they bridge the gap between high school and college, particularly when students are thought to be underprepared for collegiate study. henry and stahl (2017) noted the “effects of being underprepared and/or misprepared for the next step grow exponentially” (p. 612) and argued that bridge programs can help align students to the rigors of college-level academics. students who enter bridge programs, regardless of their level of academic preparation, must transition to life on a college campus including rigorous academics and new social dynamics. developmentally and emotionally, the college transition is difficult for most students, perhaps more so for students in bridge programs who have been conditionally-enrolled and will only continue in college pending their success in the program (covarrubias et al., 2018). yet, in terms of bridge program implementation, university time and energy are spent primarily on supporting students. less time is spent on preparing faculty to work with underprepared student populations. in my first summer of teaching, i felt as though i was thrown to the wolves. i’d taught english to first year students for many years, but the sb program dynamics made me feel as if i was a brand new teacher and highlighted issues bridge programs might present. over the next few years, i began to purposefully plan my curriculum for sb students; i practiced adapting to the needs of my students as the summer progressed. finally, i thought i had a solid curriculum around which my class would run, but i wanted to be intentional about understanding my teaching. as such, i developed a protocol for sustained, critical reflection about my sb teaching. methods spn researchers connote data differently than traditional qualitative researchers. whereas qualitative data traditionally relies on interview transcripts, document analysis, and the like, spn researchers rely on their personal experience and perspectives. nash and bradley (2011) explained: spn researchers think of what they do as giving personal testimony to make their points rather than accumulating empirical evidence [author emphasis] to prove something beyond a shadow of a doubt […] for spn researchers, scholarship is credible [author emphasis] when it flows from what writers believe and love […] with all their hearts. this meaning of credible (l. credo) comes very close to the notion of personal creed. (p. 83) pedagogical reflection: demonstrating the value of introspection 132 journal of effective teaching in higher education, vol. 4, no. 1 content for this spn was based on my personal experiences teaching, informed by voice and video reflection conducted each day after teaching. extensive journaling of my reactions to daily interactions with students, case management meetings1, and class content allowed me to reflect on experiences in a deep, critical way. journaling practices allowed me to write about the “subjective experience” defined by nash and bradley (2011) as the main design objective of spn, which seeks to “investigate, present, and analyze the inner life of the writer in order to draw insights that might be universalizable [author emphasis] for readers” (p. 83). spn’s notion of universalizability answers shenton’s (2004) call for transferability in qualitative work in that the author must give enough contextual information about the study so the reader might be able to make a transfer to their own context. throughout the writing process, i utilized members from the case-management staff (the two other teachers in our cohort, the residence life case manager2, and staff who run the sb program) to check my recollection or details about particular students. by utilizing feedback from my peers, i “overlap[ped]” methods and increased dependability (shenton, 2004, p. 71). collecting perspectives the data i collected in the form of journals, voice, and video recordings are traditional methods of data collection. spn researchers, however, eschew the term data in favor of perspectives because of its constructivist leanings (in that it allows the author to give meaning to observation) (nash & bradley, 2011). in each of the daily reflections i collected, i utilized the same set of questions to guide my reflection (similar to following an interview protocol): 1. summarize the day’s lesson (what i did, how i felt, and how students responded). 2. describe my mindset and the perspectives i anticipated addressing in the lesson (behavioral, psychological, socio-cultural, and holistic). 3. discuss my insights for adapting/changing my instruction based on the day’s lesson. while i have data on which i heavily relied, i also used introspective questions as i reviewed my previously recorded perspectives. 1 case management meetings are attended by all instructors in the cohort, a residential life case manager, the program coordinator for the sb program, and a student intern. the weekly, hour-long meetings consist of conversations about students who may be struggling both in and out of class and explore how to address the issues and support the students to complete the program. 2 the primary responsibility of a residential life case manager is to engage with first-year residents outside of class and make sure they are adapting socially and academically to college life. the case manager connects with students when behavior or other concerns arise and advises the student on how to handle any problems they may have. pedagogical reflection: demonstrating the value of introspection 133 journal of effective teaching in higher education, vol. 4, no. 1 the introspective questions came about as i identified themes that emerged in my teaching and reflection (a necessary step in the spn process that allowed me to work toward universalizability). such introspective questions required further writing and reflecting on the teaching experience. this continued reflection deviated from traditional data collection in that it was ongoing and allowed for prolonged reflection and change in viewpoint given the difference in time between initial reflection and review. further, my reflection on reflection was key because while my summer course was complete, i was an instructor to the same students the following fall. my reflection about teaching them changed as i continued to learn more about my students, their struggles, and how they developed over the summer. because of my continued interaction with these students, i came to better understand what i saw in the summer classroom with new information presented after the course was complete. while traditional forms of data collection may not allow for a constant flow of data collection, spn welcomes the changing, real-time variables that equipped me to better understand pedagogical choices i made in previous months. many spn pieces are written without formal data collection, hence the use of the term perspectives rather than data (nash & bradley, 2011), yet i chose to be methodical in my approach to collecting my perspectives. my decision to follow a specific protocol was made so i might ensure a rigorous method of perspective collecting that would still allow me to share the information in narrative form. it may also serve as a guide for teachers dedicated to doing similar self-reflective work. what good am i doing? the purpose of self-exploration in teaching teachers often get bogged down in the tasks that need to be done: prep a class, grade a quiz, complete an irb application so that data collection can begin. as a teacher and researcher, i can get so enmeshed in the day to day business of teaching, i often forget the purpose of teaching and even the people i’m hoping to teach. so many of my teacher-friends have said the same. as a result, sometimes i can lose sight of my goals: to help students understand new ideas and think in different, challenging ways. i try to equip students with the skills to solve hard problems and prepare them to interact with one another beyond the walls of our classroom. but i can forget that teaching is more than presenting information to students with their eyes and ears open. teaching is personal, relational, and can be intimate. at the same time, teaching can be guided by clear parameters: learning outcomes, time in the classroom, the classroom environment itself. when we approach the task of teaching in a personal way (building relationships with students, allowing the students to explore their own values, ideas, and perceptions in class) learning goals are met without feeling as stringent. students learn better when teachers build relationships with students in the classroom, and to do that, we (teachers and students) must share a part of ourselves. to share ourselves, we tell stories—embarrassing ones about sitting on a tack in class or difficult ones about how hard it was to focus in high school when medication ran pedagogical reflection: demonstrating the value of introspection 134 journal of effective teaching in higher education, vol. 4, no. 1 short. as we tell these stories, we learn about one another, about our community, and about ourselves both as people and as academics who have something important to share. both radical pedagogy (hooks, 1994) and liberatory pedagogy (freire, 1970/2003) call for teachers to welcome their students into the classroom and share their own experiences and beliefs. sharing stories not only allows others to better understand one’s perspective, it provides a way to make sense of one’s life and to understand how one keeps striving even when life is difficult. joan didion (1979) is famous for writing, “we tell ourselves stories in order to live.” like didion asserts, the stories i will share of my teaching have helped me understand what it means to teach new college students and, just as importantly, what it means to me to live as a teacher. the stories of triumph give me comfort when i am struggling to reach a student and help them grow. the stories of failure remind me that teaching is a craft that can always be practiced and adapted. importantly, regardless of the subject matter, stories are vital. the narrative nature of this study will convey to the reader the personal, intimate nature of teaching and the influence reflection can have on one’s pedagogy. “the institution hires you for competencies, but the whole self comes to work.” – jim gould what you do is part of who you are there are aspects of life that a teacher should not bring to her classroom, yet to act as if we are not people serves as a barrier to creating relationships with students. if we as teachers want to help students better understand themselves as learners, better engage with their education, and consider the hard questions about who they are and how they see themselves, then we must also ask those same questions of ourselves. we must share a part of ourselves with our students. what and how we decide to share is up to us, but we must bring a sense of vulnerability to the classroom if we expect our students to the same (hooks, 1994; rademacher, 2017). i have always been drawn to storytellers. the faculty members i gravitated toward in undergrad were the ones who offered up a piece of themselves in class. though i had plenty of wonderful educators who helped me understand, apply, and use course material in class, i was more engaged in class and ready to learn with the professors who showed something of themselves in the classroom. for three years of undergraduate study, i took a number of courses taught by dr. kelly lowe. he was the first professor i had who allowed us to call him by his first name, and that alone was proof to me that professors are real human beings. i had a tendency to view professors as b-list celebrities: even if i did not know exactly why they were important, i knew a lot of people really liked them, and i should give them deference. i considered professors to be the keepers of knowledge. as a first-generation student, i was not familiar with the “ivory tower” conceptions of pedagogical reflection: demonstrating the value of introspection 135 journal of effective teaching in higher education, vol. 4, no. 1 higher education, nonetheless i imagined my professors sitting in their cozy, bookcluttered offices doing the hard work of thinking. dr. lowe was incredibly intelligent and challenged his students to work at his level. his sardonic wit and dry sense of humor were a stark contrast to some of the other faculty members in the english and writing departments. over the course of many semesters, i learned more about kelly—his involvement in local theater, his love of frank zappa, and the fact that we could tell his mood based on the socks he was wearing. because kelly offered a piece of himself to the class, i found it easier to be real with him. when i considered attending graduate school, kelly was the first person i asked for advice. even after he changed institutions, he kept in touch and counseled me in my transition to graduate school. fifteen years later, if i had to choose one faculty member who most influenced my education and career trajectory, kelly would be at the top of my list. scanlon et al. (2009) found first-year students in particular seek relationships with their professors. my own experience (starting my freshman year and continuing through subsequent years) reflects these findings. each summer i taught in the sb program, i built relationships with students, and scanlon and associates’ assertions were made real. relationships matter—they matter to the student and the teacher. as i conducted my daily reflections, i realized that, having never left higher education once i began as a college freshman, my perspective on college life had changed. i had forgotten my earlier fears, because i had become a part of university life—of it, not just in it. my students (and some of their families) however, were new to it. so, i began to add lessons about what it means to be a college student into our class time. i taught them how to actively read texts, how to find help on campus for common first-year problems (study skills, homesickness, etc.), and centralized our discussions on literature including content that would resonate with them. in one scene from i am charlotte simmons, for example, students read about the title character’s overall awe at starting college; she’s overcome by the landscape and the people, and she’s painfully aware that she and her family do not fit this mold. while the lesson was on language, close reading, and discerning dense text, students also noted that their own experiences were similar. they even applied those feelings of inferiority to preparing for the lesson (see virtue, 2019, for more on student response to this instruction). reflecting on these experiences showed me the dual purpose of the sb program and how i could change my curriculum and pedagogical choices to meet the learning outcome and social and emotional needs of students. while i had a number of successful moments, there were times when i lacked confidence in my teaching. reflecting on teaching leads to many “what if” questions. after class i would journal or record my reflection and find that i merely asked myself a battery of questions. a few times, an idea would take shape, and i would ramble on or scribble a few notes down to use the next day or to table the thought for the next time i taught the course. (that list of “what to do next time” is ever-changing and never-ending.) i found myself wrestling with the questions i asked of myself. it was straight up pedagogical reflection: demonstrating the value of introspection 136 journal of effective teaching in higher education, vol. 4, no. 1 tiring to reflect on my work every day. worse, on the days when the lesson did not go as planned, or my students had twenty side conversations that distracted me and everyone else in the room, i started to feel like a failure. the “what if” questions are only one component of teaching. before i could reflect on the work of the day, i first had to complete a number of other tasks: reread the text i had assigned, prep activities, create assignments, add informational links in blackboard, grade assignments (with helpful and timely feedback), and correspond with students, staff, and colleagues in my teaching cohort. this list only reflects the demands of one class; it does not include any other responsibilities i had on any given day while also being a wife and mother. even in the moments when teaching was going well, i was tired. my exhaustion was mental and physical. while my fatigue did not always feel as though it was a direct result of teaching, teaching certainly wore on me. my questions about how to improve shifted to questions about my ability to teach at all. was i helping students? were they learning or gaining academic skills they did not have before? i did not know and felt like there was no way to tell. the only thing i knew on those days was reflecting was the pits and i hated it. birmingham (2004) would probably say to me, that’s the whole point. the fact that i felt so emotionally involved in the work, that i was willing to reflect even on the days i knew i would not be pleased with the outcome, that’s what reflecting is all about. that’s phronesis in action. being able to share this experience with others has considerable value. if stakeholders are to value the work teachers do, they must first understand the daily giving-of-oneself that goes into teaching. teachers must continue the practice of reflecting on their experience and sharing their stories with those who influence change. implications for teachers and practitioners while the format of spn requires the author to share the “subjective experience” (nash & bradley, 2011, p. 83), the goal is to take that experience and make it universalizable to others. my experiences teaching in sb were certainly context dependent, and no other teacher will have the same experiences as me. however, some of my experiences may be mirrored in classes for first-year or conditionallyenrolled students. many students struggle with issues related to the student transition including struggling to see themselves as academics and adjusting to a new environment while encountering different cultures and perspectives for the first time. thus, my recommendations below, though guided by personal experience, may inform those who encounter similar situations and contexts. as shenton (2004) argued, the reader must consider the similarities and differences in their own practices and apply the information from this research as it fits to their own circumstance. the value of reflection as teachers work toward improving their practices, they should also engage in reflection of these practices and their interactions with their students. teaching is pedagogical reflection: demonstrating the value of introspection 137 journal of effective teaching in higher education, vol. 4, no. 1 more than disseminating information, it is a personal practice that should engage the whole classroom—each student—in ways that are meaningful and developmentally appropriate. birmingham (2004) reminds us, however, that making sense of reflection will not be easy and will likely not be succinct. reflection is not meant to quantify our abilities, rather it is meant to help us understand the work we are doing and, ideally, improve upon it. sahli and benaissi (2019) noted change comes from reflection because it allows teachers, especially those focused on teaching methods, teacher-student relationships, and “the human conditions of teaching” (p. 48), to identify weaknesses and seek out support in developing quality instruction. when i first started thinking about the process of intentional reflection on my teaching, i thought about it in this sort of lofty ah yes, the life of the mind! kind of way. i imagined i would sit and think, then write—long-hand—all the encounters and conversations i had in the given day. i soon realized that was no way to research or reflect in a meaningful way. with the help of my dissertation chair and many teacher friends who are far wiser than me, i began to do what many gifted teachers do: i wrote a brief lesson plan for reflection. through a number of drafts, i whittled down a reflection protocol that guided my reflection and kept me focused. as such, i was able to maintain focus on the issues that were most salient to my daily teaching: thinking about how i designed each lesson and to what extent these lessons “worked.” the lessons that worked the best were, not surprisingly, the ones that were student focused. when students engaged with material that was reminiscent of aspects of their own lives, their conversation flourished and their writing improved (virtue, 2019). for example, as students read various fictional pieces that took place on college campuses, they were able to discuss how aspects of the story were crucial to their understanding of theme, but also how it informed their new lives as college students. such findings were incredibly helpful to me as i continued to design my curriculum and reading lists; i added more excerpts and stories of students who struggled academically (e.g., the art of fielding) or socially (i.e., fangirl; i am charlotte simmons)—my students, after all, consistently commented on these characters and wrote about identifying with them. in order for teachers to learn from their students and the work they are doing in the classroom, i suggest teachers frame reflection in a style similar to their lesson plans. regardless of the specific goal, reflection should be intentional, prolonged (i.e., over a sustained period of time), and structured enough that when the teacher engages in a second round of reflection, the perspectives before them seem to be connected. without such structure, reflections may meander through various topics and, while cathartic, may not guide one toward the goal of better understanding one’s teaching. i began my reflections in written form and later found that video recording reflection was ultimately a more productive choice. i found that video recording meant i could reflect quickly and with more detail. other researchers (mccoy & lynam, 2020; snead & frieberg, 2019) noted the success pre-service teachers had with video self-reflection, particularly when followed up with additional feedback (weber et al., 2018). pedagogical reflection: demonstrating the value of introspection 138 journal of effective teaching in higher education, vol. 4, no. 1 the structure one creates should be focused on the goal of reflection (what do you want to learn about yourself as a teacher?) and the practicality of accomplishing the goal. i started with a long list of questions to consider each day and eventually narrowed the list down to three items on which to focus my attention. for reflection to work, it need not be a chore or, at the risk of sounding blasé, a total time-suck. reflecting on teaching is likely one of the most important work tasks you’ll do each day, but it should not take the whole day. following a guide of three to four prompts will foster precise and quick summaries of the day. providing structure gives additional rigor to data collection. brookfield (2017) noted, “the lens of personal experience is probably the lens of critical reflection that’s taken least seriously” (p. 170). while brookfield’s observation may be true, if the intent of reflection is to share personal experience with others, then thoroughly attending to personal experience is essential and worthy of the time and effort required. the second-pass on reflection (phronesis) will be invaluable. reflection on reflection allows the teacher to take time to think about what the teaching experience meant. such reflection works best when it does not happen during the semester in which one is teaching. the time and space to think and rethink about teaching will likely lead to a better understanding of teaching practices and what those practices mean for you and your students. further, maintaining such focus during one semester of teaching might lead to teaching fatigue or burnout. scholars note that faculty burnout is common and comparable to those who serve in school (sabagh et al., 2018, p. 12) and healthcare settings (watts & robertson, 2011). the demands of teaching are intense and emotionally stimulating. though i have always been an early-to-bed-early-to-rise individual, i found myself going to bed promptly after my kids were asleep, sometimes before 8:00 pm. one night, i fell asleep in my son’s bed waiting for him to nod off for the night. while tiring, reflection about teaching practices and how those practices are affected by other life obligations plays a vital role in an instructor’s ability to reach their students. as a teacher takes stock of their daily interactions, they may realize, as i did, that taking on too many responsibilities prevents teachers from giving students the focus they deserved. birmingham (2004) noted the importance of validating “the personal elements of teaching” and that teaching is often “more akin to personal development than professional development” (p. 321). teaching demands a lot of the instructor: social engagement and awareness of students, attention to how students are learning, and encouraging students to work independently and with others. godsey (2016) explained that for many k-12 teachers, the fatigue associated with the socially demanding requirements of teachers has led many to change careers altogether. the need to consider teaching obligations and recharge is vital to avoid burnout, especially in teachers who are introverts (godsey, 2016). though godsey’s article focused on secondary educators, much of what he found can apply to college educators as well. reflection is key, but so too is the time away from prolonged interaction that would allow for rejuvenation and a return to focus. as we reflect on our teaching practices, we are also reflecting on our personal lives, and pedagogical reflection: demonstrating the value of introspection 139 journal of effective teaching in higher education, vol. 4, no. 1 as bruner (1987) might have suggested, the culture of teaching. indeed, reflection and autobiographical writing provides a framework for how we change and “how these accounts come to take control of our ways of life” (p. 15). not only does reflection and the writing about it reveal new learning, it also allows us to use this knowledge to change our teaching lives for the better. thinking about teaching at the end of a long day of teaching, i have a hard time getting my mind to settle into sleep. i recount what happened in class: from the short conversations with students before class starts when we sing our favorite musicals to our difficult conversations about what is happening in the world around us. i consider how i shared a bit of myself with my students and hope that they know i work hard each day to see them, listen to their needs, and find ways to support them. sometimes, i fall short. many nights, however, i lie awake thinking about getting to class in the following days, so we can continue to engage with and learn from one another. i look forward to my time with my students, i love to see them come into their own. even though for most of them their development as academics is just beginning, little glimmers of the students and academics they will someday be peek through each day. as i try to get to sleep, i remind myself that every day, i have a chance to do important work. as access to higher education increases, faculty and institutional leaders must address the changing needs of students in their classrooms. these changes can only be addressed if, like my work in this study shows, teachers take the time to think about their teaching practices and how such practices might inhibit or encourage student growth. my own work on pedagogical reflection allowed me to better understand not only my students, but my role as an instructor and mentor to students. faculty members who are willing to do the hard work of sustained and critical reflection may find opportunities to better support their students and find more enjoyment in their roles as teachers. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references birmingham, c. (2004). phronesis: a model for pedagogical reflection. journal of teacher education, 55(4), 313–324. doi:10.1177/002248704266725 blanchard, s. b., king, e., van schagen, a., scott, m. r., crosby, d., & beasley, j. (2018). diversity, inclusion, equity, and social justice: how antibias content and self-reflection support early childhood preservice teacher consciousness. journal of early childhood teacher education, 39(4), 346–363. https://doi.org/10.1080/10901027.2017.1408722 pedagogical reflection: demonstrating the value of introspection 140 journal of effective teaching in higher education, vol. 4, no. 1 brookfield, s. d. (2017). becoming a critically reflective teacher (2nd ed.). josseybass. bruner, j. (1987). life as narrative. social research, 54(1), 11–32. https://www.jstor.org/stable/40970444 civitillo, s., juang, l. p., badra, m., & schachner, m. k. (2019). the interplay between culturally responsive teaching, cultural diversity beliefs, and selfreflection: a multiple case study. teaching and teacher education, 77, 341– 351. clandinin, d. j. (1992.) narrative and story in teacher education. in t. russell & h. munby (eds.)teachers and teaching: from classroom to reflection (pp.124– 137). the falmer press. covarrubias, r., gallimore, r., & okagaki, l. (2018). “i know that i should be here”: lessons learned from the first-year performance of borderline university applicants. journal of college student retention: research, theory & practice, 20(1), 92–115. doi:10.1177/1521025116651635 didion, j. (1976. december 5).why i write. the new york times. https://www.nytimes.com/1976/12/05/archives/why-i-write-why-iwrite.html dewey, j. (1933). how we think: a restatement of the relation of reflective thinking to the educative process. d. c. heath and company. fook, j., & gardner, f. (2007). practising critical reflection: a handbook. mcgrawhill education. ebook. fook, c. y., & sidhu, g. k. (2010). authentic assessment and pedagogical strategies in higher education. journal of social sciences, 6(2), 153–161. freire, p. (2003). pedagogy of the oppressed (30th anniversary ed.). continuum. (original work published 1970) garcia, l. d., & paz, c. c. (2009). evaluation of summer bridge programs. about campus, 14(4), 30–32. https://doi.org/10.1002/abc.299 godsey, m. (2016, january 25). why introverted teachers are burning out. the atlantic. https://www.theatlantic.com/education/archive/2016/01/whyintroverted-teachers-are-burning-out/425151/ henard, f., & roseveare, d. (2012). fostering quality teaching in higher education: policies and practices. an imhe guide for higher education institutions. organization for economic co-operation and development. http://www.printel.am/uploads/page/pdfs/fostering%20quality%20teaching %20in%20he_policies%20&%20practices_2012.pdf henry, l. a. & stahl, n. a. (2017). dismantling the developmental educational pipeline: potent pedagogies and promising practices that address the college readiness gap. journal of adolescent and adult literacy, 60(6), 611–616. https://doi.org/10.1002/jaal.640 hooks, b. (1994). teaching to transgress. routledge. pedagogical reflection: demonstrating the value of introspection 141 journal of effective teaching in higher education, vol. 4, no. 1 hozebin, c. 2018. structured post-observation conversations and their influence on teachers’ self-reflection and practice. schools and societal issues, 85(1), 45– 58. maestas, r., vaquera, g. s., & zehr, l. m. (2007). factors impacting sense of belonging at hispanic serving institutions. journal of hispanic higher education, 6, 237–256. mayhew, m. j., rockenbach, a. n., bowman, n. a., seifert, t. a., & wolniak, g. c. (2016). how college affects students: 21st century evidence that higher education works (vol. 3). jossey-bass. mankey, r. c. (2014). dialogue, reflection, and learning. in k. a. kline (ed.) reflection in action. (pp. 85-100). stylus. mccoy, s., & lynam, a. m. (2020). video-based self-reflection among pre-service teachers in ireland: a qualitative study. education and information technology, 26, 921–944. https://doi.org/10.1007/s10639-020-10299-w nash, r. j. (2004). liberating scholarly writing: the power of personal narrative. teachers college press. nash, r. j. & bradley, d. l. (2011). me-search and re-search: a guide for writing scholarly personal narrative manuscripts. information age publishing. nemelka, b., askeroth, j., & harbor, j. (2017). summer start: supporting success for conditionally admitted students in a summer bridge program. summer academe: a journal of higher education, 11, 2–13. palmer, p. j. (2007). the courage to teach. jossey-bass. rademacher, t. (2017). it won’t be easy: an exceedingly honest (and slightly unprofessional) love letter to teaching. university of minnesota press. sabagh, z., hall, n. c., & saroyan, a. (2018). antecedents, correlates and consequences of faculty burnout. educational research, 60(2): 131–156. https://doi.org/10.1080/00131881.2018.1461573 sahli, n., & benaissi, f. b. (2019). using teacher self-assessment and reflection to foster change in the writing classroom. international journal of social sciences & educational studies, 5(4), 39–53. https://doi.org/10.23918/ijsses.v5i4p39 scanlon, l., rowling, l., & weber, z. (2007). ‘you don’t have like an identity…you are just lost in a crowd’: forming a student identity in the first-year transition to college. journal of youth studies, 10(2), 223–241. doi:10.1080/13676260600983684 schreiner, l. a., noel, p., anderson, e. c., & cantwell, l. (2011). the impact of faculty and staff on high-risk college student persistence. journal of college student development, 52(3), 321–338. https://doi.org/10.1353/csd.2011.0044 shenton, a. k. (2004). strategies for ensuring trustworthiness in qualitative research projects. education for information, 22(2), 63–75. pedagogical reflection: demonstrating the value of introspection 142 journal of effective teaching in higher education, vol. 4, no. 1 snead, l. o., & freiberg, h. j. (2019). rethinking student teacher feedback: using a self-assessment resource with student teachers. journal of teacher education, 70(2), 155–168. https://doi.org/10.1177/0022487117734535 strayhorn, t. l. (2012). college students’ sense of belonging: a key to educational success for all students. routledge. virtue, e. e. (2019). using fiction to support identity development and transition in conditionally-enrolled students. journal of effective teaching in higher education, 2(2), 70–84. watts, j., & robertson, n. (2011). burnout in university teaching staff: a systematic literature review. educational research, 53(1): 33–50 weber, k. e., gold, b., prilop, c. n., & kleinknecht, m. (2018). promoting preservice teachers’ professional vision of classroom management during practical school training: effects of a structured onlineand video-based selfreflection and feedback intervention. teaching and teacher education, 76, 39–49. microsoft word robinson_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi 10.36021/jethe.v4i2.200 teaching black lives amidst black death: reflections from a black visiting professor robert p. robinson, john jay college, city university of new york, rrobinson@jjay.cuny.edu abstract. in this essay the author addresses the struggles of teaching a special topics course, black freedom movement education, in the midst of a global pandemic and donald trump’s proposed ban on anti-racist training and critical race theory. the educator framed the course under the conceptual lens of stealin’ the meetin’—a black antebellum practice of creating otherwise literacy practices under repressive circumstances. this form of educational resistance continued beyond enslavement as black communities used the resources available to educate each other by any means necessary (robinson, 2020). on a smaller scale, this class carried on the resistance through critical metacognitive engagement with black education history. the author discusses how he navigated the course when, less than halfway through the quarter, a black man was killed and burned in a trench. using emails, lecture notes, student evaluations, texts, and reflections, the author shares vignettes of tension, black affinity, and communal restoration. keywords: place-making, stealin’ the meetin’, black education history, black affinity groups, anti-blackness i awakened that late september morning and groggily scrolled through emails on my phone. after deleting the usual scores of junk mail, i paused at the email from our school president: you may be aware of reports of a horrific death in our community. credible news sources have identified the victim as michael williams, 44…our hearts go out to the family and friends of mr. williams in this unspeakable tragedy. i could feel the heaviness surround my chest as my heartrate increased, but i was still unaware of the gravity of this news. i checked my missed group text notifications. a colleague who knew more of the details was even more shaken than i was. she asked if we needed anything or if we wanted to join her for a trip to town, but sadly, i looked at the time to see that i had missed her by over an hour. never in her text did she indicate what made this death particularly harrowing. as i walked downstairs to join my roommate, i was met with the details of the horrific reality. on wednesday, authorities found the charred body of the victim in a trench in the next town. on friday, reports identified the body as a black man from our town. this burned black body in an overwhelmingly white midwestern region less than two months before the election was a sign. i had gotten used to the feigned kindness and awkward stares from the people of my new environment. i knew what it was like to be one of the 300 (which felt more like 30) or so black people in this town. but this—this public lynching that arrived on the heels of a black lives in black death 100 journal of effective teaching in higher education, vol. 4, no. 2 summer of marching for black lives, after a scholar strike just moments following labor day weekend, after a move from the blackest city i had ever lived in to one of the whitest cities i had ever lived in—this felt like a declaration. one can only imagine how much more concentrated the town death was for my students in the midst of these educational contexts. as we learned about the legacy of black pedagogies in my black freedom movement education course, we highlighted black women educators—paying close attention to septima clark, ella baker, bernice robinson, ericka huggins, and gertrude wilkes—women of the black freedom movement who recognized the personal, political, and pedagogical as intertwined. in many cases, educators and students at citizenship schools, freedom schools, and early panther political education spaces encountered extreme threats of violence from white local residents and from white law enforcement. here i was, a black queer historian of black education from a working-class family in a small, white, rural town, teaching this very black education class to a virtual classroom that was half white—making visible the black radical imagination in a world of antiblackness. i wanted to call upon the strength and fortitude of the elders whose names were on our mouths every class session. that saturday, however, as i came to terms with my own hypervisibility in a neighborhood with blue lives matter flags and trump-pence signs, i wondered if i could return to the course on tuesday with my head held high. i wondered if i, too, could speak truth to power. in this reflexive essay, i consider what it was like for me to teach through this difficult moment. conversations include how i made space for all students in class. i specifically outline how i made targeted space for my black students. finally, i consider how i offered such support when i was in fear for my own safety—knowing my own hypervisibility as a black person in a white town. looking at the short quarter term overall, i question the following: • what were the joys and challenges of teaching this course before, during, and after the college community struggled with the violent murder of a local black man? • how did historical and contemporary black pedagogical approaches aid with and/or complicate teaching under these circumstances? in this essay, i will briefly outline the structure of the class, the assignments, the general backgrounds of the students, and the timeline of the course. i will highlight in-class and out-of-class conversations with students and analyze correspondence with colleagues. finally, i will discuss the strengths of the course, followed by student course reflections. ultimately, i argue that the course material, themes, and weekly classroom practices along with the self-preservation approaches and affinity group sessions for black faculty, staff, and students helped the instructor and the students to survive and thrive in the presence of black death. black lives in black death 101 journal of effective teaching in higher education, vol. 4, no. 2 course overview the course i designed was a culmination of my research and praxis: a course that merged the history of resistance during the black freedom movement with the pedagogical affordances of black education that dated back to the antebellum era. i built the course with the concept of “stealin’ the meetin’,” which was a literacy practice during enslavement. in the federal writers’ project: slave narrative project, an account from one woman by the name of elizabeth sparks outlined this practice. sparks shared, “nigguhs used to go way off in the quarters an’ slip an’ have meetins. they called it stealin’ the meetin’. the children used to teach me to read. schools! son, there warn’t no schools” for black folks (federal writers’ project, 1936, pp. 53–54). the short account of her educational history provides insight into the intergenerational dynamics of education in antebellum black communities—a space where racial hierarchies were challenged in order to provide liberation through education. sparks was enslaved in virginia. at the time, teaching a black person to read was illegal, punishable by beatings, imprisonment, or—in some cases—death. this tradition of “‘stealin’ the meetin’,” then, established a trajectory of resistance in black life—a long line of bold and sometimes life-threatening engagements with education. this concept holds heuristic importance as it conveys the extreme conditions and hurdles black communities have had to consistently overcome to establish education for themselves, their children, and future generations. this was the teleological and conceptual framework that guided our engagement with black education history for the term. the course was designed for a 7½ week quarter. part of the rationale for the short timeframe was to provide students with intense focused time—fewer courses in a short time. i had 13 students in the class: five black women, four white women, two white men, one asian man, and one latinx non-binary person. while i had long dreamed of a course that would combine black history and education, i knew that the dozens of books that served as the foundation of my dissertation would be entirely too much for any undergraduate class, especially one this short. i decided on the volume teach freedom: education for liberation in the african american tradition edited by charles m. payne and carol sills strickland (2008) and russel rickford’s (2016)we are an african people: independent education, black power, and the radical imagination. these readings helped us to trace the pedagogical throughline of stealin’ the meetin’. we met for two hours twice a week, discussing the texts in whole group and small group after i provided a small mini-lecture on the histories or theories we encountered in our readings. for half of the fridays, i had students write 200to 300-word journal entries about how they were feeling, what their learning goals were, and how they were progressing (or not) towards those goals. early on, i told the students they would design their own final based on their individual learning goals and expected outcomes for themselves. in addition to these questions, i asked students to reflect on the readings, paying attention to the pedagogical approaches, historical particularities, and usefulness for contemporary education. the purpose was to encourage students to track their black lives in black death 102 journal of effective teaching in higher education, vol. 4, no. 2 own learning even as they analyzed the learning structures of black liberatory organizations. stealin’ the meetin’: a brief historiography conceptual framework i am a black queer man from a working-class background teaching at a predominately white/historically white institution (hwi/pwi). my primary mode of inquiry is historical, which was the undercurrent of the course and the framing of stealin’ the meetin’. i am also a scholar of pedagogy, and i aim to speak to a practice of both affinity and black-specific liberatory practice connected to the historical foundations of stealin’ the meetin’. as a perpetual outsider—minoritized among the faculty, the town, and even my own classroom—i constantly felt the dearth of queer community, even as i struggled to connect to the black campus community being that none of us were on campus. despite these struggles, i constructed my class with a focus on otherwise opportunities: stealin’ the meetin’— carving out room to dream of educational opportunities when few existed (anderson, 1988; federal writers’ project, 1936; robinson, 2020). this was especially true as my students and i developed affinity meetings. moving from within outward, i engage in this reflexive exercise to heal from my own struggles, to amplify historical pedagogical approaches and their relevance to the present, and to offer the affordances of black histories, practices, and affinity spaces as sites of healing for black academics and black students. a legacy of stealin’ the meetin’ in her aforementioned oral history, formerly enslaved elizabeth sparks speaks to a practice of black folks who stole away for literacy—seeing this path of education as one congruent with liberation. she tells of a literacy community that problematizes age and gender hierarchies; children taught adults—women taught men and vice versa (federal writers’ project, 1936). in his study of antebellum and post-bellum literacy practices in the south, historian of education herbert gutman tells of many educational spaces that were created by black folks (gutman, 1992). such stories disrupt this over-told narrative of white philanthropists from the good north who traveled to the bad south to enlighten the black slave. in many of these cases, schools were established by black women. between deveaux’s secret school in 1835 savannah, miss l’s secret school in charleston, south carolina, or miss milla grandison’s covert educational space in natchez, mississippi, gutman elucidates a practice of black literacy in repressive circumstances (gutman, 1992, pp. 260– 262). scholar james anderson also highlights this practice of stealin’ the meetin’, paying homage to the oral history of elizabeth sparks and the early sunday schools that gave birth to meeting grounds for black intellectual thought in education (anderson, 1988). while the phrase is associated with the south, the spirit of stealin’ the meetin’ did not solely exist in the south. in her study of literacy communities in the north, black lives in black death 103 journal of effective teaching in higher education, vol. 4, no. 2 scholar elizabeth mchenry (2002) also tells of learning and intellectual communities with an expansive definition of literacy. those of us who work in language arts or any language-heavy discipline are aware of the importance of modes of reading, writing, listening, and speaking—such was the case in these black literacy communities in the north and south. in my research on the black freedom movement, i uphold stealin’ the meetin’ as a heuristic for understanding how black folks have tailored communities of resistance in the face of antiblackness. whereas scholar jarvis givens speaks to this practice of black liberatory educational models in the face of an antiblack existence under the teleology he refers to as fugitive pedagogy (givens, 2021), i maintain a very similar argument with added focus on black women’s voices and a term introduced to us by a black woman (robinson, 2020). black queer scholar ashon crawley refers to another covert black practice of worship known as “stealing away” as a model of “otherwise possibilities” (crawley, 2016). i argue that stealin’ the meetin’ does the same—an otherwise possibility for black literacy, affinity, and liberation where few or none exist. whereas the earlier practice was covert, the subsequent fights for black liberatory education kept that same fervent energy publicly and privately: an education by any means necessary on local black community terms (robinson, 2020). in the reconstruction era, black communities saw an increase of schools; in fact, in the early constitutional meetings, black representatives in southern districts fought for public education for all (du bois, 1998). part of this was meant to protect black students from having to work in the fields (anderson, 1988). in her oral history, a virginia-born formerly enslaved woman by the name of miss mary jane wilson speaks of how she was not among those who were lucky enough to be taught during enslavement. in the reconstruction era, however, she built a schoolhouse that educated nearly 75 students at once—carrying on the legacy of black women who founded schools that emphasized liberatory education for black people by any means necessary (federal writers’ project, 1936). in the periods that followed the first and second world wars, black educational communities thrived despite the dearth of resources. educators refined their craft at normal schools, and the intellectual vigor we saw in black life was cultivated in teacher preparation programs. still, with or without formal teacher training, a number of black artists, thinkers, and activists began their careers as teachers in black educational spaces: anna julia cooper, richard wright, ida b. wells-barnett, w.e.b. du bois, and mary church terrell are among the many. within these classroom spaces, such thinkers were creating otherwise possibilities that often contributed to their intellectual and socioemotional growth in their subsequent years. while much of the philosophy centered on the prevailing concept of “racial uplift,” black thinkers and pedagogues treated education as necessary for black social, political, and economic growth (bay, 2009; grant et al., 2016; higginbotham, 1993; morris, 2015). among those who became educators in post-reconstruction black america was septima clark, the architect of the pedagogical styles that influenced multiple groups of the black freedom movement. clark was born in charleston, south black lives in black death 104 journal of effective teaching in higher education, vol. 4, no. 2 carolina, and she began teaching in 1916 (lake, 2013). for years, she taught black students across ages, focusing on the need to connect education to the immediate political realities of black people. much of the strength of the pedagogical framework involved in the southern christian leadership conference (sclc), student nonviolent coordinating committee (sncc), and eventually the black panther party were centered on this model of a co-constructed learning space that merged student experiences, political immediacy, and multiliteracies (robinson, 2020). while clark’s approach was perfected at highlander, which was owned and directed by white organizer myles horton, the craft and hand-on work were carried out in all-black citizenship schools, freedom schools, and liberation schools. clark and her cousin bernice robinson built on a tradition of loose hierarchies and education by any means necessary (payne et al., 2008; robinson, 2020). between donations for pencils and narratives written on dry-cleaner bags, they saw education as a liberatory pathway—and they carved otherwise possibilities in these black educational spaces (lake, 2013; payne, 2007; robinson, 2020). while much of clark and robinson’s methods are explained through paulo freire’s later work, which has been coined as critical pedagogy or transformative pedagogy (freire, 2000, 2005a, 2005b), they draw on an older tradition of black education. clark and many other black women educators and black educational spaces in the black freedom movement were the foundation of our course. such spaces were often fostered by what we would now deem a black radical feminist ethos of care and community beyond the structures of race, class, and gendered hierarchies. as a black queer man with working-class roots who commits to feminist praxis, i maintain this focus, recognizing and highlighting the importance of the black women figures of my course, affinity spaces, and colleague connections. while i may have been the facilitator of the classroom setting, it was through my engagement with colleagues and students, especially black students, within and beyond our remote classroom that we truly created the spaces for us to think through our existence in a town and institution that were founded in anti-blackness. how was our classroom and our separate affinity space also a practice in otherwise? how was i drawing from black critical education foremothers to further our intellectual, social, and political liberty? the events the weekend for my entire first two months in town, i spent the bulk of my time with my roommate/colleague and another colleague—two black women who share my scholarship interests across the humanities and social sciences. as the pandemic pod, we did everything together: trips to our favorite food places, errands, lounging by the lake, in-town writing retreats, and more. when the very unexpected storm tore through town, we even split a hotel suite. it was no surprise, then, when we learned of williams’ murder, we immediately synchronized our flights. we held this grieving space together, and while i was still empowered by our meeting, i could not help but think of my students’ reactions to the news. black lives in black death 105 journal of effective teaching in higher education, vol. 4, no. 2 shortly after our flight logistics planning, i reached out to the whole class: hello everyone i wanted to update you about the proposal deadline for the final. i am also still processing the violent homicide of michael williams, particularly as a black man living in [this town]. i encourage you to take the necessary time to process this weekend. please see the president’s email for counseling services if needed. also, feel free to reach out to me if you want to talk… the letter was meant to acknowledge what students might be feeling and to locate my own specific fears and frustrations as the only black man in our classroom community. even in the space i had carved out to be our remote academic home, i felt like an outsider in this moment of heightened black fungibility (moten, 2008). early sunday afternoon, we had a bipoc faculty and staff meeting to connect, vent, and share our suggestions for moving forward. it was there that my colleagues shared the most brutal details of their experiences: loud trump-pence celebrations the night of the 2016 election, campus safety officials who wore “blue lives matter” shirts and hats, staff members with signs that read “we back the badge,” and the string of american flags in front of homes—to name a few. these were the perpetual threats to our safety and personhood which were protected by a certain brand of midwestern whiteness. we saw the looming dangers, but the politics of the very anti-black governor and half of the surrounding local town ensured that my neighbors and colleagues could post these messages without social consequence. consequently, these were the coded assaults on our mental health that were compounded by our additional unpaid service, our pedagogical contributions, our writing demands, and our quest for academic security in spaces never intended for us. we listed the immediate needs for faculty and staff to have access to transportation that took us away from town, the need for campus safety that was not tied to blatantly white supremacist organizations, recruitment of black faculty and the inclusion of a black studies program, and a plan for continued meetings and organizing moving forward. we were locating the concrete demands for our administration. later that evening, the president sent an email to the black faculty and staff that named the victim and spoke directly to his identity: dear colleagues, this evening i want to offer some words and actions to express my commitment to you. in this white town, white college, and white state, the thousands of small and large ways that you have experienced prejudice require attention and recognition. for far too long these incidents have been black lives in black death 106 journal of effective teaching in higher education, vol. 4, no. 2 unacknowledged. the death of michael williams must be recognized in its own tragedy and for its greater impact and connection to the historic trauma of the black community. for these reasons, classes tomorrow have been cancelled. i am working to take actions in the immediate future that i hope will bring needed changes to the college through our policies and practices. specifically: • local and regional transportation and formulating a plan for shuttles within grinnell and to des moines and iowa city • working with campus safety and creating a safety plan (e.g. reporting, escorting, resources) that addresses community safety vs. policing • and moving forward with a faculty vote on the spoken use of the n-word in the classroom (executive council is meeting tomorrow to discuss this process). …i care deeply about you. i will work with campus and community partners to act on it. i was at once impressed, encouraged, and frustrated by this email. an initial glance convinced me that she committed to some concrete actions. while this was heartening, especially since two of the items were on our list, i was infuriated by the third bullet. my first faculty meeting was an emergency session on “academic freedom,” called by faculty who felt threatened by the president’s critique of nonblack professors using the “n-word” in class. “academic freedom” became the coded language of whiteness—an excuse for verbal assaults on black life. these professors vehemently clung to their freedom to read the “n-word” aloud in class. it took the violent murder of a local black man to end this curricular violence (jones, 2020) against black students. while conversations of black studies and recruitment were in our meeting notes and in general conversation, the president made no mention of this. committed to the black life on campus, including my students’ experiences and the future of my own potential stay at the college, i replied to the president: dear president, thank you for this letter. as a new black term faculty member, i have cautiously navigated this very white town. on friday, before i was made aware of the violent details of michael williams’ murder, i told my department chair that i am often nervous when i run at my local park. that nervousness is heightened… i appreciate how you have outlined three concrete moves to increase safety inside and outside the classroom. i also hope that the college increases attention to black studies curriculum, recruitment of black students, and hiring of black faculty in the coming years. while i am wary of inviting black black lives in black death 107 journal of effective teaching in higher education, vol. 4, no. 2 people into very white spaces, i am simultaneously aware that there is strength in numbers and value in providing black affirmations in the curriculum. i know this to be true in my own classroom. within the first two class sessions of the semester, one of my students thanked me for being here and added “representation matters; i appreciate you.” our perceived absence in word, deed, and physical proximity incites a real threat to our very existence. students and staff know it and feel it, and we celebrate each other’s presence, feeling a little more alive. every time i encounter another black person in the community or see a new black face in a video call, my breathing eases, and my smile widens; i feel a little bit safer. i knew exactly what my student meant… my letter was an attempt to affirm her commitments and name the things we needed—to speak from what i knew from my experiences with students and my own interactions with the town. naming my experience helped me to see that it is not just representation, but also critical engagement and representation that mattered, and i had to make that known to her. class 1 more difficult than that email exchange was my first synchronous class on tuesday. school had been cancelled on monday, so students were just re-engaging in classes for the first time that week. i did not attend the vigil on campus for fear of hearing white mourning—or worse yet—of laying open my real-time grief for white audiences. on tuesday morning, i prepped for class with the weight of my fears and grief. i wanted to open space for students to process, but i also wanted to honor the direction of the class. we began with a check-in. like i had done two weeks prior, i asked them to examine a dozen or so semi-personified blob cartoon images to locate their feelings and explanations in the chat. when i asked for verbal comments, only two students spoke. after, i read some of the responses aloud and asked if it were okay for us to engage in a breathing exercise. i explained to them what i had explained to the president—the need to consider the black faces in white spaces and my fear as a black man in town. shortly after, i asked the students their next moves, reminding them that it was their class as well. students wanted to move on with the lesson for the day. i still felt the weight of their silence which spoke volumes. when they broke into small groups, i stepped out of the room to vent with my roommate. nearly in tears, i confessed, “hi, friend. that might have been the hardest class i have ever had to teach. can i have a hug?” black lives in black death 108 journal of effective teaching in higher education, vol. 4, no. 2 “of course. i’m sorry, buddy.” her embrace was the balm i needed as i stumbled through the second half of class. students returned to share their midterm ideas in the chat. again, one or two students shared verbally, and the heaviness persisted. once all groups had shared their input, we tried to review the reading in small groups and then in the larger group. as the students broke into groups again, i read the latest news on the case. four suspects had been charged. one news source reported, williams was black. all the people charged are white. but during a tuesday news conference with the iowa-nebraska chapter of the naacp, police said there was no evidence [he] had been murdered because of his race. at least one of the suspects…knew williams, they said (sahouri & smith, 2020). the response was disconcerting. the presumption that knowing someone would preclude racial motivation in the murder says much about the ignorance of the multiple layers of race and how it has operated historically. many black victims of lynching were everyday people in the community—known by the members in the town and the perpetrators of their demise. what would make her believe this was any different? while the naacp regional president’s follow-up statement was a little more fleshed out, it still did not convey the weight of racial injustice: “given that the current climate where racial justice is on the front burner for so many…we understand the fear this kind of incident evokes,” andrews said, adding that it’s understandable the discovery of a black man’s body burning in rural iowa raised red flags. “if there is something there, then we need to identify and address it. if there is not something there, we need to make sure that we have that clarity, as well. any time there is a concern about the possibility of a racist act, that’s where we are” (pratt, 2020). how much higher did these flags need to be raised for the chapter to also strongly consider this a racially motivated case? what was going to be their response for the people who had to carry this fear? what did it mean for them to “be there”? how had we gotten through an entire school year no clarity? when students returned from groups, sensing they were overwhelmed, i cut class short and sent them away with warm wishes. that night i kept replaying the experience over again in my mind. in this online space, we had somehow created community. in the weeks prior, we laughed together, affirmed responses, asked critical questions, and made connections. this week, we mourned together, shared our frustrations, and rested in a collective silence around the threats against black life. black lives in black death 109 journal of effective teaching in higher education, vol. 4, no. 2 in the press conference that took place that afternoon, law enforcement shared a similar sentiment as the naacp and with the same rationale. a special agent reported, “while it would be inappropriate to speak to a motive in this case, i can say without hesitation that no evidence has been found that show the acts against williams were motivated by race” (pratt, 2020). again, “without hesitation” removes centuries’ worth of violence against black lives—ignores the insidious nature of white supremacy and the ways in which it holds us hostage, binds us, kills us, and protects itself from any real accountability. this was anti-black white supremacy on display without systemic consequence. the next day, the nation heard the court’s verdict regarding breonna taylor’s murderers. class 2 i woke up earlier than usual that thursday morning with the dual weight of breonna taylor case and the nearly weeklong grief of michael williams’ murder. as if the difficult tuesday were not enough—the national conversation compounded the threat against black lives, and i had to share in the experience with students. earlier in the week, i presented a broad timeline of the black freedom movement, highlighting the deaths of several key figures. that thursday, i knew enough to return to this timeline to recognize its aptness. my brain was overwhelmed with connections, and to make sense of it all, i wrote. in class, we opened with our usual activities. their quick write asked “what are you going to do to protect and keep your joy this week? after they shared their heartening responses, i took a moment to read aloud my early morning meditations: sadly, death catalyzes people: members of sncc were moved to action after the death and funeral of emmitt till and the swiftly unjust trial of his murderers. yesterday marked the 65th anniversary of the 67-minute deliberation and acquittal of till’s murderers. interesting how yesterday, only one of breonna taylor’s killers was indicted, and the charge was not murder; it was “wanton endangerment.” class d felony: 1 year at the least, 5 at most.” people are understandably, justifiably angry. after the death of malcolm x, more folks began to take up nationalism in their own communities. in rickford we read about the growing organizations’ attachment to his framework. interesting how the watts rebellions/the long, hot summer happen(s) the summer after malcolm x’s assassination. black folks were pushed to the brink. a number of [rebellions] strike up around the country after mlk jr’s. death in 1968. bpp numbers soar after this historical moment. the death of bobby black lives in black death 110 journal of effective teaching in higher education, vol. 4, no. 2 hutton in oakland also catalyzed oakland youth to join the bpp and lead charges in their local high schools and colleges. death is often the precursor to new life—and in many cases, new fights. take a minute to think about the ways you have been stirred by the large scale murders of black life this year. what was your response then? what is your response now? what does new life look like for you? what does the fight look like for you? which lane of resistance do you want to occupy? (young, 2020)… what personal shifts have you seen in your life overall? what organizational or institutional shifts have you seen in your lifetime? when i finished, i could feel the students’ silence again. this time was slightly different, so i sat with it—welcomed it. i told students that they did not have to respond, that these questions were for their personal reflections—ways that they would see themselves and their responses as connected to the matrix of oppression and activism—past, present, and future. immediately after class, i spoke to a black colleague about my experiences and my fears in town. never before had i been comforted and terrified at the exact same moment. as he replayed his experiences, i internally recounted my own. he had been followed by the infamous red truck that i saw on my morning runs; he heard people speed past him and yell obscenities; he had known the dangers of having the audacity to walk in spaces reserved for oversized vehicles—show his black skin in places preserved for whiteness. when i told him about my plans to leave the town indefinitely, he completely agreed with the idea. while the words might not have been spoken, the tone of his voice nearly pleaded with me to leave and never return. professor-student black affinity space i was still concerned about my black students. for nearly a month, they engaged in readings and discussions about the history of black education approaches during the civil rights movement, and they had to experience all of the emotions of the history with seven other non-black students. in the midst of the silence, i paid the most attention to them. i spoke with a trusted friend the next day to hear her insight. she said, “the silence could just be your perceived reaction to how they are feeling. try asking them if they would like a space to process with you; we all know the importance of having affinity.” invigorated by her suggestion, i sent an email to the five black students in class: good evening, i know i asked the class to meet with me for one-on-one conversations this week, and i still want that. i also want to see if you are interested in having a black affinity space this week to talk about whatever you want to talk black lives in black death 111 journal of effective teaching in higher education, vol. 4, no. 2 about. it could be open or specific: check in, vent, ask questions, raise concerns, chill, crack jokes…you choose. i know the importance of black affinity spaces, but i also know you could very well be engaged in spaces outside the class and would prefer not to. whatever the case, i wanted to make the opportunity available. three of the students responded, and we had an affinity group meeting the following wednesday. the first student to respond was also the first student to show up in our online meeting room. her questions and insight in our previous office hours conversations were always welcomed, so i was heartened to see her in this session. eventually another student joined, but the third student was not feeling well. in our time, i checked in with the students about their classes, about the college’s black history, and about life more broadly. i learned so much about their lives and the continual emotional labor they had to perform, and i also learned about the culture of the school—at once publicly committed to social justice and completely disconnected from the social struggles black students endured. at the end of the session, we decided to meet again the following week at the same time. in all, the affinity group met four times in the first quarter and three times in the second quarter. each meeting we discussed a variety of things; we never had a set agenda. all three students were fourth years, so they shared about courses, job/grad school planning, national and global politics, and their involvement in campus leadership. occasionally, they would ask me questions about tenure and the logistics of college/university bureaucracy. i spoke about the awkward routes to tenure, politics of teaching, and general responses to their questions. in our weekly sessions, we were carving out a black space that was uniquely ours— an unofficial remote room for discussion. i was challenged to think about my practice differently, enlightened by their reflections, and encouraged by their contributions to their respective fields. while i opened the opportunity for the affinity group to speak to their needs, i feel like i gained more from our experiences than they did. i was beginning to feel at home. to a degree, we dissolved the hierarchy of professor and student, harkening to the covert communal liberatory literacy practices in elizabeth sparks’ account (federal writers’ project, 1936). finals the energy in class shifted in the positive direction the last two weeks of the course. we finished our readings and had in-depth conversations. on our scheduled finals day, all 13 students presented in-depth projects connected to our course topics and themes. one student created a game, another presented a documentary on the black panther party’s health programs. other student projects included papers, a brochure of racially conscious medical school curriculum suggestions, a paper about the music of the time, and so on. students used their liberty to creatively engage with the history of black liberation and education. these assessments and evaluations were testament to the importance of teaching the black lives in black death 112 journal of effective teaching in higher education, vol. 4, no. 2 concept of stealin’ the meetin’, even as our affinity group was metacognitively engaging in the practice of black placemaking. figure 1 evaluation comments note: this evaluation was completed anonymously by 8 of the 13 students figure 2 evaluation question response summary black lives in black death 113 journal of effective teaching in higher education, vol. 4, no. 2 discussion in a sea of whiteness: black faces in white spaces the immediate culture shock of being part of the 3.5% black population in this town had not subsided before i experienced the murder of michael williams. in the wake of the experience, i could not help but feel hypervisible. on one end, my department chair strategically boosted my public presence for the job market, and on the other, i was the face for the school’s commitment to support black lives. in our black and bipoc faculty and staff meetings, talks with students, student affinity groups, and conversations with colleagues, we were engaged in black placemaking (tichavakunda, 2020). tichavakunda (2020) asserts, “black placemaking can be used to study different groups in different higher education contexts…one might employ this approach to examine how black faculty, graduate students, and student affairs professionals create places for themselves and in conjunction with each other” (p. 23). in this overwhelmingly white campus in an even whiter town, black folks were both invisible in prominent meetings, independent research projects, and other parts of campus life. even in our blackfocused course, less than half of the students were black. this meant that in a course dedicated to black political and intellectual history in education, black students were hyper-aware of their non-black peers and thus more mindful of how their commentary could be received. in town, we were hypervisible: existing only to the extent that we were a duboisian problem (1903) or completely fungible entity (moten, 2008). in other words, we were seen as a threat to the town because of our phenotypic and cultural distance from whiteness, or we were rendered completely invisible (or irrelevant) due to our distance from whiteness. because of this paradox, black students, faculty, and staff had to strategically build remote space for ourselves. sadly, as with most black tragedies, the local murder in a year of public black deaths ushered in a series of platitudes and low-level commitments. opening spaces to grieve together, provide advice, and support each other was important. it was through my experiences with black colleagues and friends that i knew enough to create affinity spaces with my students, which opened the opportunity to hear and be heard by each other. course evaluations the course evaluations provide insight into how the students received the instruction. regardless of the struggles of the moment, we established opportunities to explore the power of the black radical imagination in education through our sessions. by centering black historical practices regardless of the perpetual threat on black life, i helped students to learn. at least two students explicitly noted how the final project they built in relation to the course themes— based completely on their learning goals—was a key contributor to their learning process. students’ responses on evaluations also indicate that the course methods, opportunities to share, and structure were useful for them. black lives in black death 114 journal of effective teaching in higher education, vol. 4, no. 2 student responses were in the “strongly agree” and “moderately agree” category for every question, which indicates that the students believed the course was informative, well-organized, and engaging. in their comments, students noted that some of the readings were dense, but they simultaneously shared how they learned from the course through readings, discussions, small groups, and final projects. three categories garnered 100%: my ability to help them understand subject matter, my course’s ability to help them in remote circumstances, and their willingness to take a course from me in the future. this speaks to an older phrasing from gloria ladson-billings (1995) about the work of culturally relevant pedagogy— that when we create opportunities for students to engage criticality and develop cultural competence, we are just practicing good teaching. recommendations for stealin’ the meetin’ and black placemaking in higher ed what happens when a remote and in-person campus community presents one level of danger, and the neighborhood in which we live poses another? the mechanisms we use to connect remotely have to then be the spaces in which we establish and/or maintain community. the challenge moving forward is to translate black affinity spaces within classes at hwi/pwi institutions to better support black student and faculty social, intellectual, and mental health. our back black channel of affinity within this course provided an outlet to dig more deeply into conversations from the text and our lived experiences. more research should be conducted into the usefulness of student-faculty intra-class affinity spaces for the potential to build sustainability for black faculty, students, and staff. placemaking happens on an institutional level largely through the labor of black students and faculty members (tichavakunda, 2020, p. 22). i suggest institutions take up the financial and personnel commitment to black affinity, so that black students and faculty can carve out this potential separate space—black education spaces (bes)— whether remotely or (as schools reopen) in person (warren & coles, 2020). while our affinity group was the placemaking needed to steal away, our virtual classroom was also the space to find connections to the black radical imagination in education. we discussed the liberatory affordances of black freedom movement organizations as we encountered the real-time threats to black life. education scholars should also create critically engaged spaces of otherwise in readings of black scholarship, small groups, whole class discussions, mini-lectures, and journals. just as one would in the physical classroom, educators should leave space in the virtual room to critique interlocking systems of domination, share frustrations, name connections to home, and celebrate a range of joys. additionally, educators should prioritize socioemotional space, celebration of black people and black traditions, student-driven projects, and metacognition. scholar gholdy muhammad’s historically responsive literacy framework encourages educators to ground pedagogy in identity, intellectual pursuits, criticality, and skills. unaware of the details of this framework at the time of teaching my black freedom movements course, i employed all four—grounding our work in the practice of black intellectual societies and of black enslaved folks who were stealin’ the meetin’ prior to the civil war (muhammad, 2020; robinson, 2020). i encourage educators to black lives in black death 115 journal of effective teaching in higher education, vol. 4, no. 2 veer away from predominately white traditional notions of school and schooling, to imagine what education could be, and to invite students to create projects that allow them to build something new. more research and practice should examine how muhammad’s framework operates in higher education. conclusion: more work through my interactions outside of class, i was able to create and sustain liberatory practices within class. outside of class, black colleagues, students, and i were building otherwise opportunities for black love, self and collective care, and intergenerational dialogue—we were stealin’ the meetin’, creating alternative spaces of dreaming through our black educational community (robinson, 2020). we navigated the pandemic, cultivated speaking spaces, and planned for our own future safety all within the small group. black faculty and staff committed to stay at the college, in the town, even if we had to temporarily leave for our sanity. our connection and commitment was a practice rooted in otherwise possibilities (crawley, 2017, pp. 8–9; robinson, 2020, p. 61) that helped us to sustain our personhood and black intellectual connections under the threat of violence. while our engagements were not illegal and did not operate in direct opposition to the college or community, they did create otherwise opportunities to dream of a school community that was in direct opposition to anti-blackness. in both colleague and student affinity spaces, we encouraged, affirmed, and informed each other in order to maintain our presence in the face of the threat of violence. moreover, we worked to build a black community in a space where communal engagement was stifled by the pandemic. in the classroom, i carried the generative energy from black affinity spaces into praxis. bipoc students and white students learned the power of stealin’ the meetin’ within the classroom, and black students and i engaged in a deep practice of it separate from the class. this critical reflection, then, is an account of that practice, leveraging a precursor to black feminist educational practice. in her famous essay “engaged pedagogy,” cultural critic bell hooks speaks of the importance of vulnerability, healing, criticality, and love in teaching (hooks, 1994). our work was embedded in this practice, and this account is a metacognitive home-building that connects the pedagogical, personal, and political for the future stealin’ the meetin’ in higher education. conflicts of interest notions of conflicts of interest rest upon an objectivity of the research and subject. as a reflexive essay this work necessarily positions the researcher as one of the subjects. mindful of this, the author declares that there is no conflict of interest regarding the publication of this article. black lives in black death 116 journal of effective teaching in higher education, vol. 4, no. 2 references anderson, j. d. (1988). the education of blacks in the south, 1860–1935. university of north carolina press. bay, m. (2009). to tell the truth freely: the life of ida b. wells (1st ed.). hill & wang. crawley, a. t. (2016). blackpentecostal breath: the aesthetics of possibility. fordham university press. du bois, w. e. b. (1998). black reconstruction in america. free press. federal writers’ project. (1936). slave narrative project (works progress administration, vol. 17, virginia, berry-wilson). library of congress. https://www.loc.gov/item/mesn170/ freire, p. (2000). pedagogy of the oppressed (30th anniversary ed.). continuum. freire, p. (2005a). education for critical consciousness. continuum. freire, p. (2005b). teachers as cultural workers: letters to those who dare teach (ed. with commentary by peter mclaren, joe l. kincheloe, and shirley). westview press. givens, j. r. (2021). fugitive pedagogy: carter g. woodson and the art of black teaching. harvard university press. grant, c. a., brown, k. d., & brown, a. l. (2016). black intellectual thought in education: the missing traditions of anna julia cooper, carter g. woodson, and alain leroy locke. routledge. gutman, h. g. (1992). power & culture: essays on the american working class (1st paperback ed.). new press. higginbotham, e. b. (1993). righteous discontent: the women’s movement in the black baptist church, 1880–1920. harvard university press. hooks, bell. (1994). teaching to transgress: education as the practice of freedom. routledge. jones, s. (2020, spring). ending curriculum violence. teaching tolerance, (64). https://www.tolerance.org/magazine/spring-2020/ending-curriculumviolence ladson-billings, g. (1995). but that’s just good teaching! the case for culturally relevant pedagogy. theory into practice, 34(3), 159–165. lake, r. (2013). sailing with septima: a curricular journey through the jim crow south. counterpoints, 412, 59–77. https://www.jstor.org/stable/42981815 black lives in black death 117 journal of effective teaching in higher education, vol. 4, no. 2 mchenry, e. (2002). forgotten readers: recovering the lost history of african american literary societies. duke university press. morris, a. (2015). scholar denied. university of california press. moten, f. (2008). the case of blackness. criticism, 50(2), 177–218. https://doi.org/10.1353/crt.0.0062 muhammad, g. (2020). cultivating genius: an equity framework for culturally and historically responsive literacy. scholastic. payne, c. m. (2007). i’ve got the light of freedom: the organizing tradition and the mississippi freedom struggle. university of california press. payne, c. m., cobb, c. e., & strickland, c. s. (2008). teach freedom: education for liberation in the african-american tradition. teachers college press. pratt, p. (2020, september 25). four grinnell residents face charges in michael williams death. newton daily news. https://www.newtondailynews.com/2020/09/23/four-grinnell-residents-facecharges-in-michael-williams-death/a2c1t26/ rickford, r. j. (2016). we are an african people: independent education, black power, and the radical imagination. oxford university press. robinson, r. (2020). stealin’ the meetin’: black education history & the black panthers’ oakland community school [doctoral dissertation, city university of new york]. cuny academic works. https://academicworks.cuny.edu/gc_etds/3707 sahouri, a. m., & smith, z. o. (2020, september 23). 4 charged in slaying of grinnell man remembered by family as gentle, loving; no racial motive seen. des moines register. https://www.desmoinesregister.com/story/news/crimeand-courts/2020/09/22/michael-williams-death-family-remembers-grinnellman-gentle-giant-burned-body-found-in-jasper-county/5862773002/ tichavakunda, a. a. (2020). studying black student life on campus: toward a theory of black placemaking in higher education. urban education. https://doi.org/10.1177/0042085920971354 warren, c. a., & coles, j. a. (2020). trading spaces: antiblackness and reflections on black education futures. equity & excellence in education, 53(3), 382– 398. https://doi.org/10.1080/10665684.2020.1764882 young, l. [@lindss_tastic]. (2020, may 30). resistance is not a one lane highway. [tweet]. twitter. https://twitter.com/lindss_tastic/status/1266763433560756225?s=20 microsoft word joseph.docx journal of effective teaching in higher education, vol. 2, no. 1 randomize it: fair procedures when constructing multiple-choice test-keys dane christian joseph, george fox university, djoseph@georgefox.edu abstract. multiple-choice testing is a staple within the u.s. higher education system. from classroom assessments to standardized entrance exams such as the gre, gmat, or lsat, test developers utilize a variety of validated and heuristicdriven item-writing guidelines. one such guideline that has been given recent attention is to randomize the position of the correct answer throughout the entire answer key. doing this theoretically limits the number of correct guesses that testtakers can make and thus reduces the amount of construct-irrelevant variance in test score interpretations. this study empirically tested the strategy to randomize the answer-key. specifically, a factorial anova was conducted to examine differences in general biology classroom multiple-choice test scores by the interaction of method for varying the correct answer’s position and student ability. although no statistically significant differences were found, the paper argues that the guideline is nevertheless ethically substantiated. keywords: multiple-choice, guessing, procedural fairness, testing, factorial anova introduction the purpose of this study was to empirically compare three strategies for varying the position of correct answers in multiple-choice test keys and to assess their impact on test scores. students arrive in u.s. higher education institutions already familiar with the multiple-choice (mc) assessment format. mc tests are in many ways more convenient for test-makers than their constructed or written-response counterparts (haladyna, 2004; haladyna & rodriguez, 2013). they can be used in both low and high stakes testing environments, are applicable across a large number of subject matter areas, and are relatively easy to administer, score, and interpret when diligently developed (downing, 2002a; drummond, sheperis, & jones, 2016; chappuis & stiggins, 2017). the decision to administer mc assessments has a mixed history in the educational psychology literature. mingo, chang, and williams (2018) reported that 161 students in an undergraduate educational psychology course least preferred mc but most preferred constructed-response (cr) or essay types out of ten assessmentformat choices. on the other hand, parmenter’s (2009) review of studies with undergraduate business students finds they prefer mc assessments even though the students admitted to being more enthusiastic when given essays. parmenter also suggests that decreasing budgets and increasing class sizes are forcing many professors who teach larger courses to incorporate mc at the expense of cr options; thus, instructors “prefer” mc as a grading-efficient convenience. yet other research suggests that some students might prefer mc assessments as they believe they can both: (1) rely on cognitive strategies such as recall and recognition to assist them in selecting the correct answer for particularly difficult randomize it 81 journal of effective teaching in higher education, vol. 2, no. 1 test content and (2) get more readily available feedback because of the ease of mc scoring (mullet, butler, verdin, von borries, & marsh, 2014). while mc items are by nature easier to score, they are notoriously difficult to design well and all too often contain low-quality items such as items that only target students’ recall of facts and not their ability to understand or apply abstract concepts. on the other hand, high-quality mc items increase the likelihood that test score interpretations will accurately reflect examinees’ domain-specific knowledge of a subject. there are massive consequences attached to test score interpretations (roediger & marsh, 2005). for instance, standardized test performance may be proportional to the funding that schools receive (tindal, 2002). good test performance might open up the possibility of earning academic scholarships to increase accessibility as well as affordability and offset personal costs (cohn, cohn, balch, & bradley jr., 2004). conversely, poor test performance might limit accessibility and affordability options. a good deal of psychometric research has demonstrated several contributing factors to test score variance beyond superficial proxies such as student ability or prior achievement. examples include the quality of the examinees’ nutrition prior to taking a test (figlio & winicki, 2005); school socioeconomic situation, classroom environment and available resources (aikens & barbarin, 2008); and knowledge of test-wise strategies that can be deployed to attain successful guesses (supon, 2004). haladyna and rodriguez (2013) also posit that item quality can influence an item’s power to discriminate between students who truly know the correct answer from those who do not. if an item has low discrimination, students without true knowledge of the item’s answer may be able to exploit some of the item’s features to guess the correct response. thus, if item quality is dependent on item-writing ability, the latter can also impact test score variance. as a result of items being too difficult or too easy, guessing contributes constructirrelevant variance (civ) to the test score. civ is a part of the score that does not represent the examinee’s true knowledge of item(s) but rather something else, such as their ability to successfully infer and guess correct items (downing, 2002b; haladyna & downing, 2004). this lowers test score reliability and hence, interpretations from one stakeholder (person or group) or test-attempt to another (bar-hillel, budescu, & attali, 2005). nolen, haladyna, and haas (1992) warn against test score interpretations that lack validity due to civ. consequences include inaccurate formative or summative feedback to students to guide their learning as well as large-scale costs such as indefensible admissions’ standards. rationale for this study researchers have proposed and defended item-writing guidelines to assist item developers because high quality mc items increase the likelihood that precise and accurate test scores will be the result (haladyna, 2004). although roughly thirty mc item-writing guidelines (haladyna & downing, 1989a, 1989b; haladyna & rodriguez, 2013; haladyna, downing, & rodriguez, 2002) have consistently been touted to increase item quality, the empirical validity of some of these guidelines randomize it 82 journal of effective teaching in higher education, vol. 2, no. 1 remains in question. one guideline lacking empirical attention suggests varying the position of correct answers among options. the rationale for this approach is to limit the number of successful guesses that examinees can make as a result of response set, the systematic or predictable patterns in item lists. three methods that are used to vary the position of correct answers are randomized, arbitrary, and balanced (attali & bar-hillel, 2003). arbitrary methods are not purely random, although they may resemble it. human beings are prone to creating patterns. they tend to systematically but biasedly over-place desired items in middle or edge-averse locations as compared to the ends or edges (ayton & falk, 1995; bar-hillel & attali, 2002; christenfeld, 1995; falk, 1975; rubinstein, tversky, & heller, 1996). examinees can then benefit from this bias within a testing scenario either by purposefully guessing middle/edge-averse options or because they themselves are also biased to choosing such positions. in a balanced key, the correct answer is intentionally placed in each possible option an equal number of times. when keys are balanced, students may use elimination strategies such as the underdog strategy to successfully guess (bar-hillel & attali, 2002). to do this, examinees answer all the questions on the test as best as they can; count the frequency of each position among the answers; select the position with the lowest frequency (the underdog position); eliminate any clear distractors (incorrect answers); and assign the underdog to all as yet unanswered items. the higher a student’s ability, or the more knowledge a student possesses of the test items, the higher the number of correct answers that can be attained before having to guess. thus, the greater the potential for underdog strategy success. bar-hillel and wagenaar (1991) discuss the differences between random and nonrandom processes. true randomization results from a process that uses a purely random device to assign the position of correct answers; examples are die, unbiased coins, or computer programs. each possible outcome has an equal chance of occurring on any given turn. yet, the pattern is unpredictable to the human brain. given that game theory suggests that one can do no better against a random move other than to play randomly; at best, students can split even by using a purely random counter-move (attali & bar-hillel, 2003; rubinstein et al., 1996). because examinees cannot purely randomize and because one can assume they are not allowed to utilize pure randomized devices in assessments, they cannot employ successful guessing strategies for a randomized answer-key. upon examining the face validity of each method, randomization seems to be the most effective method to make the answer-key pattern unpredictable to examinees and hence, the most difficult for them to exploit through successful guesses. although randomized outcomes will balance out in the long run, the pattern on any single trial is completely unpredictable to humans. thus, no successful guessing strategies can be employed on a single-trial randomized answer-key. this was the theoretical basis for the present study. randomize it 83 journal of effective teaching in higher education, vol. 2, no. 1 methodology three research questions were investigated: 1. to what extent does the method of answer-key assignment affect the examinees’ performance on a general biology classroom multiple choice test? 2. to what extent do examinees’ test scores differ across test formats for combined ability groups? 3. to what extent do high-ability examinees’ test scores differ across test formats? because student ability factors into the success of the underdog strategy in balanced keys, a proxy for ability was explored as a second independent factor to interact with the first factor, test format. this student ability was operationalized as the amount of knowledge that students would possess on the subject matter and measured as their cumulative grade-point average (cgpa). design and analysis an experiment was conducted using a between-subjects’ factorial anova design. this design was appropriate to analyze the significance of mean differences on the dependent variable (dv) between the groups or levels of the independent variables (iv) or factors (mertler & vannatta, 2016). the manipulated factor in the study was the method for varying the location of correct answers: randomized, arbitrary, or balanced. each student was assigned only one method condition. a second nonmanipulated iv was the examinee’s proximal ability or knowledge using cgpa. cgpas were collapsed into high, medium, and low non-contiguous groupings. the dv was the total test score. individual items were scored dichotomously as either 0 (incorrect) or 1 (correct); whereas, test score was aggregated on a continuous level. beyond the anova results, item difficulty and discrimination analyses were conducted to assess the quality of item responses along with cronbach alphas used to assess the consistency of item responses in each test format. item difficulty was assessed through the use of item proportions, a classical test theory approach that looks at the average proportion of correct answers over a test domain. item discrimination was computed in the form of point biserial correlation coefficients. sample and participants participants came from a large land-grant research based university located in the pacific northwest region of the united states that had an approximate annual enrollment of just under 20,000 undergraduate students. the majority of participants were freshmen and sophomores. a few juniors and seniors were also enrolled. the sampling frame was a total of 540 students from 15 sections of 36 students in each, and 369 students provided informed consent to have their test scores analyzed in this study. randomize it 84 journal of effective teaching in higher education, vol. 2, no. 1 this n was necessary in order to have adequate power for the analysis of betweengroup differences after a power analysis was conducted using an alpha level set at .05 with desired power recommended at .80 (cohen, 1988). for large effects, total sample size requirements were n = 133 and n = 107 for interaction and main effects, respectively as well as cell sample size requirements of n = 15 and n = 12 for interaction and main effects, respectively. the course professor taught all course sections in combined lectures with teaching assistants supervising each section’s lab meetings. instrumentation and administration the test instrument was a 100-level general biology mc exam. the course instructor developed the items based on his experience in writing and using classroom mc tests as well as his expertise in the subject matter given his phd in biology. the conventional mc test included 50 items and was to be taken in a 60minute testing period per psychometric recommendations of one item per minute (burton, 2006). each item was worth one point for a total possible test score of 50 points. each test item had five options, a through e. the researcher then collected the developed instrument and conducted an informal proofreading and screening of the items for style and format concerns surrounding item clarity, grammar, punctuation, and spelling. following this, each of the three test formats were created. each format had the same exact items in the same exact order. they only differed in where the position of the correct answer was located among the five options. items were included at the end of the test to ascertain examinees’ gender, race/ethnicity, and age, in order to determine the representativeness of the sample to the university population. students also provided their cgpas. microsoft excel was used to develop the randomized and balanced keys. a random number generating function was employed to assign the position of the correct option for each item in the randomized version. an assignment of correct positions was similarly used for the balanced format by specifying that each possible option represent the correct answer exactly 10 times, i.e., 10 correct answers in position a, 10 in position b, etc. it is important to note that the distribution of correct positions did not turn out to be equally represented throughout. this was because of a conflict among other item-writing guidelines that haladyna and downing recommended. specifically, the correct answer-position on one of the items had to be relocated post hoc because the guideline for ordering numerical values was violated. in other words, the possible answers were numbers that needed to be ordered by size across the options. given that the correct answer was the smallest number, it should have been placed in position a to satisfy this item-writing guideline. since this failed to occur (the correct answer was originally placed in position d), the researcher switched the two locations. randomize it 85 journal of effective teaching in higher education, vol. 2, no. 1 the course instructor developed the arbitrary key by assigning the position of the correct answer among the options. the distribution of correct options for each test format is found in table 1 below. table 1: distribution of correct options for each test format test format option randomized arbitrary balanced a .24 .18 .22 b .10 .22 .20 c .32 .18 .20 d .14 .16 .18 e .20 .26 .20 the test was administered on days and times during which the course sections typically met for lectures. each student randomly received one of the three test formats as opposed to randomly varying the formats by whole sections. its rationale was to reduce systematic error and increase the power of the design to detect treatment effects (lipsey, 1990; underdog & sax, 1990). subjects were only informed that unnecessary guessing would tend to lower their total test score. results cronbach alphas were equal to .80, .69, and .72 for the randomized, arbitrary, and balanced test formats, respectively. this indicated fair reliability in the responses within each test format. descriptive data of students’ demographics for race/ethnicity and gender showed that the sample was indeed representative of the university population. data were then screened for missing or erroneous values and to ensure that the anova assumptions would be fulfilled. a low score of 23 and a high score of 42 were observed. some outliers were deleted from the final dataset after examination of box plots. two outliers were deleted from the analysis because they did not fit the distribution of scores as both revealed extremely low raw scores (9 and 14) for two students who self-reported extremely high cgpas. to separate the ability groupings non-contiguously, the researcher used cutoffs that produced three roughly equivalent sample sizes for high-ability (cgpa = 3.7–4.0), medium-ability (3.0–3.3), and low-ability (2.3–2.7). these were arbitrarily chosen to increase cell sample size per ability-grouping while keeping the groups as distinct as possible (see table 2). table 2: cross-tabulation of cell and group sample sizes for student ability x test format test format randomized arbitrary balanced total high 19 18 15 52 student ability medium 18 20 19 57 low 19 17 18 54 total 56 55 52 163 randomize it 86 journal of effective teaching in higher education, vol. 2, no. 1 note that although 369 students consented to participate, the noncontiguous grouping strategy cut the analytical sample by more than half. although this reduced cell and group sample sizes, conditions of statistical power for large effects were still satisfied given the power analysis results in the “sample and participants” section above. descriptive statistics for this analytical group can be found in table 3, along with the average proportions of correct answers for each test form. table 3: cell sample sizes, mean test scores, standard deviations, and mean correct-item-proportions for test formats and ability groups in anova analysis test form n m s p randomized l 19 29.11 6.09 0.58 m 18 31.44 4.23 0.60 h 19 37.74 3.12 0.75 c 56 32.79 5.88 0.62 arbitrary l 17 29.41 4.98 0.56 m 20 31.45 4.50 0.61 h 18 35.83 3.68 0.71 c 55 32.25 5.08 0.61 balanced l 18 28.61 5.77 0.57 m 19 30.79 4.21 0.60 h 15 36.87 4.20 0.73 c 52 31.79 5.80 0.61 note: l = low ability, m = medium ability, h = high ability, c = combined ability examination of group scores by histograms revealed normality. levene’s test of equality of variances found no statistically significant differences, indicating homogeneity of variances across groups, f(8, 154) = 1.42, p = .19. neither interaction nor main effect results from table 4 below revealed statistically significant differences in test scores by the ability groups. the interaction of test format by ability on test score was not statistically significant, f(2, 154) = .32, p = .72, partial eta squared < .01. the main effect of test format on test score was also not statistically significant, f(4, 154) = .35, p = .84, partial eta squared < .01. although not a research question of interest, the main effect of ability level was statistically significant as expected, f(2, 154) = 39.38, p < .01, partial eta squared = .33. randomize it 87 journal of effective teaching in higher education, vol. 2, no. 1 table 4: two-way anova summary table of interaction and main effects source df sum of squares mean squares f-ratio f-prob effect size between treatments 8 1750.79 218.84 ability level 2 1685.02 842.51 39.38 < .01 .33 test format 2 13.71 6.85 .32 .72 .00 ability level x test format 4 30.22 7.55 .35 .84 .00 within treatments 154 3294.65 21.39 total 163 174979.00 discussion for the purposes of the following discussion, it is assumed that the vast majority of the examinees would have attempted at least some guesses. based on the theoretical rationale, it was expected that the combined ability randomized group mean would be lower than either of the combined ability arbitrary or combined ability balanced group means. this expectation was not supported by this study’s findings. guessing test-takers are only successful in the randomized format in the following conditions: 1) they have relatively high ability to minimize the number of guesses needed and be able to use a randomized guessing device on the few items left over; 2) they are able to use a successful guessing strategy such as edgeaversion to better effect than students would on the arbitrary format, assuming the randomized key resembled an arbitrary key; 3) they are able to use a successful guessing strategy such as the underdog than students would in a balanced format, assuming the randomized key resembled a balanced key. use of randomized devices was strictly prohibited from the test; therefore, option 1 above is unlikely and unfeasible. results from table 1 do not confirm that the randomized key was edge-averse; options a and e appear more times (0.24 and 0.20, respectively) than their neighboring options b and d (0.10 and 0.14, respectively). middle-bias seemed to be present since option c appeared more than any other option (0.32). however, inferences from the distribution of this study’s randomized key can only be made about this particular randomization outcome. in other words, another randomization trial would likely have produced a different distribution. furthermore, students were not told which method of distributing correct answers pertained to their individual tests. even more vexing is the fact that the arbitrary key produced by the instructor was atypical and revealed no significant edgeaversion. an edge-averse strategy and option 2 above was therefore also unlikely and unfeasible given this reasoning. table 1 also shows an unbalanced randomized key. although randomized keys are expected to balance out in the long run, they are not expected to be uniformly distributed on the vast majority of single trials (attali & bar-hillel, 2003). to do so randomize it 88 journal of effective teaching in higher education, vol. 2, no. 1 would require a large number of items for each test version. it follows that singletrial randomized keys might be equated more often than not with some form of a non-balanced key. the results of this study also showed that combined ability group mean test scores from the non-perfectly balanced keys (i.e., the arbitrary and randomized formats) did not significantly differ from the almost perfectly balanced key format. therefore, option 3 above is unlikely and unfeasible. several items revealed severe problems based on the item difficulty and item discrimination results. three items had almost uniformly distributed proportions (for difficulty) across options a-e, revealing possible examinee confusion over which options were plausible or implausible distractors. several items also had poor pointbiserial correlation coefficients. this means, for example, that examinees across different ability levels would answer the item correct roughly the same number of times. hence, such an item is described as non-discriminating. for those unfamiliar with psychometric theory, discriminating items are both necessary and desirable from a test interpretation perspective. this revealed poor item design as one would expect that on average higher ability examinees would answer an item correctly more so than lower ability examinees. most noteworthy was the fact that even the high-ability students scored on average 0.75, 0.71, and 0.73 of the items correctly for the randomized, arbitrary, and balanced formats, respectively. this equates to the ‘best achieving’ students attaining c-grades across each format. thus, it is plausible that the test might just have been too difficult for everyone, regardless of ability level. some of the design choices in this study were also suboptimal in retrospect. one such example was the decision to operationalize ability through an achievement proxy as imperfect as cgpa. first-year students in their spring semesters would have only had one semester’s worth of courses, typically introductory or survey courses, compared with secondor third-year students who had several semesters’ worth of courses of supposedly increasing cognitive demand. some of the analytical choices were also questionable. while classical test theory is certainly rigorous and widely used in its own right, item-response theory is the current dominant and more accurate approach to examining item difficulty and discrimination (markus & borsboom, 2013; price, 2017). finally, beyond the answer-key varying strategy, no other key patterns were examined. yet, more recent research has identified important issues in how patterns such as sequences—whether long runs or palindromes—can trick or confuse examinees (lee, 2018). tests should never confuse the examinee as this can also create civ. implications and recommendations this section describes some pertinent implications and recommendations for classroom practitioners by discussing practical ways that ethical principles, such as equity, impact item development and guessing. kane (2013) offered possible reasons for compromised validity due to civ, including the value frameworks from which stakeholders operate. for example, increased accountability from randomize it 89 journal of effective teaching in higher education, vol. 2, no. 1 administrative sources might propel test-makers to create lower difficulty (easier) items in order to inflate test scores; conversely, test-takers might adopt cheating or other less desired behavioral strategies to attain higher scores. thus, values are a key component to item development, testing, and assessment culture. as such, the ethics of item-development procedures must be addressed if tests are to serve honorably the purposes for which they are designed. based on this study’s results it might seem that more empirical attention is warranted to support the decision to randomize classroom mc tests, and that until more validity evidence is provided, test-makers may as well arbitrarily place the position of correct answers in mc items without much concern for test-takers to benefit from successful guessing. but this is far from the truth. if the item difficulty and item discrimination results have indeed revealed anything substantial, it is that questions of fairness are not solely about the distribution of outcomes but also the procedures used to generate them. this means that item quality is just as important as test score distribution. hence, item-writer training is vital to ensuring that the process of item (stem and options) creation is fair. in that sense, the process can attend to equity issues such as student-access to test-wise strategy training. because test-wise strategy training can result in higher scores (markus & borsboom, 2013; supon, 2004), some students with the resources to access such training can gain differential score advantages that have nothing to do with subject-matter knowledge of the test items but result from their strategic guessing skills. while it would be easy to then recommend that all teachers prepare their students with test-wise strategy training, those with the economic resources to get more advanced or individualized preparation will still differentially benefit over students without such economic footing. even this approach might be forgetting that the point of testing and assessment is to provide both the instructor and student with performance feedback to direct future learning and instruction endeavors. as such, perhaps it would be better—as one reviewer recommended—to incorporate low-stakes mc assessments throughout instruction as a retrieval practice exercise. this increases compatibility across what is learned and what is assessed and helps to level the playing field for students. for a balanced test with a reasonable number of items, the guessing strategy’s success still requires a good deal of knowledge of the other items to minimize the required number of guesses. but for an arbitrary test, no such assumption is necessary. and those who know to “guess c” (or at least stay away from the edges) would have an advantage over the odds of guessing correctly. in some practically significant ways then, the empirical results of this study were always going to be a moot point. instead, what is important is that test-makers and item-writers follow theoretically sound guidelines when producing testing artifacts. when these are empirically supported, all the better. yet in special circumstances where certain item-writing guidelines are not empirically supported, such as that observed in this study, a reasonable approach randomize it 90 journal of effective teaching in higher education, vol. 2, no. 1 to test development should be adopted. fair procedures such as randomizing the answer key ensures that students with limited social and economic capital, or diminished test-wise strategy training, are on equal footing with others before the test administration. leveling the playing field with sound item-writing guidelines before test administration is the only way to equitably minimize civ and also ensure that student ability and knowledge makes the only difference on the outcome. randomization is the best and fairest method in this case. note: i wish to thank and acknowledge the editors and reviewers for their time and invaluable feedback to improving the presentation of the study and its potential for impact. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references aikens, n. l., & barbarin, o. (2008). socioeconomic differences in reading trajectories: the contribution of family, neighborhood, and school contexts. journal of educational psychology, 100, 235–251. http://dx.doi.org/10.1037/0022-0663.100.2.235 attali, y., & bar-hillel, m. (2003). guess where: the position of correct answers in multiple-choice test items as a psychometric variable. journal of educational measurement, 40(2), 109–128. ayton, p., & falk, r. (1995). subjective randomness in hide-and-seek games. paper presented at the 15th bi-annual conference on subjective probability, utility, and decision making, jerusalem, israel. bar-hillel, m., & attali, y. (2002). seek whence: answer sequences and their consequences in key-balanced multiple-choice tests. the american statistician, 56(4), 299–303. bar-hillel, m., budescu, d., & attali, y. (2005). scoring and keying multiple-choice tests: a case study in irrationality. mind & society, 4(1), 3–12. bar-hillel, m., & wagenaar, w.a (1991). the perception of randomness. advances in applied mathematics, 12(4), 428–454. burton, r.f. (2006). sampling knowledge and understanding: how long should a test be? assessment & evaluation in higher education, 31(5), 569–582. chappuis, j., & stiggins, r.j. (2017). an introduction to student-involved assessment for learning (7th ed.). hoboken, nj: pearson publications. christenfeld, n. (1995). choices from identical options. psychological science, 6, 50–55. cohen, j. (1988). statistical power analysis for the behavioral sciences (2nd ed.). new york, ny: routledge publishing. downing, s.m. (2002a). construct-irrelevant variance and flawed test questions: do multiplechoice item-writing principles make any difference. academic medicine, 77 (10), s103–s104. randomize it 91 journal of effective teaching in higher education, vol. 2, no. 1 downing, s.m. (2002b). threats to the validity of locally developed multiple-choice tests in medical education: construct-irrelevant variance and construct underrepresentation. advances in health sciences education, 7, 235–241. drummond, r.j., sheperis, c.j., & jones, k.d. (2016). assessment procedures for counselor and helping professionals (8th ed.). hoboken, nj: pearson publications. falk, r. (1975). the perception of randomness. unpublished doctoral dissertation (in hebrew, with english abstract), hebrew university, jerusalem, israel. figlio, d.n. & winicki, j. (2005). food for thought: the effects of school accountability plans on school nutrition. journal of public economics, 89(2), 381–394. haladyna, t.m. (2004). developing and validating multiple-choice test items (3rd ed.). mahwah, nj: lawrence earlbaum associates. haladyna, t.m., & downing, s.m. (1989a). a taxonomy of multiple-choice itemwriting rules. applied measurement in education, 1, 37–50. haladyna, t.m., & downing, s.m. (1989b). the validity of a taxonomy of multiplechoice item-writing rules. applied measurement in education, 1, 51–78. haladyna, t.m., & downing, s.m. (2004). construct-irrelevant variance in highstakes testing. educational measurement: issues and practice, 23 (1), 17– 27. haladyna, t.m., & rodriguez, m.c. (2013). developing and validating test items. new york, ny: routledge. haladyna, t.m., downing, s.m., & rodriguez, m.c. (2002). a review of multiplechoice item-writing guidelines for classroom assessment. applied measurement in education, 15(3), 309–334. kane, m. (2013). validating the interpretations and uses of test scores. journal of educational measurement, 50(1), 1–73. lee, c.j. (2018). the test-taker’s fallacy: how students guess answers on multiplechoice tests. journal of behavioral decision making, 1–12. doi:org/10.1002/bdm.2101. lipsey, m.w. (1990). design sensitivity: statistical power for experimental research. newbury park, ca: sage publications. parmenter, d. (2009). essay versus multiple-choice: student preferences and the underlying rationale with implications for test construction. academy of educational leadership journal, 13(2), 57–71. markus, k., & borsboom, d. (2013). frontiers of test validity theory: measurement, causation and meaning (multivariate applications book series). new york, n.y.: routledge publishing. mertler, c.a., & vannatta, r.v. (2016). advanced and multivariate statistical methods: practical application and interpretation. new york, ny: routledge publishing. mingo, m., chang, a., & williams, h. (2018). undergraduate students’ preferences for constructed versus multiple-choice assessment of learning. innovative higher education, 43(2), 143–152. mullet, h.g., butler, a.c., verdin, b. von borries, r. & marsh, e.j. (2014). delaying feedback promotes transfer of knowledge despite student preferences to receive feedback immediately. journal of applied research in memory and cognition, 3(3), 222–229. randomize it 92 journal of effective teaching in higher education, vol. 2, no. 1 nolen, s.b., haladyna, t.m., & haas, n.s. (1992). uses and abuses of achievement test scores. educational measurement: issues and practices, 11, 9–15. price, l. (2017). psychometric methods: theory into practice (methodology in the social sciences). new york, ny: the guilford press. roediger, h.l., & marsh, e.j. (2005). the positive and negative consequences of multiple-choice testing. journal of experimental psychology: learning, memory, and cognition, 31 (5), 1155–1159. rubinstein, a., tversky, a., & heller, d. (1996). naïve strategies in competitive games. in w. albers, w. guth, p. hammerstein, b. moldovanu, & e. van damme (eds.), understanding strategic interaction. new york, ny: springerverlag. supon, v. (2004). implementing strategies to assist test-anxious students. journal of instructional psychology, 31(4), 292–296. tindal, g. (2002). large-scale assessments for all students: issues and options. in g. tindal & t.m. haladyna (eds.), large-scale assessment for all students: validity, technical adequacy, and implementation (pp. 1–22). mahwah, nj: lawrence earlbaum associates. trevisan, m.s., & sax, g. (1990). reliability and validity of multiple-choice examinations as a function of the number of options per item and student ability. paper presented at the annual meeting of the american educational research association (74th boston, ma. april 16-20). microsoft word 204-article text-852-1-15-20210406.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.204 outside the box: promoting creative problem-solving from the classroom to the boardroom christine miller, tennessee tech university laura cruz, pennsylvania state university jacob kelley, auburn university, jkk0019@auburn.edu abstract. can creativity be taught? multiple sources attest that the business community values creativity in potential new hires, but a signature pedagogy of teaching for creativity in business classes has not yet emerged. to contribute to a body of evidence-based practice, this study assessed the impact of several in-class activities that were deployed among undergraduate business students to see if these enhanced their creative problem-solving abilities, as assessed by preand post-intervention measures. the results were moderately encouraging and suggest domain-specific teaching and learning strategies. further, the results offer encouragement to all instructors, irrespective of any prior experience with creativity-enhancing efforts. keywords: creativity; creative problem-solving; creative thinking; business management how many uses can you think of for a stapler? according to journalist william poundstone (2003), such questions have become increasingly commonplace for organizations that place high value on creativity. further, he suggests that successful applicants are capable of producing upwards of 300 or more responses. this number may sound daunting, but it reflects a broader trend toward recognizing creativity as a desired and desirable quality to have in the contemporary marketplace (epstein et al., 2013; mareque et al., 2019). in the 2016 national association of colleges and employers (nace) employer survey (n = 201), for example, 23.6% of respondents indicated that creativity was a desired trait for college graduates, ranking just below strategic thinking and just above tactfulness. for those future students wishing to gain this competitive edge, it would seem that they would need to seek out ways to become more creative. review of literature beyond exploring the limits of a stapler, though, it can be challenging to determine what exactly are the desired skills or attributes associated with creativity. a central debate in creativity studies is the extent to which it can be considered a personal attribute and therefore, by extension, a fixed trait (feist, 1998; mumford, 2003; mumford et al., 2012), or whether it is an ability that can be fostered or developed. because of the prevalence of the former view, creativity (in and of itself) had been largely absent from classroom practice outside of the creative arts (fasko, 2001). recent research across a number of disciplines has revealed, however, that creative thinking is a skill that can be developed and strengthened over time (fekula, 2011; promoting creative problem-solving 77 journal of effective teaching in higher education, vol. 4, no. 1 perry & karpova, 2017). in other words, while creativity may be something you either have or you do not, evidence suggests that creative thinking can be taught. additionally, there seems to be a growing consensus that it should be taught. sir ken robinson, perhaps one of the most influential public figures in education, frequently calls the modern educational system to task for not only failing to foster creativity but also for being complicit in its active suppression (resnick & robinson, 2017. sir ken robinson is not alone. social science researchers, such as richard florida (2014), have identified creative thinking as a highly desired trait in the current and future labor force, a value that seems to widely shared by practitioners. a recent analysis of millions of online job postings identified creativity as a critical “human skill” (markow et al., 2018; markow et al., 2019). a 2019 linkedin study identified creativity as the top trait desired by companies who post positions on its site (petrone, 2019). the importance of creativity begs the question of where and how creative thinking should be integrated into the college curriculum (edwards et al., 2006). the terminological switch from creativity to creative thinking may facilitate a constructive approach to creativity, but it does not free us from our definitional challenges, making it unclear just exactly what creative thinking looks like. early theorists emphasized divergent or lateral thinking as indicative of creative abilities; while more recent scholarship has emphasized creative thinking as a multi-faceted process that works both in contrast to and in tandem with critical thinking (runco & akar, 2012; sawyer, 2014; sternberg, 2006). design thinking, a moniker which has received a great of public attention lately, is a prominent variation (matthews & wrigley, 2017). much of the emphasis on the pedagogy of creative thinking has focused on divergent/lateral thinking, or fostering the ability of participants (whether college students or corporate employees) to brainstorm (also known as ideate) and generate as many new ideas as possible. the ability to generate new ideas in a systematic fashion has found resonance in a number of disciplinary fields, including computer programming, creative writing, graphic design, and marketing (mccorkle et al., 2007; mcintyre et al., 2003). in marketing, especially, creativity is often contrasted with innovation, which differs largely in its application to business contexts and its emphasis on the development of new products or services that have market impact. leaders in organizational development seek to find solutions for sustaining innovation across the culture of a business, particularly to combat the marked and persistent tendency for firms and their individual employees to become less flexible, risk-oriented, and, yes, creative over time. as a result, creative thinking is now being taught in courses on or related to entrepreneurship, including non-disciplinary or transdisciplinary courses like first-year seminars (ghafar, 2020; mcmullan & kenworthy, 2015; solomon et al., 2008), as innovation tends to resonate with contemporary business models of higher education. although the practice is growing, comparatively few studies have been conducted specifically on teaching for creativity in higher education (sternburg, 2015). the literature tends to be dominated by discussions of measurement (for which promoting creative problem-solving 78 journal of effective teaching in higher education, vol. 4, no. 1 consensus is closer but has yet to be reached) rather than pedagogy. other than student self-report instruments, most robust studies have continued to use two of the long-standing, transdisciplinary measures of creative thinking, the adult version of the torrance test of creative thinking (ttct) (almeida et al., 2008; rababah, 2018), which uses drawing exercises, or variations of the alternative uses tests, which focus on divergent thinking (george & wiley, 2019; kwon et al., 2017; nix et al., 2014) for preand post-studies of various teaching-for-creativity strategies. more recent tests integrate both convergent and divergent creative thinking skills, including the evaluation of concept maps, but the approaches are otherwise conceptually similar (perry & karpova, 2017; snyder et al., 2019; urban, 2005). to date, a pattern has emerged in which creativity or creative thinking are taught in pockets of higher education, but little attention has been paid to how creative thinking might be integrated into disciplinary-specific approaches to teaching and learning. the practical reasons for this are highly varied (craft, 2005), ranging from perhaps unjustified associations with children’s play (so not for adult college students or serious disciplines) (paek & sumners, 2019) to more significant challenges of classroom assessment. one frequently evoked explanation is that instructors frequently assume that they must be both creative thinkers and experts on creativity in order to bring these ideas into the classroom. to address these assumptions, jeffery and craft (2004) draw distinctions between teaching creatively, teaching creativity, and teaching for creativity. this latter is intended to focus on strategies to facilitate creative thinking, especially when embedded in a disciplinary context. as a field, business management is well suited as a candidate for the integration of teaching for creative thinking (sunley et al., 2019). not only is the field related to both entrepreneurship and marketing, two areas where strategies for new product development have received a great of attention, but the curriculum of business management emphasizes problem-solving. recent shifts in the theoretical foundations of decision sciences away from rational-choice models (nutt, 1984) has left conceptual space for the creative problem-solving to rise to the fore (ford & gioia, 2000; marques, 2019). and, in turn, creative decision-making has been linked by researchers to the development of broader organizational cultures that support and sustain innovation (amabile et al., 1996; kwon et al., 2017; obholzer & miller, 2018; williams, 2001). for these reasons, there has been an increasing emphasis on teaching creativity in business management classes, including a 2010 report from the association to advance collegiate schools of business (aas&b), a major accrediting body, highlighting the gaps and calling for reforms. that said, much of the pedagogical research literature to date has focused on case studies or practice reports, indicating that innovative practices may be going on, but there is a clear need for more systematic research (driver, 2001; kerr & lloyd, 2008; schlee & harich, 2014; wongpinunwatana, 2019; wynder, 2004). the present study seeks to contribute to the development of a body of evidence-based practice in teaching for creativity within the context of colleges of businesses. promoting creative problem-solving 79 journal of effective teaching in higher education, vol. 4, no. 1 overall, the need for creativity in managerial decision-making has become increasingly evident, both in practice and in the research literature (basadur et al., 2014; bilton, 2007; earl & potts, 2016; ejimabo, 2015; helfat & martin, 2015; proctor, 2014). it has been under-emphasized, however, in the pedagogical literature (schmidt-wilke, 2011) and exploratory research has demonstrated that business students are lagging behind other disciplines in their exposure to thinking “outside the box” (mcintyre et al., 2003; supiano, 2020; wang et al., 2010). this study seeks to address that gap by measuring the efficacy of a pilot program focused on teaching for creativity in the business management classroom. methodology the context tennessee tech university is a stem-focused doctoral institution (high research, or r2) located in a micropolitan area within the southeastern united states. the total student body is approximately 10,000 students, inclusive of both undergraduate and graduate levels. the pilot program itself was facilitated in the college of business, an academic hallmark of tennessee tech university. over the span of two semesters, facilitators visited undergraduate business management courses, at both the introductory and advanced levels, to provide targeted interventions designed to promote creative problem-solving. during the first semester, the pilot program was facilitated with one section of an upper division course on decisionmaking for managers. during the second semester, the pilot program was facilitated with two sections of an introduction to business management course required for all business majors and minors. the article refers to these as intervention groups 1, 2, and 3, respectively. the hypothesis the null hypothesis (h0) of the study is that creative problem-solving cannot be taught to students in higher education. the alternate hypothesis (h1) is that, using targeted interventions, creative problem-solving can be taught to students in higher education (see table 1). in particular, it can be taught to undergraduate business majors. critical t values were determined for use in the hypotheses. table 1 null hypotheses and alternate hypotheses based on intervention group intervention group 1 intervention group 2 intervention group 3 h0 : t < 1.78 , p > 0.05 h0 : t < 1.70 , p > 0.05 h0 : t < 1.71 , p > 0.05 h1 : t > 1.78 , p < 0.05 h1 : t > 1.70 , p < 0.05 h1 : t > 1.71 , p < 0.05 promoting creative problem-solving 80 journal of effective teaching in higher education, vol. 4, no. 1 the interventions prior to facilitating the pilot program in the selected business management courses, the research team received approval from the institutional review board (irb) for tennessee tech university. the pilot program began by obtaining informed consent from the students, who were asked to complete a pre-test and a post-test as bookends for a set of targeted interventions to take place during regular class time. only those students enrolled and present in the class were able to participate. after informed consent was obtained, a non-instructor administered the pre-tests. those who choose not to participate remained in the classroom during the short duration of the assessment administration, but their presence was indistinguishable from those who had completed the test. soon thereafter, the interventions were facilitated with the participants. the research team intentionally selected these particular interventions due to their acceptability within teaching and learning contexts for business programs (gundry et al., 2014; oluwade & oluwade, 2015). it should be noted, however, that each of these approaches have been the subject of criticism as facilitators of creative thinking. despite these known shortcomings, the researchers deemed the benefits as outweighing potential drawbacks. further, the researchers chose to provide multiple interventions (as listed below) rather than rely on multiple iterations of a single approach, both to enhance engagement and mitigate the limitations of any single approach. example 1: design thinking the first example intervention focused on design thinking, an approach centered around the human perspective; design thinking seeks to mesh together what people need, what is possible with technology, and what is required for business success (brown, 2009). the design thinking process consists of five phases: (1) empathize, (2) define, (3) ideate, (4) prototype, and (5) test. for the intervention, participants were tasked with designing a campaign to attract more majors to the college. example 2: scamper the second example intervention focused on scamper. developed by robert eberle (1971), scamper is an acronym for a creative problem-solving strategy that supports atypical solutions to problems as well as generating new ideas for products or concepts. scamper involves seven potential methods for innovation: (1) substitute, (2) combine, (3) add, (4) modify, (5) put to another use, (6) eliminate, (7) rearrange. for the intervention, participants were tasked with running through scamper using a toilet paper roll. example 3: six thinking hats the third example intervention focused on six thinking hats. this is an exercise that asks people to view a problem or decision from different perspectives than promoting creative problem-solving 81 journal of effective teaching in higher education, vol. 4, no. 1 their usual disposition (kaya, 2013; vernon & hocking, 2014). six thinking hats assigns a different perspective based on color: (1) white for facts, (2) red for emotion, (3) yellow for benefit, (4) green for ideas, (5) blue for planning, and (6) black for judgement. for the intervention, the participants were tasked with deploying six thinking hats to address a business case related to hiring practices. the measurement the participants in the pilot program completed an assessment called guilford’s alternate uses (guilford et al., 1978; guilford, 1967) in a pre-test/post-test design. originally distributed as unusual uses (wilson et al., 1954), guilford’s revamped version asks the participants to list up to six uses for a common object beyond the given use. examples include a key (used to open a lock), a watch (used for telling time), and a chair (used for sitting). guilford’s alternate uses comes with three components: form b, form c, and scoring key. both form b and form c are divided into two parts with three objects each. for the pilot program, form b was used as the pre-test and form c was used as the post-test. participants were given two minutes to complete each part, which means a total of four minutes for the pre-test and four minutes for the post-test. findings upon completion of each assessment, two scorers not affiliated with the courses independently assessed the responses for each participant using the scoring key. the total acceptable responses from the scorers were inputted into excel in order to compute the mean for each participant, which were then used to compute the descriptive statistics for each intervention group (see table 2). table 2 descriptive statistics for intervention groups based on alternate uses scores pre-test post-test intervention group n �̅� sd n �̅� sd 1 15 14.33 8.65 13 18.85 8.63 2 37 9.28 4.20 32 11.94 4.09 3 28 10.21 4.63 27 13.70 4.92 after the descriptive statistics for all three intervention groups were computed, it was then time to determine if there were significant differences between the pretest and the post-test for each intervention group. in order to test each hypothesis, the research team decided to deploy a one-sample t test for each intervention group using spss. while a paired t test might typically be used for hypothesis testing in a pre-test/post-test design, the scores could not be matched in the case of the pilot program. as such, it was necessary to use one sample t tests for promoting creative problem-solving 82 journal of effective teaching in higher education, vol. 4, no. 1 hypothesis testing (york, 2017). the research team used the previously computed means (�̅�) of the pre-test scores as the threshold score for the comparison of the post-test scores for each intervention group (see table 3). table 3 means, one sample t tests, and effect sizes for all three intervention groups intervention group pre-test �̅� post-test �̅� difference t p g 1 14.33 18.85 4.52 1.78 0.05 0.52 2 9.28 11.94 2.66 1.70 0.04 0.64 3 10.21 13.70 3.49 1.71 0.04 0.73 *p < 0.05 the post-test mean for intervention group 1 is 18.85, which was compared to the pre-test mean of 14.33 for intervention group 1 with a one-sample t test. the results indicate support for the null hypothesis (t < 1.78 , p > 0.05). the effect size (g) is 0.52, which is considered a medium effect size. the post-test mean for intervention group 2 is 11.94, which was compared to the pre-test mean of 9.28 for intervention group 2 with a one-sample t test. the results indicate that the null hypothesis (t < 1.70 , p > 0.05) should be rejected in support of the alternate hypothesis (t > 1.70 , p < 0.05). the effect size (g) is 0.64, which is considered a medium effect size. the post-test mean for intervention group 3 is 13.70, which was compared to the pre-test mean of 10.21 for intervention group 3 with a onesample t test. the results indicate that the null hypothesis (t < 1.71 , p > 0.05) should be rejected in support of the alternate hypothesis (t > 1.71 , p < 0.05). the effect size (g) is 0.73, which is considered a medium effect size. for each of the effect sizes, the research team elected to utilize hedges’ g to account for differentials in the sample sizes (n) of the pre-tests and the post-tests (hedges & olkin, 1985). following this initial analysis, the research team decided to conduct an additional stage of analysis using a more focused sample from the original to gain further insight into creative problem-solving. it consisted of four variables as categories of creative problem-solving: (1) fluency, (2) originality, (3) flexibility, and (4) elaboration (guilford, 1967). fluency is the sum of responses for each item. originality is a comparison between the responses given by participants in the sample in which responses that only 5% gave receives 1 point and that only 1% gave receives 2 points. flexibility is the different categories across the responses for each item. elaboration is the degree of detail provided for each use on an item. the means, p values, and effect sizes were calculated for each variable to determine if there were significant differences between the pre-test (�̅� = 36) and the post-test (n = 40) (see table 4). promoting creative problem-solving 83 journal of effective teaching in higher education, vol. 4, no. 1 table 4 means, significance, and effect sizes for each variable in focused sample variable pre-test �̅� sd post-test �̅� sd p g fluency 9.89 4.32 12.54 4.89 0.007 0.81 originality 1.61 1.79 1.72 1.86 0.49 0.06 flexibility 8.58 3.24 10.05 3.75 0.03 0.4 elaboration 2.31 2.54 2.64 1.69 0.24 0.15 *p < 0.05 as with the initial analysis, the research team deployed a one-sample t test for each of the variables to determine if there were significant differences. the findings indicate that the variables fluency (p = 0.007) and flexibility (p = 0.03) significantly increased from the pre-test to the post-test. the effect size (g) for fluency is 0.81, which is considered a large effect size. the effect size (g) for flexibility is 0.4, which is considered a small effect size. these results suggest that students were not only able to significantly increase their ability to identify more uses for the common objects indicated in the test, but also to generate more categories of usage. the most common categories of usage across all objects presented included decoration, weapon, game/entertainment, and measurement. the findings indicate that the variables originality (p = 0.49) and elaboration (p = 0.24) did not significantly increase from the pre-test to the post-test. the effect sizes for originality and elaboration are nearly non-existent. these results suggest that students did not come up with a significantly higher number of unique use cases for the objects in the second test, nor did they provide more descriptors of those uses. the research team used hedges’ g to determine effect sizes due to differentials in the sample sizes (n) of the pre-tests and the post-tests (hedges & olkin, 1985). summary of findings both analyses demonstrate that creative problem-solving can be fostered in the classroom through targeted interventions. there were significant differences between the pre-test and the post-test for two of the three intervention groups in the first analysis, which speaks to the variation among the students for creative problem-solving. there were significant differences between the pre-test and the post-test for two of the four variables (fluency and flexibility) in the second analysis, which highlights that some aspects of creative problem-solving were better promoted through the targeted interventions. nevertheless, it is clear that creative problem-solving can be taught in the business classroom. discussion these findings are the result of a pilot study conducted at a single university and should be treated as suggestive rather than definitive or representative. that being promoting creative problem-solving 84 journal of effective teaching in higher education, vol. 4, no. 1 said, they do suggest that creative thinking may take on distinctive characteristics in the context of business management. creativity researchers have long noted tensions inherent in the desire to embed creativity research in domain or fieldspecific contexts (e.g., business management) versus the desire to study creative thinking as a broad, transdisciplinary lens. until recently, the latter has predominated, but the field has shifted towards a more developmental understanding of creative problem-solving, moving through ideation and toward implementation, a process which often necessitates more domain-specific expertise (amabile, 2013; an & runco, 2016; montag-smit & maertz, 2017). in other words, business managers may need to be able to generate new solutions, but those solutions also need to be actionable within the context of a given firm or industry; and both components are part of creative thinking as a process (peterson et al., 2013). the constraint of implementation may explain why the students in our study tended to raise their ability to generate new categories (flexibility) rather than fully new ideas (originality). in defining creative thinking for institutional assessment, the american association of colleges & universities (aac&u) included categories for taking risks and innovative thinking, which relate to originality; they also provide categories for solving problems and embracing contradictions, which more closely relate to flexibility (mcconnell et al., 2019; rodriguez & fekula, 2019). solving problems refers to the implementation phase of an idea. embracing contradictions is an inherently integrative task in which multiple stakeholders and perspectives are taken into account when selecting the solution, a skill that is highly valued in contemporary business decision-makers and a critical component of the corporate social responsibility (csr) model (harjoto & laksmana, 2018; rezaee, 2016). although csr has become far more mainstream in recent years, multiple studies suggest that not all business students are sold on its relationship to business. this perception may be a lingering sentiment from previous decades when business and the environment were more frequently at odds with one another. nor should it be assumed that business students are sold on the value of creativity. at each iteration, there were students who challenged the creativity exercises done in class, and, in one memorable case, a student walked into class, saw the materials laid out, and promptly walked out, muttering that he had better things to do with his time. the more effective guest facilitators did not presume that students understood the value of creative thinking. rather, they made the case for why creativity matters in the modern workplace. further, they embedded the creativity exercises into real-world case studies of business decisions. the design thinking exercise, for example, had students work as design firms hired by the college to promote enrollment and asked them to present their findings to a college administrator. the interventions faced additional metacognitive resistance. at each iteration, for example, multiple students indicated that they were simply not creative, possibly reflecting evidence of a persistent fixed mindset. this experience affirms recent insights from creativity research that posited the existence of a creativity mindset or a mental framework that enhances the process at multiple stages (hargrove & promoting creative problem-solving 85 journal of effective teaching in higher education, vol. 4, no. 1 nietfeld, 2015; tierney & farmer, 2002). the more successful facilitators navigated this resistance in two ways. first, by sharing research indicating that creativity can be learned; second, via the application of scaffolded exercises focused on building both capacity and confidence. with multiple interventions, the students did appear to become more accustomed to creative thinking over time, but the instructor and researchers did note the onset of diminishing returns. the findings indicate more robust results with the courses that integrated fewer, but more targeted, interventions. while we did not measure the impact of specific teaching strategies, our findings prompt us to join others who are calling for the development of evidence-based, discipline-specific pedagogies (murdock, 2003; weick, 2003). those pedagogies need to be approachable for an individual faculty member. there are corporate programs, for example, that provide intensive training in creative problem-solving (puccio et al., 2006; scott et al., 2004). but these require expert trainers as well as shared context to implement. in this study, the instructor lacked confidence in her own ability to facilitate creative thinking, so she chose instead to rely on the expertise of a series of experts who were both willing and available to work with her students. this necessitated considerable time devoted to scheduling and other logistical considerations, but it also brought in multiple perspectives. as her exposure increased, the instructor noted that her confidence has risen and she has been able to integrate more small-scale creative interventions on her own. this affirms scholarship noting that teaching creatively and teaching for creativity often go hand in hand (jeffrey & craft, 2010). our experience suggests that the successful implementation of shared pedagogies for creative thinking will likely need to be accompanied by appropriate faculty development programs as well. this question of appropriate pedagogical strategies further begs the question of how we are able to assess the degree to which these are effective. the assessment of creativity has leaped from a relatively moribund field of study to one of considerable attention—two of its major assessment tools, the torrance test and the guilford alternative uses test (used for this study), are both over fifty years old. this is especially true in the context of higher education, in which leaders are calling for holistic ways to measure creative problem solving for both enrollment and graduation purposes, necessitating considerations of scale, but also authenticity, which precludes self-reported measures, such as the national survey on student engagement (nsse). at the same time, disciplinary societies are exploring the development of instruments and/or rubrics that take into account the nature of creativity that takes places within their domain contexts. exploratory research on the use of case studies as assessments for colleges of business appears promising, but our study serves as a cautionary tale for those who would seek to generalize not just the nature of creative thinking, but also its desired outcomes. while such assessment instruments do not yet exist, the challenge of their creation has generated productive conversations about the complex and often open-ended nature of the creative thinking process and how it interacts with the social, economic, disciplinary, psychological, spatial, and technological layers of the learning experiences that take place in the modern college classroom. perhaps it could be said that we need to continue to think outside the box about teaching for promoting creative problem-solving 86 journal of effective teaching in higher education, vol. 4, no. 1 creativity, both within and across disciplines, and determine how we can do so in a way that is inclusive, effective, and empowering for both the students and the instructor. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. acknowledgments research is a collaborative effort. the authors express appreciation to the editorial team and the peer reviewers for the work put forth to expand knowledge. much gratitude is also given to the participants of the study. references amabile, t. m. (2013). componential theory of creativity. in e. h. kessler (ed.), encyclopedia of management theory (pp. 134–139). sage. amabile, t. m., conti, r., coon, h., lazenby, j., & herron, m. (1996). assessing the work environment for creativity. academy of management journal, 39, 1154– 1184. https://doi.org/10.5465/256995 almeida l. s., prieto prieto, l., ferrando, m., oliveira, e., & ferrándiz, c. (2008). torrance test of creative thinking: the question of its construct validity. thinking skills and creativity, 3, 53–58. https://doi.org/10.1016/j.tsc.2008.03.003 an, d. & runco, m. a. (2016). general and domain-specific contributions to creative ideation and creative performance. europe’s journal of psychology, 12(4), 523–532. https://doi.org/10.5964/ejop.v12i4.1132 aoun, j. e. (2017). robot-proof: higher education in the age of artificial intelligence. mit press. association to advance collegiate schools of business (aacsb). (2010). business schools on an innovation mission. https://www.aacsb.edu/~/media/aacsb/publications/researchreports/business-schools-on-an-innovation-mission.ashx basadur, m., gelade, g., & basadur, t. (2014). creative problem-solving process styles, cognitive work demands, and organizational adaptability. the journal of applied behavioral science, 50(1), 80–115. https://doi.org/10.1177/0021886313508433 promoting creative problem-solving 87 journal of effective teaching in higher education, vol. 4, no. 1 bilton, c. (2007). management and creativity: from creative industries to creative management. blackwell. brown, t. (2009) change by design: how design thinking transforms organizations and inspires innovation. harper collins. craft, a. (2005). creativity in schools: tensions and dilemmas. routledge. driver, m. (2001). fostering creativity in business education: developing creative classroom environments to provide students with critical workplace competencies. journal of education for business, 77(1), 28–33. https://doi.org/10.1080/08832320109599667 earl, p. e., & potts, j. (2016). the management of creative vision and the economics of creative cycles. managerial and decision economics, 37(7), 474–484. https://doi.org/10.1002/mde.2733 eberle, r. (1971). scamper: games for imagination development. d.o.k. publishers. edwards, m., mcgoldrick, c., oliver, m. (2006). creativity and curricula in higher education: academics’ perspectives. in jackson, n., oliver, m., shaw, m., wisdom, j. (eds.), developing creativity in higher education: an imaginative curriculum (pp. 59–73). routledge. ejimabo, n. o. (2015). the influence of decision making in organizational leadership and management activities. journal of entrepreneurship and organization management, 4(2), 1–13. doi:10.4172/2169-026x.1000138 epstein, r., kaminaka, k., phan, v., & uda, r. (2013). how is creativity best managed? some empirical and theoretical guidelines. creativity and innovation management, 22(4), 359–374. https://doi.org/10.1111/caim.12042 fasko, d. (2001). education and creativity. creativity research journal, 13(3– 4), 317–327. https://doi.org/10.1207/s15326934crj1334_09 feist, g. j. (1998). a meta-analysis of personality in scientific and artistic creativity. personality and social psychology review, 2, 290–309. https://doi.org/10.1207/s15327957pspr0204_5 fekula, m. j. (2011). managerial creativity, critical thinking, and emotional intelligence: convergence in course design. business education innovation journal, 3(2), 92–101. http://www.beijournal.com/images/v3n2_text.pdf florida, r. (2014). the rise of the creative class––revisited, revised and expanded. basic books. promoting creative problem-solving 88 journal of effective teaching in higher education, vol. 4, no. 1 ford, c. m., & gioia, d. a. (2000). factors influencing creativity in the domain of managerial decision making. journal of management, 26, 705–732. https://doi.org/10.1177/014920630002600406 george, t., & wiley, j. (2019). fixation, flexibility, and forgetting during alternate uses tasks. psychology of aesthetics, creativity, and the arts, 13(3), 305– 313. https://doi.org/10.1037/aca0000173 ghafar, a. (2020). convergence between 21st century skills and entrepreneurship education in higher education institutes. international journal of higher education, 9(1), 218–229. https://doi.org/10.5430/ijhe.v9n1p218 guilford, j. p. (1967). creativity: yesterday, today, and tomorrow. journal of creative behavior, 1, 3–14. https://doi.org/10.1002/j.21626057.1967.tb00002.x guilford, j. p., christensen, p. r., merrifield, p. r., & wilson, r. c. (1978). alternate uses: manual of instructions and interpretations. sheridan psychological services. gundry, l. k., ofstein, l. f., & kickul, j. r. (2014). seeing around corners: how creativity skills in entrepreneurship education influence innovation in business. the international journal of management education, 12(3), 529– 538. https://doi.org/10.1016/j.ijme.2014.03.002 hargrove, r. a., & nietfeld, j. l. (2015). the impact of metacognitive instruction on creative problem solving. the journal of experimental education, 83(3), 291– 318. https://doi.org/10.1080/00220973.2013.876604 harjoto, m., & laksmana, i. (2018). the impact of corporate social responsibility on risk taking and firm value. journal of business ethics, 151(2), 353–373. https://doi.org/10.1007/s10551-016-3202-y hedges l. v., & olkin, i. (1985). statistical methods for meta-analysis. academic press. helfat, c., & martin, j. (2015). dynamic managerial capabilities: a perspective on the relationship between managers, creativity and innovation. in c. e. shalley, m. a. hitt, & j. zhou (eds.), the oxford handbook of creativity, innovation and entrepreneurship (pp. 421–433). oxford university press. jeffrey, b., & craft, a. (2010). teaching creatively and teaching for creativity: distinctions and relationships. educational studies, 30(1), 77–87. https://doi.org/10.1080/0305569032000159750 kaya, m. f. (2013). the effect of six thinking hats on student success in teaching subjects related to sustainable development in geography classes. educational sciences: theory and practice, 13(2), 1134–1139. promoting creative problem-solving 89 journal of effective teaching in higher education, vol. 4, no. 1 http://oldsite.estp.com.tr/pdf/en/08c6cee249cc2a3ea9e17cd16b6c84bayaen g.pdf kerr, ch., & lloyd, c. (2008). pedagogical learnings for management education: developing creativity and innovation. journal of management and organization, 14(5), 486–503. https://doi.org/10.5172/jmo.837.14.5.486 kwon, j., bromback, l., & kudrowitz, b. (2017). divergent thinking ability + interest = creative ideas: exploring the relationships between cognitive creativity assessments and product design idea generation. proceedings of the asme 2017 international design engineering technical conferences and computers and information in engineering conference. volume 7: 29th international conference on design theory and methodology. cleveland, ohio, usa. august 6–9, 2017. v007t06a002. asme. https://doi.org/10.1115/detc2017-67261 kwon, m., de prada, e., & pino-juste, m. (2019). creativity among business and tourism management university students: determining sociodemographic factors. creativity studies, 12(2), 258–279. https://doi.org/10.3846/cs.2019.6925 markow, w., hughes, d., & bundy, a. (2018). the new foundational skills of the digital economy: developing the professionals of the future. business-higher education forum. https://www.burning-glass.com/wpcontent/uploads/new_foundational_skills.pdf markow, w., hughes, d., & bundy, a. (2019). future skills, future cities: new foundational skills in smart cities. business-higher education forum. https://files.eric.ed.gov/fulltext/ed596449.pdf macdonald, a., clarke, a., & huang, l. (2019). multi-stakeholder partnerships for sustainability: designing decision-making processes for partnership capacity. journal of business ethics, 160(2), 409–426. https://doi.org/10.1007/s10551-018-3885-3 marques, j. (2019). creativity and morality in business education: toward a transdisciplinary approach. the international journal of management education, 17(1), 15–25. https://doi.org/10.1016/j.ijme.2018.11.001 matthews, j., & wrigley, c. (2017). design and design thinking in business and management higher education. journal of learning design, 10(1), 41–54. https://files.eric.ed.gov/fulltext/ej1127723.a mcconnell, k. d., horan, e. m., zimmerman, b., & rhodes, t. l. (2019). we have a rubric for that: the value approach to assessment. association of american colleges and universities. https://www.aacu.org/publicationsresearch/publications/we-have-rubric-value-approach-assessment promoting creative problem-solving 90 journal of effective teaching in higher education, vol. 4, no. 1 mccorkle, d. e., payan, j. m., reardon, j., & kling, n. d. (2007). perceptions and reality: creativity in the marketing classroom. journal of marketing education, 29(3), 254–261. https://doi.org/10.1177/0273475307306892 mcintyre, f. s., hite, r. e., & rickard, m. k. (2003). individual characteristics and creativity in the marketing classroom: exploratory insights. journal of marketing education, 25(2), 143–149. https://doi.org/10.1177/0273475303254014 mcmullan, w. e., & kenworthy, t. p. (2015). creativity and entrepreneurial performance: a general scientific theory. springer. montag-smit, t., & maertz, c. p. (2017). searching outside the box in creative problem solving: the role of creative thinking skills and domain knowledge. journal of business research, 81, 1–10. https://doi.org/10.1016/j.jbusres.2017.07.021 mumford, m. d. (2003). where have we been, where are we going? taking stock in creativity research. creativity research journal, 15(2–3), 107–120. https://doi.org/10.1080/10400419.2003.9651403 mumford, m. d., medeiros, k. e., & partlow, p. j. (2012). creative thinking: processes, strategies and knowledge. journal of creative behavior, 46, 30– 47. https://doi.org/10.1002/jocb.003 murdock, m. c. (2003). the effects of teaching programmes intended to stimulate creativity: a disciplinary view. scandinavian journal of educational research, 47(3), 339–357. https://doi.org/10.1080/00313830308597 national association of colleges and employers. (2016). job outlook: the attributes employers want to see on new college graduates’ resumes. http://www.naceweb.org/career-development/trends-and-predictions/joboutlook-2016-attributes-employers-want-to-see-on-new-college-graduatesresumes/ nix, a., arlitt, r., immel, s., lemke, m., & stone, r. (2008). investigating divergent thinking in creativity exercises through alternative uses tests. proceedings of the asme 2014 international design engineering technical conferences and computers and information in engineering conference. https://doi.org/10.1115/detc2014-35335 nutt, p. c. (1984). types of organizational decision processes. administrative science quarterly, 29, 414–550. https://doi.org/10.2307/2393033 obholzer, a., & miller, s. (2018). leadership, followership, and facilitating the creative workplace. in c. huffington, w. halton, d. armstrong, & j. pooley (eds.), working below the surface (pp. 33–48). routledge. promoting creative problem-solving 91 journal of effective teaching in higher education, vol. 4, no. 1 oluwade, d. o., & oluwade, b. (2015). an assessment of the role of creativity in business studies: its implication to business management. journal of emerging trends in economics and management systems, 6(1), 68–76. https://hdl.handle.net/10520/ejc168388 paek, s. h., & sumners, s. e. (2019). the indirect effect of teachers’ creative mindsets on teaching creativity. the journal of creative behavior, 53(3), 298–311. https://doi.org/10.1002/jocb.180 perry, a., & karpova, e. (2017). efficacy of teaching creative thinking skills: a comparison of multiple creativity assessments. thinking skills and creativity, 24, 118–126. https://doi.org/10.1016/j.tsc.2017.02.017 peterson, d. r., barrett, j. d., hester, k. s., robledo, i. c., hougen, d. f., day, e. a., & mumford, m. d. (2013). teaching people to manage constraints: effects on creative problem-solving. creativity research journal, 25(3), 335–347. https://doi.org/10.1080/10400419.2013.813809 petrone, p. (2019, january 1). why creativity is the most important skill in the world. linkedin learning blog. https://learning.linkedin.com/blog/topskills/why-creativity-is-the-most-important-skill-in-the-world. poundstone, w. (2003). how would you move mount fuji?: microsoft's cult of the puzzle––how the world's smartest companies select the most creative thinkers. little, brown and company. proctor, t. (2014). creative problem solving for managers: developing skills for decision making and innovation. routledge. puccio, g. j., firestien, r. l., coyle, c., & masucci, c. (2006). a review of the effectiveness of cps training: a focus on workplace issues. creativity and innovation management, 15(1), 19–33. https://doi.org/10.1111/j.14678691.2006.00366.x rababah, l. (2018). an adapted version of torrance test of creative thinking (ttct) in efl/esl writing: a rubric scoring and a review of studies. international journal of english and education, 7(2), 128–136. http://www.ijee.org/yahoo_site_admin/assets/docs/9luqman.9883442.pdf resnick, m., & robinson, k. (2017). lifelong kindergarten: cultivating creativity through projects, passion, peers, and play. mit press. rezaee, z. (2016). business sustainability research: a theoretical and integrated perspective. journal of accounting literature, 36, 48–64. https://doi.org/10.1016/j.acclit.2016.05.003 rodriguez, l. c., & fekula, m. j. (2019). the convergence of aac&u value rubrics, assessment, and experiential learning. business education innovation promoting creative problem-solving 92 journal of effective teaching in higher education, vol. 4, no. 1 journal, 11(1), 208–213. http://www.beijournal.com/images/v11n1_draft_.pdf runco, m. a., & acar, s. (2012). divergent thinking as an indicator of creative potential. creativity research journal, 24(1), 66–75. https://doi.org/10.1080/10400419.2012.652929 sawyer, k. (2012). explaining creativity: the science of human innovation. oxford university press. schlee, r. p., & harich, k. r. (2014). teaching creativity to business students: how well are we doing? journal of education for business, 89(3), 133–141. https://doi.org/10.1080/08832323.2013.781987 scott, g., leritz, l. e., & mumford, m. d. (2004). the effectiveness of creativity training: a quantitative review. creativity research journal, 16, 361–388. https://doi.org/10.1080/10400410409534549 schmidt-wilk, j. (2011). fostering management students’ creativity. journal of management education, 35(6), 775–778. https://doi.org/10.1177/1052562911427126 snyder, h. t., hammond, j. a., grohman, m. g., & katz-buonincontro, j. (2019). creativity measurement in undergraduate students from 1984–2013: a systematic review. psychology of aesthetics, creativity, and the arts, 13(2), 133–143. https://psycnet.apa.org/record/2019-20312-002 solomon, g., hamidi, d. y., wennberg, k., & berglund, h. (2008). creativity in entrepreneurship education. journal of small business and enterprise development, 15(2), 304–320. https://doi.org/10.1108/14626000810871691 sternberg, r. j. (2006) the nature of creativity. creativity research journal, 18(1), 87–98. http://dx.doi.org/10.1207/s15326934crj1801_10 sternberg, r. j. (2015). teaching for creativity: the sounds of silence. psychology of aesthetics, creativity, and the arts, 9(2), 115–117. https://psycnet.apa.org/record/2015-14747-001 sunley, r., harding, l., & jones, j. (2019). realising creativity in management education: putting student energy into action. the international journal of management education, 17(2), 172–181. https://doi.org/10.1016/j.ijme.2019.02.007 supiano, b. (2020). the creativity challenge. chronicle of higher education report. https://store.chronicle.com/products/the-creativity-challenge promoting creative problem-solving 93 journal of effective teaching in higher education, vol. 4, no. 1 tierney, p., & farmer, s. m. (2002). creative self-efficacy: its potential antecedents and relationship to creative performance. academy of management journal, 45(6), 1137–1148. https://doi.org/10.2307/3069429s urban, k. k. (2005). assessing creativity: the test for creative thinking-drawing production (tct-dp). international education journal, 6(2), 272–280. https://files.eric.ed.gov/fulltext/ej854980.pdf vernon, d., & hocking, i. (2014). thinking hats and good men: structured techniques in a problem construction task. thinking skills and creativity, 14, 41–46. https://doi.org/10.1016/j.tsc.2014.07.001 wang, s., zhang, x., & martocchio, j. (2010). thinking outside of the box when the box is missing: role ambiguity and its linkage to creativity. creativity research journal, 23(3), 211–221. https://doi.org/10.1080/10400419.2011.595661 weick, c. (2003). out of context: using metaphor to encourage creative thinking in strategic management courses. journal of management education, 27(3), https://doi.org/10.1177/1052562903027003004 williams, s. (2001). increasing employees’ creativity by training their managers. industrial and commercial training, 33(2), 63–68. https://doi.org/10.1108/00197850110385642 wongpinunwatana, n. (2019). enhancing students' creative thinking skills through the process of teaching business research. journal of management policy and practice, 20(2), 135–142. https://doi.org/10.33423/jmpp.v20i2.2100 wynder, m. (2004). facilitating creativity in management accounting: a computerized business simulation. accounting education, 13(2), 231–250. https://doi.org/10.1080/09639280410001676639 york, r. o. (2017). statistics for human service evaluation. sage. microsoft word howtonmcgrewliustaplesray_final1.docx journal of effective teaching in higher education, vol. 3, no. 1 the influence of syllabus tone on student engagement in a wellness course amy howton (ahowton@kennesaw.edu), mandy mcgrew, liyuan liu, lauren l. staples, and herman e. ray kennesaw state university abstract. this study examines the impact of changing the tone of a traditional syllabus to make it more learner-centered. the researchers examined a required wellness course that serves over 8,000 students per year at a large, four-year public university in the southern united states. concerns about student progression through programs and graduation rates had prompted a number of faculty to look into ways to increase these numbers. the investigators changed the tone of the syllabus to determine if a more learner-centered document would increase the chances of students interacting with the syllabus and, in turn, increase the success rates in the course. the results indicated that few students spent sufficient time reading the syllabus to take in all the information regardless of the tone used within the document, so the tone, by itself, did not impact student success. keywords: syllabus, tone, learner-centered, online course, student success syllabi are frequently provided to students in courses, and many institutions of higher education require that a syllabus be provided to students and then kept on file in the department offering the course. syllabi are used to impart information to students, including expectations, schedules, due dates, grading policies, and more. sometimes, they are also used to invite students to learn. however, there is no current consensus about the best way to create syllabi, and there is little research on the topic aside from a handful of studies undertaken in a laboratory environment. these have suggested that a formal, traditional syllabus is less effective at engaging students than is a learner-centered syllabus that creates a welcoming invitation to learning in the course. this study examines the impact in the classroom of changing the tone of a traditional syllabus to make it more learner-centered, attempting to convey to the students an invitation to learn through engagement with the course materials. we also consider the limitations of a learning-centered syllabus if students do not carefully read the document. literature review as millennial and younger students are matriculating through higher education, the shift to thinking of college from a consumer standpoint has grown immensely. millennial students are more likely to view higher education as a venue for gaining credentials in order to find jobs (grunert o’brien et al., 2008). if crafted and written intentionally, the syllabus can actually help students become more intentional learners, shifting them away from this idea and opening their minds to the joys and benefits of learning for the sake of learning (grunert o’brien et al., 2008). syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 2 as stated by grunert o’brien, millis, and cohen (2008): “the syllabus becomes an invitation to share responsibility for successful learning” (p.22). helping students feel that they have control over their learning begins with a syllabus that sends a message to students and establishes a “point of connection” between them and their professor (p. 25). concerns about student retention, progression and graduation rates are driving new policies at higher education institutions nationwide (cook & pullaro, 2010). various aspects of the teaching process are being examined and redesigned to address these concerns. however, much of this research is being conducted in the laboratory setting which does not necessarily translate to success in the classroom. the role of sotl (the scholarship of teaching and learning) is to examine how the theories and methodologies developed in the lab environment play out in the classroom setting (kern, 2015; miller-young & yeo, 2015). the limited research conducted on syllabus language argues for using inclusive, positive language to better engage students with the course through the document (canada, 2013; harnish & bridges, 2011; lund dean & fornaciari, 2014; palmer et al., 2016). however, we found that there is a dearth of research concluding whether or not this theory is applicable within the functioning classroom. in one of the few studies on syllabus effectiveness, palmer, wheeler, and aneece (2006) suggest that the syllabus’s “primary function should be as a learning tool, one that is carefully crafted through a systematic course design process” (p. 37). palmer’s team argue that a syllabus that includes detailed information on the course, including detailed course description, explanation of assignments and reasoning behind them, a detailed course calendar, and written in a positive and motivating tone has the potential to enhance student learning and engagement in the course (palmer et al., 2016). palmer’s team created two separate syllabi for the same course, one traditional and focused on content and the other focused on student learning. they randomly gave a group of sample students one of the two documents to read and then asked them to complete a survey asking them questions about the syllabus. their results showed that the students who received the learning-focused syllabus had a more positive image of the course, they judged the document to be more in depth (but also difficult to navigate), and they thought the syllabus was more relevant to them personally (palmer et al., 2016). these results are compelling, but this research only interrogated students who were made to read the syllabus as part of the study and fails to shed light on the impact these syllabi have on students when implemented in a course. the wellness syllabus study attempts to determine whether or not the tone of the syllabus affects student performance when applied in the classroom setting. according to grunert o’brien et al. (2008), the syllabus should accomplish the following goals: · convey to your students what matters to you about learning · set a tone for learning and how to learn that students will accept · send a message about what students can expect from you and the campus community to support their learning during the term. syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 3 a search of the limited literature indicated that the syllabus tone could influence students to engage more with the document and even with the course itself. for the purpose of this study, we applied the learner-centered syllabus framework developed by palmer, bach, and streifer (2014). this perspective reflects the belief that a syllabus is not a contract, but an invitation to learn, share the course plan, and establish a relationship between the students, faculty member, and course content. “a well-crafted syllabus can be the beginning of a promise fulfilled and part of the difference between just another course and one that changes lives” (canada, 2013, p. 37). as students today are entering our courses asking themselves: “why do i have to take this class?” and “what’s in it for me?” it is important for instructors to answer these questions clearly and directly in their syllabus (canada, 2013). a syllabus serves many purposes, but these can be grouped into three primary categories: motivation, structure, and evidence (littlefield, 1999 as cited in ludwig et al., 2011). the syllabus should motivate students to expel effort to excel in the course. it should provide a structure for the course so both student and instructor understand the goals of the course and the way students will go about achieving these goals. finally, the syllabus provides evidence to the students of what the instructor expects from them, as well as providing evidence to the institution of what the course contains (ludwig et al., 2011). canada (2013) encourages instructors to write their syllabi in a way that fulfills the needs of modern students. this includes using simple and clear language that is understandable to a student who is new to the course material and may not be familiar with disciplinary terms yet. he suggests writing in a friendly tone to enhance students’ positive impressions of their instructor while encouraging them that they can be successful in the course (canada, 2013). lund dean and fornaciari (2014) argue that instructors are prone to neglect considering how their syllabus signals and communicates expectations and assumptions to students. they advocate for instructors to use “more authentically inclusive language and policy construction” in their syllabi in an effort to signal mutual respect to their students (p. 725). philosophically, these authors see learning as a partnership between teacher and student, in contrast to the traditional hierarchy of professor and students. developing a positive rapport between professor and students is critical to increasing student performance (richmond et al., 2016). according to harnish and bridges (2011), the tone of a syllabus has a significant influence over how students view the course and the instructor. their foundational study shows that when a syllabus is written in a friendly tone the instructor was rated as more approachable and the course was presumed to be less difficult (harnish & bridges, 2011). their research, however, like that of palmer, wheeler & aneece (2016), was completed in a controlled environment in which students were given either a friendly or unfriendly syllabus (covering the exact same information and only changed for tone), told to read the syllabus, and then answer questions about the professor and the course. though the results of this study encourage a more positive tone be used when writing syllabi, the research falls short in that the syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 4 students were made to read the syllabus in a controlled, experimental environment and were given a reward for doing so; the conditions within a classroom or course do not offer the same amount of control. the importance of maintaining a positive tone when communicating with students is evident in dickinson’s (2017) study on the impact of tone in email communication. in her online course, dickinson found that being more open about herself personally and adding small “personal touches” to emails resulted in a 17% increase in students’ success in the course (dickinson, 2017). though dickinson’s study is not focused on the syllabus, it does validate the importance of building personal connections with students, particularly in online courses, through the use of positive tone. putting forth a supportive and accessible tone in one’s syllabus enhances student performance in the course (tokatli & kesli, 2009). while some recommend throwing out the syllabus entirely (singham, 2007), it is evident that many instructors carefully consider the structure and content of their syllabi and feel it is useful to students (thompson, 2007). further, though thompson claims teachers who include welcoming messages, frame their syllabi in positive language, promote their passion for the course material, and use inclusive language in their syllabi make all learners feel more comfortable in the class, there are no interventional studies to support these assertions, only laboratory-based studies. what is unclear is whether or not students are engaging with the syllabus at various points throughout the course—or at all. as canada (2013) states, “even the best syllabus is worthless to the student who never reads it” (p. 41). additionally, there is not a universally-accepted alternative to providing students with the information they need to succeed in the course. thompson (2007) affirms that instructors cannot presume that their students read the syllabus carefully or refer back to it throughout the term. therefore, many faculty members present the most important information to their students in class, anticipating that otherwise the students will never interact with the syllabus at all. “capturing students’ attention during the presentation is critical because teachers worry students do not read the syllabus,” (thompson, 2007, p. 68). in face-to-face classes, instructors can use technology and presentation skills to emphasize the most important aspects of the syllabus (thompson, 2007), but online class platforms do not provide the same opportunities to focus attention on specific information contained within the syllabus. in addition, even classroom instructors rarely return to the syllabus multiple times, so it is left to the document itself to entice the students to engage with it throughout the course. the syllabus study in this study, conducted at a large, four-year public university in the southern united states, we examined a required wellness course that serves over 8,000 students per year, foundations for healthy living, to determine the impact of redesigning the syllabus to make it more learner-centered through the use of more inclusive language. concerns about student progression through programs, dwfi rates, and graduation rates prompted a number of university faculty across syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 5 disciplines to look into ways to improve these numbers. one problem identified was students not submitting work accurately or on time, resulting in lower grades. this was especially noted in the online sections, where students are not forced by an attendance policy to be physically present in the classroom where they can be reminded of assignments due and deadlines imposed, but must instead be selfdirected in their acquisition of information and learning and submission of assignments. because the schedule was included in the syllabus, it became apparent that students were not utilizing this resource effectively. the research team a collaborative investigation team was formed between four entities to study whether or not a change in the syllabus tone would influence student outcomes: a faculty member from the department delivering the course, the educational specialist for part-time faculty support at the center for excellence in teaching and learning (cetl), two graduate students and their advisor from the department of statistics and analytical sciences, and the analytics director from the publisher that delivers the course e–textbook and course materials. description of the course the foundations for healthy living course is taught in three formats: face-to-face, fully online, and a hybrid of the two. however, as a result of a paperless initiative on campus, all students utilize an e-text, receive course documents (such as the syllabus and assignments) online, submit their work online, and receive feedback and grades online. content and course objectives are the same for all formats. the online platform is developed and supported by the e-text publisher through the canvas learning management system. students are required to view certain documents, including the syllabus and a course orientation, in order to proceed to the course materials. a traditional, generic syllabus has long been used in all sections because a large number of part-time faculty members teach the course, with individual faculty members adding their own contact information and the specific schedule for assignment due dates within the set dates for each module. this meant it was easy to adjust the syllabus used by any given course section. study design the learner-centered syllabus designed for this study guided students through the course, beginning with simple steps and working toward more complex tasks. bandura’s (1997) social cognitive theory was applied to encourage the students to increase their own self-efficacy through the mastery experience provided by the course design. this scaffolding of learning allowed the students to succeed in lowstakes assignments early in the semester, building their confidence in their own abilities as they completed segments of a comprehensive project that served to illustrate the culmination of their learning in the course. using the syllabus to model metacognitive strategies not only develops students’ self-efficacy in this course, but syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 6 they may apply these newly developed skills to subsequent learning experiences (bandura, 1997; palmer et al., 2016). students are responsible for their own learning, but the instructor can support this learning through careful construction of the course syllabus. research questions and hypotheses the problem of lack of interaction by students with syllabi is a known issue across institutions. in this study we attempted to address the following research questions: 1. does a learner-centered syllabus written with inclusive language improve student engagement in a required general education wellness course? 2. do students who are more engaged with the course materials provided through a learning management system achieve higher grades in the course? the research hypotheses were: h1: students who encountered a learner-centered syllabus with more inclusive language would be more engaged with the course throughout the semester. h2: students who are more engaged with the course materials will achieve higher grades in the course. method participants students (n=1046) who were enrolled in the required wellness course and met specific criteria were recruited for the study in spring 2017. the students had to be at least 18 years of age and enrolled in one of two sections taught by the same faculty member using the same format (face to face, hybrid, or online). there were a total of 24 sections, taught by 12 faculty members, recruited for the study. a consent form was delivered to all enrolled students through the online learning management system (lms). students (n=479) enrolled in the sections utilizing the traditional syllabus served as the comparison group; the students (n=567) enrolled in the sections utilizing the revised, learner-centered syllabus served as the experimental group. procedure the original, traditional syllabus (see appendix) was used as a basis for sectionspecific syllabi by all faculty teaching foundations for healthy living at a large, public, four-year institution of higher education. the course was required of all students except those pursuing three majors with high credit-hour requirements. syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 7 the original syllabus was written in traditional, third-person (“the instructor”/“the student”) language and had been previously utilized in its current iteration for 5 semesters. the syllabus was revised to reflect more learner-centered, inclusive language as described by palmer et al. (2014). the revised syllabus used first and second-person point-of-view. student engagement with the syllabus was measured by counting the number of times students clicked on the syllabus through the online platform (i.e. lms). student engagement with the course was measured using clicks in the course content. instructors teaching multiple sections of the course in the same format (two sections in the classroom, online, or hybrid) were asked to use the revised syllabus in one of their sections and the traditional syllabus in the other section. other than posting the two syllabi and letting the investigators know which section received each version the teaching faculty had no further obligations to the study. all students enrolled in the 24 sections were prompted to complete the online consent form through the lms. until the consent form was completed, the student could not progress further into the course. if a student declined to participate in the study, no data on that student was recorded as part of the study, but by completing the form the student was able to move forward in the course with no penalty. students could withdraw their consent at any time, with their data being removed upon receipt of notice by the investigators. contact information for the investigators was provided and could be accessed online throughout the semester. these study and informed consent processes were approved by the university’s institutional review board. data analysis no identifying information was gathered from students and all data were aggregated. the data were not examined until after the semester ended and final grades were calculated and reported, to avoid influencing the instructors and the way they taught either section by referencing the syllabi differently in either section. after the close of the grade reporting period, click data (i.e., the number of times students clicked on various pages within the learning management system) was collected by the analytics team at the publisher and delivered directly to the researchers for statistical analysis. two separate multivariable linear regression models were employed to examine the relationship between the dependent variables (i.e., number of on-time submissions or final score) and the independent variables collected. this included the number of pages viewed within in the course, the number of syllabus views as well as other variables collected. multicollinearity between the independent variables was assessed using pearson’s correlation coefficient. in the case that variables were highly correlated, defined as a correlation greater than +.80, then a representative variable was selected. the relationship between the independent variable and the dependent variable was assessed using backwards selection which considers the significance of the relationships in the presence of the other variables in the model. syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 8 multivariable linear regression models were designed to determine if there was a relationship between the independent variables and the final grade recorded as a percentage. a few of the instructors allowed the final percentage to be greater than 100% so their scores were rescaled by dividing the final percentage by the maximum percentage that was possible. the independent variables considered include the version of the syllabus, the number of syllabus views, the number of page views of the content, participation points awarded to the student, and the number of submissions submitted on time. results the course was delivered in two modalities: 60.32% of students were enrolled in either fully face-to-face or hybrid sections. the other 39.68% of students were enrolled in a fully-online section. of n =1046, 29.41% of the students were offered extra credit which was a binary variable reflecting the professor’s choice to allow extra credit. there were two different versions of the syllabus. approximately 54.68% of the students received a syllabus written in first person while 45.32% received a syllabus written in third person. descriptive statistics of the variables are displayed in table 1. the number of syllabus views was captured by the system and reflect the number of times over the course of the semester that the student examined the syllabus. the mean number of syllabus views is 2.81 (sd = 2.12). the number of page views is a similar variable that is captured by the system and represents the number of times the student engaged with any page contained within the system. the mean number of page views is 566.99 (sd = 200.96). the online system also captured the number of on time submissions, represented as “on time” in the table below. the mean number of on time submission is 31.7 (sd = 7.69). the outcome variable was the scaled final score (m = 82.92, sd = 17.22). sas version 9.4 along with jmp version 12 were used to construct the models. a backward selection technique was employed and the variables with a significant relationship, after adjusting for other variables remaining model, with the outcome variable were retained. the independent variables considered in the model are included in table 1. the p-value to enter the model was 0.05 while the value to stay in the model was 0.15 or less. table 1 descriptive statistics of the variables under investigation minimu m first quartil e media n third quartil e maximu m mean standard deviatio n syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 9 final score (scaled) 0 79.4 87.78 92.95 100 82.92 17.22 on time 0 30 34 36 47 31.7 7.69 participatio n 2 24 28 33 184 30.98 17.2 page views 41 445.5 542 675 1696 566.9 9 200.96 syllabus views 1 1 2 3 26 2.18 2.12 we also examined the relationships between the independent variables and the number of documents submitted on time. the independent variable total number of submissions was excluded from the analysis. the resulting model found that for each 100 page views there was 1.74 additional documents submitted on time (p < 0.001). the final model explains approximately 20.6% of the variation in the number of assignments submitted on time. in both models, the number of syllabus views was included as a potential independent variable. there is not sufficient evidence to claim a linear relationship between syllabus clicks and the final score because there were so few syllabus clicks (p = .1553). the same is true of the of the dependent variable, on time submissions (p = .6559). discussion based on the few available studies of the impact of syllabus language on student engagement (canada, 2013; harnish & bridges, 2011; lund dean & fornaciari, 2014; palmer et al., 2016), we anticipated that the students enrolled in the learnercentered syllabus group would exhibit higher levels of engagement as measured by their clicks within the platform. however, the data did not support this hypothesis. syllabus click counts did not prove to be a significant indicator of student engagement, probably because students did not utilize the syllabus after the first, required view. this brings into question the applicability of lab studies to the classroom environment. perhaps the hawthorne effect, which suggests that observation of the participants influences their behavior, or a similar effect had influenced the results of the lab studies (wickström & bendix, 2000). whereas, in our research, students were not being directly observed during their semester-long interactions with the syllabus and course materials. when subjects are made to read a document and answer questions about it, their interaction with that document is required. conversely, when students are given a syllabus in a course, there is no assurance that students will utilize that document as intended, if at all. syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 10 student engagement with the course materials did produce a statistically significant effect on their course grades. unsurprisingly, the total clicks by students in the etext and accompanying course materials was telling, with total page views translating to .044 grade points per click. there was no evidence that this effect was influenced by the syllabus language. while the number of total page clicks was correlated with student grades (i.e., the more clicks, the higher the grade), there were insufficient syllabus clicks for that to be an influencing factor. because the final model explained only approximately 20.6% of the variation in the number of assignments submitted on time, it would imply that there are many other factors that are related to the ability to submit documents online in a timely fashion. the most obvious conclusion obtained from the study is that it does not matter how learner-centered the syllabus is if the students do not read it. the click results made it clear that few students, in face-to-face, hybrid, or online courses, were thoroughly reading the syllabus to take in all the information, and even fewer returned to the document to check deadlines or review information. we hoped that by revising the syllabus to make it more learner-centered, students would be more likely to review the document carefully and return to it throughout the semester to determine the expectations of the course. although students indicated by clicking a button on an online form that they had fully read and understood the syllabus, student performance did not reflect full understanding. for example, the number of late submissions suggested that they did not notice that late work would not be accepted and would therefore earn a grade of zero. additional research is recommended to identify more successful ways of increasing student engagement with the syllabus and the course. while a syllabus that conforms to the university’s generic template is required in all courses, the investigative team wonders if the traditional syllabus model, in its various forms, has become obsolete. in this study, the positive tone of the syllabus was clearly not enough to entice the student to carefully read the document or to return to it throughout the semester. a more colorful, interactive syllabus that presents information in a format that models the way students encounter information in social media and other online platforms might now be required in order to produce engagement. the researchers are currently completing a follow-up study to investigate this question. limitations further research on the influence of syllabus language on student engagement is needed due to a large number of confounding variables, some of which were only identified after examining the data. the type of institution and its admission requirements, class size, institutional and faculty cultures, level of course, topic of course, and depth of instructor training might influence the results of this study or future studies into this topic. while students could have downloaded the syllabus and referred to it outside the platform where they could not be tracked, the average time spent in the syllabus syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 11 module reported by the publisher (<30 seconds) does not make downloads likely in the majority of cases. the university where this study was conducted requires that certain information from the student handbook be inserted into the syllabus verbatim. this information is written in third-person language, so portions of the revised syllabus were still in that format. this information included such items as the university’s academic honesty, plagiarism, and sexual misconduct policies. this shift in language may have influenced the students who were given the learner-centered version of the syllabus. the nature of required general education courses leads many students to reject the value of the material covered in the courses as it does not apply directly to their chosen major. this student attitude could have negated the learner-centered language in the syllabus. therefore, researchers may be interested in conducting further studies on syllabus language and student engagement in major courses and free elective courses, where students self-select for enrollment, to determine if a change in language is more effective in those cases. though three different modalities were included in the sample, the difference in delivery method was not accounted for in the data analysis. students who are in a classroom with a teacher at least part of the time are more likely to receive reminders to submit work promptly, and are more likely to have had the syllabus explained to them. students who study fully on-line are solely responsible for the information in the syllabus. examining the variations between delivery modalities would also be recommended for future study. conclusion the hypothesis that students who are more engaged with their course materials provided through a learning management system achieve higher grades in the course was shown to be true. however, increased engagement due to the use of more inclusive language in the writing of the syllabus was not proven. the evidence indicated that the students were not engaging with the syllabus sufficiently to produce a statistically significant result. we believe the value of this study is to highlight the fallacy that learner-centered language will necessarily produce higher levels of engagement in a course. the authors believe that further research is needed to determine if there is a different way to present the syllabus or the material contained in the syllabus in a way that will result in more fully engaged learners. conflicts of interest the author(s) declare(s) that there is no conflict of interest regarding the publication of this article. syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 12 references bandura, a. (1997). self-efficacy. freeman. canada, m. (2013). the syllabus: a place to engage students’ egos. new directions for teaching and learning, 135, 37–42. cook, b., & pullaro, n. (2010, september). college graduation rates: behind the numbers. american council on education. washington, d.c, 1–3. dickinson, a. (2017). communicating with the online student: the impact of e-mail tone on student performance and teacher evaluations. journal of educators online. https://files.eric.ed.gov/fulltext/ej1150571.pdf grunert o’brien, j., millis, b. j., & cohen, m. w. (2008). the course syllabus: a learning-centered approach (2nd ed.). jossey-bass. harnish, r. j., & bridges, k. r. (2011). effect of syllabus tone: students’ perceptions of instructor and course. social psychology education, 14, 319– 330. https://doi.org/10.1007/s11218-011-9152-4 kern, b., mettetal, g., dixson, m., & morgan, r. (2015) the role of sotl in the academy: upon the 25th anniversary of boyer’s scholarship reconsidered. journal of the scholarship for teaching and learning, 15(3), 1–14. https://doi.org/10.14434/josotl.v15i3.13623 ludwig, m.a. , bentz, a. e., & fynewever, h. (2011). your syllabus should set the stage for assessment for learning. journal of college science teaching, 40(4), 20–23. lund dean, k., & fornaciari, c. (2014). the 21st century syllabus: tips for putting andragogy into practice. journal of management education, 38(5), 724–732. https://doi.org/10.1177/1052562913504764 miller-young, j., & yeo, m. (2015) conceptualizing and communicating sotl: a framework for the field. teaching & learning inquiry: the issotl journal, 3(2), 37–53. palmer, m., bach, d., & streifer, a. (2014). measuring the promise: learningfocused syllabus rubric. to improve the academy: a journal of educational development, 33(1), 14–36. palmer, m., wheeler, l., & aneece, i. (2016). does the document matter? the evolving role of syllabi in higher education. change, 48(4), 36–47. richmond, a., slattery, j., mitchell, n., morgan, r., & becknell, j. (2016). can a learner-centered syllabus change students’ perceptions of student-professor syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 13 rapport and master teacher behaviors? scholarship of teaching and learning in psychology, 2(3), 159–168. singham, m. (2007). death to the syllabus! liberal education, 93(4), 52–56. thompson, b. (2007). the syllabus as a communication document: constructing and presenting the syllabus. communication education, 56(1), 54–71. https://doi.org/10.1080/03634520601011575 tokatli, a. & kesli, y. (2009). syllabus: how much does it contribute to the effective communication with students? procedia social and behavioral sciences, 1(1), 1491–1494. wickström, g., & bendix, t. (2000). the "hawthorne effect" — what did the original hawthorne studies actually show? scandinavian journal of work, environment & health, 26(4), 363–367. syllabus influence on student performance journal of effective teaching in higher education, vol. 3, no. 1 14 appendix sample language from the online traditional syllabus and the online studentcentered syllabus: traditional: communicate with the professor through the ksu email system. replies will be sent to student email. the professor will reply within 72 hours. if the professor has not replied within 72 hours, resend through the kennesaw state university email system. the ksu email is listed above. do not email through d2l. revised: when you have a question, the best way to communicate with me is through the ksu email system, shown above. i will reply to your student email account as soon as possible, but at least within 72 hours. if you have not heard from me within 72 hours, please resend your email or call my office. you may use my ksu email at any time, or email through the bearface platform. please do not email through d2l, as i check that system only rarely after the first two weeks of class, and your questions are important to me. microsoft word boss_final.docx journal of effective teaching in higher education, vol. 4, no. 2 doi 10.36021/jethe.v4i2.222 on black mattering and (un)framing the preparation of higher education/student affairs administrators ginny jones boss, phd, university of georgia, ginnyboss@uga.edu tiffany j. davis, phd, university of houston christa j. porter, phd, kent state university abstract. the purpose of this critical content analysis study was to examine higher education/student affairs (he/sa) preparation toward a racial framing that centers and honors black mattering. we explored linkages between black literacies and epistemic credibility as indicators of black mattering by analyzing 24 syllabi of foundational courses in he/sa graduate preparation programs using muhammad’s (2020) historically responsive literacy (hrl) framework. the hrl framework is a four-layered equity framework with the following learning goals: (a) identity development; (b) skill development; (c) intellectual development; and (d) criticality. across the four layers, we found little evidence of black mattering in our data. to meaningfully situate black mattering within curriculum development, we suggest instructors use their syllabus to begin unframing white supremacy and framing the curriculum with black mattering. keywords: black mattering; literacy; content analysis; student affairs; teaching and learning higher education/student affairs (he/sa) preparation programs play a crucial role in training and socializing collegiate administrators (boss et al., 2018). foundational courses serve to set the context for the broader curriculum within he/sa graduate programs. the problem is that many of these programs have operated in a white racial frame (feagin, 2013), obscuring legacies of racism and granting white ways of knowing an epistemic credibility (alcoff, 1999) not afforded to black ontoepistemologies. parson and major (2020) argued teaching and learning approaches should be responsive to the sociocultural contexts in which learning takes place. this level of responsiveness necessitates “understanding how our institutions have historically silenced or harmed persons of color” (parson & major, 2020, p. 31). specifically for black students, educators must understand the explicit and insidious nature of anti-black racism on campus (haynes & bazner, 2019). historical antiblack racism and its present-day manifestations have inspired numerous leaders in higher education to issue statements of solidarity, but college and university systems and structures remain oppressive of black students. one of the two leading umbrella professional organizations in student affairs created a framework for racial justice and decolonization for use by he/sa professionals (acpa, 2018). despite the existence of this framework, approaches for racial justice and decolonization have not been widely adopted; nor have there been conversations about categorically addressing anti-blackness and black mattering outside a small collective within the profession. on black mattering and (un)framing 28 journal of effective teaching in higher education, vol. 4, no. 2 the purpose of this critical content analysis study was to examine he/sa preparation toward a racial framing that centers and honors black mattering. we explored linkages between black literacies and epistemic credibility as indicators of black mattering. namely, we used muhammad’s (2020) historically responsive literacy (hrl) framework to analyze content within foundational course syllabi in he/sa graduate preparation programs. our findings showcase how the present-day racial framing of he/sa foundational courses fails to address black mattering. on the basis of our findings, we offer recommendations for unframing he/sa foundational curriculum toward a greater focus on black mattering. our study makes a significant contribution to teaching and learning literature about the importance of centering black mattering in higher education in the u.s., where a focus on blackness has been unwelcomed or ignored in curricular planning outside of black ethnic studies programs or historically black colleges and universities (hbcus). literature review blackness is a collection of onto-epistemologies that represents the rich and varied cultures of the african diaspora. we use the term onto-epistemology to represent the mutually reinforcing bond between ontology (reality) and epistemology (knowing). blackness has not mattered in higher education (tuitt et al., 2018). black exclusion began with the enslavement of black laborers, segregation during the jim crow era, and racial animus in response to desegregation (anderson, 1999; fleming, 1984; perkins, 2002; wilder, 2013). despite the establishment of hbcus and forced desegregation into predominantly white institutions, white people’s perceptions of black students as intellectually inferior have remained intact. black student enrollment has increased over time, but white-centered ideologies of success, learning, and development have relegated black onto-epistemologies to the margins. black scholars have insisted on the relevancy of black ontoepistemologies in higher education (okello, 2018; patton, 2016; porter et al., 2020). throughout our literature review, we highlight scholars who have made the case for blackness and black mattering as part of a liberatory curriculum in education. scholarship of teaching and learning and onto-epistemic credibility within the growing body of scholarship on teaching and learning, only a small contingent of literature has acknowledged or presented issues of identity in a meaningful way. even within identity-conscious literature, few scholars address anti-blackness and the ways it manifests in teaching and learning in higher education. across the disciplines, higher education curricula have been constructed within white racial frames that misrepresent, dismiss, or erase blackness and black people (okello, 2020). haynes (2021) described the college classroom as “a racialized space, which cultivates white supremacy through the teaching of white normalcy” (p. 2). anti-blackness undergirds the teaching of white normalcy through the erasure and dehumanization of black onto-epistemologies (haynes & bazner, 2019). black, indigenous, people of color (bipoc) and white faculty may reject black onto-epistemic credibility unknowingly because of racial unconsciousness or on black mattering and (un)framing 29 journal of effective teaching in higher education, vol. 4, no. 2 knowingly through a conscious alignment with white supremacy. haynes and bazner (2019) argued that the work of countering anti-blackness in teaching and learning begins with recognizing and honoring black onto-epistemic credibility. within formal college classrooms, the syllabus is a critical artifact that serves multiple functions for (re)socialization. previous research on the syllabus has shown it to be a site of first contact between students and the instructor, a document that conveys power structures and disciplinary norms, a plan for the semester, a contract between faculty and students, a signal for campus climate, and a tool to facilitate learning (parkes & harris, 2002; priester et al., 2008; sulik & keys, 2014). sulik and keys (2014) examined the syllabus as a pedagogical tool for (re)socialization. they described how syllabi signal processes of socialization not only to the learning environment but also to the discipline at large and suggested “syllabi might help instructors address elements of the broader cultural environment” (p. 158). sulik and keys’ findings were useful in the present study, which examined he/sa graduate preparation programs with the aim of socializing graduate students into the profession. relatedly, liao (2015) suggested faculty construct their syllabi in a way that invites student co-construction of the classroom and shared power over learning. throughout the literature on the syllabus, much emphasis is placed on language’s function of conveying power and commitments to honor the ontoepistemologies of the students in the classroom. (black) literacies literacy and language are indelibly linked to each other, and given the interconnectedness of linguistic and racial hierarchies (baker-bell, 2020), the study of literacy cannot be race-neutral. much of mainstream literacy literature examines text literacy and its expansions, such as media literacy, information literacy, and digital literacy. muhammad (2020) articulated additional literacies as a means of liberatory educational practice. she offered critical literacy (i.e., leveraging an understanding of power, inequality, oppression, and social justice to understand text and other media), racial literacy (i.e., seeing, naming, and interpreting the world with an awareness of race, racialization, and racism), and agitation literacy (i.e., ”reading, writing, thinking, and speaking that are connected to the intention and action to upset, disturb, disquiet, or unhinge systemic oppression”) (p. 125). critical, racial, and agitation literacies create a direct link between literacy and alcoff’s (1999) writings on epistemic credibility. alcoff argued epistemic credibility is concerned with whose writing, speech, and other forms of knowledge sharing is legitimized. muhammad’s (2020) research on historical black literary societies’ critical literacy practices showcases how literacy is inextricably bound with black mattering and liberation. muhammad’s (2020) work nods to a growing body of literature on black literacies and their importance in addressing anti-black racism and centering black mattering in education (coles, 2019; johnson et al., 2017; kelly, 2020; young, 2020). black literacies involve the use of literacy practices, such as those described in the preceding paragraph, to enliven the onto-epistemologies of black people and black on black mattering and (un)framing 30 journal of effective teaching in higher education, vol. 4, no. 2 culture. black literacies are “grounded in black liberatory thought, which supports and empowers the emotional, psychological, and spiritual conditions of black people throughout the diaspora” (johnson et al., 2017, p. 63). the engagement in black literacies has played a crucial role in understanding and addressing the antiblackness embedded in u.s. society (coles, 2019). black literacies have also played a crucial role in black liberation and black mattering as acts of resistance to white (language) supremacy (baker-bell, 2020). historically, enslaved black folks resisted educational suppression through their covert development of literacy as a pathway to freedom. even after enslavement, black literary societies continued to bolster black literacies in the jim crow era (muhammad, 2020). as coles (2019) argued, black literacies remain an important pursuit toward black mattering in a neoliberal educational system, which would rather operate in a color-neutral space than address anti-blackness. his argument is shared by love (2019), who asserted educational systems and institutions often dispose of and erase black people and their experiences, histories, and knowledge. theoretical framework muhammad (2020) argued that starting with a focus on blackness provides the strongest foundation for inclusive education and offered a historically responsive literacy (hrl) framework modeled after the approaches of 19th-century black literary societies. the hrl framework has the following learning goals: (a) identity development, (b) skill development, (c) intellectual development, and (d) criticality. these four layers of the framework offer insight into how curricular decisions can respond to historical legacies of anti-blackness and center black ontoepistemologies. the first layer, identity development, focuses on providing opportunities for black students to reclaim black onto-epistemologies that have been historically excluded from institutionalized education. the second layer, skill development, recognizes the role of multiple literacies in facilitating meaning making. the third layer, intellectual development, involves cultivating intellect through asset-based approaches. lastly, the fourth layer, criticality, involves building curriculum using critical theories that respond to racism and other oppressions in learning. instructors develop criticality by guiding students to “read, write, and think in ways of understanding power, privilege, social justice, and oppression” (p. 119). through muhammad’s framework, which forefronts black onto-epistemologies, we sought insight into how to center black mattering within the he/sa curriculum. interaction with black mattering can be liberatory for black students and also expand the imaginative and cognitive capacity of other students as they are (re)socialized to he/sa professional practice. methods given that black mattering is not addressed within collegiate syllabi literature, we tested muhammad’s (2020) work as a potential framework in higher education. in this study, we examined 24 he/sa course syllabi using a blended content analysis approach to determine the extent to which such syllabi manifest the four learning pursuits of the hrl framework. we sought to answer the following research question: what do he/sa foundational course syllabi reveal about how the structure on black mattering and (un)framing 31 journal of effective teaching in higher education, vol. 4, no. 2 and content of the courses are meant to: (a) help black students learn something about themselves and others; (c) help black students develop literacy skills; (d) help black students develop capacity for transforming information into knowledge; and (e) engage black students’ thinking about power and equity and the disruption of oppression? data collection syllabi were solicited via email through csptalk, a national listserv for student affairs preparation program faculty. we requested syllabi from those who had taught foundational he/sa courses. of the 69 syllabi received, we excluded 19 because they were either duplicates or did not meet our inclusion criterion: masterlevel syllabus from the years 2017–2020. we reviewed the assigned textbooks, course descriptions, and learning outcomes of the 45 remaining syllabi, and we identified two distinct categories. one set of syllabi focused on the history of higher education (n = 21), and the other set focused on an introduction to student affairs (n = 24). the introductory syllabi covered both history and the other areas most closely aligned with our theoretical framework (identity development, skill development, and intellectual development) whereas the history syllabi were more limited in scope. we chose the broader sample for analysis and provide a profile of the data in (see table 1). table 1 he/sa syllabi sample profile syllabus number naspa region program degree institutional control average cohort size institutional diversity year 1 iv-east med public 16–20 pwi 2020 2 iv-east ma public 26+ pwi 2020 3 iii ma public 21–25 pwi 2019 4 iv-east ma public 26+ pwi 2020 5 iv-east med public 26+ pwi 2017 6 iv-east ma public 26+ pwi 2020 7 iv-east med private 26+ pwi 2020 8 iv-east ma public 21–25 pwi 2020 9 v med public unknown pwi 2020 10 vi med public 16–20 pwi 2020 11 iii ma public unknown pwi 2019 12 v med private 26+ pwi 2020 13 iv-west med private 10–15 pwi 2020 14 ii ms public 16–20 pwi 2018 15 iii med public 16–20 hsi 2020 16 iii ma public > 10 pwi 2020 17 iv-east med public 21–25 pwi 2020 on black mattering and (un)framing 32 journal of effective teaching in higher education, vol. 4, no. 2 18 i ms private 10–15 pwi 2020 19 i ma public 16–20 pwi 2020 20 iii med public 16–20 pwi 2020 21 iii med public 16–20 pwi 2020 22 iv-east ms public 16–20 pwi 2019 23 iii ma public 16–20 pwi 2020 24 iv-east ma public 21–25 pwi 2020 data analysis content analysis employs a family of analytic approaches that seek to make meaning of textual data, ranging from more interpretative qualitative analyses to objective quantitative analyses (hsieh & shannon, 2005). we blended two content analysis approaches: qualitative directed content analysis (hsieh & shannon, 2005) and conceptual content analysis (christie, 2007). qualitative content analysis is a method of systematically coding and identifying themes and patterns within text data for subjective interpretation. the goal of this directed approach to content analysis is to “validate or extend a theoretical framework or theory” (hsieh & shannon, 2005, p. 1281). in the directed content analysis phase, author 1, ginny, created a codebook informed by muhammad’s (2020) four-layer equity framework, which included subcategories, each with accompanying definitions and examples from a sample syllabus. conceptual content analysis involves quantifying and counting the presence of both explicit and implicit concepts (christie, 2007). in this phase, author 2, tiffany, created the coding categories, which the research team reviewed and confirmed. we coded for existence (related to learning activities for the four pursuits) and frequency (related to course texts and media) within the course syllabi. the conceptual analyses were completed by two graduate assistants using a qualtrics form, and author 2 reviewed the results to ensure syllabi attributes were accurately coded. in this study, the directed and conceptual content analysis approaches, which were blended, complemented each other for the interpretation of syllabi content. trustworthiness we ensured trustworthiness by using analyst triangulation and reflexivity (merriam & tisdell, 2015). in analyst triangulation, two or more analysts independently review the data to support analytic conclusions. all three authors independently analyzed the data and engaged in collective analyses. throughout the study, we carried out a variety of reflexive processes to assess the ways in which we were influencing the study, and vice versa (thurairajah, 2015). this study was very closely aligned with many of our positionalities as black women, critical pedagogues, and faculty in he/sa graduate preparation programs. we tracked our reactions to data through critical conversations with one another and other faculty in our field. we used these conversations to identify any potential barriers that might have limited our understanding of the data. on black mattering and (un)framing 33 journal of effective teaching in higher education, vol. 4, no. 2 findings our findings showcase our two goals. our first goal was to validate or extend muhammad’s hrl framework using directed content analysis. in this section, we present our findings from this qualitative component of our analysis through a narrative overview, with supporting examples taken directly from syllabi. our second goal was to capture implicit and explicit messages within the syllabi by quantifying the presence or absence of hrl framework signifiers (e.g., learning activities, requirements, or descriptors that align with the framework) using conceptual content analysis. what follows is an overview of he/sa programs’ current approaches as determined through our analyses of the syllabi in search of identity development, skill development, intellectual development, and criticality. identity development muhammad (2020) explained, “[i]dentity is composed of who we are, who others say we are (in both positive and negative ways), and whom we desire to be” (p. 67). identity development in the classroom has a discursive element tied both to who students are when they show up and the messages they receive about themselves reflected in the curriculum and in their relationships with the instructor and their peers. muhammad argued that instructors need to use asset-based frames and provide opportunities for learners to explore multiple facets of selfhood. she described how black students are flooded with images and messages of antiblackness and how instructors cannot separate identity from learning goals. she explained that identity-conscious teaching must be the primary pursuit of teachers because when identity is ignored, it can create barriers and opportunity gaps for those who do not fit the prevailing white framing of the curriculum. within syllabi, we observed how instructors either distanced themselves through third-person language (e.g., “you are welcome to ask the instructor about the course content, . . .when you have initial questions about assignments, due dates, or readings, please, first check the syllabus and then contact your classmates”) or humanized themselves and students through firstand second-person language (e.g., “. . . but if you have initial questions about assignments, due dates, or readings, please first check the syllabus and then contact your classmates”). most of the instructors used humanizing firstand second-person language, such as, having been a graduate student myself twice, i recognize that you are making a degree of sacrifice in order to pursue your studies. honoring the fact that you are choosing to be in class rather than somewhere else you may also need to be, i am willing to excuse two sessions during the semester for unavoidable conflicts that you are able to anticipate during the semester. (syllabus 8) muhammad (2020) suggested that instructors further humanize themselves and students by providing statements regarding their positionalities and identities. in our content analysis, we found that 33% of syllabi contained such statements. while most instructors did not explicitly share much about themselves across on black mattering and (un)framing 34 journal of effective teaching in higher education, vol. 4, no. 2 syllabi, 79% required students to engage in autobiographical learning activities. in further humanizing students, most instructors highlighted different learning styles, the importance of individual identities, and multiple perspectives of the world as valuable to the class’ learning community. often, these identity-conscious statements were interwoven with classroom learning expectations, as shown in the example below. the classroom will be a safe space for all learners regardless of race, gender, class, (dis)ability, cultural background, or sexuality. we will promote and embrace active engagement, critical dialogue, self-reflection, active listening, and authenticity. each voice is valued in this space as we honor the experiential and content knowledge of every individual. (syllabus 20) we found very few representations specifically of blackness, with notable exceptions: instructors who required readings and texts written by black scholars who centered blackness, or instructors who included hbcu environments as part of foundational understandings of he/sa. when it comes to identity and skills, the notion of representation becomes a critical touchstone of black mattering (bakerbell, 2020). the presence or absence of black authors, creators, and histories within the curriculum is a bellwether of black mattering. skill development muhammad (2020) emphasized the importance black literary societies placed on literacy and engaging with a variety of texts across genres. in contemporary learning environments, literacy education has expanded beyond text literacy and has come to encompass digital and web-based texts and other media, which provide access to an incalculable amount of information and opportunities to engage across genres and cultures. this expansion of literacy also presents increased opportunities for instructors to incorporate a variety of texts in the curriculum to serve as mentor texts in literacy development. muhammad (2020) argued against a focus on skill development in a vacuum, claiming skills alone are not enough to address mattering if taught exclusively through hegemonic perspectives. she suggested literacy instruction should incorporate more culturally relevant approaches through the use of learning activities such as case study analysis, current/critical-issue presentations, peer review, literature review, and choose-your-own-adventure or student choice assignments. through our content analysis, we found that less than 2% of classes offered opportunities for students to engage in case study analysis. we also found that approximately 80% required literature reviews; 71%, current-issue presentations; 54%, student choice assignments; and 33%, peer review. much of the engagement across multiple literacies and most of the associated assessment practices were constructed in a dominant frame lacking culturally responsive practices. additionally, we found although instructors were using a range of literacy practices overall, very few were critical literacies. when critical literacies were present, they represented only a small portion of the overall curriculum of the course. most classes required students to engage with text and information on black mattering and (un)framing 35 journal of effective teaching in higher education, vol. 4, no. 2 literacies, and some classes required students to engage in digital and media literacies, such as constructing, a student affairs scholar/practitioner cajita (creative knowledge canvas). the cajita represents a cultural autobiographical story/student journey told in carefully selected artifacts such as family photos, letters, personal belongings, newspaper articles, etc. a short, 1-3 page paper which may be in the form of a poem, letter, essay, etc. should be included with the cajita. (syllabus 4) intellectual development muhammad (2020) explained, “historically, it was necessary to cultivate intellectualism because society and oppressors did not see black people as useful, smart, or capable of rigorous learning” (p. 102). she argued that intellectual development must involve asset-based, learner-centered pedagogies that include engagement with topics, concepts, and paradigms that situate learning in its historical context and that highlight the cultural impact of history. she also claimed that intellectual development requires critical self-reflection and that this is the natural next step to identity and skill development, offering students the facility to “express their ideas, work through justice-centered solutions to the world’s problems, and expand their mental capacities” (p. 104). our analyses revealed that most syllabi (96%) included descriptions of asset-based, learner-centered pedagogies that included engagement with topics, concepts, and paradigms that situate learning in its historical context and highlight the cultural impact of history. for example, syllabus 1 instructed students to, “draw conclusions about whose stories are left out of the history of higher education and student affairs to better advocate for these populations” and syllabus 11 included the following statement, “we learn through intra-action (barad, 2007) and dialogic exchange (freire, 1974/2013; hooks, 1994, 2003). we will engage heavily in communal practices to honor each individual’s contributions to our scholarly community.” however, few syllabi (29%) explicitly offered students opportunities to engage specifically with black histories or to critically explore the current-day cultural impacts of anti-blackness on college campuses and within the profession. although the syllabi revealed the pedagogical approaches instructors used to support intellectual development, the predominance of white racial framing (feagin, 2013) of such approaches can have a chilling effect on students’ willingness and capacity to employ justice-centered solutions to the world’s problems, such as antiblackness. criticality muhammad (2020) defined criticality as “the capacity to read, write, and think in ways of understanding power, privilege, social justice, and oppression, particularly for populations who have been historically marginalized in the world” (p. 120). to analyze how criticality surfaced in the syllabi, we focused on two areas: the evidence of a critical conceptual framework and the extent to which learning on black mattering and (un)framing 36 journal of effective teaching in higher education, vol. 4, no. 2 activities were structured to embed the development of critical, racial, and agitation literacies. although not all syllabi contained overarching critical conceptual frameworks, a few indicated critical framing through the presence of language acknowledging issues of power and oppression, such as diversity, equity, or inclusion statements (46%); land acknowledgements (1%); statements of the disparate impacts of covid-19 on communities of color (1%); encouragement to use bias-free language in submitting assignments (<1%); and the use of critical text and media (55%). for example, statements such as the following prompted us to also look for examples of critical literacy beyond texts within assignments and learning activities: i believe we learn best when pushed outside our comfort zones; therefore, challenging readings/videos/assignments, etc. and perspectives are included in this course. these readings are meant to be challenging and push you into a state of cognitive dissonance and disequilibrium. (syllabus 11) muhammad (2020) argued that teaching criticality “helps students assume responsibility for the ways in which they process information—to avoid being passive consumers of knowledge and information” (p. 122). in our search for criticality within syllabi, we coded for the existence of student reading journals, media reviews, position papers, and minute papers, as each of these types of activities includes an emphasis on critique as a part of reflection. only 42% of syllabi required student position papers, and other assignment types that required critique made up less than 1% of assignments represented across syllabi. our findings show that despite the articulated commitment to an active learning environment (100%), there were few requirements for critically and actively engaging with ideas related to racial justice. lastly, we looked for examples of racial or agitation literacy. we found little evidence of racial literacy across syllabi. outside the use of critical media that provided limited exposure to issues of race and racism, very few assignments required engagement with racial literacy, and only one syllabus suggested antiblackness as a possible topic students could explore. we analyzed syllabi for actionoriented projects, and of the 46% of syllabi with a diversity, equity, and inclusion (dei) assignment requirement, only one syllabus had an agitation literacy assignment requirement. discussion and implications an increasing number of he/sa preparation faculty are building their curriculum around the council for the advancement of standards in higher education (cas) standards for master’s level higher education and student affairs professional preparation programs (cas, 2019) and the acpa/naspa professional competency areas for student affairs educators (acpa & naspa, 2015). the cas (2019) standards state, “programs must provide learning opportunities so graduates are able to reference historical and current documents that state the philosophical foundations of the profession and to communicate the relevance of these on black mattering and (un)framing 37 journal of effective teaching in higher education, vol. 4, no. 2 documents to current higher education and student affairs practice” (p. 10). foundational courses provided a useful starting place for examining curricular and teaching and learning practices in he/sa preparation programs. in using he/sa syllabi, we join an ongoing scholarly conversation about the critical functions of the syllabus within teaching and learning. as several scholars have argued, the syllabus is a multifaceted tool that establishes expectations and (re)socializes students to the learning environment and discipline of focus (liao, 2015; parkes & harris, 2002; priester et al., 2008; sulik & keys, 2014). parkes and harris (2002) also argued that constructing a syllabus can be an opportunity for instructors to engage in reflexivity. in (re)imagining a higher education that centers black mattering, critical reflexivity is a necessary pursuit. through critical reflexivity, instructors can better recognize their tendency to cultivate “white supremacy through the teaching of white normalcy” (haynes, 2021, p. 2). using directed and conceptual content analyses, we set out to affirm muhammad’s (2020) hrl framework as a useful tool in exploring black mattering in he/sa foundational courses. we found muhammad’s work, which is rooted in black literacies and ontoepistemic credibility, a useful framework both to root out indicators of black mattering in he/sa syllabi and to offer recommendations for centering black mattering in course syllabi and teaching practices. unfortunately, we found little evidence of black mattering within our syllabi data across the four layers of muhammad’s (2020) framework. with regard to identity development and black mattering, our findings suggest very few instructors design their courses to help black students explore blackness in positive and affirming ways. for skill and intellectual development, despite the engagement of a variety of literacies, little attention was given to addressing the historical and cultural impacts of anti-blackness. the continued marginalization of literacy materials and activities that center blackness has been referred to as epistemic apartheid (evans-winters, 2020; rabaka, 2010). the inclusion of student choice and current-issue assignments as student requirements in most of the syllabi we reviewed can offer an opportunity for black students to challenge this apartheid, but students may be reticent to leverage their learning in this way when the structure of the class suggests onto-epistemic credibility is granted only to white literacy practices. love (2019) referred to this dampening of the ambition and hopes of black students due to white supremacy and other injustices as spirit murder. throughout our data, white literacy practices were embedded in and evidenced through a predominance of white authors, traditional grading structures, and rigid course policies as well as a lack of historically responsive and culturally relevant and sustaining pedagogies. when it came to criticality, we found little evidence of the kinds of critical, racial, and agitation literacies that would support black mattering and liberation. graduates of he/sa preparation programs serve in administrative positions within colleges and universities. the imperative for black mattering within he/sa programs is important for the liberatory impact it may have not only on black graduate students within these programs but also on the black students served by graduates of these programs. as several black literacy scholars have argued, black literacies enliven and empower students to reclaim cognitive, on black mattering and (un)framing 38 journal of effective teaching in higher education, vol. 4, no. 2 emotional, and spiritual resources that have been depleted by anti-blackness (coles, 2019; baker-bell, 2020; johnson et al., 2017). the imperative is also crucial to galvanizing white and bipoc professionals to promote black mattering through the use of critical, racial, and agitation literacies toward recognizing the onto-epistemic credibility of black students, faculty, and staff. given the (re)socialization potential of course syllabi (sulik & keys, 2014), we have directed our recommendations for (un)framing he/sa teaching and learning practices starting with the syllabus as an answer to the imperative for black mattering. unframing white supremacy and framing onto-epistemic credibility black literacy is a powerful approach to centering black mattering and unframing curriculum built within a white racial frame. student engagement with text, media, digital, and information literacies can provide a useful starting place for learning how to read, write, and speak across multiple modalities. to be liberatory, they must be combined with critical, racial, and agitation literacies (muhammad, 2020). the literature on black literacies provides a model for unframing white supremacy and increasing the mental, linguistic, and existential freedom of black students (baker-bell, 2020). within he/sa graduate preparation programs, black literacies can manifest in a number of ways. we organized our suggested pathways for unframing through black literacies according to muhammad’s four-layer framework. identity development muhammad (2020) emphasized the importance of identity to black liberation. within the he/sa graduate preparation program syllabi, a focus on identity should begin with the inclusion of instructor statements of teaching philosophy and a statement of a conceptual framework of criticality (we have provided an example statement in the section on criticality). in centering black mattering, instructors should also include labor acknowledgements that recognize the critical role of enslaved black laborers in building and sustaining higher education institutions (stewart, 2021). we encourage instructors to refer to the example statement stewart (2021) included in his article in diverse issues in higher education. throughout learning activities, instructors should include historical and contemporary works by black creators rather than silo works to only 1 or 2 weeks out of the entire semester. skills and intellectual development when it comes to skill and intellectual development, instructors must avoid reifying white supremacy in both their use and articulation of assessment practices and how they historicize and describe the cultural impact of history in higher education. student learning activities should involve not only the engagement of black literatures but also an expressed focus on countering anti-black racism. instructors may enhance recognition of black onto-epistemic credibility through their use of student choice and justice-focused assessment practices such as labor-based grading. labor-based grading is an approach that flattens the power structure inherent in assessment practices steeped in white racial frames and involves on black mattering and (un)framing 39 journal of effective teaching in higher education, vol. 4, no. 2 assigning grades based on student labor (inoue, 2019). finally, instructors can expand onto-epistemic possibilities by considering their syllabus an outline open to collective revision with students once the course begins. criticality syllabi should be built on a conceptual framework of criticality. criticality is a hallmark of black literacy in which black folks critically engage traditional literacy practices toward liberation. within their syllabi, he/sa instructors should include a statement that details how their course is framed by criticality. below is an example of such a statement that centers black mattering. in this course, we will engage critical, racial, and agitation literacy practices as articulated by muhammad (2020). i consider this syllabus a skeleton that we will enflesh and enliven through a collective identification of text, media, and learning activities that honor the onto-epistemic credibility of those who have been historically excluded in higher education and who presently remain on the margins. we will prioritize engagement in racial literacy (i.e., seeing, naming, and interpreting the world with an awareness of race, racialization, and racism) and agitation literacy, which involves “reading, writing, thinking, and speaking connected to the intention and action to upset, disturb, disquiet, or unhinge systemic oppression” (muhammad, 2020, p. 125). we will do this work in recognition of the legacies of anti-blackness that persist across higher education in the curriculum and co-curriculum. conclusion in this study, we examined he/sa preparation foundational courses to understand their current racial framing and to propose a racial framing toward black mattering. we chose to focus on foundational courses because of their function of socializing graduate students to the field and work of he/sa administration. through directed and conceptual content analyses, we found that most foundational courses operate within white racial frames and perpetuate a legacy denying or ignoring black mattering. we argue instructors should use their syllabus as a first step in unframing white supremacy and reframing foundational courses in criticality with a special focus on black mattering. this work joins a body of works that have come before us to uplift black mattering and promote loving black flesh (okello, 2020). our work has the potential to guide he/sa graduate preparation faculty toward a curriculum that honors the onto-epistemic credibility of black people, from those whose labor built and sustained many of our colleges and universities to those who are entering colleges and universities with the hope of enriching their lives. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. on black mattering and (un)framing 40 journal of effective teaching in higher education, vol. 4, no. 2 references acpa–college student educators international. (2018). a bold vision forward: a framework for the strategic imperative for racial justice and decolonization. acpa–college student educators international & naspa—student affairs administrators in higher education. (2015). acpa/naspa professional competency areas for student affairs educators. alcoff, l. m. (1999). on judging epistemic credibility: is social identity relevant? philosophic exchange, 29(1), 73–93. anderson, j. d. (1999). race in american higher education: historical perspectives on current conditions. in p. g. altbach, k. lomotey, & w. a. smith (eds.), the racial crisis in american higher education: continuing challenges for the twenty-first century (pp. 3–22). suny press. baker-bell, a. (2020). linguistic justice: black language, literacy, identity, and pedagogy. routledge. boss, g., linder, c., martin, j. a., dean, s. r., & fitzer, j. r. (2018). conscientious practice: post-master’s student affairs professionals’ perspectives on engaging social justice. journal of student affairs research and practice, 55(4), 373–385. http://dx.doi.org/10.1080/19496591.2018.1470004 christie, c. (2007). content analysis. in r. baumeister, & k. vohs (eds.), encyclopedia of social psychology (p. 176). sage. coles, j. a. (2019). the black literacies of urban high school youth countering antiblackness in the context of neoliberal multiculturalism. journal of language and literacy education, 15(2). council for the advancement of standards in higher education (cas). (2019). cas professional standards for higher education (10th edition). evans-winters, v. (2020, july 15). interviewer: now, that you've gotten promoted to full professor and your book is doing really well, what's next for you [tweet]. twitter. https://twitter.com/drvevanswinters/status/1283551662104346624 feagin, j. r. (2013). the white racial frame: centuries of racial framing and counter-framing (2nd ed.). routledge. fleming, j. (1984). blacks in college: a comparative study of students’ success in black and in white institutions. jossey-bass. haynes, c. (2021). the susceptibility of teaching to white interests: a theoretical explanation of the influence of racial consciousness on the behaviors of white on black mattering and (un)framing 41 journal of effective teaching in higher education, vol. 4, no. 2 faculty in the classroom. journal of diversity in higher education, online first. https://doi.org/10.1037/dhe0000256 haynes, c., & bazner, k. j. (2019). a message for faculty from the present day movement for black lives. international journal of qualitative studies in education, 32(9), 1146–1161. https://doi.org/10.1080/09518398.2019.1645909 hsieh, h.-f, & shannon, s. e. (2005). three approaches to qualitative content analysis. qualitative health research, 15(9), 1277–1288. inoue, a. (2019). labor-based grading contracts: building equity and inclusion in the compassionate writing classroom. the wac clearinghouse. johnson, l. l., jackson, j., stovall, d. o., & baszile, d. t. (2017). “loving blackness to death”: (re)imagining ela classrooms in a time of racial chaos. english journal, 106(4), 60–66. kelly, l. l. (2020). exploring black girls' subversive literacies as acts of freedom. journal of literacy research, 52(4), 456–481. liao, h.-c. (2015). what are course syllabi telling students: a critical discourse analysis of classroom power relationships. transnational curriculum inquiry, 12(1), 13–27. love, b. l. (2019). we want to do more than survive: abolitionist teaching and the pursuit of educational freedom. beacon press. merriam, s. b., & tisdell, e. j. (2015). qualitative research: a guide to design and implementation (4th ed.). jossey-bass. muhammad, g. (2020). cultivating genius: an equity framework for culturally and historically responsive literacy. scholastic teaching resources. okello, w. k. (2018). from self-authorship to self-definition: remapping theoretical assumptions through black feminism. journal of college student development, 59(5), 528–544. okello, w. k. (2020). “loving flesh”: self-love, student development theory, and the coloniality of being. journal of college student development, 61(6), 717– 732. parkes, j., & harris, m. b. (2002). the purpose of a syllabus. college teaching, 50(2), 55–61. parson, l., & major, c. (2020). learning theory through a social justice lens. in l. parson, & c. c. ozaki (eds.), teaching and learning for social justice and equity in higher education (pp. 7–38). palgrave macmillan. on black mattering and (un)framing 42 journal of effective teaching in higher education, vol. 4, no. 2 patton, l. d. (2016). disrupting postsecondary prose: toward a critical race theory of higher education. urban education, 51(3), 315–342. https://doi.org/10.1177/0042085915602542 perkins, l. m. (2002). the impact of the “cult of true womanhood” on the education of black women. in l. g. goodchild & h. s. wechsler (eds.), the history of higher education (2nd ed.) (pp. 183–190). pearson custom publishing. porter, c. j., green, q., daniels, m., & smola, m. (2020). black women’s socialization and identity development in college: advancing black feminist thought. journal of student affairs research and practice, 57(3), 253–265. priester, p. e., jones, j. e., jackson-bailey, c. m., jana-masri, a., jordan, e. x., & metz, a. j. (2008). an analysis of content and instructional strategies in multicultural counseling courses. journal of multicultural counseling and development, 36(1), 29–39. rabaka, r. (2010). against epistemic apartheid: w.e.b. du bois and the disciplinary decadence of sociology. lexington books. stewart, t. j. (2021, february 24). on labor acknowledgements and honoring the sacrifice of black americans. diverse issues in higher education. https://diverseeducation.com/article/206161/ sulik, g., & keys, j. (2014). “many students really do not yet know how to behave!”: the syllabus as a tool for socialization. teaching sociology, 42(2), 151–160. thurairajah, k. (2015). practicing reflexivity: balancing multiple positionalities during fieldwork. in sage research methods cases (pp. 1–13). sage. tuitt, f., haynes, c., & stewart, s. (2018). transforming the classroom at traditionally white institutions to make black lives matter. to improve the academy, 37(1), 63–76. https://doi.org/10.1002/tia2.20071 wilder, c. s. (2013). ebony and ivy. race, slavery, and the troubled history of america’s universities. bloomsbury press. young, v. a. (2020). black lives matter in academic spaces: three lessons for critical literacy. journal of college reading and learning, 50(1), 5–18. microsoft word litton_final.docx journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.247 videos in online courses: viewing patterns and student performance eric litton, coker university, elitton@coker.edu abstract. many instructors use videos to support their teaching in online courses to convey course content that would normally be taught in a traditional setting. prior studies have shown some connection between utilizing online videos and student performance but do not always support their findings statistically or consider the nuance of the online videos, such as if the videos are required and how long the videos are. this article uses various quantitative analysis techniques, such as course-level multivariate regressions and a fixed effects model by week, to investigate the relationship between grades, video length, and student videoviewing patterns. it is one of the first studies to analyze student video-viewing patterns by tracking exactly how many minutes of each video the students watch. the findings indicate that videos should stay within a certain length to encourage student engagement with the videos and course assignments. also, watching online videos is only positively related to grades when students are not required to watch, a result that is consistent across course-level and student-level models. student viewing patterns also differ for courses that require watching videos versus those that do not. the article concludes by discussing the relevance of these results and how instructors can best utilize online videos in their courses. keywords: requiring videos; video length; online courses; quantitative analysis; edpuzzle online or distance learning is becoming increasingly popular, and even necessary, in the united states. there were almost 7 million online education courses in the united states, enrolling 35.3% of all higher education students (nces, 2019). then, with the covid-19 pandemic, virtually all colleges and universities transitioned to online instruction in spring 2020. unfortunately, there was little time to build out online courses or for educators to transfer their in-person lesson plans to an online environment (means et al., 2020). in fall 2020, 81.7% of all u.s. higher education institutions planned to offer their instruction completely online, primarily online, in hybrid format, or with online options; by spring 2021, 60.9% of institutions were primarily or fully online or in a hybrid/hyflex model (davidson college, n.d.). unfortunately, this change resulted in a dramatic shift in student satisfaction. according to a survey of 1,008 undergraduate students by digital promise and langer research associates, only 19% of students reported being very satisfied with their courses after moving to online learning, as compared to 51% before (means et al., 2020). this was a result of a decrease in student satisfaction in multiple aspects of online courses, including quality of course content, quality of instruction, and overall learning. students’ dissatisfaction was mainly driven by students’ inability to collaborate with other students on coursework and the teacher not being able to keep students’ interests in course content. survey respondents videos in online courses: viewing patterns and student performance 95 highlighted difficulty in staying motivated in the course and missing the presence of the instructor. additionally, students appear to get lower grades in online courses. when compared with face-to-face courses, taking an online course lowers student grades by 0.3 points, on a 4-point scale (xu & jaggars, 2013). furthermore, students in online courses have lower completion rates, and online courses increase the performance gap for disadvantaged students and students from other socioeconomic groups that tend to underperform in traditional courses (protopsaltis & baum, 2019). the covid-19 pandemic further lowered student grades in online courses; 85.4% of college students noted that the pandemic negatively affected their performance (oneclass, 2020). despite students’ reported dissatisfaction with and lower grades in these courses and the disproportionate effects on some populations, online education is the fastest growing segment of higher education (protopsaltis & baum, 2019). online courses are likely to pervade in higher education as they allow schools to reach more students while being flexible to those students’ needs and interests. consequently, it is in educators’ interests to find ways to keep students motivated while delivering high-quality instruction. recorded videos are one of the main means that educators have used to deliver content in online courses. since the appearance of covid-19, 55% of courses used pre-recorded videos as part of their lesson plans (means et al., 2020). even though live video sessions were more popular since the onset of covid-19 than recorded videos, live videos are not as sustainable. live videos were common because teachers already had class times scheduled and a general lesson plan for the semester planned out; live video sessions allowed them to maintain that schedule. online courses are more commonly asynchronous in which students and teacher are not expected to attend sessions together from separate locations. recorded videos similarly allow educators to visually teach content, but they are scalable since online teachers can utilize the same recorded videos for multiple courses or semesters. this article examines the effectiveness of recorded videos in online teaching by comparing video-viewing patterns with grades using multiple quantitative techniques. video-viewing patterns are the percentage of total minutes that a student watches a video or a group of videos. first, the article analyzes videoviewing patterns based on video length. in addition, the article compares videoviewing habits in courses that included watching these videos as a requirement of the course grade versus courses that did not include video viewing as part of student grades. this required-versus-optional setup is analogous to including (or not including) attendance or participation as part of a grade in standard in-seat college courses. it is one of the first studies to compare required and optional online videos. the findings suggest that the effectiveness of watching recorded videos, as measured by student grades, is dependent on if the videos are required or not. optional or non-graded videos have a significant effect on grades while graded videos do not. further, the results are consistent when measuring at the course-level and a within-student design. the article uses these results to describe videos in online courses: viewing patterns and student performance 96 implications for educators teaching online courses, which can depend on the teacher’s preference for engaging students in online learning. literature review there is a general belief that shorter online videos are better because students are more likely to fully watch and comprehend them. an analysis of massive open online courses (moocs) shows that students fully watch videos of six minutes or less, and total viewing time per video decreases as the length of the video increases (guo et al., 2013). that sentiment has been challenged, however, as that study was done in free online courses where students have less incentive to watch the videos and to perform well. follow-up studies show that students are willing to watch much longer videos, even videos more than 50 minutes, by breaking the videos up into shorter sessions. on average, students watched 17 to 20 minutes of video per session with median minutes watched of 13 to 16 (lagerstrom et al., 2015). this is consistent with earlier observations of traditional in-seat courses that found students’ attention spans tend to be best for the first 15 to 20 minutes of class (middendorf & kalish, 1996). additionally, including interactive questions that students can respond to while watching the videos increases mean and median viewing time as well as the percentage of students who completely watch the videos (geri et al., 2017). so, longer videos may be productive if the students stay engaged or can view them in multiple sessions. connections have been made between watching content-related videos of various lengths and student performance in online courses. for very short videos, under 5 minutes, there are mixed results. a large study of undergraduate biology students showed that including very short videos increased test scores by 2% across all students and that lower gpa students benefitted the most from the videos, increasing their grades by 6.2% (dupuis et al., 2013). on the other hand, undergraduate math students’ grades did not vary based on how many very short videos they watched; however, withdrawal and fail rates did decrease when the courses included these short videos (hsin & cigas, 2013). also, online nursing students did not change their engagement in online courses after watching very short summary videos, nor did the students believe the videos helped their learning (luo & kalman, 2018). aragon and wickramasinghe (2016) used longer videos, 10–15 minutes, to support students in undergraduate statistics courses. they found all students watched at least some of the videos and that watching videos had a positive effect on grades— on average, watching one video increased a student’s grade by 4.07 points. many studies, however, do not indicate the length of their videos. studies using undergraduate and graduate students have preliminary results that students watch videos in online courses more than once and those students believe the videos help them understand the content better (pan et al., 2012; rose, 2009); on the other hand, no statistical analysis was done to make connections between video-viewing patterns and student surveys. other studies have made varied statistical videos in online courses: viewing patterns and student performance 97 connections. evans (2014) first found that undergraduate students felt better prepared for exams, but test scores were about the same for students in online courses with lecture videos in comparison to students in online courses with just powerpoint slides. in a follow-up study, though, students in online courses with videos did significantly better on three out of four exams (evans & cordova, 2015). kuo et al. (2015) were able to measure video watching patterns at different time periods during an undergraduate psychology course. they found that accessing videos was positively correlated to higher final grades. further, students who watched videos earlier prior to the assigned due date and again later to review earned better grades than students who only watched videos once before the assigned completion date. students who sporadically or infrequently watched videos received the worst grades. the authors noted, however, that they only measured video hits on their website, not the amount of time spent on each video. there are some consistent limitations across studies in addition to the length of videos not always being explicit. as kuo et al. (2015) mentioned, most studies do not measure the amount of time spent on each video. instead, studies utilize student-reported views, number of website hits, or a comparison of courses with videos to those without. moreover, studies that include multiple video views are only able to use student-reported views or website hits. no studies that were found look at how videos of varying lengths relate to students’ performance and videoviewing practices, nor do any studies look at students’ viewing patterns and compare their performance between courses in which the videos are required or optional. research questions and hypotheses this study adds to the prior research by measuring student viewing patterns more precisely by tracking each student’s viewing patterns down to the minute for each video. consequently, the study adds to the research on video viewing and student performance by answering two main research questions related to students’ detailed viewing patterns. research question 1: how does video length relate to students’ videoviewing patterns, and do video-viewing patterns change when videos are part of their grades? students appear to watch longer videos if they are engaged (geri et al., 2017). on the other hand, students prefer to break up longer videos into shorter, more manageable sessions (lagerstrom et al., 2015). other studies used videos of varying length but did not analyze the specific amount of time students spent on each video. as a result, it is expected that students will watch videos of a moderate length, though, viewing patterns will start to decrease once video length reaches 13 to 16 minutes. student viewing patterns appear to increase if they have an incentive to watch the videos (guo et al., 2013; lagerstrom et al., 2015). consequently, it is expected that students will watch more video-minutes when doing so is part of their course videos in online courses: viewing patterns and student performance 98 grade. additionally, the variability in the amount of time students spend on videos is expected to decrease for courses in which they are graded since students are expected to watch up until they receive full credit (i.e., watching 100% of each video). research question 2: how do video-viewing patterns affect student grades? studies comparing control courses (no videos) to courses with videos mostly indicate a positive connection between watching videos and student performance (aragon & wickramasinghe, 2016; dupuis et al., 2013; evans & cordova, 2015; kuo et al., 2015) or students’ perception of their performance (pan et al., 2012; rose, 2009); however, these results are not universal. some studies found no connection between watching videos and student performance (evans, 2014; luo & kalman, 2018). based on prior studies that found a connection between online videos and student performance, it appears that the videos are helpful because they are another way to teach or reinforce course content. because of this, it is expected that student course grades will increase as they watch more of the total course videos; and student assignment grades will increase as they watch more videos related to the specific assignment. to the contrary, this connection is expected to be mitigated or disappear in courses that include the video as part of their grade since video-viewing patterns are expected to be less variable. data collection data was gathered from seven masters-level business courses, five negotiation courses, and two ethics courses at a regional university in the southeastern united states. each course was offered completely online for six weeks. during each course, the instructor created content-relevant videos to help students better understand course topics and assignments. videos focused on topics that would normally be covered in-person during a live lecture, such as summarizing key points of case studies as well as preparing for assignments. in four courses (three negotiation courses and one ethics course), watching the videos was not part of students’ grades, hereafter referred to as “non-graded courses.” in three courses (two negotiation and one ethics), the videos represented a small portion of the overall grade, hereafter referred to as “graded courses.” in these courses, a student received full credit if they watched 100% of the video. the number of minutes watched per video by student was tracked using edpuzzle, an online platform for hosting educational videos. edpuzzle tracks student viewing of videos, including how much of the video each student has watched and how many times the student has viewed each section of video (each video, regardless of length, is divided into 10 sections). for all courses, students were not informed prior to the course that their viewing patterns would be tracked. data was extracted from edpuzzle into excel, then converted to stata data files to analyze. all data was anonymized before conducting the analysis and the data collection method was approved through the institution’s irb process. videos in online courses: viewing patterns and student performance 99 measuring video-viewing patterns in this manner, down to the minute for each student on each video, is a substantial improvement over past studies. many studies did not collect any video-viewing patterns and simply compared courses with videos to those without videos (dupuis et al., 2013; evans, 2014; evans & cordova, 2015; hsin & cigas, 2013; pan et al., 2012). other studies collected data on video viewing based on students’ self-reporting (aragorn & wickramasinghe, 2016; luo & kalman, 2018; rose, 2009) or by counting hits on the video’s website (kuo et al., 2015). no other studies that were found look at how much time each student spent on each video. there were 78 videos across all courses or almost two per week per course. two videos were removed from the analysis because some students had problems with the videos in edpuzzle, and viewing data was not properly tracked. a total of 76 videos are used in the analysis. there is a total of 1,065 minutes of videos, an average of 14.01 minutes per video (sd = 6.42). the videos range from 4 minutes to 31 minutes. since the total minutes of videos per course varies, the data is normalized by converting minutes watched to percentages of total minutes, which is used as the variable for students’ video-viewing patterns. there were 132 total students in the seven courses. data from two students who did not finish a course was removed, resulting in data from 130 students: 84 students in non-graded courses (62 negotiations, 22 ethics) and 46 students in graded courses (32 negotiations, 14 ethics). slightly over half of the students (52.3%) are female. the number of students and number of courses analyzed in this study are, with one exception (dupuis et al., 2013), greater than all of the prior cited studies that looked at online video-viewing habits and student performance. analysis and results first, video-viewing patterns are analyzed. then student performance is analyzed by looking at the course-level data then, in more detail, by analyzing potential patterns between video viewing and grades by week. video-viewing patterns when pooling the 130 students and the videos across courses, there are 1,390 total observations of online videos: 870 in non-graded courses and 520 in graded courses. figure 1 shows a graph of how much students are watching videos of different lengths. when comparing video-viewing patterns between non-graded and graded courses, the percentage of video minutes watched appears to diverge once videos become longer than 15 minutes. after this point, students in graded courses tend to view longer videos only once, and students in non-graded courses are less likely to watch the videos. videos in online courses: viewing patterns and student performance 100 figure 1 average percentage of video minutes watched by video length pooled ordinary least squared (ols) regressions are conducted for each course type to analyze if the relationship visualized in figure 1 is statistically significant, with control variables for gender and course topic (table 1). pooled ols is used to capture the differences across all videos instead of global differences in length that would result from a normal linear ols. for non-graded courses, the wooldridge test shows the presence of autocorrelation (f(1, 83) = 20.24) and the breusch– pagan/cook–weisberg test shows the presence of heteroscedasticity (chi2(1) = 26.61). for graded courses, there is no indication of autocorrelation in the data (f(1, 45) = .73), but there is heteroscedasticity (chi2(1) = 5.24). as a result, robust standard errors are utilized in the table 1 regressions (mehmetoglu & jakobsen, 2017). when looking at the overall regressions (across videos of all lengths), video length is negatively related to percentage of video minutes watched for both non-graded courses, r2 = .07, f(3, 83) = 20.48, p < .001, and graded courses, r2 = .02, f(3, 45) = 2.36, p = .021. a different story unfolds, however, when separating the data between longer videos (greater than 15 minutes) and shorter videos (15 minutes or less). for both segments in graded courses, the length of videos is not significant although there is a substantial drop in the percentage of video minutes students are watching as shown in the difference in the constant values between the two models. 0 20 40 60 80 100 120 140 160 4 6 8 10 12 14 16 18 20 22 24 26 28 30p e rc e n ta g e o f t o ta l v id e o m in u te s w a tc h e d video length (minutes) non-graded graded videos in online courses: viewing patterns and student performance 101 table 1 pooled ols regressions for percentage of video minutes watched (dependent variable) for courses with non-graded videos and graded videos variable overall length ≤ 15 minutes length > 15 minutes beta se beta se beta se non-graded length -2.672*** .362 4.526** 1.378 -2.489*** .664 female 9.978 8.480 8.354 11.231 12.848 7.834 ethics 23.595* 10.762 13.624 14.426 29.123** 9.203 constant 123.871*** 10.008 50.444** 17.350 109.397*** 16.091 graded length -.650* .272 1.082 .553 -.852 .748 female -5.390 5.289 -5.936 6.636 -3.895 7.109 ethics 1.875 6.660 7.721 8.269 -7.411 8.852 constant 124.967*** 5.859 106.894*** 8.492 126.476*** 18.146 note: ***p < .001; ** p < .01; *p < .05. for non-graded courses, there is a shift in video-viewing patterns. when videos are 15 minutes or less, students tend to watch more of a non-graded video as the length of the video increases, r2 = .03, f(3, 83) = 4.52, p = .001. on the other hand, for videos longer than 15 minutes, students tend to watch less of a nongraded video as it becomes longer, r2 = .08, f(3, 83) = 9.57, p < .001. course grades and video viewing the comparison of grades and video-viewing habits by course type are included in table 2. as would be expected, students in graded courses watched a greater percentage of total video minutes, t(128) = -4.16, p = .0001. they also fully viewed more videos, that is, watching 100% of the video minutes or more, t(128) = -8.28, p < .0001. students in graded courses received higher overall grades, but the difference is only moderately significant, t(128) = -1.71, p < .1000. table 2 comparison of grades and video watching patterns between courses with nongraded videos and graded videos non-graded graded t-test diff. m sd m sd grade 83.90 7.14 86.15 7.27 p = .0900 % of total video minutes watched 87.40 39.39 112.77 16.75 p = .0001 videos in online courses: viewing patterns and student performance 102 % of total videos fully watched 64.77 26.63 97.91 6.79 p < .0001 note: grades are out of 100. a linear regression was done for both non-graded courses and graded courses with the course grade as the dependent variable and independent variables for percentage of video minutes watched and percentage of videos fully watched along with control variables for gender and course topic. table 3 shows the detailed results. even though the variables for percentage of video minutes watched and percentage of videos fully watched are somewhat correlated (rho = .8145), the regression vif values are low (2.21 and 1.34, respectively). additionally, the coefficients and significance of the remaining predictors are unchanged when removing the variable for percentage of videos fully watched from the regressions. as a result, both variables are included in the regression models (mehmetoglu & jakobsen, 2017). table 3 linear regressions for course grades (dependent variable) for non-graded courses and graded courses variable beta se 95% ci p ul ll non-graded % minutes .0925 .0339 .025 .1599 .008 % videos -.0141 .0482 -.110 .0820 .771 female -2.156 1.441 -5.025 .7124 .139 ethics -2.128 1.729 -5.570 1.314 .222 constant 78.393 1.968 74.476 82.310 <.001 graded % minutes -.029 .075 -.181 .123 0.701 % videos .286 .186 -.090 .661 0.132 female -1.249 1.991 -5.271 2.772 0.534 ethics 7.039 2.176 2.645 11.433 0.002 constant 60.001 15.030 29.646 90.355 <.001 in non-graded courses, the percentage of total video minutes watched predicts the final grade, r2 = .20, f(4, 79) = 5.00, p = 0.008; while the percentage of total videos that were watched fully does not predict final grade, p = .771. for each additional percentage point of videos watched, on average, a student’s grade increases by .093% (±.067%). in other words, to increase a final grade by 5%, a student watches an additional 54.07% of total video minutes. in graded courses, the percentage of total video minutes watched does not predict the final grade, r2 = .25, f(4, 41) = 3.37, p = .701, and the percentage of total videos that were watched fully also does not predict final grade, p = .132. videos in online courses: viewing patterns and student performance 103 weekly grades and video viewing next, video-viewing habits and the effect on grades is analyzed by week using longitudinal data where the videos and grades are split into the weeks that they were covered in each course. a comparison of video-viewing habits between graded courses and non-graded courses is shown in figure 2. the figure illustrates that students in non-graded courses watched more videos as the course progressed; whereas, students in graded courses watched about the same amount of the video minutes each week. notably, graded courses are consistently above 100%, which indicates that students are watching the graded videos fully plus about 10–30% extra. figure 2 average percentage of total video minutes watched per week, with 95% confidence intervals students’ weekly viewing habits are reinforced with a simple time-series regression with the week number as the independent variable and the percentage of total video minutes watched as the dependent variable. for non-graded courses, the percentage of minutes watched increased significantly each week, beta = 7.169, r2 = .03, f(1, 502) = 16.18, p < .001. on the other hand, for graded courses, the percentage of minutes watched did not significantly change each week, beta = 1.141, r2 = .00, f(1, 274) = .88, p < .350. 60 70 80 90 100 110 120 130 140 150 1 2 3 4 5 6 p e rc e n ta g e o f t o ta l v id e o m in u te s w a tc h e d week non-graded courses graded courses videos in online courses: viewing patterns and student performance 104 moreover, students watched fewer videos earlier in non-graded courses. the average values are significantly different for week 1 (t(128) = -4.75, p < .0001), week 2 (t(128) = -3.60, p = .0004), and week 3 (t(128) = -3.08, p = .0025). the average values converge and are not statistically different in weeks 4 through 6. table 4 compares the weekly assignment grades for non-graded courses and graded courses (not including the points for watching the videos). for the first four weeks, grade averages were higher in graded courses and the difference is statistically significant in weeks 2, 3, and 4. similar to the video-viewing habits, grades converge in the final week of the courses. table 4 comparison of weekly grades between courses with non-graded videos and graded videos week non-graded graded t-test diff. m sd m sd 1 88.33 15.40 90.23 8.41 p = .4404 2 82.56 13.57 87.55 10.24 p = .0313 3 87.00 11.70 91.95 12.39 p = .0256 4 86.04 11.30 90.20 9.23 p = .0349 5 89.76 7.87 88.75 6.90 p = .4695 6 80.71 10.63 79.55 13.59 p = .5916 course grades tend to be better earlier in the course as each masters-level course begins with topics that are more familiar to the students, then progresses to newer, more difficult topics, and concludes with a challenging final project. because of this, using weekly raw grade percentages will likely not truly reflect each student’s viewing patterns over time. as such, mean difference grades are used by first calculating the average grade for each course by week and then subtracting the students’ grades from the weekly course average. table 5 reflects models using this mean difference grade as the dependent variable and time series viewing patterns as the independent variables. a fixed effects model is used for each because all analyses are interested in an individual student’s viewing habits and grades (i.e., variations within students), there is no group grading in these courses (i.e., variations between students), and the results from some of the hausman’s tests of the variables indicates covariation between the error term and the explanatory variables (mehmetoglu & jakobsen, 2017). for each set of variables, the random effects model results in similar coefficients with the same statistical significance. videos in online courses: viewing patterns and student performance 105 table 5 panel data using fixed effects models to predict grades (compared to average) based on percentage of video watched and percentage watched in excess of 100% course within r2 beta se 95% ci p variable ll ul non-graded % minutes .022 .0034 .0011 .0012 .0057 .002 constant -.3209 .1218 -.5602 -.0815 .009 non-graded above 100% .048 .0047 .0019 .0009 .0085 .015 constant -.0468 .1396 -.3232 .2296 .738 graded % minutes .016 .0040 .0021 -.0001 .0082 .057 constant -.4656 .2517 -.9616 .0304 .066 graded above 100% .020 .0039 .0028 -.0016 .0094 .164 constant -.0318 .1261 -.2823 .2186 .801 note: control variables for gender and class topic are omitted because they are constant within weeks. the fixed effects model for non-graded courses indicates that the percentage of total video minutes a student watches each week has a positive impact on the student’s grade for that week, r2 = .02, f(1, 419) = 9.29, p = .022. similarly, if a student watches more than 100% of the video minutes, the amount of the video they rewatch has a positive impact on their grade for that week, r2 = .05, f(1, 121) = 6.09, p = .015. for graded courses, there is some indication that percentage of total video minutes watched has a positive impact on the student’s grade for that week, r2 = .02, f(1, 229) = 3.65, p = .057), but the number of videos watched above 100% is not statistically significant, r2 = .02, f(1, 95) = 1.97, p = .164. discussion and implications the results of this study are able to provide detailed yet nuanced answers to both of the research questions. the variations in videos used through the courses in this study allowed a comparison of video length and video-viewing patterns, which answers research question 1. the results support other studies on video length, specifically that students are less likely to watch videos that are longer than 15 minutes. interestingly, this study showed that students were more likely to watch nongraded videos that were slightly longer as long as the videos stayed under that 15minute threshold. once videos in non-graded courses exceeded 15 minutes, the percentage of minutes watched by students reduced significantly. videos in online courses: viewing patterns and student performance 106 graded videos under 15 minutes had consistent viewership, with students tending to rewatch small portions of the videos. with longer graded videos (longer than 15 minutes), students were essentially watching the video once and not rewatching sections. when comparing the two course types over time, students in non-graded courses watch more of the online videos as the course progresses while students in graded courses watch about the same amount each week. interestingly, both the videoviewing habits and student grades are virtually the same at the end of each course type (in week 6). this initial difference could be a result of students’ overconfidence and/or procrastination. in non-graded courses, students may feel that they sufficiently learned the content using other resources, such as the textbook; alternately, they may have waited too long to work on the assignments, and watching videos would use too much of the little time they have remaining. in graded courses, on the other hand, a small incentive was enough for students to watch the videos throughout the course. lastly, looking first at research question 2, the results show a positive connection between instructor-made online videos and student grades for courses in which the videos are optional—that is, the videos are not required as part of the students’ grade. these results were found to be robust as they were true at both the overall course level and at the student level when comparing each student’s performance and video-viewing patterns across weeks. for non-graded courses, even though the connection between video viewing and grades is statistically significant, video viewing should not be thought of as a silver bullet to improving student performance. first, the coefficients for the course-level and student-level models are low, indicating that a student would have to spend a lot of time watching videos to change their letter grade. the results suggest that for a student to increase their grade by 10%, they would need to watch (on average) more than 100% of the video minutes; in other words, watch all the videos again and then some. of course, students on the margins may just need to review key concepts in the videos to bump up their grade. second, other model outputs, such as the r2 values, indicate that there are other factors also contributing to student performance. for graded courses, none of the models (course-level or student-level) showed a significant relationship between video-viewing patterns and grades. this does not necessarily mean that requiring students to watch videos in an online course is not worthwhile. students in graded courses did earn higher grades than students in non-graded courses, but the difference was only statistically significant for certain weeks and not statistically significant for overall course grades. this grade difference may be more pronounced in other course types (e.g., undergraduate courses or courses in other subjects). videos in online courses: viewing patterns and student performance 107 implications for practice these findings have numerous implications for higher education instructors. the implications below highlight how instructors can apply these results with respect to the following pedagogical choices: 1. selecting the optimal length for online videos 2. getting students to watch more videos related to their course, especially earlier in the course, by including video viewing as part of the grade 3. using videos for more difficult online courses 4. increasing video-viewing patterns without making it part of the course grade whether requiring videos or not, instructors should keep their videos to less than 15 minutes. students are more likely to watch these videos more than once if they are graded. if videos are optional in the course, students are more likely to watch these shorter videos and even watch them more than once. as a result, only using videos under 15 minutes, required or not, makes it more likely for students to engage with the videos and, consequently, better understand the content. if instructors are having trouble getting students to watch videos, they should require them as part of the grade. even having videos be a small portion of the grade (total video grades 4.5% to 6% per course were used in this study) is enough to incentivize students to fully watch videos. the difference between how much of each video students are watching in graded versus non-graded courses is especially apparent early in the courses. it appears that when videos are not part of the grade, students do not realize the value of the videos until they after they begin receiving assignment grades. then, they begin watching more and more of the videos as the course progresses. instructors can help fill this gap by requiring videos right up front or by explaining to students that watching these videos has a positive impact on their grade. some courses are considered difficult at an institution or have new topics that students may not have seen yet. requiring videos in these courses could help students from unnecessarily falling behind early in the online course. graded videos can also help students to start the course strong by taking advantage of the online videos as a resource to better learn the difficult content. requiring videos can be troublesome for some instructors. they may not have a way to check or verify that a student has watched a video. instructors can use online resources, such as edpuzzle, or they can utilize other ways to engage students while watching the video. for instance, professors can embed questions into the videos they make (e.g., asking students to pause and consider the question). the embedded questions can help students think about the early steps for a project or help students to slow down when considering difficult concepts or practice problems. another way instructors can increase engagement with videos is by including worksheets that students can complete while they watch the videos and then having the students turn in the worksheets once they are done with the videos in online courses: viewing patterns and student performance 108 video (or series of videos). studies have found that engaging students like this makes them more likely to watch longer videos and can increase their course grades through spending more time on online assignments (lagerstrom et al., 2015; aragon & wickramasinghe, 2016). if an instructor chooses to make videos optional for their online course, then they should reinforce the benefits of the videos to the students. instructors can do this by informing students of the connection between watching content-related videos and grades. they can also encourage students to rewatch videos before a test or before starting a big assignment by posting an announcement on the online platform with links to the relevant videos. limitations this study utilized masters-level business courses that are relatively short, only 6 weeks. most other studies use undergraduate courses that take place over 8or 16week terms. while the different course level, discipline, and duration helps support the connection between video viewing and grades, some results could be different for undergraduate students or students in other disciplines. the lack of a statistical relationship between grades and video viewing in graded courses might be surprising to some people. this could be due to the lack of variability in the dependent variable (percentage of minutes watched). the range and standard deviation were both less in graded courses than non-graded courses. additionally, grades were higher for courses that required videos, but not by a significant margin. both may become statistically significant in other course settings. conclusions online learning is becoming more prevalent in higher education, and contentrelated videos are a common way for teachers to communicate lessons and ideas that they would normally discuss in a traditional, in-seat class. this article is one of the first studies to utilize detailed student viewing pattern data to analyze the connection between watching online videos and student performance. the results illustrate that watching instructor-made videos positively impacts student grades, but only when the videos are not required as part of the course. these results held at both the course level and the within-student level; that is, individual students in non-graded courses improved their grades when watching more video minutes in one week as compared to other weeks. nevertheless, requiring videos could be helpful in courses if students are not utilizing the videos to better understand course content. lastly, longer videos (more than 15 minutes) can be counterproductive as students are less likely to watch them, which could negatively impact their grades. instructors can take advantage of these results in a variety of ways to improve their online courses. videos in online courses: viewing patterns and student performance 109 conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references aragon, r., & wickramasinghe, i. p. (2016). what has an impact on grades? instructor-made videos, communication, and timing in an online statistics course. journal of humanistic mathematics, 6(2), 84–95. https://doi.org/10.5642/jhummath.201602.07 davidson college. (n.d.). the college crisis initiative dashboard. retrieved july 26, 2021, from https://collegecrisis.shinyapps.io/dashboard/ dupuis, j., coutu, j., & laneuville, o. (2013). application of linear mixed-effect models for the analysis of exam scores: online video associated with higher scores for undergraduate students with lower grades. computers & education, 66, 64–73. https://doi.org/10.1016/j.compedu.2013.02.011 edpuzzle help center. (n.d.). edpuzzle. retrieved july 31, 2021, from https://support.edpuzzle.com/hc/en-us/sections/360001671011-gettingstarted evans, h. (2014). an experimental investigation of videotaped lectures in online courses. techtrends: linking research & practice to improve learning, 58(3), 63–70. https://doi.org/10.1007/s11528-014-0753-6 evans, h. k., & cordova, v. (2015). lecture videos in online courses: a follow-up. journal of political science education, 11(4), 472–482. https://doi.org/10.1080/15512169.2015.1069198 geri, n., winer, a., & zaks, b. (2017). challenging the six-minute myth of online video lectures: can interactivity expand the attention span of learners? online journal of applied knowledge management, 5(1), 101–111. http://dx.doi.org/10.36965/ojakm.2017.5(1)101-111 guo, p. (2013, november 13). optimal video length for student engagement. edx blog. https://blog.edx.org/optimal-video-length-studentengagement/?track=blog hsin, w.-j., & cigas, j. (2013). short videos improve student learning in online education. journal of computing sciences in colleges, 28(5), 253–259. kuo, y.-y., luo, j., & brielmaier, j. (2015). investigating students’ use of lecture videos in online courses: a case study for understanding learning behaviors via data mining. in f. w. b. li, r. klamma, m. laanpere, j. zhang, b. f. manjón, & r. w. h. lau (eds.), advances in web-based learning—icwl 2015 (pp. 231– 237). springer international publishing. https://doi.org/10.1007/978-3-31925515-6_21 videos in online courses: viewing patterns and student performance 110 lagerstrom, l., johanes, p., & ponsukcharoen, u. (2015). the myth of the sixminute rule: student engagement with online videos. 2015 asee annual conference & exposition, 26.1558.1-26.1558.17. https://peer.asee.org/24895 luo, s., & kalman, m. (2018). using summary videos in online classes for nursing students: a mixed methods study. nurse education today, 71, 211–219. https://doi.org/10.1016/j.nedt.2018.09.032 means, b., neisler, j., & langer research associates. (2020). suddenly online: a national survey of undergraduates during the covid-19 pandemic. digital promise. https://doi.org/10.51388/20.500.12265/98 mehmetoglu, m., & jakobsen, t. g. (2017). applied statistics using stata: a guide for the social sciences (1st ed.). sage publications. middendorf, j., & kalish, a. (1996). the “change-up” in lectures. national teaching & learning, 5(2). https://docstull.files.wordpress.com/2013/11/the-generalistintegration.pdf nces. (n.d.). digest of education statistics, 2019. national center for education statistics. retrieved july 26, 2021, from https://nces.ed.gov/programs/digest/d19/tables/dt19_311.15.asp oneclass. (n.d.). did college students perform worse during covid-19? retrieved july 26, 2021, from https://oneclass.com/blog/featured/184334-did-collegestudents-perform-worse-during-covid-193f.en.html pan, g., sen, s., starrett, d. a., bonk, c. j., rodgers, m. l., tikoo, m., & powell, d. v. (2012). instructor-made videos as a learner scaffolding tool. journal of online learning and teaching, 8(4). https://jolt.merlot.org/vol8no4/pan_1212.htm protopsaltis, s., & baum, s. (2019). does online education live up to its promise? a look at the evidence and implications for federal policy. george mason university. https://jesperbalslev.dk/wp-content/uploads/2020/09/onlineed.pdf rose, k. k. (2009). student perceptions of the use of instructor-made videos in online and face-to-face classes. journal of online learning and teaching, 5(3). https://jolt.merlot.org/vol5no3/rose_0909.htm xu, d., & jaggars, s. s. (2013). the impact of online learning on students’ course outcomes: evidence from a large community and technical college system. economics of education review, 37, 46–57. https://doi.org/10.1016/j.econedurev.2013.08.001 microsoft word iluzada_final.docx journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.229 personal technology in the classroom: evaluating student learning, attention, and satisfaction christina long iluzada, baylor university, christina_iluzada@baylor.edu robin l. wakefield, baylor university allison m. alford, baylor university abstract: college instructors desiring classrooms free from learning distractions often enforce personal-technology-use policies to create what they think is an optimal learning environment, but students tend not to favor restrictive personal technology policies. which type of personal technology classroom environment maximizes student satisfaction, learning, and attention? we surveyed 280 business communications students in two types of classrooms: a personal technology-restricted environment and a free-use environment. we evaluated student perceptions of cognitive learning, sustained attention, and satisfaction with the course as well as the technology policy governing their classrooms. students believed they achieved greater cognitive learning in non-restricted personal technology classrooms and perceived no significant difference in sustained attention. although students may be more satisfied with a free personal-technologyuse policy in the classroom, overall satisfaction with the course did not significantly differ according to the classroom environment. we discuss the importance of sustained attention and policy satisfaction for enhancing student course satisfaction in classrooms with both technology policy types. keywords: classroom environment; cognitive learning; course satisfaction; personal-technology-use policy; sustained attention ask any professor about distractions in the undergraduate classroom and they will likely respond with a tale of woe related to students’ personal technology use. in her book, reclaiming conversation, mit sociologist sherry turkle (2015) laments the misuse of communication devices such as smart phones and asserts that due to distractions caused by students’ personal technology, many professors enforce a device-free classroom policy (p. 15). turkle believes that students appreciate restrictive personaltechnology policies and admits to enforcing these policies in her classes (p. 121). indeed, college instructors are concerned that students’ use of personal technology (e.g., laptops, smart phones, and tablets) unrelated to the course content distracts them and diminishes their ability to learn. we investigate students’ cognitive learning, sustained attention, and course satisfaction in restricted and unrestricted personal-technology-use personal technology in the classroom 112 classrooms and offer recommendations to manage the tension between students’ preferences and instructors’ desire for effective learning environments. the relationships between personal technology, attention, and learning confirming instructors’ anecdotal experiences, research indicates students’ personal technology use may hinder focus and attention in the classroom. multitasking often impairs sustained attention, especially during complicated tasks (wood et al., 2012), and students who frequently text during class time have difficulty sustaining attention with in-class activities, which could impede learning (wei et al., 2012). texting during lectures appears to lower students’ ability to recall information (barks et al., 2011; clayson & haley, 2012; rosen et al., 2011). distracting devices may also challenge students’ ability to maintain attention while studying course materials. researchers found students study on task an average of five to six minutes before switching to a technological distraction (rosen et al., 2013). it is likely that technology distractions contribute to attention residue (i.e., lingering cognitive activity impairing performance) when students switch between schoolwork and their devices (leroy, 2009). students’ lack of focus is particularly problematic because learning generally occurs when students exhibit sustained attention (wei et al., 2012). conversely, multi-tasking with personal technology in class hinders or, at least, slows learning (rosen, 2010; sana et al., 2013; wood et al., 2012) because attention is not sustained. grades, which are used as a measure of cognitive learning (bloom et al., 1956), often suffer when a student is distracted. multiple studies show student grades decline when students text, post to social media, or multi-task during schoolwork, due to the distractions inherent in these activities (barks et al., 2011; kuznekoff & titsworth, 2013; martin, 2011; sana et al., 2013; wood et al., 2012). together, the research suggests personal technology use in undergraduate classrooms can be detrimental to student learning. students are not uniform in their beliefs about their personal technology use and distraction. some admit to being distracted and having their learning compromised while texting or when nearby classmates text (tindell & bohlander, 2012), and others indicate laptop and cell phone use in class negatively affects their concentration (attia et al., 2017). in contrast, students also believe smart phone use in class is not detrimental to their academic performance (berry & westfall, 2015; braguglia, 2012; tindell & bohlander, 2012), and laptop use supports their learning, citing twice as many benefits as challenges (kay & lauricella, 2011). indeed, mobile technology has also been found to serve educational purposes (villena personal technology in the classroom 113 martinez et al., 2021) and has potential to enhance academic performance (qi, 2019). students can use their devices for course-related purposes such as note-taking, searching for examples or images of something discussed in class, or defining a term. for students with disabilities, personal devices can be a vital aid for learning. these devices enable students, for example, to view the professor’s presentation (camacho et al., 2017), create audio recordings (barbetta et al., 2021), or cope with attention disorders (ariel & elishar-malka, 2019). potentially, the differences in student perceptions and outcomes may be attributed to the goal of technology usage, be it personal or classroom oriented. when students are directly surveyed about perceptions of their learning and attention related to their devices and prohibitive technology policies, they respond that they prefer to access their personal devices during class. prohibitive policies are typically unacceptable to students (gikas & grant, 2013; jackson, 2013; stowell et al., 2018) and negatively affect students’ opinions about their instructors and classes. professors enforcing prohibitive policies are regarded as less relevant (gikas & grant, 2013), and this attitude compromises professors’ rapport with students (stowell et al., 2018). although some students favor policies limiting device use to prevent distractions (mccoy, 2013), others indicate that no-technology environments are boring (rosen, 2010). students also value their autonomy and prefer not to turn off their devices in class (santos et al., 2017), particularly if they are not distracting (tindell & bohlander, 2012), and believe their devices contribute to learning (braguglia, 2008; kay & lauricella, 2011; kuznekoff & titsworth, 2013). while students favor policies that allow access to personal technology, their reasons may not be wholly related to learning. students’ reliance on mobile devices reflects a burgeoning shift in forces outside of instructors’ control. recent research has shown that humans desire quick access to their mobile devices, even when inactive (schaposnik & unwin, 2018), for reasons such as maintaining immediate connection to friends and family as well as mitigating feelings of anxiety and security. not unlike adult pacifiers, technology owners often turn to their smartphones for comfort and stress-relief (melumad & pham, 2020). it is no wonder, then, that students’ preference is for access to personal technology during class time. with mixed recommendations, instructors struggle to create effective personaltechnology-use policies for the classroom. instructors may not know the best personal technology policy to implement to maximize the effectiveness of the learning environment. if they enforce a prohibitive personal-technology-use policy to create an environment more conducive to attention, they may suffer professionally from negative student evaluations, personal technology in the classroom 114 and students may miss opportunities for note-taking and other productive personal technology behaviors. we explore this tension by evaluating student perceptions of the effect of their classroom environment on their cognitive learning, sustained attention, and satisfaction. some research questions guiding our study include the following: which type of personaltechnology classroom environment promotes attention, cognitive learning, and student satisfaction? should instructors implement a free personaltechnology-use policy to increase student attention and satisfaction? do students believe their attention and learning suffer in classes with a free personal-technology-use policy? are students less satisfied with a course when a restrictive technology-use policy governs the classroom? theory and hypotheses stimulus-organism-response theory (sor) (mehrabian & russell, 1974) proposes that an environmental stimulus elicits both affective and cognitive reactions that influence individuals’ attitudes and behaviors. that is, a stimulus prompts both emotions and beliefs, and these affect individuals’ attitudes and actions. sor theory is applicable to our study because a personal-technology-use policy in the classroom, whether restrictive or unrestrictive, is an environmental stimulus that prompts students’ reactions specifically (i.e., toward the policy) and more broadly (i.e., toward the course). the study of students’ perceptions is important because they influence students’ evaluations of the overall learning experience. thus, students’ feelings and beliefs about a technology use policy will influence satisfaction toward the policy as well as other aspects of the course such as learning and attention. researchers find learning occurs when students are focused (wei et al., 2012), distractions are minimal, attention is engaged, and the environment is optimized for learning. however, students indicate that they experience boredom in no-technology environments (rosen, 2010), which may trigger distracting thoughts and daydreaming to relieve boredom. thus, if technology engages students’ attention, a classroom governed by an unrestricted personal-technology-use policy may be viewed as more interesting and stimulating. students may perceive their cognitive engagement with personal technology during class as participation in class content or learning. if a technology-inclusive classroom is deemed more appealing, stimulating, and engaging, students are likely to believe greater cognitive learning occurs. also, students with disabilities can use apps and browser extensions on personal devices to facilitate their learning (ariel & elishar-malka, 2019; barbetta et al., 2021; camacho et al., 2017). therefore, we propose the following hypothesis: personal technology in the classroom 115 h1: students in a personal-technology-inclusive classroom will perceive greater cognitive learning compared to students in a personal-technology-prohibited classroom. research indicates personal technology use in class is a hindrance to students’ focus and attention. text messaging, for example, introduces stimuli that interfere with sustained attention and learning (wei et al., 2012). students themselves believe personal technology devices negatively affect sustained attention (attia et al., 2017), and some students favor policies limiting device use to prevent distractions (mccoy, 2013). additional studies find student grades suffer when they multitask with personal technology because it distracts them (barks et al., 2011; kuznekoff & titsworth, 2013; martin, 2011; sana et al., 2013; wood et al., 2012). personal device use also challenges students’ ability to maintain attention while studying (rosen et al., 2013), and when students shift attention between schoolwork and technology use, cognitive ability may be impaired and hinder academic performance (leroy, 2009). prior research suggests free access to personal technology will impede focused, sustained attention. thus, we hypothesize students will believe their ability to achieve sustained attention is negatively affected in permissive personal-technology classrooms: h2: students in a permissive personal-technology classroom will perceive lower sustained attention compared to students in a prohibited personal-technology classroom. in a study conducted in an online classroom setting, bradford (2011) found cognitive load and course satisfaction were moderately correlated. as learning becomes less burdensome, students are more satisfied with the course. it is likely that personal-technology-inclusive classrooms contribute to perceptions of greater cognitive load because students’ sustained attention is compromised by their technology use (attia et al., 2017) as it interrupts their attention (rosen et al., 2013) and contributes to multitasking behavior. sustained attention is associated with the state of flow, or effortless attention, and is significantly related to enjoyment (csikszentmihalyi & nakamura, 2010). thus, when one is easily and continuously attentive, the experience is enjoyable and likely perceived as satisfying. teachers who facilitate students’ sustained attention generally achieve greater numbers of positive student evaluations (titus, 2008), an indication of student satisfaction. because sustained attention is associated with content engagement, ease of learning, and enjoyment, we believe students will be less satisfied with classrooms inclusive of personal technology. thus, we hypothesize the following: personal technology in the classroom 116 h3: students in a personal-technology-inclusive classroom will indicate lower course satisfaction compared to students in a personaltechnology-prohibited classroom. instructor-created policies governing personal technology use in the classroom are initiated to reduce student distraction and create an environment conducive to student focus and learning. instructors generally determine what personal technologies are permissible and the allowed uses of those technologies. for example, smart phones may be allowed in class in an inactive state and laptops may be banned altogether. in general, students perceive the freedom to use their devices in class enriches and contributes to their learning (braguglia, 2008; kay & lauricella, 2011; kuznekoff & titsworth, 2013; qi, 2019; villena martinez et al., 2021). also, studies have shown that students who need learning accommodations may benefit from access to their devices (ariel & elishar-malka, 2019; barbetta et al., 2021; camacho et al., 2017). therefore, students are likely more satisfied with a liberal personal-technology-use policy that provides them autonomy. in contrast, lower policy satisfaction is likely if students perceive unnecessary restrictions or bans on their freedom to use personal technology in class. thus, we hypothesize the following: h4: students in a personal-technology-inclusive classroom will indicate greater satisfaction with their technology use policy compared to the students in a personal-technology-prohibited classroom. methods participants business communications instructors at a large, private, southwestern university were involved in this cross-sectional field study. the study took place over two semesters in which half of the classes enforced a prohibitive personal-technology-use policy and the other half a non-prohibitive personal-technology-use policy. at the outset of each semester, students in both classroom environments participated in the same discussion regarding the potential disadvantages of personal technology use in the workplace, specifically its hindrance to interpersonal communication, concentration, and attention. research has shown that the language used in higher education, particularly in syllabi, can impact the way students learn (howton, et al., 2020; kenney & sreckovic, 2019). therefore, the syllabus for each class stated the personal-technology-use policy governing the class, and this statement was emphasized on the first day of the course. personal technology in the classroom 117 in classes prohibiting personal technology use, students in violation of the policy received a written notification containing the sanctions for policy violation, which was a warning email for the first offense and a tardy on each day of each additional offense. the attendance policy counted two tardies as an absence and stipulated that students would automatically fail the course if they exceeded the maximum allowed absences. the policy also detailed the link between personal technology use and distraction in the workplace and the value of technology-free zones for strengthening interpersonal communication and relationship-building with colleagues. no sanctions existed in the free personal-technology-use policy classes, and students were merely cautioned to use their personal technology wisely. we developed a survey in qualtrics software and provided students with online access to it during class at the end of the semester (see appendix a). the survey took approximately 10 minutes to complete, and participation was optional with no associated benefits for completion. the data were downloaded into spss software for analysis. a total of 280 students participated in the survey with 137 in the technology-restricted classes and 143 in the unrestricted classes. the descriptive statistics of the students in each classroom environment are shown in table 1. participants completed the “revised learning indicators scale” (frymier & houser, 1999) to measure cognitive learning and the sustained attention scale (wei et al., 2012). each of these used a 1–7 likert type scale anchored by strongly disagree to strongly agree. the generalized satisfaction scale measurement (crosby & stevens, 1987) used a 1–7 polar opposite scale of adjectives such as unfavorable-favorable and dissatisfied-satisfied. table 1 respondent descriptive statistics personal-technology-use policy type prohibitive non-prohibitive n % n % gender male 72 53% 94 66% female 65 47% 49 34% age 19-20 years 27 20% 37 26% 21-22 years 89 65% 91 64% >22 years 21 15% 15 10% classification sophomores 28 21% 31 22% juniors 91 67% 92 64% seniors 18 12% 20 14% ethnicity personal technology in the classroom 118 asian/pacific isle 11 8% 14 10% african american 10 7% 9 6% hispanic/latino 11 8% 14 10% white 102 75% 101 71% other 3 2% 5 2% results first, we evaluated the measurement items to determine the reliability and validity of the research constructs. table 2 shows construct reliability, validity, and correlations. construct composite reliabilities (cr) range from 0.89 to 0.98 and cronbach’s alpha (α) range from 0.84 to 0.98. each reliability measure exceeds the recommended 0.70 threshold (nunnally & bernstein, 1994). we evaluated convergent validity and discriminant validity using the average variance extracted (ave) and fornell-larcker criterion, respectively. each ave exceeded the recommended 0.50 threshold (fornell & larcker, 1981), and each bolded element on the diagonal in the correlation table is larger than its associated off-diagonal elements. the factor loadings of each item and descriptive statistics are shown in appendix a. overall, the constructs demonstrate good internal consistency, reliability, and validity. table 2 reliability, validity and construct correlations variable cr α ave 1 2 3 4 1. cognitive learning .91 .88 .55 0.739 2. sustained attention .89 .84 .67 0.688 0.818 3. policy satisfaction .98 .98 .91 0.321 0.383 0.955 4. course satisfaction .97 .96 .82 0.363 0.495 0.406 0.903 note: cr = composite reliability; α = cronbach’s alpha; ave = average variance extracted. bolded numbers on the diagonal are the square root of the ave. we used analysis of variance (anova) to evaluate the hypotheses by analyzing the differences in student responses between those in the prohibitive and non-prohibitive personal-technology-use classrooms. the results are shown in table 3. h1 is supported in that the mean for cognitive learning in classes prohibiting personal technology (m = 5.11, sd = 1.21) is personal technology in the classroom 119 significantly lower (f = 7.002, p = 0.01) compared to non-prohibitive classes (m = 5.46, sd = 0.97). however, the data do not support h2. there is no difference in perceptions of sustained attention between students in the restricted classes (m = 5.00, sd = 1.37) compared to those in the unrestricted classes (m = 5.08, sd = 1.15). h3 was also not supported. perceptions of course satisfaction were not statistically different (f = 2.276, p = 0.13) among students in the personal-technology-restricted (m = 5.22, sd = 1.32) and unrestricted (m = 5.46, sd = 1.30). h4 is supported in that student satisfaction with their classroom technology policy is significantly different (f = 20.405, p = 0.000). students in the unrestricted classes viewed their technology use policy more favorably (m = 5.24, sd = 1.49) compared to students with restrictive policies (m = 4.84, sd = 1.55). table 3 hypotheses results and anova tests prohibitive personaltechnologyuse policy (n = 137) non-prohibitive personaltechnology-use policy (n = 143) significance test hypothesis support m sd m sd h1: cognitive learning 5.11 1.21 5.46 0.97 f = 7.002 p = 0.01 supported h2: sustained attention 5.00 1.37 5.08 1.15 f = 0.284 p = 0.59 not supported h3: course satisfaction 5.22 1.32 5.46 1.30 f = 2.276 p = 0.13 not supported h4: policy satisfaction 4.84 1.55 5.24 1.49 f = 20.405 p = 0.00 supported discussion the objective of our study was to understand student perceptions to manage the tension between students’ preferences to use personal technology in class and instructors’ commitment to classroom environments conducive to learning. personal technology in the classroom 120 our first hypothesis (h1) questions whether students believe they learn better in a classroom environment where they may access their personal technology. the findings suggest this is the case. students in classes allowing unprohibited personal technology use believe they achieved greater cognitive learning compared to students in the technology-restricted classrooms. as shown in appendix a, the means for each of the eight items measuring cognitive learning are greater for students in non-restricted classrooms. students in non-restricted classes were also asked about the use of specific technologies. on a 1–7 likert-type scale anchored by strongly disagree to strongly agree, the mean scores for “using a smartphone in this class facilitated my learning in class” was 4.03 and “using a laptop in this class facilitated my learning in class” was 4.15, both above the midpoint. it appears that students’ assessment of their learning was not diminished in classes with a free personal-technology-use policy. based on students’ discernment, there is no evidence that a technologyprohibitive classroom contributed to greater cognitive learning. it may be that students’ freedom to access their devices during class contributes to how well they learned. students who used personal technology in class to look up terms, phrases, and concepts for greater clarity, for example, are likely to achieve greater learning. additional research is necessary to determine how and why students used their personal technology in-class to determine its effect on actual learning. nevertheless, students with in-class accessibility to their devices, whether they used them or not, are more likely to think they achieved greater learning. it should be welcome news to instructors that students generally do not believe their sustained attention is compromised in classrooms with free access to personal technology. the results of h2 indicate no evidence to conclude that students believe their sustained attention is greater in a technology-prohibited classroom. while instructors are concerned with diminished attention caused by personal technology use in the classroom, students in a permissive-technology environment did not perceive more difficulties related to attention and focus compared to students in a restricted-technology environment. additionally, no support was found for differences in students’ course satisfaction between the two classroom environments (f = 2.276, p = 0.13). at a liberal level of significance (p = 0.13), students in classes with freedom to use personal technology may tend to be more satisfied with their course (m = 5.46) compared to students in technology-restricted classes (m = 5.22). we expected students in the technology-restricted classes would be more satisfied because the absence of technology distractions would increase their focus and attention, creating a more enjoyable learning personal technology in the classroom 121 environment. however, the presence or absence of personal technology did not appear to hinder course satisfaction in our study. unsurprisingly, students prefer an unprohibited personal-technology-use policy governing the classroom as stated in our fourth hypothesis (h4). students are significantly more satisfied with a policy in which they have freedom to choose to use their devices rather than a policy with sanctions for using them. it may be that students prefer having control over their inclass actions rather than restrictions and sanctions that remove control. post-hoc regression analysis regression analyses were performed to determine the factors significantly contributing to course satisfaction among students in classes governed by the restricted and unrestricted personal-technology-use policies. course satisfaction was input as the dependent variable and cognitive learning, sustained attention, and policy satisfaction were input as determinants of course satisfaction for each classroom environment. in the technologyrestricted classes (n = 137), sustained attention (t = 3.937, p < .001) and policy satisfaction (t = 3.407, p < .001) were significantly related to course satisfaction, explaining 31.7 percent (adjusted r2) of the variance in course satisfaction. in the unrestricted classes (n = 143), sustained attention (t = 2.763, p < .01) and policy satisfaction (t = 2.991, p < .01) were also significantly related to course satisfaction. however, these factors explained only 19.7 percent (adjusted r2) of the variation in students’ satisfaction. this indicates that in technology-restricted classrooms, students give more weight to the technology constraints and how well their attention is maintained. this may put added pressure on instructors in technologyrestricted environments to develop policies acceptable to students and design courses that sustain students’ attention. implications the results of this study provide clear depictions of students’ perceptions regarding personal technology use in the classroom. some of the key takeaways from our study include the following: 1. students believe they achieve greater cognitive learning in a classroom environment with unfettered access to personal technology. for students with differing abilities, access to personal devices may provide a vital learning aid. 2. in both classroom environments, students indicated similar ability to focus and sustain attention. classrooms allowing the free use of personal technology in the classroom 122 personal technology do not appear to compromise student attention. 3. students in restricted and non-restricted personal-technology-use environments tend to be similarly satisfied with their course. 4. students are more satisfied with a non-prohibitive personaltechnology-use policy compared to a classroom policy restricting personal device use. 5. in both types of classroom environments, sustained attention and the technology policy are important factors in students’ satisfaction with the course. however, they are more important determinants of course satisfaction in classes with restrictive policies. 6. among the constructs we measured, the dominant driver of course satisfaction in either classroom environment is students’ sustained attention. the results show that the classroom technology environment has important consequences for instructors. achieving balance in creating a classroom that maximizes student learning and accommodates students’ need for control over personal technology use is an ongoing challenge. an important practical application tip for instructors to consider when crafting a personal technology policy and creating a course’s learning environment is to shift the thinking from “getting their attention back from devices” to “keeping their attention through sustained engagement.” students desire to take classes with professors who can hold their attention and rate those courses more favorably (titus, 2008). attention is connected to enjoyment, and enjoyment affects students’ evaluations of their learning (titus, 2008). learnercentered classrooms, those that place utmost importance on the needs of the students, have been shown to create a positive impact on evaluations of teaching-presence and encourage students to participate in their own learning (stover et al., 2019). the more a teacher can design a course to engage and maintain the attention of the students, the better. yet, our results provide no indication that students perceive one type of personaltechnology environment as more attention sustaining than the other. other research indicates students negatively view prohibitive personaltechnology-use policies and/or the teachers who enforce them (gikas & grant, 2013; jackson, 2013; stowell et al., 2018), and our findings likewise show that students are more satisfied with a free-use policy. however, a prohibitive personal-technology-use policy is not necessarily detrimental to course satisfaction if students believe their instructor is effective in maintaining their attention. an instructor’s effectiveness in sustaining student attention may overcome a student’s dissatisfaction with the personal technology in the classroom 123 personal-technology-use policy. additional research is warranted to examine this relationship. however, we suggest it would be wise to accommodate students’ desire for freedom and control in a classroom policy. rather than technology bans, instructors should consider de-emphasizing restrictions on personal technology usage while maintaining effective instructor-directed, application-oriented classroom activities, which help get—and sustain— students’ attention (samson, 2010), an important contributor to students’ course satisfaction. our analyses did not indicate that students believed they learned more in a setting with a restricted personal-technology-use policy. pink (2009) asserts that successful management in the twenty-first century "resists the temptation to control people” and instead “reawaken[s] their deep-seated sense of autonomy" (p. 89). trusting adult students to know their best learning style and use personal technology to their benefit creates a shared responsibility between instructor and student. providing students with control over personal technology use in class is a superior pedagogical move if instructors optimize in-class activities to maintain and sustain student attention for enhanced learning. limitations and recommendations for further research the present study had notable limitations due to its scope. these data were based solely on multiple sections of a business communications course and students’ perceptions of satisfaction, sustained attention, and cognitive learning. though an established scale was used to measure cognitive learning, students’ actual grades were not included in the analyses. past research indicates that when students multitasked using personal devices, their exam grades were lower, though their perceptions of their learning did not reflect that reality (ravizza et al., 2014), a phenomenon we were not able to capture. future studies could include grades in the evaluation of cognitive learning in conjunction with students’ perceptions. additionally, future research might compare grades between classes prohibiting the use of personal technology and classes with lenient personal-technology policies. if greater internet usage in class is associated with poorer scores (ravizza et al., 2014), it would be useful to know if a prohibitive personal-technologyuse policy positively affects students’ grades. additional research could also clarify effective boundaries for in-class technology use that contributes to student learning. implementing restrictive policies may not actually prevent students from using their devices but will remove opportunities to incorporate these devices into teaching methodologies (santos et al., 2017) and reduce autonomy that empowers students to choose learning over distraction. presumably, students will use personal technology in the classroom 124 personal technology regardless of policy. for example, clayson and haley (2012) found 49 percent of students texted in a class despite a prohibitive technology policy. further research is also necessary to determine the types of courses in which prohibitive personal-technology-use policies support learning. while the hands-on, activity-based business communications classroom may benefit from a lenient personal-technology-use policy, this result is not generalizable to all types of classroom environment and curricula. depending on a course’s structure, the ability to use personal technology in the classroom may influence students’ satisfaction with the course. conclusion as college instructors consider how to create the best classroom environment to facilitate learning and student satisfaction, we suggest they de-emphasize personal technology restrictions. instead, instructors should devote their energies to vying for students’ sustained attention by encouraging student participation, creating engaging classroom activities, and providing students with meaningful opportunities to learn while balancing their need for autonomy related to personal technology use. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references ariel, y., & elishar-malka, v. (2019). learning in the smartphone era: viewpoints and perceptions on both sides of the lectern. education and information technologies, 24(4), 2329–2340. attia, n. a., baig, l., marzouk, y. i., & khan, a. (2017). the potential effect of technology and distractions on undergraduate students’ concentration. pakistan journal of medical sciences, 33(4), 860–865. https://doi.org/10.12669/pjms.334.12560 barbetta, p. m., bennett, k. d., & monem, r. (2021). academic technologies for college students with intellectual disability. behavior modification, 45(2), 370–393. https://doi.org/10.1177/0145445520982980 barks, a., searight, h. r., & ratwik, s. (2011). effects of text messaging on academic performance. signum temporis, 4(1), 4–9. https://doi.org/10.2478/v10195-011-0039-0 personal technology in the classroom 125 berry, m. j., & westfall, a. (2015). dial d for distraction: the making and breaking of cell phone policies in the college classroom. college teaching, 63(2), 62–71. https://doi.org/10.1080/87567555.2015.1005040 bloom, b. s., englehart, m. d., furst, e. j., hill, w. h., & krathwohl, d. r. (1956). taxonomy of educational objectives: the classification of educational goals. mckay. bradford, g. r. (2011). a relationship study of student satisfaction with learning online and cognitive load: initial results. the internet and higher education, 14(4), 217–226. https://doi.org/10.1016/j.iheduc.2011.05.001 braguglia, k. h. (2008). cellular telephone use: a survey of college business students. journal of college teaching & learning, 5(4), 55–61. clayson, d., & haley, d. (2012). an introduction to multitasking and texting: prevalence and impact on grades and gpa in marketing classes. journal of marketing education, 35(1), 26–40. https://doi.org/10.1177/0273475312467339 camacho, b. m., lopez-gavira, r., & moriña díez, a. (2017). the ideal university classroom: stories by students with disabilities. international journal of educational research, 85, 148–156. https://doi.org/10.1016/j.ijer.2017.07.013 csikszentmihalyi, m., & nakamura, j. (2010). effortless attention in everyday life. in b. bruva’s (ed.), effortless attention: a new perspective in the cognitive science of attention and action. mit press, 179–190. crosby, l., & stephens, n. (1987). effects of relationship marketing on satisfaction, retention and prices in the life insurance industry. journal of marketing research, 24(4), 404–411. https://doi.org/10.1177/002224378702400408 fornell, c., & larcker, d. (1981). evaluating structural equation models with unobservable variables and measurement error. journal of marketing research, 18(1), 39–50. personal technology in the classroom 126 frymier, a. b., & houser, m. l. (1999). the revised learning indicators scale. communication studies, 50(1), 1–12. https://doi.org/10.1080/10510979909388466 gikas, j., & grant, m. m. (2013). mobile computing devices in higher education: student perspectives on learning with cellphones, smartphones, & social media. internet and higher education, 19, 18– 26. https://doi.org/10.1016/j.iheduc.2013.06.002 howton, a., mcgrew, m., liu, l., staples, l. l., & ray, h. e. (2020). the influence of syllabus tone on student engagement in a wellness course. journal of effective teaching in higher education, 3(1), 1–14. https://jethe.org/index.php/jethe/article/view/25 jackson, l. (2013). is mobile technology in the classroom a helpful tool or a distraction?: a report of university students’ attitudes, usage practices, and suggestions for policies. the international journal of technology, knowledge, and society, 8(5), 129–140. https://doi.org/10.18848/1832-3669/cgp/v08i05/56335 kay, r. h., & lauricella, s. (2011). exploring the benefits and challenges of using laptop computers in higher education classrooms: a formative analysis. canadian journal of learning and technology, 37(1), 1–18. https://doi.org/10.21432/t2s598 kenney, c. k., & sreckovoc, m. (2019). examining language in higher education courses: creating opportunities for critical reflection. journal of effective teaching in higher education, 2(2), 85–101. https://doi.org/10.36021/jethe.v2i2.22 kuznekoff, j. h., & titsworth, s. (2013). the impact of mobile phone usage on student learning. communication education, 62(3), 233–252. https://doi.org/10.1080/03634523.2013.767917 leroy, s. (2009). why is it so hard to do my work? the challenge of attention residue when switching between work tasks. organizational behavior and human decision process, 109(2), 168–181. https://doi.org/10.1016/j.obhdp.2009.04.002 martin, l. r. (2011). teaching business statistics in a computer lab: benefit or distraction? journal of education for business, 86(6), 326–331. https://doi.org/10.1080/08832323.2010.529958 personal technology in the classroom 127 mccoy, b. (2013). digital distractions in the classroom: student classroom use of digital devices for non-class related purposes. faculty publications, college of journalism & mass communication, 71. retrieved from http://digitalcommons.unl.edu/journalismfacpub/71 melumad, s., & pham, m. t. (2020). the smartphone as a pacifying technology. journal of consumer research, 47(2), 237–255. https://doi.org/10.1093/jcr/ucaa005 mehrabian, a., & russell, j. (1974). an approach to environmental psychology. mit press. nunnally, j., & bernstein, i. (1994). psychometric theory. mcgraw-hill. pink, d. h. (2009). drive. riverhead books. qi, c. (2019). a double-edged sword? exploring the impact of students’ academic usage of mobile devices on technostress and academic performance. behaviour & information technology, 38(12), 1337– 1354. https://doi.org/10.1080/0144929x.2019.1585476 ravizza, s., hambrick, d., & fenn, k. (2014). non-academic internet use in the classroom is negatively related to classroom learning regardless of intellectual ability. computers & education, 78, 109–114. https://doi.org/10.1016/j.compedu.2014.05.007 rosen, l. (2010). rewired: understanding the igeneration and the way they learn. palgrave macmillan. rosen, l., carrier, l., & cheever, n. (2013). facebook and texting made me do it: media-induced task-switching while studying. computers in human behavior, 29(3), 948–958. https://doi.org/10.1016/j.chb.2012.12.001 rosen, l., lim, a., carrier, l., & cheever, n. (2011). an empirical examination of the educational impact of text message-induced task switching in the classroom: educational implications and strategies to enhance learning. psicologia educativa, 17(2), 163–177. https://doi.org/10.5093/ed2011v17n2a4 samson, p. j. (2010). deliberate engagement of laptops in large lecture classes to improve attentiveness and engagement. computers in education, 20(2), 1–18. personal technology in the classroom 128 sana, f., weston, t., & cepeda, n. j. (2013) laptop multitasking hinders classroom learning for both users and nearby peers. computers & education, 62, 24–31. https://doi.org/10.1016/j.compedu.2012.10.003 santos, i. m., bocheco, o., & habak, c. (2017). a survey of student and instructor perceptions of personal mobile technology usage and policies for the classroom. education and information technologies, 23, 617–632. https://doi.org/10.1007/s10639-017-9625-y schaposnik, l. p., & unwin, j. (2018). the phone walkers: a study of human dependence on inactive mobile devices. behaviour, 155(5), 389–414. https://doi.org/10.1163/1568539x-00003496 stover, s., heilmann, s. g., & hubbard, a. (2019). learner-centered design: is sage on the stage obsolete? journal of effective teaching in higher education, 1(1), 1–19. https://doi.org/10.36021/jethe.v1i1.16 stowell, j. r., addison, e. a., & clay, s. l. (2018). effects of classroom technology policies on students' perceptions of instructors: what is your syllabus saying about you? college teaching, 66(2), 98–103, https://doi.org/10.1080/87567555.2018.1437533 tindell, d., & bohlander, r. (2012). the use and abuse of cell phones and text messaging in the classroom: a survey of college students. college teaching, 60, 1–9. https://doi.org/10.1080/87567555.2011.604802 titus, j. j. (2008). student ratings in a consumerist academy: leveraging pedagogical control and authority. sociological perspectives, 51(2), 397–422. https://doi.org/10.1525/sop.2008.51.2.397 turkle, s. (2015). reclaiming conversation: the power of talk in a digital age. penguin press. villena martínez, d., perez-garcia, p., & munoz-garcia, a. (2021). is the smart mobile phone transforming university educational reality? interactive learning environments, 29(5), 835–847. https://doi.org/10.1080/10494820.2019.1707694 wei, f. f., wang, y. k., & klausner, m. (2012). rethinking college students’ self-regulation and sustained attention: does text messaging during personal technology in the classroom 129 class influence cognitive learning? communication education, 61(3), 185–204. https://doi.org/10.1080/03634523.2012.672755 wood, e., zivcakova, l., gentile, p., archer, k., de pasquale, d., & nosko, a. (2012). examining the impact of off-task multi-tasking with technology on real-time classroom learning. computers & education, 58, 365–374. https://doi.org/10.1016/j.compedu.2011.08.029 personal technology in the classroom 130 appendix a survey items, descriptive statistics, and factor loadings survey item prohibitive personaltechnologyuse policy (n = 137) nonprohibitive personaltechnologyuse policy (n = 143) factor loading m sd m sd factor 1: cognitive learning 1. i actively participated in class discussions. 5.50 1.51 5.91 1.15 .691 2. i talked with my family and/or friends about what i was doing in the class. 4.72 1.79 5.08 1.68 .732 3. i explained course content from the class to other students. 4.63 1.67 5.03 1.54 .751 4. i volunteered my opinion in class. 5.43 1.58 5.59 1.42 .632 5. i thought about the course content outside of class. 5.12 1.70 5.50 1.31 .809 6. i see connections between the course content and my career goals. 5.54 1.55 5.84 1.16 .797 7. i review the course content for the class. 4.63 1.56 4.98 1.55 .679 8. i compare the information from this class with other things i have learned. 5.30 1.39 5.71 1.27 .804 factor 2: sustained attention 1. i feel i have learned a lot in this class. 5.35 1.59 5.59 1.29 .831 2. i never shifted my attention to other non-task oriented activities during the class. 4.26 1.77 4.10 1.75 .734 3. i could sustain my attention to learning activities during this class. 5.17 1.59 5.28 1.46 .833 4. i paid full attention to class discussions. 5.24 1.54 5.36 1.27 .871 factor 3: class impressions 1. dissatisfied…satisfied 5.34 1.38 5.63 1.40 .899 2. displeased…pleased 5.23 1.44 5.55 1.35 .904 personal technology in the classroom 131 3. unfavorable…favorable 5.31 1.62 5.55 1.43 .933 4. unpleasant…pleasant 5.12 1.48 5.54 1.45 .904 5. i didn’t like it at all…i liked it very much 5.12 1.52 5.24 1.46 .905 6. frustrated…contented 5.12 1.57 5.38 1.62 .890 7. terrible…delighted 5.07 1.31 5.31 1.31 .889 factor 4: policy impressions 1. dissatisfied…satisfied 4.98 1.63 5.61 1.41 .955 2. displeased…pleased 4.80 1.63 5.66 1.38 .972 3. unfavorable…favorable 4.83 1.66 5.61 1.44 .958 4. unpleasant…pleasant 4.83 1.67 5.70 1.41 .957 5. i didn’t like it at all…i liked it very much 4.71 1.71 5.51 1.41 .943 6. frustrated…contented 4.98 1.58 5.67 1.35 .944 7. terrible…delighted 4.74 1.54 5.56 1.39 .956 microsoft word burteubank_optimismresiliencehealthfactorscovid19.docx journal of effective teaching in higher education, vol. 4, no. 1 doi: https://doi.org/10.36021/jethe.v4i1.206 optimism, resilience, and other health-protective factors among students during the covid-19 pandemic kate g. burt, phd, rdn, lehman college, city university of new york, katherine.burt@lehman.cuny.edu jacob m. eubank, edd, lehman college, city university of new york abstract. black, indigenous, and people of color (bipoc) suffer disproportionately from coronavirus-related illness, death, and financial loss. the aim of this retrospective, qualitative study was to better understand the experiences of bipoc students at a bronx-based public university during the covid-19 pandemic. data was collected from a reflective final exam in a health sciences course in may 2020. responses (n = 28) were coded and analyzed using the social determinants of health (sdh) framework. several themes were identified in structural and intermediary determinant areas, including occupation, education, social cohesion, and psychosocial factors. participants demonstrated optimism, resilience, and perseverance—protective factors against exposure to adverse sdh. findings indicate that covid-19 negatively impacted bipoc students in multiple sdh areas which may have a compounding effect, hindering equity and justice. providers of social and academic support are critical levers in addressing sdh barriers and helping students strengthen protective factors to reduce adverse impacts of healthdamaging determinants. keywords: social determinants of health; social support; minority students; covid19; anchor institutions the covid-19 pandemic hit new york city (nyc) especially hard at the onset (march 2020) in the united states and within less than three weeks of its first case, new yorkers were ordered to shelter-in-place (opam, 2020; van dorn et al., 2020). it became clear that black, indigenous, and people of color (bipoc) suffered disproportionately from coronavirus-related illness, death, and financial loss (garcia, 2020; van dorn et al., 2020). in the bronx, a borough home to many impoverished bipoc residing in poor housing and suffering from poor health, the devastation of the pandemic (in terms of hospitalizations and death) was especially evident (freytas-tamura et al., 2020). during the early months of the pandemic, bronx residents were also experiencing the loss of income as unemployment rates soared 2,000% as compared to the previous year; many residents were forced to work as essential employees, putting them at greater risk of contracting coronavirus (freytas-tamura et al., 2020). at this time, some new yorkers were also enrolled in higher education. the rapid switch from in-person to online or remote learning caused concern about equity and inclusion, particularly for bipoc students (maloney & kim, 2020). in general, students experienced heightened anxiety and stress and have lost job and internship offers (aucejo et al., 2020; husky et al., 2020). as a result of covid, low-income students may be as much as 55% more likely than higher-income peers optimism, resilience, and other health factors during pandemic 2 journal of effective teaching in higher education, vol. 4, no. 1 to delay graduation (aucejo et al., 2020). despite research accumulating about the medical and health impacts of the covid-19 pandemic, little is known about the socio-emotional impact on bipoc students during the intense, early months. some quantitative research indicates that college students experienced higher anxiety, stress, and depression during the early pandemic and that students from low socioeconomic backgrounds or who knew someone with covid experienced worse psychological impact (browning et al., 2021; molock & parchem, 2021; son et al., 2020). this evidence also indicates that financial and emotional support for students from higher education institutions may be beneficial, particularly for students of color (molock & parchem, 2021). however, due to the quantitative study designs, these data provide little insight into the lived experiences of lowincome students of color, in particular. understanding the impact of public health crises on marginalized groups who suffer disproportionally from health disparities is critical to improving preparedness, strengthening communities, and promoting health. the aim of this retrospective qualitative study was to better understand the experiences of bipoc students at a bronx-based public university during the covid-19 pandemic. methods this study explored the experiences of health sciences students at a bronx-based public college located in nyc while it was the epicenter of the covid-19 pandemic. participants in this study completed a retrospective qualitative final examination in may 2020 at the end of the spring semester at lehman college, city university of new york. setting lehman college is the only bronx-based, hispanic-serving, four-year public university. it is an anchor institution whose mission and strategic vision are aligned with serving the low-income community in which it is situated (cruz, 2019). the bronx has the poorest health outcomes of any new york state county and is home to the poorest congressional district in the nation with a poverty rate of 29.1% (more than double the national average of 13.1%) (united states census bureau, n.d.; freytas-tamura et al., 2020). bronx residents were disproportionally impacted by the covid-19 pandemic with more cases, hospitalizations, and deaths than any other borough (freytas-tamura et al., 2020). the bronx has been cited as one of the worst places for economic mobility, ranking better than only about 5% of counties nationwide (cruz, 2019). on the other hand, lehman college is consistently ranked in the top five us public institutions for the highest rates of economic mobility and was most recently ranked first (“colleges with the highest student-mobility rates, 2014,” 2018; espinosa et al., 2018). about 80% of lehman college students are bipoc, 50% are from households earning $30,000 or less, and 41% speak a language other than english at home (cruz, 2019). about 60% of lehman college students are from the bronx (lehman college interactive fact book, n.d.). to serve these students and the surrounding community, lehman college is home to important support services optimism, resilience, and other health factors during pandemic 3 journal of effective teaching in higher education, vol. 4, no. 1 such as a small business incubator, a wellness center, and a food pantry. lehman college also hosts various cultural and arts events. participants participants in this study were senior health sciences students, majoring in dietetics, foods, and nutrition, and enrolled in a seminar-style course in professional practice. many students were healthcare workers while enrolled in the bachelor of science program for professional advancement to become registered dietitian nutritionists. data collection the data for this research was collected through a mandatory online final exam. it consisted of 9 open-ended questions, primarily designed to reflectively gauge learning about course objectives during the spring 2020 semester (january–may). most questions related to how the course content contributed to students’ skill development and learning. the two final questions gathered information unrelated to the course material: 1) is there anything you want me to know about your experience this semester? 2) how are you, really? i haven’t asked you this in about a month. i want an update. you do not have to share personal information if you don’t want to. demographic information was obtained from enrollment data and was not linked to exam responses to protect participant confidentiality. the final exam was not initially intended to be used for research; the sole purpose was to assess cumulative knowledge as an alternative to a typical final exam. however, upon reviewing responses, the wealth of information about students’ experiences was apparent. expedited irb approval was obtained (#2020-0424) to gather and analyze student responses. analysis responses from the two questions were anonymized and exported into dedoose, an online coding software (dedoose, 2016). in vivo coding was conducted by the two authors independently. interrater reliability was analyzed using cohen’s kappa and found to be .66, indicating substantial agreement (mchugh, 2012). discrepancies in coding were discussed and a final codebook was generated. next, both authors (together) conducted a thematic analysis by mapping codes to components of the world health organization’s social determinants of health (sdh) framework (world health organization, 2010). the sdh framework depicts the social, economic, and political context that influence an individual’s socioeconomic position (figure 1) (world health organization, 2010, p. 6). there are both structural and intermediary determinants described within the framework. structural determinants refer to the “interplay between the socioeconomic-political context, structural mechanisms generating social stratification and the resulting socioeconomic position of individuals” (world health organization, 2010, p. 28). as such, structural determinants also include optimism, resilience, and other health factors during pandemic 4 journal of effective teaching in higher education, vol. 4, no. 1 one’s socioeconomic position in relation to both fixed and modifiable characteristics. fixed characteristics include social class (e.g., ownership and control of resources), gender, and race/ethnicity, while modifiable characteristics include education, occupation, and income. intermediary determinants are context specific to individuals and impact individual health outcomes. social cohesion and social capital are considered “cross cutting” determinants, influencing both structural and intermediary determinants (world health organization, 2010, p. 43). for this study, social cohesion and capital refer to the communitarian approach, defined as a “psychosocial mechanism” and “extension of social relationships” (world health organization, 2010, p. 41). this study focused on individual-level factors, including structural determinants related to socioeconomic position (education, occupation, and income), intermediary determinants (material circumstance, behaviors, and psychosocial factors) and social cohesion and capital. a policy analysis of other structural determinants is outside the scope of this study. figure 1 the world health organization’s social determinants of health framework, used as the conceptual framework for this study (social determinants of health, 2010) results sample characteristics participants in this study were 75% female, 93% identified as bipoc, and the mean age was 32.6 years old. of note, as this study is primarily concerned with bipoc, optimism, resilience, and other health factors during pandemic 5 journal of effective teaching in higher education, vol. 4, no. 1 only 2 of the 9 participants who identified as white, solely identified as white, the others identified as two or more races/ethnicities. several participants were employed as essential health care workers while nyc was the epicenter of the pandemic, and others lost employment. many students have children and/or care for other family members, which compounded their responsibilities when schools, senior centers, and other social gathering spaces were closed during shelter-inplace orders. table 1 participant demographics (n = 28) demographics n (%) gender male 6 (21.4) female 21 (75.0) not reported 1 (3.6) agea 18-24 4 (14.3) 25-34 16 (57.1) 35-50 6 (21.4) over 50 2 (7.1) race/ethnicityb asian 1 (3.6) black 12 (42.9) hispanic 15 (53.6) white 9 (32.1) amean age is 32.6 years old bthe total proportion is greater than 100% because participants were able to identify as more than one race/ethnicity. several themes emerged from the data, using the social determinants of health framework as a guide to provide context in structural and intermediary determinants of health. primary themes observed in this study include stress, anxiety, optimism, resilience, and perseverance. stress, anxiety, and fear were related to loss of employment or income, being an essential healthcare worker, balancing coursework with employment, and managing unstructured time. within intermediary determinants, the importance of social cohesion and social capital was especially prominent. each is described in more detail in the following sections. direct quotes are reported with a participant number indicated at the beginning of each quote (e.g., p1 for participant 1). optimism, resilience, and other health factors during pandemic 6 journal of effective teaching in higher education, vol. 4, no. 1 structural determinants of health participants described stressors related to loss of employment/income and to the stress of being an essential healthcare worker during a viral pandemic. physical and emotional safety was of concern for many participants who were required to be physically present at work or feared contracting coronavirus in public spaces (i.e., the subway). others were scared for their safety at work in healthcare settings or for family members with whom they lived. one participant described his stressful experience living in a multi-generational household, with an elderly family member coincidently visiting when the pandemic hit. p25: when all this pandemic started, i have to admit i was scared…i was afraid of my grandmother's safety due to my brother and i work[ing] in a hospital. she is visiting from el salvador, but el salvador shut the airport a week before her return flight, so she is staying with us until it is safe to go back…it doesn't help the stress levels when co-workers [become] infected with the virus; one of them got intubated for over twenty days… unfortunately, we also lost two staff members to the virus. my stress level…is high but is slowly coming down. several participants discussed the challenges working in hospitals and living with the fear of becoming sick or dying from covid-19. many spoke of co-workers or family members who developed covid-19, such as one whose father was “in the hospital…making a slow recovery” (p14) or another who “lost a family member to covid” (p5). working in the healthcare field or as essential workers further exacerbated the impact of the pandemic on these students. participants frequently described balancing employment with coursework during the final semester of their studies. students enrolled in the major must earn a high gpa in order to best position themselves to match with a dietetic internship, which is highly competitive and the next step to become a registered dietitian nutritionist. one participant recalled her experience trying to balance employment with coursework: p5: this semester was going to be very exciting for me, i was waiting to take [a high level, intensive course] for so long. i learned a lot from it but i don’t think i was able to focus as hard [after the semester moved online] as i did during in person lecture[s]. my grades are not as high as usual and i’m worried about my gpa. a lot slipped past me as i was working in the hospital and dealing with that fear. similarly, many participants discussed experiencing difficulty focusing on or completing coursework. some did not have internet access or a computer at home. some were expected to transition coursework and two or more part-time jobs online, while caring for children or older dependents. one participant recalled the cumulative effect of managing competing priorities: optimism, resilience, and other health factors during pandemic 7 journal of effective teaching in higher education, vol. 4, no. 1 p18: i am actually starting to feel the effects of being quarantined. from the crazy supermarket lines to not really being able to go outside and to living a remote life it has been a bit challenging to keep up with. one of the hardest parts is creating work-life boundaries. before it was easier because when i was at work i worked, but now work and life are all mixed together. despite everything, i am still able to keep up with all of schoolwork, work from home for my two jobs, and continue my sports training at home but it has been overwhelming. the difficulty managing unstructured time while having family around was impactful for many participants. they described the importance of trying to stay busy and the pressure or desire to stay productive. p8: this semester was different, i had to learn how to adjust to online classes. it was much harder to keep the motivation compared to going to class in person. to be honest i thought it would be cool cause now i can just take classes online, but it is not the same. students leaving their mics on during class, my family talking loudly and me losing my concentration and so many distractions it was not the same. intermediary determinants of health participants’ material circumstances changed greatly at the onset of the pandemic. many elaborated on safety concerns of public spaces, described above, as a place where they could contract coronavirus. access to groceries and necessary household goods (e.g., toilet paper), were of concern. these separate conditions (public spaces and household goods) are inexorably linked for those without a car because it meant frequent small trips, sometimes to multiple stores. a participant described their situation as follows: “i do have some concerns about going on the trains…[two] supermarkets have most of the groceries that i need, however paper towels, toilet paper, and cleaning goods still seem to be in short supply [so i also need] to get toilet paper from bjs and paper towels from target” (p9). in addition to these difficult circumstances, participants described many impacts of the pandemic on behaviors and psychosocial factors. the most commonly discussed behaviors were time management/productivity and behaviors related to leisure, recreation, and self-care. a participant described incorporating self-care into her new routine: p18: living with multiple family members, it is hard to be at home all the time having to accommodate with everyone’s different schedules and routines. nonetheless, i make sure to stay grounded and focus on the positive and simple little things to be grateful for every day. i have been enjoying different types of self-care practices that i have incorporated into my daily routine such as meditation, gratitude, yoga, listening to podcasts and enjoying tea before bed. optimism, resilience, and other health factors during pandemic 8 journal of effective teaching in higher education, vol. 4, no. 1 others discussed cooking more frequently, spending time with family, talking to friends or family, praying or meditating, singing, and/or taking long walks as practices important to relax. participants described the relief these activities provide, such as “one great benefit of being quarantined is home-cooked meals. cooking is like my form of therapy and one of the simple joys of life. i truly enjoy playing with different recipes/dishes and being able to eat mindfully everyday” (p18). another participant, who commuted from a rural town in new york state described their environment: “[my town] is a very quiet, small, farming town but we are glad that the pandemic finds us here, where we can go for walks without seeing many people” (p12). participants frequently mentioned stress, anxiety, and fear. nearly all participants discussed the negative mental health impact of covid-19, including helplessness, uselessness, anger, depression, devastation, disappointment, and feeling overwhelmed, scared, lost, and out of control. p22: i could feel the spring coming and things felt better when all of a sudden, we were hit with covid-19. the restaurant where i work as a server closed, and my world just turned upside down… it is scary to go outside, look around and realize that everything has changed. it feels like i am in a neverending nightmare and cannot wake up from it. i stopped watching the news because i got sad, stressed, depressed, anxious…my future feels so uncertain, even typing this makes me tear up and it fills me with an emptiness. some participants even described anticipating a less stressful time while acknowledging their current mental state. for these participants, faith (as a religious practice or a sense of connectedness to a greater universal force) emerged as important. p5: i am not doing fine at all, but i keep smiling every day and hoping for a better tomorrow. these months have been tough for me. i lost my job and i also lost a family member due to covid 19…i also take advantage of this free time to get closer to god and my loved ones. i believe that these things will go away soon, and everything will back to normal. i pray to find a job soon hopefully and pursue my master’s degree. despite emotional, physical, and financial stress, most participants privileged positive aspects of their experiences. participants were optimistic, resilient, and determined to persevere. many remained hopeful, grateful, and even-keeled. some even spoke explicitly about their emotional journey. p6: honestly, i am exhausted after this semester, but i am relieved and proud of myself for sticking it through to the end. this has been such a whirlwind for all of us, dealing with the effects of the pandemic, and to have successfully completed/ at the end of our classes, i am happy. this has strengthen[ed] my level of resilience as a person, and taught me that life will throw anything at you, so be ready to adjust and adapt to anything. i am optimism, resilience, and other health factors during pandemic 9 journal of effective teaching in higher education, vol. 4, no. 1 happy i had a chance to get to learn so much from [dr. a], looking forward to learning much more in the future. future-oriented participants described personal goals as something they continued to look forward to and plan for. some participants described looking forward for opportunities to travel, earning more academically, or gaining employment in nutrition, such as “i am so ready to work!!! i have been applying for job positions…i honestly can’t wait to work and feel productive again” (p15). social cohesion and social capital participants emphasized the importance of their social networks during the covid19 pandemic. despite the ways that living with family members complicated their schedules, many participants discussed the important role that family and friends played in helping them cope with the impacts of the pandemic. they also discussed support they received from classmates, professors, the college, and university system as a whole. p23: i want to say this semester has been the most stressful semester during my years of study at lehman [college]. this virus and the sudden change of everything (moving class online) etc. has made me realize that no matter how hard life gets, there will always be a way out, you just need to find it. the support provided by the [city university of new york] and you professors were heartwarming. this experience has taught me the love you all have for us and for me i want to say thank you and i’m very grateful. feeling connected to the academic community and empathy from professors was of particular importance for many participants. some even described how the support they received expanded their own capacity and compassion: “my empathy for professors and students alike has greatly increased. our vulnerabilities in many respects are exposed—and yet i see all of us rising to the occasion and finding ways through this” (p4). in more extreme examples, participants explicitly connected relief from depressive symptoms to the support they experienced from faculty. p14: i am still hurting and restless. right now, as i type this, it is may 27th, 2020 and the time is 3:40 am. i can’t sleep and i haven’t eaten properly [for weeks]. i am grateful that both you and [other professors] has heard me out and gave me advice that has caught me from falling into a deeper state of sadness and depression. i have shared everything with you more so [than other professors] and i don’t regret talking about it. i felt more relieved once i typed out my feelings and more ready to return to my schoolwork. discussion this study describes experiences of students (n = 28) at a bronx, new york-based, hispanic-serving institution while nyc was the national epicenter of the covid-19 pandemic. the sdh framework was used as a lens to better understand how optimism, resilience, and other health factors during pandemic 10 journal of effective teaching in higher education, vol. 4, no. 1 participants’ experiences fit with social determinants of health. as anticipated, there were clear negative impacts of the covid-19 pandemic in this sample of mostly bipoc living in the unhealthiest county in new york state. however, researchers did not anticipate the degree to which participants demonstrated optimism, resilience, and perseverance, which contributed to an overall positive mental state. several participants were employed as essential healthcare workers in nyc hospitals, a burden assumed disproportionately by bipoc (van dorn et al., 2020). thus, they may have perceived a more intense, continuous threat of covid-19 than non-essential/non-healthcare workers. stress and anxiety were almost palpable in how participants described experiences at work, confirming findings indicating high levels of stress and anxiety in college students (and college students of color, in particular) (molock & parchem, 2021; son et al., 2020). in addition, close living quarters in multi-generational households, a phenomenon more common to bipoc due to systemic racial inequity (airhihenbuwa et al., 2020), increased the risk of contracting coronavirus and, for participants in this study, made accomplishing tasks at home especially difficult. however, setting goals and staying busy during stressful events like covid-19 are promoted by leading health organizations as important self-care and stress-reduction techniques (covid-19, 2020, p. 19). remaining focused on academic achievement likely helped participants in this study create structure for managing time and gave them a sense of productivity while sheltering-in-place. it may have also served to reduce the negative psychosocial impact of being quarantined. the social support participants derived from the academic community (i.e, peers, faculty) was not anticipated and is not well described in current research. for the purposes of this study, social support is defined as behavioral action between individuals to offer emotional, instrumental, informational, or appraisal help for one individual from the other (glanz et al., 2015). this is in contrast with student services/support services, which are services provided by colleges and universities to students and are critical for student success (ciobanu, 2013). in this study, emotional support from the academic community seemed more consequential than the typical academic (i.e., learning) support provided by faculty to students. social support is important for physical and psychological health and can enhance resiliency to stress (ozbay et al., 2007). yet, the role of higher education anchor institutions as a provider of social support is not clear. more commonly, anchor institutions benefit communities by developing and hiring a local workforce, supporting local businesses, enriching local culture with arts and entertainment, contributing to community development, or increasing capacity for child and youth development (cruz, 2019; rubin & rose, 2015). this study is the first to reveal the potential of educational anchor institutions to provide significant and important social support for students. it is also possible that the academic environment became a critical lever of social support because the entire community moved online in mere days. other communities of support, like religious or faith-based communities, may not have had tools or resources to transition online and support congregants as quickly as the academic community did. in academics, communication between students and optimism, resilience, and other health factors during pandemic 11 journal of effective teaching in higher education, vol. 4, no. 1 faculty is expected in face-to-face interaction and incorporating that social interaction when transitioning online has been cited as a best practice, particularly during covid-19 (rapanta et al., 2020). that many faculty members already utilize online educational platforms to collaborate or communicate with students may have been an unanticipated benefit to bipoc. it is likely that continued regular contact with classmates and faculty members was important to participants’ reception of social support. shelter-in-place orders during covid-19 has been associated with increased anxiety and loneliness (tull et al., 2020), likely making increased social support on an already established platform even more important. ultimately, the support provided in academic communities may make educational anchor institutions serving high need communities even more impactful and important during public health crises. the results of this study indicate a need to thoroughly examine the potential for educational institutions to improve social support among bipoc communities. interestingly, income was not frequently discussed by participants in this study and there may be several reasons why not. first, many participants were healthcare workers that continued to earn income during the pandemic. secondly, many participants were enrolled full-time in order to be eligible for financial aid and may not be the primary income earners in their households. a third reason why income may not have been a focus is because lehman college offered financial relief for students in the form of monetary stipends to pay bills, rent, or other basic necessities. this support service may have alleviated some of the financial hardship participants experienced. finally, participants adopted new behaviors in response to sheltering-in-place and had new emotional experiences that had both positive and negative impacts. participants described implementing new time management and self-care practices, which likely positively contributed to coping and stress (covid-19, 2020). the negative psychosocial impact of covid-19 was more anticipated than positive psychosocial impacts because of early warnings from public health officials (centers for disease control (cdc), 2020, p. 19). however, stressors remain of concern, particularly for bipoc communities, who suffer disproportionately from stress as compared to their white peers (bulatao & anderson, 2004). bipoc experience a greater number of life stressors in general, the response to which is mediated, in part, by socioeconomic status. that is, low-income bipoc experience more distress than their higher income bipoc peers (bulatao & anderson, 2004). since access to health care, including mental health services, for bipoc is more limited than in affluent or whiter communities (world health organization, 2010), the negative psychosocial impact of public health crises, like covid-19, in bipoc communities warrants more attention. an unanticipated but important finding of this study are participants’ positive attitudes. resilience, optimism, and perseverance were prevalent throughout participants’ responses, which may have a protective effect against the negative impacts of sdh (palmer et al., 2019). yet, health promotion initiatives are often focused on reducing risks or exposure rather than promoting resilience (davis et al., 2005). examining resiliency has recently been prioritized in bipoc communities optimism, resilience, and other health factors during pandemic 12 journal of effective teaching in higher education, vol. 4, no. 1 for its importance to mitigate the adverse effects of sdh on individuals (palmer et al., 2019). while this study did not pre-suppose any positive psychosocial impacts of the covid-19 pandemic, it was apparent that most participants experienced these protective factors. individual resilience related to recovery in combat veterans experiencing post-traumatic stress disorder (meyer et al., 2019), and community resilience following weather-related disasters (chan et al., 2015) is fairly common. however, the potential for resilience to be protective during public health crises is not well explored. bipoc, in general, demonstrate more optimism than their white peers at all socioeconomic levels (graham, 2020). while the reasons for this aren’t entirely understood, optimism is a strength found in the current study and the potential to capitalize on optimism to promote resilience warrants attention. future communitybased participatory research should measure community resilience using available tools, like the thrive toolkit, to assess communities’ built environment, social capital, services and institutions, and structural factors (davis et al., 2005). doing so may serve to highlight communities’ strengths and reveal areas of weakness that can be used to identify and prioritize goals to create lasting structural changes to reduce health disparities (brown et al., 2019; davis et al., 2005). limitations despite having a relatively high number of participants for a qualitative study, the results from the specific population studied are not necessarily generalizable. participants in this study were undergraduate health sciences students in a dietetics major; other students’ experiences may differ. while this study was primarily concerned with the experiences of bipoc students, two students who identified as white were included (as indicated in the methods section). in addition, the data used for this study was from a final exam measuring course content. a data collection method more focused on gathering information about the impacts of covid-19 on the student experience may reveal additional information. future quantitative and qualitative research is needed to better understand the depth, breadth, and generalizability of the findings. lastly, research that explores the potential impact of institutional or faculty policies to support students and promote engagement and learning would also benefit the student experience at higher education institutions, particularly those that serve low-income bipoc. implications for practice this study identifies the potential impact of educational anchor institutions as critical levers of social support for students and found that optimism, resilience, and perseverance may be observed in low-income bipoc during shelter-in-place orders. the mental health impact of covid-19 on bipoc and healthcare student workers (and other essential workers) should be considered by academic institutions when seeking to support students through public health crises using a two-pronged approach addressing institutional policies and pedagogy. optimism, resilience, and other health factors during pandemic 13 journal of effective teaching in higher education, vol. 4, no. 1 educational anchor institutions should consider practical supports that students need to ease the burden of remote learning and reduce financial hardship via technology assistance and access, monetary support, mental health services, and academic flexibility. access to adequate computers and high speed internet are critical to efficient and effective remote learning. implementing laptop loan programs or providing internet hotspots to students can provide them with the technology they need to support their learning. particularly for low-income students of color, financial strains that result from decreased opportunities to work as a result of shutdowns and stay-at-home orders can add undue stress. examples of strategies to provide financial relief include offering small monetary stipends for immediate, basic needs (e.g., housing, food, bills) or forgiving holds on students’ accounts (which may typically prevent students from persisting in their educational pathway). ensuring that financial aid offices are adequately staffed during times of financial strain may improve processing times. mental health services, like other college services, should be made available to students remotely and flexibly; that is, there should be flexible cancellation policies, pay options (e.g., a sliding scale or free sessions), and/or timing of appointments. finally, offering academic flexibility through policies such as extended course withdrawal deadlines and/or an option to convert a course from a letter grade to pass/fail are institutional strategies to support student learning. in addition to institutional supports, individual faculty have opportunities to support students by demonstrating empathy, prioritizing self-care, increasing flexibility, and becoming familiar with best practices in online pedagogy. as found in this study, faculty who demonstrate empathy for their students provide an opportunity for students to articulate how they’re doing and to feel like someone is listening. it gives faculty the opportunity to identify students who are struggling and liaise with mental health services, technology assistance, student success services, or other college services and resources. moreover, faculty should prioritize self-care and talk about stress management with students, as it may be advantageous to forge stronger relationships and to model positive behaviors. in essence, talking about stress and stress management techniques may help normalize and ultimately reduce it. faculty should also consider ways that they can create flexibility in courses, such as with assignment deadlines (which may be of particular help to students who struggle with reliable internet or who are unexpectedly caring for children who are also learning remotely). pedagogical practices that promote learning in a remote environment should be considered and adopted. while this may create additional work for faculty to implement (e.g., to record lectures in advance or learn new technology, like voicethread), adjusting course delivery methods and expectations for synchronous class time may be more supportive and promote student engagement. these suggestions, based on the current study, are limited, and there are likely numerous ways that institutions and faculty can support students. there may be opportunity for educational anchor institutions and faculty to expand their capacity and capability to support students in unconventional ways, particularly during unconventional, unprecedented times. optimism, resilience, and other health factors during pandemic 14 journal of effective teaching in higher education, vol. 4, no. 1 conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references airhihenbuwa, c. o., iwelunmor, j., munodawafa, d., ford, c. l., oni, t., agyemang, c., mota, c., ikuomola, o. b., simbayi, l., fallah, m. p., qian, z., makinwa, b., niang, c., & okosun, i. (2020). culture matters in communicating the global response to covid-19. preventing chronic disease, 17. https://doi.org/10.5888/pcd17.200245 aucejo, e. m., french, j., ugalde araya, m. p., & zafar, b. (2020). the impact of covid-19 on student experiences and expectations: evidence from a survey. journal of public economics, 191, 104271. https://doi.org/10.1016/j.jpubeco.2020.104271 brown, a. f., ma, g. x., miranda, j., eng, e., castille, d., brockie, t., jones, p., airhihenbuwa, c. o., farhat, t., zhu, l., & trinh-shevrin, c. (2019). structural interventions to reduce and eliminate health disparities. american journal of public health, 109(suppl 1), s72–s78. https://doi.org/10.2105/ajph.2018.304844 browning, m. h. e. m., larson, l. r., sharaievska, i., rigolon, a., mcanirlin, o., mullenbach, l., cloutier, s., vu, t. m., thomsen, j., reigner, n., metcalf, e. c., d’antonio, a., helbich, m., bratman, g. n., & alvarez, h. o. (2021). psychological impacts from covid-19 among university students: risk factors across seven states in the united states. plos one, 16(1), e0245327. https://doi.org/10.1371/journal.pone.0245327 bulatao, r. a., & anderson, n. b. (eds.). (2004). stress. in understanding racial and ethnic differences in health in late life: a research agenda. national academies press (us). http://www.ncbi.nlm.nih.gov/books/nbk24685/ centers for disease control (cdc). (2020, february 11). coronavirus disease 2019 (covid-19). https://www.cdc.gov/coronavirus/2019-ncov/daily-lifecoping/stress-coping/index.html chan, j., dubois, b., & tidball, k. g. (2015). refuges of local resilience: community gardens in post-sandy new york city. urban forestry & urban greening, 14(3), 625–635. https://doi.org/10.1016/j.ufug.2015.06.005 ciobanu, a. (2013). the role of student services in the improving of student experience in higher education. procedia social and behavioral sciences, 92, 169–173. https://doi.org/10.1016/j.sbspro.2013.08.654 optimism, resilience, and other health factors during pandemic 15 journal of effective teaching in higher education, vol. 4, no. 1 colleges with the highest student-mobility rates, 2014. (2018, august 19). the chronicle of higher education. https://www.chronicle.com/article/collegeswith-the-highest/244094 covid-19 and your mental health. (2020, april 2). mayo clinic. https://www.mayoclinic.org/diseases-conditions/coronavirus/indepth/mental-health-covid-19/art-20482731 cruz, j. l. (2019). lehman college: a mission anchored in our history. journal on anchor institutions and communities, 2, 4–12. davis, r., cook, d., & cohen, l. (2005). a community resilience approach to reducing ethnic and racial disparities in health. american journal of public health, 95(12), 2168–2173. https://doi.org/10.2105/ajph.2004.050146 dedoose (7.0.23). (2016). [computer software]. sociocultural research consultants, llc. http://www.dedoose.com espinosa, l. l., kelchen, r., & taylor, m. (2018). minority serving institutions as engines of upward mobility (p. 24). american council on education, center for policy research and strategy. freytas-tamura, k. de, hu, w., cook, l. r., & bhaskar, g. (2020, may 26). ‘it’s the death towers’: how the bronx became new york’s virus hot spot. the new york times. https://www.nytimes.com/2020/05/26/nyregion/bronxcoronavirus-outbreak.html garcia, s. e. (2020, june 17). where did bipoc come from? the new york times. https://www.nytimes.com/article/what-is-bipoc.html glanz, k., rimer, b. k., & viswanath, k. (eds.). (2015). health behavior: theory, research, and practice (5th ed.). jossey-bass. graham, c. (2020). happiness for all?: unequal hopes and lives in pursuit of the american dream. princeton university press. husky, m. m., kovess-masfety, v., & swendsen, j. d. (2020). stress and anxiety among university students in france during covid-19 mandatory confinement. comprehensive psychiatry, 102, 152191. https://doi.org/10.1016/j.comppsych.2020.152191 lehman college interactive fact book. (n.d.). tableau software. https://public.tableau.com/views/lehmancollegeinteractivefactbook/startme nu?:embed=y&:showvizhome=no&:host_url=https%3a%2f%2fpublic.table au.com%2f&:tabs=yes&:toolbar=no&:animate_transition=yes&:display_stat ic_image=no&:display_spinner=no&:display_overlay=yes&:display_count=y es&:loadorderid=0 optimism, resilience, and other health factors during pandemic 16 journal of effective teaching in higher education, vol. 4, no. 1 maloney, e. j., & kim, j. (2020). the challenge of equity in higher education under covid-19. inside higher ed. https://www.insidehighered.com/blogs/learning-innovation/challenge-equityhigher-education-under-covid-19 mchugh, m. l. (2012). interrater reliability: the kappa statistic. biochemia medica, 22(3), 276–282. meyer, e. c., kotte, a., kimbrel, n. a., debeer, b. b., elliott, t. r., gulliver, s. b., & morissette, s. b. (2019). predictors of lower-than-expected posttraumatic symptom severity in war veterans: the influence of personality, self-reported trait resilience, and psychological flexibility. behaviour research and therapy, 113, 1–8. https://doi.org/10.1016/j.brat.2018.12.005 molock, s. d., & parchem, b. (2021). the impact of covid-19 on college students from communities of color. journal of american college health: j of ach, 1– 7. https://doi.org/10.1080/07448481.2020.1865380 opam, k. (2020, march 24). it’s not ‘shelter in place’: what the new coronavirus restrictions mean. the new york times. https://www.nytimes.com/article/what-is-shelter-in-place-coronavirus.html ozbay, f., johnson, d. c., dimoulas, e., morgan, c. a., charney, d., & southwick, s. (2007). social support and resilience to stress. psychiatry (edgmont), 4(5), 35–40. palmer, r. c., ismond, d., rodriquez, e. j., & kaufman, j. s. (2019). social determinants of health: future directions for health disparities research. american journal of public health, 109(suppl 1), s70–s71. https://doi.org/10.2105/ajph.2019.304964 rapanta, c., botturi, l., goodyear, p., guàrdia, l., & koole, m. (2020). online university teaching during and after the covid-19 crisis: refocusing teacher presence and learning activity. postdigital science and education. https://doi.org/10.1007/s42438-020-00155-y rubin, v., & rose, k. (2015). strategies for strengthening anchor institutions’ community impact (p. 11). policylink. social determinants of health. (2010). world health organization. license cc bync-sa 3.0 igo. son, c., hegde, s., smith, a., wang, x., & sasangohar, f. (2020). effects of covid19 on college students’ mental health in the united states: interview survey study. journal of medical internet research, 22(9), e21279. https://doi.org/10.2196/21279 optimism, resilience, and other health factors during pandemic 17 journal of effective teaching in higher education, vol. 4, no. 1 tull, m. t., edmonds, k. a., scamaldo, k. m., richmond, j. r., rose, j. p., & gratz, k. l. (2020). psychological outcomes associated with stay-at-home orders and the perceived impact of covid-19 on daily life. psychiatry research, 289, 113098. https://doi.org/10.1016/j.psychres.2020.113098 united states census bureau. (n.d.). bronx county, new york. https://data.census.gov/cedsci/profile?g=0500000us36005&q=bronx%20co unty,%20new%20york van dorn, a., cooney, r. e., & sabin, m. l. (2020). covid-19 exacerbating inequalities in the us. lancet (london, england), 395(10232), 1243–1244. https://doi.org/10.1016/s0140-6736(20)30893-x world health organization. (2010). a conceptual framework for action on the social determinants of health: debates, policy & practice, case studies. world health organization. retrieved june 27, 2020, from http://apps.who.int/iris/bitstream/10665/44489/1/9789241500852_eng.pdf microsoft word yeo+et+al_jethe_final.docx journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.232 a play in three acts: confused, conflicted, and convinced (learning to reflect in athletic therapy) michelle yeo, mount royal university, myeo@mtroyal.ca mark lafave, mount royal university abstract. in some fields, written reflection is commonplace whereas in others it is uncommon. while athletic therapy education aims to produce reflective practitioners, written reflection is not a typical pedagogy employed. in 2014, the athletic therapy program at our institution began the implementation of a clinical presentation (cp) approach to facilitate competency-based curriculum requirements. this innovation to pedagogy required a reimagined approach to teaching, learning, and assessment. we describe one aspect of a larger sotl study on this transformation, inquiring into the development of reflective practice through reflective writing. students were asked to regularly reflect on their experiences in the clinic or field as part of their program. in this qualitative component of the study, we were able to gain insight into how students perceived the reflective process, how that evolved over their program, what were enablers and barriers to their reflection, and what was the role of feedback in their learning. the characteristics of student perceptions in each year, which followed a learning arc which we describe sequentially as “confused, conflicted, and convinced,” is explored, along with implications for pedagogy in assisting students to develope reflective professional practice. keywords: reflection; reflective writing; professional education; qualitative research; athletic therapy. in recent years, there has been a great deal of interest in reflection in education in medical and allied health professions. the importance of reflective practice is consistently noted as “an essential characteristic of professionally competent practice” (wald & weiss, 2018, abstract). wald (2015) pointed to reflection as a “key element” to professional identity development, and this is even more important for increasingly complex patients and environments (uygur et. al, 2019). in order to encourage reflective habits and skills, reflective writing is often the vehicle educators turn to in assigning students structured reflection. in a systematic review, chen and forbes (2014) noted that despite program variability in which reflective writing is used, “the review illustrated consistent, positive outcomes of self-reflection on student well-being and clinical skills” (p. 4). they also pointed out the widespread use of reflective writing across programs and that reflective writing can be used as “both intervention and metric…reflective writing can be seen as both a test that teaches and measures empathy changes” (p. 4). reflective writing is not a common pedagogical tool in athletic therapy (at) programs. this study is a qualitative look at the implementation of reflective writing across multiple years of an at program and students’ perceptions of their learning. the study takes a close look at 10 students who participated in interviews about the a play in three acts: confused, conflicted and convinced 133 journal of effective teaching in higher education, vol. 4, no. 3 reflective process and analyzes their final yearly reflections as a second source of data. backstage: literature on reflection why reflection, and why reflective writing? the literature consistently asserts that reflective writing can be a powerful tool to teach some of the non-technical knowledge in the medical and allied health professions. for example, reflective writing is thought to improve awareness of self and others while also increasing powers of observation (reisman et al., 2006). it is thought to help students to process difficult experiences and encourage a growth mindset. reflective traits can be cultivated and learned with effort (teunissen & bok, 2013). research has shown a link between reflective capacity and clinical decision making as well as enhanced academic performance (tsingos-lucas et al., 2017). reflection “fosters empathy and practical wisdom” and helps practitioners to live with complexity (karkabi et al., 2014, p. 44). as charon et al. (2016) point out, “to write is not only to report or record but also to discover” (p.4). they suggest that reflective writing and other creative activities can help students with the quality of their attention: “teaching clinicians the skills of the close reading of literature, creative writing, and the viewing of fine arts can strengthen their habits of ‘close listening’ or ‘slow looking,’ thereby improving their quality of perceptive attention” (p. 4). yet students struggle with reflective writing: why? students’ struggles with reflection are well documented and not unique to the at discipline (e.g., wald, 2015), particularly in the beginning. one possible reason for this might be how they approach processing experiences and seeking feedback. for example, teunissen and bok (2013) describe differences in learner approaches to difficult challenges, comparing those with a learning orientation (who actively seek constructive feedback) to those with performance goal orientation (who exhibit more limited feedback seeking behaviour). despite the challenge, the literature also supports the idea that many students find the reflection process valuable (tsingoslucas et al., 2016). in particular for at students, there may be a learning preference element. using a multiple intelligences framework, kutz et al. (2013) found that at students tend to be drawn to the profession due to its kinesthetic components and scored lowest on verbal intelligence, which may contribute to their struggles with written reflection. coker (2000) notes a learning preference in clinical settings for hands-on learning. while the “learning-style” theory has been debunked (brown, 2014), there are still preferences in individuals that may draw them to specific professions as they prefer certain modalities, while avoiding others. a play in three acts: confused, conflicted and convinced 134 journal of effective teaching in higher education, vol. 4, no. 3 how to support students in learning to write reflectively and to help them learn from reflection? various methods are proposed both in terms of structuring the writing for students and in terms of assessing it. some programs have experimented with aesthetic responses to artistic forms, such as opera or paintings, to evoke emotion. other programs work to create rubrics or other systematic approaches to assessing and responding to reflective writing (wald et al., 2009). tsingos-lucas et al. (2017) suggest that “perhaps the greatest pedagogical challenge or educators of health professionals is how we can help them to write reflectively” (p. 6), and driessen et al. (2008) articulate that “students do not adopt reflective learning habits spontaneously, so teachers must help them” (p. 827). however, these kinds of reflections must be carefully structured to have the desired effect. driessen et al. (2005) found that four conditions must be met for success: good coaching; structure and guidelines; adequate experiences and material for reflection; and summative assessment. they point out that “students simply completing reflective assignments is no guarantee that reflection will occur” (p. 1230). in summary, reflection is a critical aspect of effective clinical practice. however, we cannot count on students intuitively knowing how to do it nor that it is intuitive for instructors to teach and assess effectively. in this study, we gained insight into how students perceive the process, how that evolves over their program, what are enablers and barriers to their reflection, and what is the role of feedback in their learning. setting the stage: context of the study in 2014, athletic therapy (at) faculty members at our institution agreed to implement a clinical presentation (cp) approach to facilitate competency-based curriculum requirements (lafave et al., 2016). this innovation to pedagogy required a reimagined approach to teaching, learning, and assessment (yeo et al., 2017). we describe one aspect of a larger study inquiring into the development of reflective practice through reflective writing, and, thus, this sotl study will be of interest to professional programs and disciplines beyond at. in addition to the classroom pedagogical change in how material is presented (starting with an injury and working backwards to learning about the body’s structure and function), students were asked to keep logbooks throughout their program. the logbooks contained lists of clinical presentations, and students kept track of what they were exposed to in their classes, clinical, and field placements and rated their confidence and competence in treating those injuries (lafave & yeo, 2019). key for the present discussion, they were also asked to regularly reflect on their experiences in the clinic or field. this culminated in a final reflection for each year. this article focuses on what we learned about the development of student reflection across the program. in the early phases of the implementation of the new pedagogy, instructors noticed students struggling with the reflective activity. we wondered about potential causes of this resistance, such as cognitive overload, difficulty in understanding the value a play in three acts: confused, conflicted and convinced 135 journal of effective teaching in higher education, vol. 4, no. 3 and process of reflective practice, dominant kinaesthetic orientation to learning for many at students (kutz, et al., 2013), affective aspects (middendorf et al., 2015), and cultural factors within the profession of at. in our interviews, we asked students about their reflective process and have begun to learn about their challenges. for example, participants felt unsure about what was expected from the reflective process, sometimes had difficulty identifying appropriate events to reflect about, and were challenged to dedicate time to reflection while engaged in practical learning experiences. we noticed a clear progression in the students’ responses to the reflective practice over the three years, demonstrating a learning arc over several years. we will describe this learning arc and make subsequent recommendations for practice based on these results. set design: methodology the participants were drawn from three successive cohorts of athletic therapy students that we started following in their second-year practicum course. ten students were included in the study because they agreed to be interviewed (in addition to providing written work as described below). in other words, it was a convenience sample—all students had equal opportunity to participate. the data was collected over three years; thus, for the first cohort, we have interview data following them to graduation (please see table 1 for summary). all students were required to use a student logbook wherein they tracked their exposure to a wide range of clinical presentations, along with their confidence in handling these specific injuries in practicum courses each semester. confidence was self-administered twice a semester, once at the start and once at the end, using a 100 millimetre visual analogue scale with “extremely confident” at one anchor and “not confident at all” at the other end. confidence was not a focus of this study but rather was reported previously (see lafave & yeo, 2019). students, as part of their coursework, were regularly asked to reflect on their experiences and the meaning they were making of them. for example, they might reflect on an injury or treatment, they might consider elements of practice such as interprofessional communication, or they may reflect about the interaction of those two topics. topics were often directed by where the course was placed in the curriculum so there was alignment between theory courses and the practicum course that was intended to bridge theory to practice. based on feedback from students, the number of reflections students were required to complete were adjusted over time. originally, students were being asked to reflect on three clinical presentations each week in clinic and field settings. this was overwhelming and, therefore, reduced to a submission of 10 clinical presentations over a 4-month span. all students were required to complete these logbooks and reflections as part of their program; only the students who consented to participate in the research component were asked to submit their logbooks for analysis. the instructors of the courses were unaware of which students participated and which did not. this a play in three acts: confused, conflicted and convinced 136 journal of effective teaching in higher education, vol. 4, no. 3 methodology was approved by mount royal university’s human research ethics board. the present qualitative analysis focuses specifically on the reflective element, and specifically, the student perceptions of the process. this element of the study uses a basic qualitative approach, as defined by merriam and tisdell (2016). it is a common approach in applied fields of practice such as education, implementing a constructivist lens. qualitative research assumes that “individuals construct reality in interaction with their social worlds” and researchers tend to be interested in how participants “make sense of their lives and their experiences” (merriam & tisdell, 2016, p. 24). quantitative results for the cp logbooks are presented elsewhere (e.g., lafave & yeo, 2019). at the end of each year of data collection, participating students were invited to take part in a research interview. a smaller number of students (10 unique individuals with a total of 12 interviews) came forward for interviews then participated in the larger study; however, this data proved to be qualitatively rich. the present analysis focuses on the interview data, along with the summative written reflections produced by the students interviewed at the end of each year. the following table summarizes the interviews conducted. as shown in table 1, two of the students were interviewed more than once. the letter codes are not personally identifying initials, rather they are codes given to participants according to an internal system to help track their year and program. table 1 summary of interviews conducted first cohort second cohort third cohort winter 2016 aw (2nd year) winter 2017 bb (3rd year) bh (3rd year) ck (2nd year) cl (2nd year) cn (2nd year) winter 2018 bb (4th year) bh (4th year) an (4th year) el (2nd year) eq (2nd year) er (2nd year) totals 7 second-year interviews, 2 third-year interviews, 3 fourth-year interviews the 12 interviews were recorded and transcribed, and the researchers independently analyzed the interviews thematically and met to compare. we agreed that saturation was achieved in the interviews, indicating a sufficient qualitative data set (merriam & tisdell, 2016, p. 101). while initially looking for a common set of themes across all interviews, the researchers agreed once they conducted their analysis that they noticed a clear progression evident over the three years of the a play in three acts: confused, conflicted and convinced 137 journal of effective teaching in higher education, vol. 4, no. 3 students’ program in terms of their perceptions of the reflective process, and their ability to reflect. the characteristics of each year, which we describe sequentially as “confused, conflicted, and convinced,” along with increasing sophistication in the reflections themselves were well evident across the data, along with participants’ growing sense of independence as professionals. this progression is more prominent than any consistent themes across all three years, other than one specific theme around feedback which transcended stage in program. additionally, the final written reflections of the students interviewed were reviewed to triangulate what was said in the interview with the reflections themselves. this provided additional insight. students’ relationship with reflection — a play in three acts act i: confusion the students interviewed in the second year about the reflective process seemed confused overall about the intention and benefit of writing reflections and were frustrated in attempting to understand what their course instructors “wanted” in order to receive full marks for the reflection pieces. as found in the literature, students initially found this type of writing challenging (tsingos-lucas et al., 2017). they were uncertain about the core purpose of the logbooks and reflections, many thinking that they were primarily for research or curriculum development purposes. in fact, all students were asked to do this as a core pedagogy of the program intended to help them track and confirm their progress as well as build reflective practice. students expressed consternation about the marks they received, and much of their response to interview questions focused on the assessment aspect of the interviews. they saw a disconnect between what they understood they were being asked to do and how they perceived that they were evaluated. they saw the reflections as highly subjective and wondered if it was fair to be “marked” on a personal experience. cn expressed the following: i would say some of the reflections can be good, but, even then, it is hard when you are getting marked on a personal experience, like, to be marked on your opinions and your thoughts and whatnot, and you have a grade put on top of that. it is frustrating, i guess, because you are not sure if…eventually you just start putting down what they want to hear, not as much as like, just what you are seeing or your thoughts and feelings of the situation is. i think it can be a bit difficult in that sense. at this stage of the reflective process, students interviewed often used the term “frustrating” to describe their reflective experience. several students talked about trying to “figure out” or “put down” what “they” [their instructors] wanted to hear. this observation supports the contention in the literature that “…reflection is not necessarily intuitive, and educational interventions are warranted” (wald & reis, 2010, p. 747). we discuss this further in the section regarding feedback and assessment. a play in three acts: confused, conflicted and convinced 138 journal of effective teaching in higher education, vol. 4, no. 3 because the course utilized the reflection journal in the second year related to “soft” skills (for example effective communication, contrasting with the “hard” skills of assessing and treating injuries), students also expressed annoyance at having to focus exclusively on soft skills for reflection purposes. there was a sense that this was not where the “real” athletic therapy happened. they found the restriction to soft skills challenging, seeing these skills as peripheral (eq, cl). they would make a general acknowledgement that soft skills were “important,” yet seemed anxious to turn their attention to what seemed more exciting to them—the hard skills. they often expressed that they had difficulty choosing something “meaningful” (the assignment direction). they expressed that they were often “making stuff up” (cl)—stretching an event to write about it and trying to make things meaningful that were not really meaningful to them. at times, this seemed related to frustration at their placements, because as novices, they were often expected to “stand around” since they did not have sufficient skill yet to treat. they perceived that the “reflection is more for you guys, but also us,” demonstrating their general confusion regarding the purpose of the reflections. in some interviews, contradictory statements were made. they were beginning to acknowledge the potential benefit of reflection, however housed within frustration: oh, i think so, yeah, definitely. whether it was in the fact it made me reflect on my experiences or whether it was because you guys now had something to maybe work to improve the program on, either way i think it definitely does help our learning… sometimes you are writing these reflections and you are just like, “oh, this is so stupid, why do i have to do this?” but really it did make us think really deeply about what we learnt from that experience and… it really made me realize what i still have to learn, definitely. and it also made me recognize how important those soft skills are in this career, because i know i went into it being like, “oh practicum? i am going to learn how to do all this cool stuff,” like the hard skills, and it was totally more soft skills that were the big takeaway. (cl) in this quote we can see the kind of circular statement that was typical in the second-year cohort interviews—confusion about the purpose of the reflections, expressions of frustration, but also acknowledgement of value. students described getting “the hang” of writing the reflections over the semester, particularly those that paid close attention to feedback and were motivated to improve their mark on the reflections. those that took the time to meet with the instructor seemed to have breakthrough insights in terms of what was being asked, for example, providing more detail and making connections to course material. the reflections themselves in the second-year cohort are characterized by direct observations and thoughts about what was correct or incorrect but little in terms of self-aware reflections about themselves. finally, this phase is marked by a lack of agency over their own learning by the students. the emphasis was on “what the instructors want” and how clearly that a play in three acts: confused, conflicted and convinced 139 journal of effective teaching in higher education, vol. 4, no. 3 was communicated, rather than on the students’ ability to see reflection as part of the learning itself and as a process by which they would become reflective practitioners and come to their own insights about their experiences. act 2: conflicted the third-year interviews were characterized by a sense of internal conflict. students contradicted themselves frequently—first expressing frustration with the reflective process and then, sometimes in the same statement, acknowledging its value. in this sense, the third year seemed a transitional phase between “confused” and “convinced,” with students rocking back and forth between the two. sometimes it is a bit of a pain, but i think it was a good thing to be able to do that so that you can really learn from what you did, and learn from things that you might have missed. did you miss it because you just simply missed it, or you are already taking those steps into, “okay, i don’t have to do this because i know this already”? so it kind of gets more feedback to yourself as to how you are learning through the year. (bb) in these interviews, there was less of a focus on what the instructor “wanted” but still not a fully internalized sense of agency in learning. the focus seemed to shift to the rubric rather than the inside of the instructor’s mind. it is just hard to…like, the reflections are real life—or they should be based off real life—and the rubric is structured. i get why and i understand what they want, but it is hard for you to grab each one of those points out of the rubric out of each experience. sometimes you have to think back, like, “maybe i was thinking this?” or, “maybe this is why i did this?” “maybe i know this from class because…” like some things don’t go through your head when you are doing things, and you don’t get the marks from the rubric. (bh) this same student, however, who continued to struggle with identifying what was most meaningful, also spoke about how the reflections can help with the learning process: i think every once in a while, yes, some things just, like, it just clicks. i felt like especially early on you could think back when you did something and it was completely wrong, or you felt something and you were like, “i don’t know what that was, that was weird,” and then two months down the road you are like, “oh, that is what that was!”(bh) bb made a very similar statement: well sometimes there was stuff i didn’t really understand maybe as i was writing it, but as we learned it in class material, putting those pieces together that i didn’t know before made it a lot easier to understand why that was happening…it is good to be able to look back at that and be like, “okay, i didn’t know that then, but i might know it now.” (bb) a play in three acts: confused, conflicted and convinced 140 journal of effective teaching in higher education, vol. 4, no. 3 this student also made a comment that reflecting became a useful record for later, a kind of way to jog learning that had occurred earlier but was submerged and not easily accessible, and realized, “okay i do know this… this is something i might have forgot, and this probably one of the more important things you are supposed to do!” this comment speaks to the importance of having a structured mechanism for having students go back and review their own reflections to help consolidate the learning. a final aspect of the “conflicted” nature of this stage was that while students valued the feedback they received, they commented that it was difficult to know how to apply it. they started to realize the importance of including theoretical components from class, rather than making their reflections purely observational. for example, bb wrote in reflecting about assessing a shoulder injury: in our discussion…it was evident that i had selected some good tests, my performance however wasn’t always top notch. this doesn’t come as a surprise for me, i haven’t seen many shoulder injuries, and have thus not practiced as much as i should be. i am definitely going to make this a goal (if i haven’t already) for this semester to improve on. note to self: do not fear the shoulder, embrace it openly. in these reflections, wonderings about the ramifications around professional decision-making were more apparent: this was the first time i had felt what i thought was a positive meniscus test, which was cool to see and feel, but was surprised that i could not produce a positive valgus test. this was also the first time i had had to decide whether or not someone was alright to wrestle during a competition, and i am always very cautious to put them back in, but especially for wrestling because of the way their joints get torqued on. i was quite happy that he was fine during his match, but wonder if anything would have changed if i found the positive meniscus test earlier. (bh) this was an example of a melding of the description of what was done and felt by the student, including their excitement of feeling something they hadn’t been exposed to before, with the weight of responsibility of deciding whether an athlete can play or not. while the students themselves may feel conflicted about the value of the reflections, in reading them from an educational perspective, their value is clearly apparent. as pointed out by tsingos-lucas et al. (2017), “research has shown that fostering reflective skills in health professional education can assist students to improve their clinical decision-making skills and enhance academic performance…the ability to reflect on a deeper level is a desirable attribute for all health professionals” (p. 1). the value is not only for the student to articulate and deepen their understandings but to provide a window into their learning for the instructor as the basis for a conversation. a play in three acts: confused, conflicted and convinced 141 journal of effective teaching in higher education, vol. 4, no. 3 act 3: convinced fourth-year interviews have a different tone and character with students appearing to have crossed through the liminal space to practitioner. while still aware of their early career status, they seem to think about their own learning as something under their own agency and think about how to continue to progress regardless of external expectations. there seems to be a full realization of the purpose of the reflection process: interviewer: seeing the benefits from this, do you think it might be something to carry forward, that reflective practice into professional practice? p: it would be a really good idea! i know for sure, the first little while, i won’t do it because like, i had a love/hate relationship with those reflections, but it would be a useful thing, for sure…i might not be getting feedback…but reflecting on it, for sure, helped me learn. like even as i am writing i would say, like, yeah this is what i found and this is what i did, and then i would be like, oh, wait a second, why didn’t i do this? and then halfway through the reflection, like ninety percent of the time, i would say, “upon reflection i realize i should have…” blah, blah, blah and so that is just me learning on my own, which is probably exactly what it is designed to do. (an) an reflected in multiple statements in his interview about how far he had come in his learning. he makes the statement that in terms of his practice, “i feel confident in anything that walks through the door.” students could see their own growth when they went back over their reflections to write cumulative statements. an shared, “i actually spent time trying to reflect and learn something from it.” there is a sense that they are surprised by their own growth and surprised that, despite the frustrations, they could, in hindsight, see the value of the reflective process. in these interviews, we can see a leap in terms of students’ sense of agency and responsibility. while appreciating positive feedback and guidance from mentors and instructors, there is a distinct looking inward for answers, even if it’s not perfect. bh comments: i know we are not amazing right now, but we are not supposed to be, and it still seems like there is so much we don’t know. but at this point we have done everything and practiced everything at least a little bit, yeah, like we can probably do most things, just not amazing. it is just practice, experience, time… bb wrote in the final reflection both about this growing sense of agency and excitement at becoming more independent while, at the same time, continuing to ask for guidance when needed from mentors and peers: i plan on writing [the canadian athletic therapists association certification exam] in june, i am excited to have this opportunity and hope to pass so that i can get out in the real world and really start practicing on my own. this does a play in three acts: confused, conflicted and convinced 142 journal of effective teaching in higher education, vol. 4, no. 3 somewhat terrify me because i won’t have someone to relay my information to and have them okay it or challenge me to dig deeper. however, i think that contributes to the thrill of it all. i can’t say that i will never have help again because let’s face it throughout this program i believe i have been exposed to many great mentors that i’m sure will help with guiding me should i ask for it in the future. even in our peers, i can always bounce ideas off of them so long as some of us stay in contact and can meet or even just communicate regularly. which, i am more than sure we will. this student demonstrated the sense of having become convinced of the value of reflection when asked for advice to students coming in to the fourth year: keep doing them. you might not like it, but keep doing it. well it is always hard because you don’t want to write about the same thing every time so it is figuring out the one thing that was different in that week…don’t be afraid to be jumping on a spine because that is the only way you are going to learn how to do it, and you have to write about it anyways, so don’t be afraid of it. you might not like it, you might get confused all the time, but you will figure it out! in this sense, the written reflections can act as a prompt for action rather than as a post-learning from action taken. students throughout the interviews mention how knowing they will have to reflect on certain things acts as an incentive for observation—focusing their attention in particular directions. in this sense, reflective assignments can become a powerful pedagogical tool. as charon et al. (2016) point out, “to write is not only to report or record but also to discover.” (p. 4) another aspect that was visible in an’s final written reflection was the meta-level learning of pattern recognition and how that led to the confidence expressed above: “i feel confident in anything that walks through the door.” in his reflection, an writes the following: each week as i have gone through the list of cp’s [clinical presentations] in my logbook checking off what i’ve been exposed to, i’ve seen myself not only get exposure to more and more of the cp’s, but grow in confidence in each one. one of the biggest things that has grown my confidence is the fact that many rehabilitation techniques and strategies are common/relevant for a wide range of assessment findings. as i’ve been exposed to more and more real-life cp’s, i have realized that i have enough tools, strategies, confidence, and skills to handle any combination/presentation of the cp’s i’ve learned about over the past two years… thus, over the three years, we argue that the reflections have immediate value in deepening the learning in the moment, making visible the growing meta-cognitive learning and agency, and, finally, acting as a prompt to seek out and observe learning opportunities. there is a real sense in which reflecting both is a measure and instigator of learning. a play in three acts: confused, conflicted and convinced 143 journal of effective teaching in higher education, vol. 4, no. 3 theme through all acts: feedback and assessment throughout all three years, an overarching theme from the interviews with students was the role of feedback and assessment of their reflections. for these students, the motivation to complete the written reflections was the mark that they received. for many, improving the mark was a high priority for them—they were not satisfied with a mediocre grade. they struggled to understand what the instructors were looking for and struggled to align experience with the rubric. they felt frustrated that subjective personal experience and opinion was subject to a mark. it was clear that good assessment practices, such as timely, detailed, constructive feedback was critical. those students who took the initiative to meet with the course instructor regarding their reflections seemed to gain a lot of understanding from oral feedback. students seemed to need prompting to go back and review their own previous reflections, such as through an end-of-year summative reflection. providing helpful feedback on assessment is challenging for instructors, and they can use development in this process so they can become, in karkabi et al.’s (2014) terms, more effective “reflective coaches” (p. 44). as they point out, “reflection is not necessarily intuitive for learners or teachers” (p. 44). our data agrees with the four conditions discussed in the literature review found by driessen et al. (2005) for effective reflection: good coaching; structure and guidelines; adequate experiences and material for reflection; and summative assessment. the students struggled most when they did not feel they had experiences that warranted reflection. since the students were always in clinic or field experiences, with events happening all the time, it also may be that they needed practice in identifying what experiences might be interesting to write about and observing in a particular way. developing professionals need practice in reflection, but they also hone their ability to observe in a meaningful way—to quote the fictional character sherlock holmes, “seeing” is not the same as “observing.” in the data, the feedback component was key as was the students’ willingness to engage with the feedback and seek further direction if unclear. there sometimes also appears a disconnect between the feedback provided and the students’ ability to take it in and make meaning of it. this too, appears to be part of the “learning to reflect” process that takes months and years to hone, not the weeks comprising a typical university course. all the world’s a stage: implications for practice we learned through this qualitative examination of interview data, and related student written reflections, that reflective practice is a process not learned overnight. in the case of these students, it took three years of regular, guided reflection to make meaning of the process and understand the value as they moved through identifiable stages. students progressed through states of confusion and conflict, finally becoming convinced of the value of narrative reflection. in reviewing the reflections, the value of the reflective process in terms of learning was visible as was the students’ developing ability to meaningfully delve into their experiences by the final year in the program. it is apparent that part of the learning process is reflection, but additionally students need to learn to observe and interpret a play in three acts: confused, conflicted and convinced 144 journal of effective teaching in higher education, vol. 4, no. 3 experiences in a reflective way—in other words, they need to “see” as a reflective practitioner as well as write about it. clearly, there is an important role for instructors in teaching developing practitioners to reflect on their learning. our results underline the value of reflective writing in contributing to the development of reflective practitioners but not in a vacuum. instructors need to be present in the process, providing structure and regular, detailed, and timely feedback. as demonstrated, athletic therapy students expressed a lot of frustration at the beginning. it may help to set an expectation for students for a long-term learning trajectory; on the other hand, it may not be possible to circumvent the frustration they feel in the beginning. indeed, it is possible that the frustration is a predictable part of the process that may even be integral to the learning arc. in this regard, instructors need to be patient and expect that it will take time far beyond the scope of one course for students to become proficient at the process. we recommend that reflective writing become a coordinated process throughout a professional program. while this qualitative study took place in a specific context, the benefits of reflective writing are demonstrated consistently enough to warrant serious consideration by any program, especially when considered in the context of the literature which supports these benefits despite differences between programs (chen & forbes, 2014). future directions in the students’ journals, a transformative process is evident, with students crossing a liminal space from “student” to “professional.” this transformation, as evidenced in the journals, is worth further investigation and mapping. it would be worth repeating a similar study in other athletic therapy programs, or other professional programs altogether, and further incorporating reflection as a formal learning process and one method of assessing students’ development. finally, further work is required to determine the most effective reflective writing structures, practices, and interventions for students in professional programs. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references brown, p. c. (2014). make it stick: the science of successful learning. the belknap press of harvard university press. charon, r., hermann, n., & devlin, m. j. (2016). close reading and creative writing in clinical education: teaching attention, representation, and affiliation. academic medicine, 91(3), 345–350. https://doi.org/10.1097/acm.0000000000000827 a play in three acts: confused, conflicted and convinced 145 journal of effective teaching in higher education, vol. 4, no. 3 chen, i., & forbes, c. (2014). reflective writing and its impact on empathy in medical education: systematic review. journal of educational evaluation for health professionals, 11(20), 1–6. https://doi.org/10.3352/jeehp.2014.11.20 coker, c. a. (2000). consistency of learning styles of undergraduate athletic training students in the traditional classroom versus the clinical setting. journal of athletic training, 35(4), 441–444. driessen, e., van tartwijk, j., overeem, k., vermunt, j., & van der vleuten, c. (2005). conditions for successful reflective use of portfolios in undergraduate medical education. medical education, 39(12), 1230–1235. https://doi.org/10.1111/j.1365-2929.2005.02337.x driessen, e., tartwijk, j., & dornan, t. (2008). the self critical doctor: helping students become more reflective. british medical journal, 336, 827. https://doi.org/10.1136/bmj.39503.608032.ad karkabi, k., wald, h. s., & cohen castel, o. (2014). the use of abstract paintings and narratives to foster reflective capacity in medical educators: a multinational faculty development workshop. medical humanities, 40, 44–48. https://doi.org/10.1136/medhum-2013-010378 kutz, m., dyer, s., & campbell, b. (2013). multiple intelligence profiles of athletic training students. the internet journal of allied health science and practice, 11(1), 1–9. lafave, m., & yeo, m. (2019). the relationship between the volume of clinical presentation exposures, hours, and student self-rated confidence. journal of competency-based education, 4(4), e01204. https://doi.org/10.1002/cbe2.1204 merriam, s. b., & tisdell, e. j. (2016). qualitative research: a guide to design and implementation (4th ed.). jossey-bass. middendorf, j., mickute, j., saunders, t., najar, j., clark-huckstep, a. e., & pace, d., with eberly, k., & mcgrath, n. (2015). what’s feeling got to do with it? decoding emotional bottlenecks in the history classroom. arts and humanities in higher education, 14(2), 166–180. reisman, a. b., hansen, h., & rastegar, a. (2006). the craft of writing: a physician-writer’s workshop for resident physicians. journal of general internal medicine, 21(10), 1109–1111. https://doi.org/10.1111/j.15251497.2006.00550.x teunissen, p. w., & bok, h. g. j. (2013). believing is seeing: how people’s beliefs influence goals, emotions and behaviour. medical education, 47, 1064–1072. https://doi.org/10.1111/medu.12228 a play in three acts: confused, conflicted and convinced 146 journal of effective teaching in higher education, vol. 4, no. 3 tsingos-lucas c., bosnic-anticevich, s., & smith, l. (2016). a retrospective study on students’ and teachers’ perceptions of the reflective ability clinical assessment. american journal of pharmaceutical education, 80(6). https://doi.org/10.5688/ajpe806101 tsingos-lucas, c., bosnic-anticevich, s., schneider, c. r., & smith, l. (2017). using reflective writing as a predictor of academic success in different assessment formats. american journal of pharmaceutical education, 81(1), 8. https://doi.org/10.5688/ajpe8118 uygur, j., stuart, e., de paor, m., wallace, e., duffy, s., o’shea, m., smith, s., & pawlikowska, t. (2019). a best evidence in medical education systematic review to determine the most effective teaching methods that develop reflection in medical students: beme guide no. 51. medical teacher, 41(1), 3– 16. https://doi.org/10.1080/0142159x.2018.1505037 wald, h. s. (2015). professional identity (trans)formation in medical education: reflection, relationship, resilience. academic medicine, 90(6), 701–706. wald, h., reis, s., & borkan, j. (2009). reflection rubric development: evaluating medical students' reflective writing. medical education, 43(11), 1110–1. doi: 10.1111/j.1365-2923.2009.03470.x wald, h., & weiss, b. (2018). making it “more real”: using personal narrative in faculty feedback to a medical student’s reflective writing—an illustrative exemplar. mededpublish, 7, 171. https://doi.org/10.15694/mep.2018.0000171.1 yeo, m., lafave, m., westbrook, k., mcallister, j., valdez, d., eubank, b. (2017). impact of decoding work within a professional program. new directions for teaching and learning, 150 (summer), 87–96. yeo, m., lafave, m., westbrook, k., valdez, d., eubank, b., mcallister, j. (2016). curricular change: deepening professional community. transformative dialogues, 9(1), 1–12. microsoft word kenneysreckovic_finalversion.docx journal of effective teaching in higher education, vol. 2, no. 2 examining language in higher education courses: creating opportunities for critical reflection christine k. kenney, university of michigan-flint, ckmeyer@umich.edu melissa sreckovic, university of michigan-flint abstract. words and labels used about a person inevitably have an impact, whether intended or unintended. this article discusses the need for intentional and overt, critical reflection and discussion in higher education regarding the language faculty and students use. the authors provide language examples commonly used in courses that could have varied meanings. instructional strategies which support reflective conversations surrounding language in higher education courses are provided. keywords: instructional practices, reflective teaching, higher education given ever-increasing diversity found on college campuses the task for instructors to cultivate an inclusive climate becomes essential where diversity (i.e., the range of human differences) is celebrated and respected. all students, regardless of gender, race, ethnicity, culture, sexual orientation, identity, age, and ability should be welcomed and valued in the higher education learning environment. numerous discipline-specific professional organizations echo this sentiment of respect and inclusiveness (american psychological association, 2016; american sociological association, 2018; council for exceptional children, 2015; national association for the education of young children; 2011). in an effort to create an inclusive learning environment where all students are welcomed and valued (moriña, 2017), faculty and instructors should examine the words, phrases, gestures, and cues used within classrooms and course work as they often send messages, both intended and unintended, which inevitably leave impressions on a diverse set of students. one may argue the language used within higher education learning environments has lasting social, educational, and policy implications. what an instructor says and how students internalize language transcends the higher education classroom context. observational learning occurs every day with people of all ages (bandura, 1977; gaskins & paradise, 2010). therefore, the language and strategies instructors model in the college classroom can generalize into students’ professional practices. therefore, careful reflection and examination of the language used within higher education learning environments demands attention. it is essential for instructors to foster and cultivate critical reflection in students by providing strategies and opportunities to approach language in a sensitive manner. the purpose of this article is to review specific language which, when used within higher education classrooms, intentionally or inadvertently relays messages to students. in addition, this article examines the role of deliberate and overt reflective practice surrounding language usage both by instructors and examining language in higher education courses 86 journal of effective teaching in higher education, vol. 2, no. 2 students and the use of transformative learning in the higher education context. finally, strategies to encourage language reflection are offered. it is important to note, that the discipline of the authors is teacher preparation and many of the examples within this article focus on this area. however, the strategies provided are applicable to all disciplines. regardless of the content area being taught, critical reflection and discussion of language usage is essential as students may begin to use the same language in the work force without recognizing the unintentional message it sends. reflection as a learning and teaching skill prominent educational philosopher and reformer john dewey (1916) argued that we do not learn from experience; rather, we learn from reflecting on experience. the act of reflection may be seen as the bridge between experience and learning. schon (1983) examines reflection within professional practice where individuals understand why they do certain things because they think about what it is they are doing. this adds an element of intentionality and consciousness into experience and learning. this intentionality and consciousness in learning brought about by reflection is especially important in teacher preparation where students are learning how to teach through being taught. the strategies, techniques, and language within the learning context take on a new element of importance, given that the way in which students learn about how to teach may impact their pedagogical decisions when they themselves are teachers (berry, 2007). the same can be said in other disciplines such as medicine, nursing, business, social work and so forth. while the content is imperative, other elements of the learning environment will inevitably impact the type of professional the student becomes. zeichner and liu (2010) take a look at the “reflection movement” wherein the concept of reflection is seen as mandatory but is often abstract, undefined, or misunderstood (p. 69). this may equate to the belief that reflection is good but what exactly is meant by reflective teaching and learning may be unclear. within the literature, researchers and philosophers alike have attempted to define what reflection may look like. for example, freire (1968) argues against the notion of “banking education” with instructors viewed as holding all knowledge and students as passive recipients with the sole job of consuming what is provided by their instructors (p. 58). instead, freire calls for reciprocal action in learning; reflective dialogue where learning is shaped by interactions with others and their ideas, perspectives, etc. critical reflection is at the heart of this learning where students and instructors discuss, reflect, discuss some more, and so on. both instructor and student have a pivotal role in the reflection and learning that take place within the classroom context. larrivee (2000) speaks of critical reflection and the link to transforming teaching practices arguing that critical reflection involves both critical inquiry and selfreflection. educators must not simply learn and implement instructional strategies in isolation from reflecting upon these instructional strategies and how they mesh with or challenge their personal beliefs, experiences, and biases. larrivee posits that educators make overt connections for themselves and their students between examining language in higher education courses 87 journal of effective teaching in higher education, vol. 2, no. 2 the instructional strategies they employ and their own beliefs regarding how teaching and learning occur. larrivee offers several strategies for this discovery process including setting time aside for isolated personal reflection and questioning the status quo. smith (2011) reviews the role of critical reflection in higher education and the conditions for such reflection to be successful. the author maintains “teaching critical reflection in [higher education] requires creating conditions for intellectual challenge – which is a challenge” (p. 14), but goes further to argue that teaching critical reflection is a sound step in supporting higher education students to become professionals who can navigate complex ethical issues. understanding and acknowledging implicit bias is key in the type of critical reflection discussed above. social psychologists greenwald and banaji (1995) are credited with coining the term implicit bias and argue it is possible to unconsciously hold attitudes and stereotypes toward an individual or group which impact how we behave toward or refer to others. because these attitudes and stereotypes are often automatic and not readily accessible, self-awareness and even critical reflection may not be enough to fully understand the messages instructors pass along in their language and behaviors in the higher education learning context. staats (2015) reviewed research on mitigating the effects of implicit bias in education. suggestions include taking the free online implicit association test (iat) designed by greenwald, mcghee and schwartz (1998) and “exposure to counterstereotypical exemplars…[including] male nurses [and] female scientists” (staats, n.d., paragraph 31). participating in actions such as taking the iat test and then engaging in critical reflection can support instructors and students in identifying and challenging their implicit bias. one would hope that purposeful critical reflection within higher education coursework along with becoming aware of and challenging implicit bias might lead to a shift in thinking and learning. however, instructors need to take into consideration that their students are adults and have already formed frames of reference that define their world. in other words, they have developed assumptions, values, and feelings through which they understand their experiences (mezirow, 2012). adults often reject ideas that do not fit into their already developed frames of reference (mezirow, 1994). this emphasizes the need to understand our experiences as well as implicit biases so we can strive toward viewpoints that are more inclusive and functional (mezirow, 2012). creating conditions in which transformative learning can occur is important, and often necessary, in order for adults to meaningfully engage with colleagues and other professionals (king, 2004). transformative learning transformative learning has been defined as “the social process of construing and appropriating a new or revised interpretation of the meaning of one’s experience as a guide to action” (mezirow, 1994, p. 222-223). when one’s beliefs are no longer functional, one reflects on the unexamined assumptions of one’s beliefs (mezirow, 1994). critical reflection occurs when one experiences a disorienting dilemma. for examining language in higher education courses 88 journal of effective teaching in higher education, vol. 2, no. 2 example, a student may volunteer in an unfamiliar community and experience a disorienting dilemma related to culture and family structures of community members that did not fit into their frame of reference. when critical reflection occurs, one examines the origin, nature, and consequences of the assumption to determine if the long-held belief is still functional (mezirow, 1994; mezirow, 2012). research indicates higher education instructors can create conditions in which transformative learning occurs. for example, bell and colleagues examined the experiences of 150 college students from the united states who participated in an interdisciplinary, experiential study abroad experience (bell et al., 2016). throughout the program students were provided opportunities that were outside of their “comfort zones” (e.g., new country, new cultural context, new food) and encouraged to change the way they understand their worldview and themselves. analysis of student written reflection indicated participants experienced various disorienting dilemmas (e.g., first time leaving family, first time flying) and recognized ways their assumptions and beliefs changed and made new habits and identified plans of action. follow up conversations indicated some students did follow through with their new plans of action. while incorporating activities and opportunities for critical reflection in higher education is imperative, guerra and pazey (2016) argue that higher education instructors must first start with themselves and examine their own values, beliefs, and implicit biases that may be portrayed to their students. faculty must critically reflect on their own long held assumptions and the language they use that can impact students. critical reflection on language there are times in one’s life when a simple word, phrase, gesture or encounter can lead to a transformation where a shift in thinking can create a shift in pedagogical and instructional practice. a transformative shift of this nature occurred for us when we first became aware of the terminology “children with special rights” spoken by i. soncini, a psychologist and pedagogista for the reggio-emilia municipality (personal communication, march 21, 2016). we long considered ourselves to be advocates for children with disabilities both having worked with children with disabilities for years as classroom teachers and currently teaching undergraduate and graduate courses in special education and inclusion. however, the introduction of the reggio-emilia philosophy of working with children with special rights led to a series of reflective conversations surrounding the meaning behind and messages associated with children with special needs vs. children with special rights. rights vs. needs? this is not a new question, as scholars and professionals who have traveled to and studied about reggio-emilia, italy and their approach to education have written about the culture of inclusion surrounding children with special rights for some time (soncini, 2012; vakil, freeman, & swim, 2003). building from the reggio-emilia perspective, runswick-cole and hodge (2009) discuss the “power of language” where language used within the educational system can “create positive and negative images of children” (p. 198). runswickexamining language in higher education courses 89 journal of effective teaching in higher education, vol. 2, no. 2 cole and hodge argue that these images may influence educational practices as well as have social and policy implications. they call for a shift in discourse surrounding special education in order to more accurately discuss the rights of individuals with disabilities without also attaching a message of need. we began to discuss whether a similar shift in discourse might be warranted for other common terminology within higher education courses. for example, the words “family” and “parent” are often used interchangeably within the literature, course content, and assignments. however, many argue that with ever-changing family dynamics, the word “family” may be more inclusive than “parent.” who might feel omitted when a teacher schedules a parent activity night? who is included when a teacher refers to the same event as a family activity night? spurred by reflective conversations of this nature, we began to ask how we might inadvertently shape our students with the words we use to describe individuals within our teaching practices. as brookfield (2017) suggests “one can never be sure of the effect you’re having on students or the meanings people take from your words and actions” (p. 2). reflective conversations between instructors and students surrounding terminology common in any discipline area are critical to provide the opportunity for a transformative shift to the use of more sensitive and representative language. throughout the remainder of this article we highlight several language examples which might be opportune topics for reflective conversations between instructors and students. strategies for fostering reflective practice around language and discourse with students in higher education courses are also included. language examples it is helpful to first identify common language which might be prone to misunderstanding, exclusion, or overall ambiguity before instructors can foster reflective conversations with students regarding language use. similar to how children with special rights vs. children with special needs motivated a reflective conversation for us, other labels, terminology, or phrases may foster thoughtful and productive conversation leading to reflective and sensitive teaching practices. throughout this section, particular words are examined with supporting literature. the examples we provide are drawn from the education field; however, they are applicable across disciplines. for example, when speaking about people with disabilities, what unintentional message is sent if the speaker uses the terms “handicapped”, “disabled”, “exceptional” or “differently-abled”? all students regardless of their academic discipline will come in contact with people with disabilities in their education, workplace and community. therefore, all instructors should be mindful of the language they use when speaking about people with disabilities. the examples provided are not an exhaustive list as there are endless interpretations for language. we simply offer a beginning from which instructors and students might build. differences. different does not mean deficient. however, historically within multiple contexts, differences among individuals may be recast as deficits. examining language in higher education courses 90 journal of effective teaching in higher education, vol. 2, no. 2 cummings (2003) discusses how the differences children bring to school (e.g., racial identity, language, ability, class, gender) are often translated as deficits when compared to a dominant group (p. 39). the deficit perspective has a long history within the literature in describing various groups based on ability, racial identity, class, gender, and more (annamma, 2016; gorski, 2010; valencia, 2010). gorski (2010) describes the deficit perspective as a “symptom of larger sociopolitical conditions and ideologies borne out of complex socialization processes” (p. 2). educators who take on a deficit perspective may point to a child’s family or culture for the child’s school failure (bartolomé & balderrama, 2001). educators must examine the myths that underlie the deficit perspective (e.g., “families from marginalized communities do not value education”; volk & long, 2005, p. 14). given that the deficit perspective is linked heavily to ideologies, socialization processes, and misperceptions (annamma, 2016; bartolomé & balderrama, 2001; gorski, 2010; valencia, 2010), it would make sense that reflecting on the language surrounding differences would be ripe for critical reflection, personal awareness, and transformative learning. the matter of different vs. deficient has been explored throughout the literature including with women in stem fields (kenney, mcgee, & bhatnagar, 2012) and through the perspective of gender and racial identity (emdin, 2012). the power of labeling is problematic as the simple act of using a label can create associated stereotypes. negative connotations attached to the word difference may denote lacking, inadequacy, or inferiority. on the other hand, differences can be celebrated and appreciated. instructors must be cognizant when discussing differences as to not perpetuate stereotypes. deliberate conversations about differences may be excellent opportunities to discuss intention behind language as well as the messages specific words may send. at-risk. the term at-risk is certainly not a new one within the literature. swadner and lubeck (1995) cite the presence of thousands of journal articles and conference papers focusing on the topic. they posit the term has become a “buzzword” where those described as at-risk are marginalized and portrayed as lacking or deficient (p. 1). swadner and lubeck focus on how the term at-risk is often associated with low income, non-english speakers, or those of color. similar to the deficit perspective discussed above, blanket at-risk labeling may lead to oversimplifications, overgeneralizations, and unexamined biases. swadner and lubeck call for in-depth analysis of the intentions behind using the term at-risk and offer alternative terminology, “at-promise” to convey a message of possibility, potential, and uniqueness (p. 4). in the united states educational system, the term at-risk is associated with federal funding for students who are at-risk of falling behind in developmental or learning areas. in order to receive federal and state financial support, schools must prove a risk is evident. in this situation, the term at-risk is necessary in order for students to receive supports. thus, an examination of the term at-risk becomes complex. some individuals may indeed be at risk and in need of support. others may erroneously be referred to as at-risk based on their economic or racial background. the intention behind the use of the term and the implications attached then examining language in higher education courses 91 journal of effective teaching in higher education, vol. 2, no. 2 become impactful, signaling the necessity for careful and deliberate reflection on language use. striving, thriving, and developing understanding. in content areas, such as math and literacy, among others, commonly used terminology may perpetuate unintended messaging as well. for example, johnson and keier (2010) focus on the complicated process of learning to read and suggest rethinking the term struggling reader. they note that the literacy-learning process from the beginning is not without difficulty for most and at one point or another everyone was considered a struggling reader. therefore, they suggest discussing the literacy-learning process in terms of striving instead of struggling. greenleaf and hinchman (2009) went further to say that “too many young readers leave schools with identities as poor readers and failures” (p. 11). they call for continued reform in instructional practices with a focus on allowing readers to see themselves as thriving while learning to read. they argue that when students are offered tailored literacylearning experiences built upon their strengths and are referred to and refer to themselves as thriving they may be better positioned to persevere throughout the reading acquisition process. the terminology used both by teachers and the students themselves is important. lewis (2014) examines labeling in terms of mathematics and calls for “reconceptualizing” how we look at and refer to children throughout the mathematical learning process (p. 351). the author suggests that many individuals face difficulty while learning mathematical concepts but the reasons for this difficulty vary. therefore, it may be more accurate to look at learning differences rather than learning deficits. likewise, in a discussion with one of our mathematics education colleagues, it was suggested that individuals in the midst of the mathematical learning process may be developing understanding vs. misunderstanding (e. cunningham, personal communication, october 12, 2016). developing understanding suggests a movement toward understanding, while misunderstanding may denote a lack of knowledge or confusion. while the distinction is slight, the messages attached are strong. across the content areas, the terminology used to describe individuals within the learning process may have intended or unintended messages. a discourse surrounding labels within the content areas would then seem to be an advantageous exercise. person-first vs. identity-first language. the american psychological association publication manual (2019) acknowledges that the language concerning disability is “evolving…[and] the overall principle for using disability language is to maintain the integrity (worth and dignity) of all individuals as human beings” (p 136). the publication manual stresses that individuals who are writing about disabilities take into consideration the preferences of people with disabilities; both person-first language and identity-first language should be used as appropriate. snow (2012) defines person-first language as placing a person before their disability and offers examples such as “people with disabilities” instead of “handicapped or disabled” or an individual “with a learning disability” instead of “learning disabled” (p. 4). the argument being that having a disability is only part examining language in higher education courses 92 journal of effective teaching in higher education, vol. 2, no. 2 of a person and should not be the definition of that person. however, within disability studies and the disability culture more generally, others offer the identityfirst perspective where the disability is seen as a point of pride and individuals choose their identity instead of others doing so for them (e.g., dunn & andrews, 2015). for example, brown (2011) offers an example where, within the autism community, some self-advocates prefer the term “autistic” over a “person with autism.” however, she goes further to say that not all agree and there are others who prefer the latter. dunn and andrews (2015) review the arguments for using both person-first and identity-first language. in their review, they state that language surrounding disability has changed considerably and is ever-evolving. they call for “advancing cultural competence through disability language” (p. 262) where discussions relating to disability and disability terminology may result in opportunities for education and impeding the spread of stereotypes and intolerance. whether one chooses person-first or identity-first language may be determined by context, those involved, and so forth. perhaps the most important elements are for one to be knowledgeable, educated on the topic of disability language, sensitive to perspectives, and reflective. disability labels. disability, impairment, and handicap are often used interchangeably and are the most frequently used disability labels in the english language (baglieri, 2017). impairment refers to the characteristics of a person’s body, disability is the experience of having the impairment compared to those who do not have the impairment, and handicap is the resulting experience when an individual with a disability interacts with their environment and experiences some sort of disadvantage because of their disability (baglieri). while some advocacy groups prefer the term “disability”, several euphemisms for disability are commonly used in community and educational settings. for example, the terms exceptional, physically challenged, and differently-abled are frequently used to describe individuals with disabilities. while some individuals who use such language may do so to challenge stereotypes of disability, individuals with disabilities may find those terms patronizing (baglieri). it is imperative, therefore, that instructors and students examine their own opinions, biases, and potential misconceptions about disability labels and how such labels are reflected in their language. as sutherland (1984) stated, “it is for people with disabilities to decide how we choose to define ourselves, and few of us choose to do so according to the prejudices of people who consider themselves able-bodied” (p. 14). when examining the use of disability language, it is also important to recognize particular terms are associated with special rights. for example, the term disability is used in both the americans with disabilities act amendments act (adaaa, 2008) and the individuals with disabilities education improvement act (2004) affording individuals with disabilities specific rights within the united states. the terms exceptional and differently-abled do not have the same rights associated with them. baglieri (2017) argues that individuals should use the terms disability, impairment, and handicap directly without trying to soften the language as “it is not shameful or hurtful to describe bodies, minds, their strengths, and their limits in direct examining language in higher education courses 93 journal of effective teaching in higher education, vol. 2, no. 2 language” (p. 29). however, garland-thomson (2002) states that disability is a broad term that encompasses categories such as deformed, abnormal, and sick “which disadvantage people by devaluing bodies that do not conform to cultural standards” (p. 74). this highlights the vast perspectives of disability labels and how intention behind labels can vary from person to person. while there may not be a clear answer for what language to use, the act of assisting students to become aware that disability labels are complicated may help to create a profession of individuals who think before they speak. moving forward: where to go from here? for some, it may seem overwhelming to consider how to engage in conversations with students which are sensitive, educated, and responsive. the point is not to cause doubt, fear, and a desire to no longer engage in thoughtful discussions about individuals and associated labels. quite the opposite, the point is to bring awareness to how we talk about others and the words we put forth in class conversations and elsewhere. the ability to design reflective conversations within university classroom contexts, where instructors and students partake in thoughtful, respectful discussion surrounding terminology that may otherwise be taken for granted, opens the door to fostering a multitude of professionals primed to consider the language they use and reflect upon why they are using it and the messages it inadvertently sends. ideally, and more than likely, instructors are currently reflecting upon their own use of the language examples highlighted above or other common language specific to their discipline or field of study. the key is that instructors make their critical reflection apparent and overt for their students as part of their everyday instructional practice. equally important is for instructors to cultivate a habit in students to critically reflect upon their own implicit biases, the language they use, and the messages they are portraying. instructors can design the learning context to include opportunities for in-depth, reflective conversations regarding specific language and meaning. discussed below are strategies that higher education faculty can use in order to foster critical reflection and conversations surrounding language usage and associated meanings. fostering reflective discussions: strategies to use with students self-reflection and preparation for critical reflection to occur within higher education settings, instructors must first reflect upon their own experiences, thoughts, opinions, beliefs, and language use (guerra & pazey, 2016). critical reflection and inquiry involve the “conscious consideration of the moral and ethical implications and consequences of classroom practices on students’’ (larrivee, 2000, p. 293). in this case, language modeled by instructors may have lasting implications for students and therefore, instructors must be cognizant about what it is they are saying and what messages their language carries. careful and intentional self-reflection of one’s own language is important. included in this self-reflection must be an examination of biases. examining language in higher education courses 94 journal of effective teaching in higher education, vol. 2, no. 2 furthermore, making this reflective process visible to students in the higher education setting can have lasting implications on how students approach their own learning and future professional work. for example, we examined our use of the words rights vs. needs in terms of the special education system and discussed this reflective process with our students. this deliberate self-reflection, and the act of sharing our reflective process with students allowed for conversations about language choice and usage. it permitted students to witness the process of critical reflection and also to realize that their instructors engage in this type of deliberation. instructors must acknowledge the consequences of critically reflective conversations in the higher education classroom context. discussions of this nature take time and therefore, there are technical decisions that must be attended to. when should conversations such as these take place? how much time can and should be dedicated to reflective conversations surrounding language and messaging? we suggest it is necessary to devote ample time to this reflective process in order to cultivate professionals who approach language critically and sensitively. to that end, it is advantageous for reflective conversations to take place in every class period. students should feel free to pose alternate ideas, question the intention behind specific language, and share their thoughts or perhaps anxiety about specific topics. this requires time for open-ended discussion in both small and large groups. in addition, students may need time to reflect individually prior to discussing with a group. reflective conversations may open up difficult conversations where instructors and students feel unprepared. a judgement free atmosphere should be created by fostering a classroom learning environment which values open and respectful communication. learning experiences which foster transformation once the environment is primed, instructors can foster reflective discussions and carve out space and time for critical conversations to take place. mezirow (1994; 2012) discusses that for transformative learning to occur, students need to be faced with a disorienting dilemma which leads to self-reflection, discussion with others, and potentially a reconceptualization of thoughts and beliefs. instructors may incorporate elements into the course which allow for disorienting dilemmas to occur. for example, class activities and fieldwork may be strategically designed in order to foster the development of new and challenging perspectives. case studies. the use of case studies as an instructional practice allows for students to read about a specific individual or situation and have a common foundation from which to discuss. for example, including a case study where a child is identified as at-risk for emotional and/or behavioral disorders based on her class and racial background may generate opportunities for critically reflective conversations about previous experiences, preconceived ideas, beliefs, and language. students and instructors may draw upon the case study in order to examining language in higher education courses 95 journal of effective teaching in higher education, vol. 2, no. 2 support, respectfully challenge, or disagree with one another. they may question whether the child is truly at-risk and what the child might be at-risk for. they may ponder whether the at-risk label is necessary for this child. they may examine the intentional and unintentional messages associated with labeling the child at-risk. they may discuss personal and societal biases. they may look at overgeneralizations and oversimplifications of at-risk in light of viewing the term through the lens of one specific child. from this case study, the instructor is able to introduce potentially sensitive topics and support students throughout the conversations. case studies can be used across disciplines to encourage reflective conversations (e.g., a social work case study about a child in foster care; a medical case study on a child with an autoimmune disease; a criminal justice case study about an adult being detained). fieldwork. numerous and varied field-placements also support opportunities for reflective and transformative learning. university programs can hold classes in prek-12 school settings, invite practicing teachers to co-teach university classes, or embed field experience into the class structure where the entire class and instructor venture together into school settings (zeichner, 2010). as another example, in the medical profession students engage in fieldwork in various stages including, but not limited to, simulated patients, apprenticeship, and residency, all of which contribute to the process of professional identify formation (sharpless et al., 2015). these learning opportunities allow students to gain first-hand knowledge of the education/medical process but also provide common experiences for the instructor and class to reflect upon and discuss. experiences in the field, especially experiences that create a disorienting dilemma for students, (e.g., students are placed in settings with people who are different from themselves) enable college students to continue to gather varied perspectives and practical knowledge. however, it is important for students and their instructors to reflect upon and examine the language and perceptions they encounter in the field. they can participate in conversations about their field placement encounters in order to capitalize on the potential for critical reflection and potential shifts. service learning/community engagement. service learning and community engagement also allow students to move away from the perceived safety of the university learning environment into contexts perhaps less familiar and out of their comfort zone. bell and colleagues (2016) incorporated service learning and research into the curriculum of an interdisciplinary study abroad experience for students and found students not only transformed their assumptions and beliefs, but made changes to their everyday routines because of their experiences (e.g., drove their car less, conserved water). students who are given the opportunity to encounter community issues where they exist and make an impact are exposed to new perspectives on which they may reflect and discuss. it is possible to focus community engagement or service learning projects on examining language and unintended bias. for example, a community engagement project may involve students creating and offering educationally enriched activities at a public park in a city setting. through working with community members, families, and their children to plan the event, students will have the opportunity to better understand the needs of children and their families from various backgrounds. rich conversations examining language in higher education courses 96 journal of effective teaching in higher education, vol. 2, no. 2 around family make-up, family expertise, and children’s needs can occur during the planning process, while implementing the activities, and following the project itself. creating space for students to reflect throughout the planning and event process is critical for students to fully benefit from the community engagement activity and examine the impact of their language. student self-reflection journaling. the aforementioned activities and instructional strategies may provide students with new-found perspectives from which they can draw upon throughout class discussions. however, students must also engage in solitary self-reflection. journal writing is supported within the literature as an activity to develop reflective practices (boud, 2001; larrivee, 2000; peoples, thompson, & murphy, 2016). through journaling, students can organize their thoughts and beliefs, revisit certain events and recapitulate specific occurrences, contemplate new or varying perspectives, critique alternating viewpoints, and prepare comments in a safe and personal space. through journal entries, students internally grapple with what they believe, feel, agree, or disagree with. journaling also provides students the opportunity to coordinate their thoughts prior to engaging in a more public class discussion. for example, an environmental science instructor may include student self-reflection journaling as an assignment after visiting urban areas to identify alternative energy solutions. students may reflect on the environment, challenges the community faces, strengths of the community, and preconceived notions they had before visiting the site. students may journal how their views have changed and why. while it is an assignment, the instructor does not collect the journals; instead, the instructor provides time in class for students to share their thoughts if they feel comfortable. instructors may offer journal prompting questions which focus students on contemplating language and terminology. prompting questions may include: reflect on specific labels which you use in your everyday life. why do you use them? are they necessary? what might be the intended or unintended messages attached to these labels? if the intention is to allow students time to process and prepare for reflective conversations surrounding language use, it is possible that journals should remain private and not attached to grades (boud, 2001). implications and conclusions throughout this article, we posit that intentional and supported conversations surrounding language usage are important aspects of the higher education learning context. an important principle to teach higher education students is that reflection on one’s language is a skill that will enhance all aspects of their professional disposition. to achieve this goal in our own courses we are participating in regular self-reflection of the language we use with our students. we are creating welcoming and caring environments where students feel safe to take risks, empowered to ask questions, and respectfully disagree. we are modeling reflective and critical processes with our students during class discussions and fostering in our students a duty to do the same. we are carving out space and time in class to engage in difficult conversations surrounding bias (implicit and explicit), prejudice, inequities, and unintended messages. we are participating in activities and assignments that examining language in higher education courses 97 journal of effective teaching in higher education, vol. 2, no. 2 stretch students to think beyond language and labels as generalizations and see people as individuals who deserve the respect to be identified in a way they feel most comfortable. further, having discussions about the implications of language usage opens the door for a broader conversation about unintended messages sent in our gestures and actions as well as unrecognized biases. we do all of this in an open and intentional manner so that we may embolden a generation of individuals who look beyond labels and question historically accepted terminology and messages as status quo. and, we engage in ongoing examination of our practices to better understand how they are impacting our students. we believe the same critical reflection of language must occur in other fields and disciplines in the higher education context. for example, a finance instructor who only uses the pronouns he/his when speaking about business owners may send a message to their students that business owners are primarily male. a nursing instructor who speaks about the nursing workforce using the pronouns she/her may offend male nursing students and also send unintended messages about the nursing profession as a whole. in the nursing and finance examples, when students then enter the work force they may use that same language and alienate potential clients and colleagues. while each discipline and field of study is unique, it is necessary to consider the commonly used language within that field or discipline, contemplate what intended and unintended messages may be relayed with the use of such language, and decide if there is room to adapt language to be more inclusive. it is difficult to determine the right language to use in every case. context, reflection, and examining intention are all important elements to consider. as instructors, we must help our students to approach language and labels with forethought. the questions to ask are: what is the intention behind the language being used? and how might the person receiving this language feel? when individuals approach language with intention and empathy, they create a more inclusive, representative, and sensitive context for successful relationships to form and strengthen. intentional, overt, and reflective discussions surrounding language and implicit bias within higher education learning contexts are a step toward fostering and nurturing a community of professionals across numerous disciplines. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references american with disabilities act amendments act of 2008. pl. 110-325. stat. 3406. (2008). american psychological association. (2019). publication manual of the american psychological association (7thed). washington, dc: author. examining language in higher education courses 98 journal of effective teaching in higher education, vol. 2, no. 2 american psychological association. (2016). ethical principles of psychologists and code of conduct. retrieved july 30, 2019. retrieved from https://www.apa.org/ethics/code/ american sociological association. (2018). american sociological association code of ethics. retrieved july 30, 2019 from https://www.asanet.org/sites/default/files/asa_code_of_ethics-june2018.pdf annamma, s.a. (2016). discrit: disability studies and critical race theory in education. new york: teachers college press. baglieri, s. (2017). disability studies and the inclusive classroom: critical practices for embracing diversity in education (2nd ed,). new york: routledge. bandura, a. (1977). social learning theory. morristown, nj: general learning press. bartolomé, l.i., & balderrama, m. v. (2001). the need for educators with political and ideological clarity. in m. de la luz reyes & j.j. halcón (eds.), the best for our children: critical perspectives on literacy for latino students (pp. 4864). new york: teachers college press. bell, h. l., gibson, h. j., tarrant, m. a., perry, l. g., stoner, l. (2016). transformational learning through study abroad: us students’ reflections on learning about sustainability in the south pacific. leisure studies, 35(4), 389405. berry, a. (2007). tensions in teaching about teaching: understanding practice as a teacher educator (vol. 5). springer science & business media. boud, d. (2001). using journal writing to enhance reflective practice. new directions for adult and continuing education, 90, 9-19. brookfield, s. d. (2017). becoming a critically reflective teacher. hoboken, nj: wiley. brown, l. x. z. (2011, august 4). the significance of semantics: person-first language: why it matters [blog post]. retrieved from http://www.autistichoya.com/2011/08/significance-of-semantics-personfirst.html council for exceptional children. (2015). code of ethics. retrieved on july 30, 2019 from https://www.cec.sped.org/standards/ethical-principles-and-practicestandards cummings, j. (2003). challenging the construction of difference as deficit: where are identity, intellect, imagination, and power in the new regime of truth. pedagogies of difference: rethinking education for social change, 41 60. dewey, j. (1916). democracy and education: an introduction to the philosophy of examining language in higher education courses 99 journal of effective teaching in higher education, vol. 2, no. 2 education. new york: macmillan. dunn, d., & andrews, e. (2015). person-first and identity-first language: developing psychologists’ cultural competence using disability language. american psychologist, 70(3), 255-264. emdin, c. (2012). yes, black males are different, but different is not deficient. phi delta kappa, 93(5), 13-16. freire, p. (1968). the pedagogy of the oppressed. new york: continuum press. garland-thomson, r. (2002). the politics of staring: visual rhetorics of disability in popular photography. in s. l. snyder, b. j. brueggemann, & r. garland thomson (eds.), disability studies: enabling the humanities (pp. 56–75). new york: modern language association. gaskins, s., & paradise, r. (2010). learning through observation in daily life. in d. f. lancy, j. bock, & s. gaskins (eds.), the anthropology of learning in childhood (pp. 85-118). lanham, md: altamira. gorski, p. c. (2010). unlearning deficit ideology and the scornful gaze: thoughts on authenticating the class discourse in education. counterpoints, 402, 152-173. greenleaf, c. l., & hinchman, k. (2009). reimagining our inexperienced adolescent readers: from struggling, striving, marginalized, and reluctant to thriving. journal of adolescent & adult literacy, 53(1), 4-13. greenwald, a., mcghee, e., & schwartz, l. (1998). measuring individual differences in implicit cognition: the implicit association test. journal of personality and social psychology, 74(6), 1464-1480. greenwald, a., & banaji, m. (1995). implicit social cognition: attitudes, self esteem, and stereotypes. psychological review 102(1), 4-27. guerra, p. l., & pazey, b. l. (2016). transforming educational leadership preparation: starting with ourselves. the qualitative report, 21(10), 1751 1784. individuals with disabilities education improvement act of 2004, p.l. 108 446, 20 u.s.c. $ 1400 et seq. johnson, p., & keier, k. (2010). catching readers before they fall: supporting readers who struggle, k-4. portland, me: stenhouse publishers. kenney, l., mcgee, p., & bhatnagar, k. (2012). different, not deficient: the challenges women face in stem fields. journal of technology, management & applied engineering, 28(2), 2-9. examining language in higher education courses 100 journal of effective teaching in higher education, vol. 2, no. 2 king, k. p. (2004). both sides now: examining transformative learning and professional development of educators. innovative higher education, 29(2), 155-174. larrivee, b. (2000). transforming teaching practice: becoming the critically reflective teacher. reflective practice, 1(3), 293-307. lewis, k. e. (2014). difference not deficit: reconceptualizing mathematical learning disabilities. journal for research in mathematics education, 45(3), 351-396. moriña, a. (2017). inclusive education in higher education: challenges and opportunities. european journal of special needs education, 32(1), 3-17. mezirow, j. (1994). understanding transformation theory. adult education quarterly, 44(4),222-232. mezirow, j. (2012). learning to think like an adult: core concepts of transformation theory. in e. w. taylor, p. cranton & associates (eds.), the handbook of transformative learning: theory, research, and practice (pp. 73–95). san francisco: jossey-bass. national association for the education of young children. (2011). code of ethical conduct and statement of commitment. retrieved july 30, 2019 from https://www.naeyc.org/sites/default/files/globallyshared/downloads/pdfs/resources/positionstatements/ethics%20position%20statement2011_09202013update.pdf peoples, b., thompson, m., & murphy, b. (2016). the journey of matriculation: a tool for developing learning through reflection. transformative dialogues: teaching & learning journal, 9(2), 1-4. runswick-cole, k., & hodge, n. (2009). needs or rights? a challenge to the discourse of special education. british journal of special education, 36(4), 198-203. schon, d. (1983). the reflective practitioner: how professionals think in action. new york: basic books. sharpless, j., baldwin, n., cook, r., kofman, a., morley-fletcher, a., slotkin, r., & wald, h. (2015). the becoming: student’s reflections on the process of professional identity formation in medical education. academic medicine, 90(6), 1-5. smith, e. (2011). teaching critical reflection. teaching in higher education, 16(2), 211-223. snow, (2012). people first language. retrieved from http://www.inclusionproject.org/nip_userfiles/file/people first in depth.pdf soncini, i. (2012). the inclusive community. in c. edwards, l. gandini, & g. forman (eds.),the hundred languages of children: the reggio emilia experience in transformation (187-212). santa barbara, ca: praeger. examining language in higher education courses 101 journal of effective teaching in higher education, vol. 2, no. 2 staats, c. (2015). understanding implicit bias: what educators should know. american educator, 39(4), 29-33. sutherland, a. (1984). disabled we stand. bloomington, in: indiana university press. swadner b.b, & lubeck, s. (1995). children and families ‘at promise’: deconstructing the discourse of at-risk. albany: ny: suny press. valencia, r. r. (2010). dismantling contemporary deficit thinking: educational thought and practice. routledge. vakil, s., freeman, r., & swim. t. (2003). the reggio emilia approach and inclusive early childhood programs. early childhood education journal, 30(3), 187-192. volk, d., & long, s. (2005). challenging myths of the deficit perspective: honoring children’s literacy resources. young children, 60(6), 12-19. zeichner, k. (2010). rethinking the connections between campus courses and field experiences in college-and university-based teacher education. journal of teacher education, 61(1-2), 89-99. zeichner, k., & liu, k. y. (2010). a critical analysis of reflection as a goal for teacher education. in. n. lynons (ed.), handbook of reflection and reflective inquiry (pp. 67-84). new york: springer. microsoft word c-calkins_revised_final.docx journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.249 the jigsaw design challenge: an inclusive learning activity to promote cooperative problem-solving susanna c. calkins, northwestern university, s-calkins@northwestern.edu jonathan rivnay, northwestern university abstract. this article highlights an innovative take on the jigsaw format, an inclusive and cooperative active learning strategy, implemented in an upper-level engineering elective course. after students complete the usual two steps of the jigsaw method—first gaining mastery in “expert groups” and then collaboratively teaching their peers in “jigsaw groups”—they then complete a third step in their jigsaw groups, in which they work together on an authentic design problem, offering a practical take on applying course content. this activity was implemented in three courses offered both in person and remotely (online only). we share how this innovation can promote learning, problem-solving, perspective sharing, and teamwork in contexts with students from different backgrounds and levels of experience. keywords: cooperative learning; jigsaw; active learning; design challenge although the traditional lecture-based format is still pervasive in many disciplines, active learning strategies have increasingly been recognized as beneficial for deeper student learning (prince, 2004), longer-term retention of concepts and ideas (laal & laal, 2012), improved ability to apply concepts to new contexts (roehl et al., 2013), improved collaboration and communication skills (minifie & davis, 2013), improved social presence and engagement (minifie & davis, 2013), and promotion of inclusive learning environments (johnson, 2019). notably, performance is significantly improved using active learning approaches compared to traditional lecturing. in one meta-study, for example, that focused on undergraduate science, technology, engineering, and mathematics (stem) courses, active learning techniques varied significantly (type, duration) and included activities such as group problem-solving, worksheets or tutorials completed during class, use of personal response systems, and studio or workshop course designs (freeman et al., 2014). students engaged in traditional lecture-based formats were 1.5 times more likely to fail than those in active learning environments. exam scores in active learning courses showed a 6% increase compared to lecture format courses. more specifically, a subset of active learning, cooperative learning, has shown positive outcomes in many fields. cooperative learning is based on the premise that cooperation is more effective than competition among students for producing positive learning outcomes (desai & kulkarni, 2016; slavin, 1980). courses integrating more interactive classroom formats (including cooperative activities) showed higher learning gains and better conceptual understanding (knight & wood, 2005). cooperation further promotes interpersonal relationships and effective the jigsaw design challenge 20 teamwork (desai & kulkarni, 2016), improves self-esteem (johnson et al., 1998a; johnson et al., 1998b), and enhances motivation (tran, 2019). one specific cooperative approach, jigsaw, is particularly well-suited for multidisciplinary settings or settings where students must be exposed to different theoretical or methodological approaches, especially in those courses that are more applied. the jigsaw technique was originally developed by elliot aronson in the 1970s to enhance empathy in racially and socioeconomically diverse high school settings (aronson, 1978; aronson & bridgeman, 1979). this approach involves peer-teaching and cooperative interactions that minimize the culture of competition and enhance learning and summative performance on both group and individual levels and can increase individual self-confidence (aronson, 1978; crone & portillo, 2013). it also encourages problem-solving and learner accountability through peerto-peer instruction (goolsarran et al., 2020) and can enhance individual construction of knowledge, improve individual comprehension of texts (booker, 2021; namaziandost, 2020), and improve retention of key concepts (nolan et al., 2018). the jigsaw technique has since been generalized and adapted numerous times in many disciplines across higher education with relevant examples in engineering (desai & kulkarni, 2016), chemistry (knight & wood, 2005), medical education (goolsarran et al., 2020; kumar et al., 2017), language learning (namaziandost et al., 2020), and psychology (crone & portillo, 2013; nolan et al., 2018). commonly, the jigsaw method employs two steps: (1) students first learn together in “expert groups,” gaining mastery over a specific content area and (2) the groups are reconfigured into “jigsaw groups” so that there is one expert from each area in each new group who will then teach the other students (aronson, 1978). in this article, we take a case study approach to describe how we expanded on the jigsaw activity in one biomedical engineering course, detailing how we developed and implemented an innovative third step as a “twist” to the more common jigsaw format. rather than completing the jigsaw activity after the first two steps, jonathan rivnay (the course instructor) asked the jigsaw groups to synthesize the distinct topics to complete a unique and randomly assigned design problem, thus building on the cooperative nature of the activity with an authentic task. authentic learning experiences situate tasks and skills for future use, help students develop deeper knowledge, and allow students a means to transfer concepts and knowledge to new contexts (herrington & herrington, 2006). as we describe below, students were asked to reflect on their experience of completing the activity and compare their approach for the design challenge with published works tackling the same engineering problems. the combination of a cooperative group activity involving peer-teaching (jigsaw) with problem-solving borrows from multiple aspects of cooperative learning (desai & kulkarni, 2016) while giving a practical and authentic take on applying course content. the method also fosters an inclusive learning environment, where all students are involved and help one another learn the content effectively. the jigsaw design challenge 21 while we focused on the implementation of this innovation in a specific upper-level engineering context, which is typically attended by students of varying majors or training backgrounds, we contend that this modified jigsaw activity could be useful in any course that features multidisciplinarity or is comprised of students from different educational backgrounds (or majors) or who possess varying levels of experience. we think that this approach would be beneficial in any learning context in which understanding multiple perspectives, teamwork, and problem-solving are essential tasks, especially in courses that feature authentic or real-world questions and problems. course context and description this pedagogical innovation was carried out in a new upper-level engineering elective course at northwestern university, a private research-intensive university located in the midwest united states that was co-taught by rivnay. the course is an elective with no prerequisites that met twice weekly over a 10-week term. the jigsaw innovation was carried out in three separate iterations of the course over three years, with only minor modifications. the first two iterations occurred during traditional in-person lecture periods while the third was online-only, mixed asynchronous/synchronous due to the covid-19 pandemic. the course also coordinated with an associated lab course that was assessed separately but tightly integrated with the main course. the course content focused on materials and device design considerations for wearable and implantable bioelectronic devices for medical diagnostics and therapeutics. by the end of the course, students were expected to be able to (1) identify and appreciate the multidisciplinary, collaborative environment needed to design and implement bioelectronic devices; (2) understand the basics of bioelectronic diagnostics and therapeutics; (3) identify the importance of core engineering topics for designing effective and long lasting bioelectronic devices; and (4) analyze scientific primary literature, perform literature searches, and synthesize new ideas from them. within the learning objectives, there were sub-objectives focusing on teamwork, cooperation, and collaboration through problem-solving and practical applications of core concepts. the course is interdisciplinary by design, each year drawing a mixture of undergraduates and graduate students from different engineering and stem fields. across the three course offerings, the balance between male and female and between undergraduates and graduate students (masters/phd) was roughly equal (first offering: n = 34, 16 male/18 female, 18 undergrads/16 grads; second offering: n = 18, 9 male/9 female, 10 undergrads/8 grads; third offering: online only, n = 39, 24 male/15 female, 20 undergrads/19 grads). the training levels of students spanned sophomore undergraduates through second year phd students, with most enrolling in senior year of undergraduate or first year in the graduate program. such demographic information was determined from enrollment information; additional information on race and/or ethnicity as well as gender was not available nor collected for this work. the most represented disciplines were biomedical engineering and materials science and engineering with some the jigsaw design challenge 22 participation from mechanical, electrical, computer engineering, and biology. from an introductory survey, it was determined that approximately one to two thirds of students reported that they are currently or have in the past engaged in research that they would classify “bioelectronics” and that fewer than ten percent of students had taken a formal course in this subject matter in the past. the activity and assessment were considered part of regular coursework and, as such, are not considered human subjects research by our university. student responses were examined in aggregate. individual comments were anonymized and de-identified and only used to illustrate larger points. the jigsaw activity preparing the students to prepare the students for the jigsaw activities, rivnay made a concerted effort to establish the importance and need for group interaction early in the course, help students feel comfortable and at ease with teamwork, and manage their expectations. furthermore, he highlighted the multidisciplinary nature of the subject matter (bioelectronics) to draw parallels with the diversity of majors and scholarship of the student cohort itself. finally, he noted that such diverse teamlike settings are a norm in industry, government laboratories, and research laboratories. to help students be more comfortable with collaborative active learning strategies, he implemented think-pair-share activities and small group discussions into his lectures from the outset. he also incorporated discussion-based literature critiques into the course to build up the expectation for group interaction. early on, students were split into small groups and assigned an academic article on a specific topic. individually, students submitted a 1–2-page critique of the paper. that same day, the students who were assigned the same article met to discuss the reading and their critiques. afterwards, the different papers were compared, contrasted, and discussed in either a whole-class setting or in individual groups in which each student read a different paper. this mirrored the general format of the jigsaw assignment below, priming the students for the more complex activity. jigsaw implementation the 4–5-person expert and jigsaw groups were preassigned manually based on major and year in the program to ensure that at least one topical expert was in each jigsaw group. jigsaw groups were not revealed until after the expert group portion of the activity was completed. on the first day, rivnay spent about 30 minutes outlining the components, describing expectations and assessments, highlighting practical and real-world parallels, and generally motivating the students, explaining how cooperative peer learning would be beneficial (see table 1 for a breakdown of the schedule and activities). he also briefly described the findings and results found in jigsaw-based literature and placed four relevant articles (desai & kulkarni. 2016; doymus, 2008; kumar, 2017; davidson & major, 2014) on the learning management system (lms) for the interested student. the jigsaw design challenge 23 table 1 first iteration modified jigsaw activity flow, including timing/duration, in class activities, and assignments due day, duration activities, flow assessments due thur [1] – end of class, 30 min • motivate activity • describe flow, assessments • assign expert groups • pass out expert prompts • allow group 10 min to meet and plan tue [1] – entire period, 90 min • assign jigsaw groups • take turns peer teach/learn • hand out jigsaw quiz expert group write-up thur [2] – entire period, 90 min • draw design prompts randomly • free brainstorm 15 min in jigsaw groups • complete design challenge prompt • hand out debrief assignment jigsaw quiz (individual) jigsaw group design writeup (in class) tue [2] – beginning of class, 15-30 min • in class debrief: share with class what groups came up wit • open reflection debrief writeup canvas survey in the first step, students were placed into expert groups in order to deeply learn about one specific topic area as “experts” (figure 1). for this project, the topics are fabrication/materials, mechanical properties, bio-signal transmission, and biological response, all areas requiring special attention when designing a bioelectronic probe/device. as experts, students needed to identify key terms, processes, and governing equations associated with their specific topic and prompt, which will guide their researched content and approach to peer-teaching. in addition, they are asked to highlight key tradeoffs and application-specific weights for these fundamental topics as they relate to the global topic (bioelectronic devices) depending on factors ranging from use case, required lifetime, type of signal/implementation, etc. the nature of these topics would be specific to the course and activity envisioned, providing suggested starting points to guide the the jigsaw design challenge 24 expert groups’ research including a short list of general, required, and topical bullet points with associated papers and review reference citations. figure 1 expert (or topic) groups—tasked with researching and teaching a topic in-depth students were placed in the 4–5 person teams preformed by the instructor for all group activities (8 groups in the first and third iterations and 4 groups in the second iteration). initial expert (or topic) groups (see figure 1) were assigned to ensure diversity of academic class; however, where possible, the student’s major was used to match them to the closest possible expert topic to ensure a level of topical familiarity by at least one or two group members (e.g., mechanical engineers in the mechanical properties group or biomedical engineers/biologists in the biosignals/biological response group). experts from each expert/topic group come together to form a jigsaw group in which each member has in-depth knowledge in a complementary area. within each group, peer-teaching facilitates knowledge transfer, so everyone learns about the other topic areas (see figure 2). figure 2 jigsaw groups—peer teaching/learning with expertise from different topical areas the innovation (design challenge) rather than ending the cooperative team interactions with the jigsaw group peer teaching and learning, students met for a second session with their jigsaw group and were randomly assigned a design challenge. the challenge consisted of a short prompt describing a use-case of a bioelectronic device. for example, students might be asked to design a skin-worn device to record ecg signals and temperature the jigsaw design challenge 25 to monitor babies in the neonatal intensive care unit, a skin patch to monitor local skin impedance and biochemical cues during wound healing, or a device to monitor intracranial pressure after trauma or neurosurgery. the design challenges are all different and nuanced but are linked by a similar set of design considerations related to the expert topics. the groups spend time brainstorming their design prompt and then collaborate to complete a prompt. this part of the activity highlights the multidisciplinary and teamwork aspects of bioelectronics in an applied way. rather than learning about abstract concepts such as those in the expert topic groups, students must apply these concepts as a team to solve the design challenge (see figure 3). figure 3 jigsaw design challenge—cooperative teamwork to solve distinct practical design problems, which have a common foundation in the expert group topics finally, working individually or in their jigsaw groups, students were asked to look into literature and assess how their approach compared to current and relevant bioelectronics research and development. they then individually reflected on the entire activity through a written prompt and survey. the entire activity progressed as visualized in figure 4 over the course of 2–3 weeks (see figure 4). the class-period by class-period breakdown (timing, duration), specific activities, and associated assignments are described in table 1. the jigsaw design challenge 26 figure 4 flow and modified jigsaw activity from the perspective of one student (denoted as orange puzzle piece “1”) variance in later offerings in the second and third iterations, the instructor extended the time allotted for expert group preparations (week 1) and allotted more time for debrief and reflection. the third iteration was entirely remote, adding additional complications. to account for this, the jigsaw quiz assessment was removed, more time was allotted for jigsaw group teaching and learning, and the jigsaw design challenge was accomplished over 24 hours (rather than in one 90-minute class period). students were asked to abide by the honor code and not use outside resources so that they did not base their design on a published approach. the extended time was especially useful in a remote setting where technical issues and non-ideal working environments preclude students from an ideal collaborative virtual work environment during the scheduled class time. assessment students were informed that the jigsaw activity would be in lieu of a formal, written midterm assignment/exam. in the first iteration, this activity was called a “jigsaw midterm” while later it was simply referred to as a “jigsaw activity.” the aggregate scoring made the entire jigsaw activity worth 25% of the course grade with the following breakdown: expert group summaries (30%), individual quiz (10%), jigsaw design challenge write-up (40%), and debrief write-up/survey (20%). participation in the jigsaw groups in class was largely confirmed by presence, and absence or severe tardiness was docked on an individual basis as it adversely affects the team. otherwise, group portions were graded as a group (same grade). the jigsaw design challenge 27 the expert group documents were assessed based on a provided prompt, looking for concepts and depth of understanding but also some mention or discussion of how the students in the expert group will approach the task of teaching their peers. in the current iteration, this task is meant to guide the students’ outside research such that they address at minimum a set of preassigned topics that are required. they are asked to go beyond this, however, and identify other interesting topics to cover within their expert group and to cite their sources. the individual jigsaw quiz (first and second iteration) is an eight-question takehome quiz covering the topics “required” of the expert groups and thus expected of all jigsaw group members after their peer-teach sessions. this is akin to the assessment in the original aronson implementation. class notes and a “cheat sheet” from their jigsaw peers is allowed, and the total value of the quiz is low (encouraging completion, but not penalizing students if their jigsaw group faltered in peer teaching/learning). as noted above, this assessment was removed in the later iterations due to the pandemic, and the results of the activity as a whole were not affected. the design challenge write-up is the core component of this assignment. it highlights how the students worked as a team, applying core theoretical concepts to a specific applied task. the nature of the questions encourages creativity in design, but there is no “right” or “wrong” design solution. the students are asked to justify and defend their design choices based on the expert topic areas. this approach adheres to real-world engineering design and, as an authentic assessment, (villarroel et al., 2018) also models how expertise is identified in the disciplines that the students are studying (sternberg, 2003). finally, the debrief write-up is meant to be brief; however, in the end, this is actually a long activity, and the initial fear on the part of rivnay was that such a multi-part activity was risky and cumbersome. the debrief was assessed based on the students’ reflection on their own design and how it compared with those in current literature, allowing students to draw parallels between the priorities and tradeoffs they navigated with those of published researchers. evaluation and findings in each iteration, we collected information about student perceptions of the jigsaw process using both formal and informal measures in all three iterations of the course. in addition to the 1–2 page reflection described above, students completed a classroom assessment technique (cat) (angelo & cross, 1993), which consisted of an anonymous survey administered through the learning management system course site. the survey consisted of 10 quantitative and qualitative questions (see below for examples) related to collaborative group work, peer-to-peer learning, and perceptions around their own learning. rivnay also collected informal feedback from the teaching assistants from end-of-term standardized student evaluations and made notes about his observations of the group interactions and discussions. in the third iteration, which occurred during the pandemic, the activities were carried out via online zoom groups, which were not readily observable. observations consisted the jigsaw design challenge 28 of the perceived level of engagement and participation of team members, total duration of discussion, and depth of discussion beyond key concepts (in peer teaching, for example). overall, the survey, the debrief write-up, the informal feedback from students, and the observation during the jigsaw component collectively indicate that students appreciated the goals and purpose of the cooperative activities and learned from the experience although they certainly noted aspects that could be improved upon or changed. informal observation and feedback the instructor’s in-class observation indicated that students were initially hesitant to engage with each other at the start of the course but became more conversant when they were asked to regularly engage in small group discussions throughout the term. in the jigsaw activity, most students seemed engaged, even enthusiastic, in both the peer teaching and design challenge portions. some groups seemed to fly through the peer teaching and then would sit quietly for 20–30 minutes or more, requiring some guiding questions to maintain conversation through instructor or teaching assistant intervention. these informal observations were confirmed by what the students noted in the reflective debrief and the survey. reflective debrief as prompted, students compared their own group’s design project with similar engineered devices or approaches discussed in the scholarly literature. students commented on the feasibility of their own designs, noting places where improvements or enhancements could be made given sufficient time and resources. a few individuals noted crucial components or factors that they had left unexplored or otherwise failed to consider. many distinguished what would have led to more optimal solutions. all were able to critically evaluate their own work, contextualizing their application of key concepts in their own design to research activities in the field. class survey quantitative findings the survey results from all 92 students indicated that the group work component was overwhelmingly positive. greater than 92% of students agreed (a) or strongly agreed (sa) when asked about the groups working well together, and 85% valued learning from peers. students were less comfortable as topic experts in their jigsaw groups, with only 78% reporting that they felt comfortable (a+sa). most notably when asked if the jigsaw activity was helpful in demonstrating the multidisciplinary and cooperative nature of bioelectronics, there was a positive response (a+sa) of 91% (2% d+sd). the jigsaw design challenge 29 these findings resulted from the following questions where respondents were asked to rank their agreement: 1. i felt that my expert group worked well together 2. i felt that my expert group divided up tasks evenly/fairly 3. i felt comfortable as a topic expert in my jigsaw group 4. i felt that my contributions were valued by my peers 5. i found it valuable to learn from my peers 6. my jigsaw group worked well as a team to solve the design problem 7. the jigsaw activity was helpful in demonstrating the multidisciplinary and cooperative nature of bioelectronics research and development. the tabulated responses to all questions are presented graphically in figure 5. figure 5 quantitative debrief survey results note: results from year to year were in good agreement. color coding as noted in legend, from right to left: strongly agree (sa), agree (a), neither agree nor disagree (n), disagree (d), and strongly disagree (sd). n = 91 qualitative findings expectations. students were first asked about their expectations for the jigsaw activity and how their experience compared in practice. out of the 91 responses (34, 18, 39), two-thirds of the students (n = 59) noted that they found the experience to be better than expected, commenting that the activity helped them learn the material, and they found the collaboration with peers more enjoyable as well. the jigsaw design challenge 30 i honestly thought the jigsaw activity was going to be the bane of my existence. i dreaded it the moment [the professor] brought it up in class. however, and i'm happy to see... this activity really was engaging and pretty fun actually. to not just be tested on materials we have to study but be able to freely apply our knowledge that we've gained from our time at [the university] but also from learning about the topic in our topic groups. the culmination of it was really inspiring to me and i feel like i learned a lot about how to think of different aspects when creating a biomedical device. it felt really gratifying. fourteen students (4, 4, and 6) noted that the activity met their expectations, with nearly all indicating this in positive terms. fifteen students (7, 5, and 3) indicated that the activity did not meet their expectations, with most noting that the activity was harder or more time-consuming than expected. several students gave responses that could be described as “mixed,” with some parts positively exceeding their expectations, and some parts not meeting them. the workload in [the] expert group is heavier than my expectation before, because we got a lot of things to study and we also have to generate handout, slides, as well as cheat sheet to teach our jigsaw groupmates. i think it’s really cool to become an expert in one area, though the process can be overwhelming. perceptions of cooperative aspects. students were also asked to explain the extent to which cooperating with their peers helped them learn. several gave responses that could be coded in more than one category. of the 91 students (34, 18, 39), 80 students (31, 16, 33) commented positively on the collaborative aspect of learning from their peers and being able to talk over ideas. one student noted the following: it was useful to discuss the different topics in the expert group, as it helps you to think further than what's written in the papers. it was interesting to see the group dynamic during the design approach. we were also able to have constructive discussions about certain aspects. defending your point of view and explaining its advantages is an incredibly good way of learning in my opinion. while another student suggested these benefits: explaining concepts/information to others was a great way to cement it in my mind and force me to think critically about what i learned. it also helped to see how others in my expert and jigsaw group thought about the subject material. with a lecture you only observe the profs. [sic] way of thinking; with individual projects you only observe your own way of thinking; but with this project i was able to observe seven other people's approaches to researching/understanding subject material. the jigsaw design challenge 31 additionally, 18 students (9, 1, 8) noted that having to teach the material to a peer helped them learn the material more deeply. having to teach my peers about my topic motivated me to learn the material well before the activity. also, having a practical outlet for the information right after learning it helped cement some of the knowledge. however, it should be noted that 17 students (7, 2, 8) said that they found it difficult to learn from their peers although several of these students were the same who commented on learning more deeply from teaching the material. i did not feel [i] became an expert for my peers. my group did what we could to find information, but i personally felt we hit a brick wall with finding information and didn't know how to get around it. also, to be honest, i felt like i was flailing around the entire process not fully sure what to do but no idea how to ask for help either. i conceptually understand but when i got to physically doing it, i became stuck. improvements and revisions while the majority of the students felt the peer-teaching session was valuable for discussing ideas and learning from others, about 10% of the students indicated that they felt their peers did not take the activity seriously. a few students also commented that they did not feel that they were “experts” and did not feel comfortable teaching their peers. additionally, students also noted aspects of the activity that could be improved, encouraging rivnay to make the following changes: 1. allocate more time to the activity: many students thought more time could have been allocated to the first step of the activity, allowing expert groups to do research and formulate a teaching plan as well as to the third step with the design challenge. however, a few students indicated the opposite, wishing that less time had been devoted to the activity overall. 2. provide more explicit instruction: some students indicated that more direction in the expert topics would have been beneficial. others commented that the knowledge they had gained while exploring a very broad area in the topic/expert group was not effectively used owing to their specific design prompt. 3. make quizzes ungraded: some students were concerned about being evaluated individually, particularly for the handful who viewed the teaching by their peers to be suboptimal. in the original jigsaw activity of aronson, this component is meant to provide motivation for the peerteaching tasks, which is why the instructor employed it in the first iteration. however, the addition of another cooperative jigsaw session (design challenge) seemed to motivate the students, suggesting that the quiz did not play an overly important role in motivating the students to the jigsaw design challenge 32 participate fully. indeed, when the instructor removed the quiz component in later offerings, there was little qualitative change in performance or student interaction. the instructor used the student feedback to make several crucial changes to the structure, assessment, and communication of the jigsaw activity. additionally, he recognized the need to do the following: 1. get buy-in from students: the entire activity requires buy-in that can be built over the course of 3–4 weeks using discussion-based activities and group work and through adequate motivation in an introductory lecture or activity introduction. 2. clarify the rationale underlying group formation: after some students conveyed displeasure about not being able to select their own groups, rivnay spent more time explaining the importance of forming the groups around their relative experience. he was more transparent and intentional about the learning objectives associated with the manually formed groups, emphasizing the cooperative nature of the activity (desai & kulkarni, 2016). 3. revise language: rivnay also rethought using the term “expert” (in “expert group”) because it seemed to set unrealistic expectations for the students, even when using this term in quotations and despite qualifying with the use of the word relative. in the second iteration we began using the term “topic groups” instead, with each student in the jigsaw group a “topic lead.” comments related to concerns over the need to be an expert decreased following this change. discussion the consistent and positive response to the jigsaw activity was striking across a three-year span, particularly since the third iteration was administered fully online within the context of the pandemic. even though students spoke informally about the difficulties of engaging with their teammates during this third iteration (an idea that had not been expressed in the pre-pandemic in-person iterations), they were careful to attribute their disengagement to being online (and to the pandemic) rather than to the jigsaw activity itself, which they still viewed as mostly positive. given that their overall performance did not decline in comparison to the previous two iterations, this suggests that the jigsaw activity—including the design challenge twist—is portable across teaching modalities (for both in-person and online contexts). the jigsaw activity, with its design challenge twist, provides a crucial cooperative learning opportunity within an authentic real-world context (herrington & herrington, 2006). offering far more than a simple discussion-based group learning activity, this jigsaw innovation could be readily extended to other courses, particularly in those in which multifaceted design and system considerations the jigsaw design challenge 33 spanning disciplines may be necessary or current research and development efforts are rapidly evolving. such an approach would also be fruitful in those learning contexts where instructors wish their students to develop expertise around key topics or concepts (nolan et al., 2018) and, even more critically, develop confidence around that expertise (crone & portillo, 2013). while rivnay did not specifically seek to measure improvement in self-esteem or confidence in implementing this activity, research on jigsaw, as conducted in a variety of contexts, indicates that employing such cooperative-based activities can elicit positive changes in confidence (crone & portillo, 2013), self-esteem (johnson et al., 1998a) and motivation (tran, 2019). additionally, the debrief on the design challenge will allow students to make connections to real-world literature and examples in more authentic ways than what they might experience through simply writing a research paper or taking an exam (herrington & herrington, 2006). similarly, the debrief, during which students share their design approaches with other jigsaw teams, is intended to demonstrate how the same set of core expertise and rationale could be applied towards vastly different applications and use-cases. this type of debrief could be adapted to other contexts, where the expertise of a given field or discipline is modelled as is the notion that cooperation in a real-world context relies on experts sharing and making sense of their own and one another’s knowledge. while the jigsaw activity with design innovation requires thought, care, and transparency to implement effectively, it provides an invigorating way to help students acquire and communicate key concepts and ideas and offers an authentic opportunity for students to learn cooperatively and deeply. conflict of interest statement the authors declare that there are no conflicts of interest regarding the publication of this article. acknowledgements the authors gratefully acknowledge the organizational and intellectual support of northwestern’s searle fellows program as well as financial support from the national science foundation grant no. nsf dmr-1751308. references angelo, t. a., & cross, k. p. (1993). classroom assessment technique examples. classroom assessment techniques: a handbook for college teachers (2nd ed.). jossey-bass. aronson, e. (1978). the jigsaw classroom. sage. aronson, e., & bridgeman, d. (1979). jigsaw groups and the desegregated classroom: in pursuit of common goals. personality and social psychology bulletin, 5(4), 438–446. https://doi.org/10.1177/014616727900500405 the jigsaw design challenge 34 booker, t. a. (2021). taking a (modified) jigsaw to it: an in-class method to teach students to write a literature review. college teaching, 69(1), 58–60. https://doi.org/10.1080/87567555.2020.1809984 crone, t. s., & portillo, m. c. (2013). jigsaw variations and attitudes about learning and the self in cognitive psychology. teaching of psychology, 40(3), 246– 251. https://doi.org/10.1177/0098628313487451 davidson, n., & major, c. h. (2014). boundary crossings: cooperative learning, collaborative learning, and problem-based learning. journal on excellence in college teaching, 25. http://celt.muohio.edu/ject/fetch.php?id=592 desai, t. s., and kulkarni, p. (2016). “cooperative learning” tool for optimizing outcomes of engineering education." journal of engineering education transformations, 30. doymus, k. (2008). teaching chemical equilibrium with the jigsaw technique. research in science education, 38(2), 249–260. https://link.springer.com/article/10.1007/s11165-007-9047-8 freeman, s., eddy, s. l., mcdonough, m., smith, m. k., okoroafor, n., jordt, h., & wenderoth, m. p. (2014). active learning increases student performance in science, engineering, and mathematics. proceedings of the national academy of sciences, 111(23), 8410–8415. https://doi.org/10.1073/pnas.1319030111 goolsarran, n., hamo, c. e., & lu, w. h. (2020). using the jigsaw technique to teach patient safety. medical education online, 25(1). https://doi.org/10.1080/10872981.2019.1710325 herrington, a., & herrington, j. (2006). what is an authentic learning environment? in a. herrington & j. herrrington (eds.), authentic learning environments in higher education (pp. 1–14). information science publication. johnson, k. m. (2019). implementing inclusive practices in an active learning stem classroom. advances in physiology education, 43(2), 207–210. https://ro.uow.edu.au/cgi/viewcontent.cgi?article=2212&context=edupapers johnson, d., johnson, r., & smith, k. (1998a). active learning: cooperation in the college classroom (2nd ed.). interaction book co. johnson, d., johnson, r., & smith, k. (1998b). cooperative learning returns to college: what evidence is there that it works? change: the magazine of higher learning, 30(4), 26–35. https://doi.org/10.1080/00091389809602629 knight, j. k., & wood, w. b. (2005). teaching more by lecturing less. cell biology education, 4(4), 298–310. https://doi.org/10.1187/05-06-0082 the jigsaw design challenge 35 kumar, v. c. s., kalasuramath, s., patil, s., kumar, r. k. g., taj, s. k. r., jayasimha, v. l., ... & chacko, t. (2017). effect of jigsaw co-operative learning method in improving cognitive skills among medical students. international journal of current microbiology and applied sciences 6(3), 164–173. https://doi.org/10.20546/ijcmas.2017.603.018 laal, m., & laal, m. (2012). collaborative learning: what is it? procedia-social and behavioral sciences, 31, 491–495. https://doi.org/10.1016/j.sbspro.2011.12.092 minifie, j. r., & davis, k. (2013). ensuring gen y students come prepared for class; then leveraging active learning techniques to most effectively engage them. american journal of business and management, 2(1), 13–19. namaziandost, e., gilakjani, a. p., & hidayatullah. (2020). enhancing preintermediate efl learners’ reading comprehension through the use of jigsaw technique. cogent arts & humanities, 7(1). https://doi.org/10.1080/23311983.2020.1738833 nolan, j. m., hanley, b. g., divietri, t. p., & harvey, n. a. (2018). she who teaches learns: performance benefits of a jigsaw activity in a college classroom. scholarship of teaching and learning in psychology, 4(2), 93–104. https://doi.org/10.1037/stl0000110 prince, m. (2004). does active learning work? a review of the research. journal of engineering education, 93(3), 223–231. https://doi.org/10.1002/j.21689830.2004.tb00809.x roehl, a., reddy, s. l., & shannon, g. j. (2013). the flipped classroom: an opportunity to engage millennial students through active learning strategies. journal of family & consumer sciences, 105(2), 44–49. slavin, r. e. (1980). cooperative learning. review of educational research, 50(2), 315–342. sternberg, r. j. (2003). what is an “expert student?” educational researcher, 32(8), 5–9. https://doi.org/10.3102/0013189x032008005 tran, v. d. (2019). does cooperative learning increase students' motivation in learning? international journal of higher education, 8(5), 12–20. villarroel, v., bloxham, s., bruna, d., bruna, c., & herrera-seda, c. (2018). authentic assessment: creating a blueprint for course design. assessment & evaluation in higher education, 43(5), 840–854. https://doi.org/10.1080/02602938.2017.1412396 microsoft word editorletter_final.docx journal of effective teaching in higher education, vol. 3, no. 1 dear contributors, reviewers, and readers: spring 2020 has been an uncertain and difficult time for higher education around the world. the effects of covid-19 have been far-reaching, and we hope that during these times you have been able to stay safe and healthy as you support your students, colleagues, and communities. despite the obstacles, we are grateful for the efforts of reviewers and authors to help us produce a spring issue of the journal of effective teaching in higher education (jethe). although originally intended to be published in april 2020, the current issue was slightly delayed. we want to thank our colleagues from around the world who have submitted articles for consideration and served as reviewers— especially in the midst of the global pandemic. thank you! we also want to thank those who downloaded and read the articles published in our first three issues (november 2018, april 2019, & november 2019). the interest in publishing with jethe has been incredible, and the overall acceptance rate for articles is 30.3%. interest in articles published in jethe has been equally impressive with over 3100 full-text downloads of articles published in our first three issues. on average, articles have been downloaded approximately 150 times each. we believe that our current issue demonstrates our continued commitment to high quality, data driven scholarship as represented in the four pieces included in volume 3 issue 1. the four articles focus on a range of topics relevant to teaching in higher education today. the first two articles in this issue examine the syllabus and coteaching, respectively, while the third article analyzes the implementation of metacognitive note-taking (mnt) at an institution in vietnam. we chose to close this issue with a dialogic reflection on online teaching—a topic that resonates with the current climate within higher education. each piece provides meaningful implications for the practice of teaching across disciplines—an important contribution that is emphasized in jethe. we invite you to engage with the pieces in our may 2020 issue and consider making your own contributions to this work. we also invite you to become a reviewer for jethe. if you are interested in reviewing, please send an email to the editors at jethe@uncw.edu indicating the subjects related to sotl in higher education that most interest you and about which you have expertise. please also attach your cv to that email. our next issue will be published in november 2020, and we look forward to working with some of you as authors and reviewers in production of that and future issues of jethe. to submit an article or download full-text copies of current and past issues please visit http://jethe.org. please stay safe and well! james devita and colleen reilly, coeditors, jethe microsoft word dandotkar_etal_final.doc journal of effective teaching in higher education, vol. 5, no. 1 https://doi.org/10.36021/jethe.v5i1.251 knowing about knowing: assessing how reflective writing influences undergraduate students’ epistemic beliefs srikanth dandotkar, university of southern indiana, sri.dandotkar@gmail.com laura e. cruz, pennsylvania state university jeffrey r. stowell, eastern illinois university m. anne britt, northern illinois university abstract. epistemic beliefs are one’s assumptions about knowledge and knowing. given the research in educational psychology that established epistemic beliefs as reliable predictors of student success, we devised a pedagogical intervention to improve students’ epistemic beliefs. in this study, we examined the effectiveness of the reflective writing task as a means of changing how students think about what is known. students from two upper-level psychology classes (cognitive and research methods) took a general epistemic belief survey by rating their agreements with 38 items at three different times in a semester (first-day, pre-reflective-writing task, and post-reflective-writing task). day 1 responses were utilized to validate the survey items using principal component analysis—three variables (knowledge construction and modification-kcm, structure of knowledge-kst, and meaning of successful students-ss) emerged. the intervention successfully improved students’ beliefs specific to structure of knowledge-kst and meaning of successful studentsss, beliefs that predict student learning. this study suggests that even short interventions have the potential to influence students’ beliefs about knowledge, which have been shown to have demonstrable effects on their academic success. keywords: epistemic beliefs; epistemic reflection; non-cognitive factors; reflective writing in his 2020 essay entitled “what’s the problem now?,” randy bass (2020) argues that learning is a sufficiently complex construct and that we (as scholars and instructors) should shift from viewing it as a problem to be solved to viewing it as an opportunity to exercise our collective imagination to find novel pathways through it. his approach was affirmed by a recent meta-analysis of published scholarship of teaching and learning (sotl) articles that concluded that a majority of the studies under-theorized learning, focusing primarily on assessing cognitive outcomes related to specific course material (manarin et al., 2021). this disconnect has not occurred because of a lack of theory from which to draw. educational theorists have long recognized that learning has multiple dimensions beyond the cognitive (e.g., behavioral, affective), and the more recent scholarship has articulated additional layers, with particular interest in the metacognitive and socalled non-cognitive factors that significantly influence learning (kautz et al, 2014). the term “non-cognitive” is perhaps a misnomer as the components (e.g., grit, growth mindset, curiosity, self-concept, goal orientation) are not completely distinct knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 2 from cognitive or other thinking processes, but they do share a common positionality in that they often foreground how students learn in the classroom (barrett, 2014). several high-profile studies have established strong theoretical and applied linkages between the cultivation of positive non-cognitive traits and improved student outcomes, especially in, but not limited to, stem fields (cavigliaharris & maier, 2020). carol dweck’s (dweck, 2008; yeager et al., 2019) studies of fixed versus growth mindset in middle school mathematics students and duckworth et al.’s (2007) work with grit in high-achieving college students are perhaps two of the best-known examples. that said, recently both grit and growth mindset have been criticized as weak predictors of academic success (maddi et al., 2017), and the search continues for a fuller understanding of the interplay between non-cognitive factors and student learning (stankov & lee, 2014). several scholars have posited the significance of epistemic beliefs as non-cognitive factors that influence student learning. epistemic beliefs are defined as a student’s assumptions about knowledge (hofer & pintrich, 1997; schommer, 1990) and knowing (schommer, 1990). according to schommer (1990), epistemic beliefs are a system of independent beliefs, and these beliefs vary on a continuum of sophistication (see table 1 for the list of beliefs and the unsophisticated and sophisticated views about these beliefs). as the table reflects, epistemic beliefs fall into five major categories: speed (how long will it take to learn), structure (how complex the subject matter is), construction (how new knowledge is created), success (what they need to do to master the subject), and objectivity (how they view the objectivity of the subject matter). table 1 general epistemic beliefs on a continuum of sophistication beliefs unsophisticated view sophisticated view b1. speed of knowledgeacquisition-ksp quick learning slow learning b2. structure of knowledge-kst simple knowledge complex-knowledge b3. knowledge construction & modification-kcm passive-learning active learning by questioning b4. meaning of successful students-ss innate fixed ability acquired incremental ability b5. obtaining objective truth-ot certainty-knowledge probabilistic knowledge note: taken from schommer (1990). a number of seminal studies in educational psychology have established the validity of epistemic belief as a construct. schommer-aikins et al. (2005) found that middle knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 3 school students’ epistemic beliefs directly predicted their overall gpa and their mathematics performance. in a series of studies, schommer (1990; 1993) and schommer et al. (1992) demonstrated that high sophistication in each of the speed of knowledge acquisition-ksp, structure of knowledge-kst, and obtaining objective truth-ot subscales (see table 1) was significantly and independently correlated to secondary student learning across multiple disciplines, as measured either by mastery tests or student artifact analysis. more recently, scholars have engaged in secondary analysis of large education datasets (e.g., the program for international student assessment or pisa) to identify statistically significant linkages between epistemic beliefs and student learning in a variety of national contexts (khine et al., 2020; she et al., 2019). karakolidis et al. (2019), for example, identified epistemic beliefs as significant mediators between high motivation and low achievement for 5,532 greek students. taken collectively, these studies of secondary students have served not only to establish the viability of epistemic belief as a construct but also to link it to student success. to date, however, there have been few, if any, commensurate studies of college students’ general epistemic beliefs. rather, the scholarly emphasis had fallen largely on domain-specific ways of knowing, and studies of disciplinary epistemologies (e.g., beliefs about science or history) are prevalent in some fields (donald, 1990; park & lee, 2004; wiley et al., 2020). this perspective follows the disciplinary orientation of much work in sotl (huber, 2013), but the recent shift in emphasis towards integrated, cross-disciplinary learning as a critical component of general education has challenged the efficacy of this compartmentalization. in this context, scholars have worked to identify schemas that incorporate broad epistemic beliefs along with one, or more, domain-specific ways of knowing. the theory of integrated domains in epistemology (tide) model, for example, classifies beliefs as either domain-specific, general, personal, or all three (muis et al., 2006; muis et al., 2016), leading some scholars to argue that the respective constructs may serve different, though perhaps complementary, functions (buehl & alexander, 2006; schraw, 2001). accordingly, the current study explores the domain-general nature of epistemic beliefs, particularly in the less studied population of college students. in addition to debates over the relative salience of general versus domain-specific beliefs, researchers also question the extent to which such beliefs can be learned, and, if so, what appropriate developmental models may apply. the well-known perry taxonomy (1970), for example, suggests levels of epistemic development across the four-year undergraduate experience. schommer’s work (1993) affirms the latter stages of perry’s model, indicating that beliefs characterized as sophisticated were prevalent among students graduating from four-year institutions. further, king and kitchener (1994) found that epistemic assumptions underlying reflective judgments of participants varied as a function of their number of years in college. similarly, schommer (1992) found that participants’ educational levels predicted their beliefs about the nature of knowledge: the higher the level of one’s education the less likely that the person would believe in simple (structure of knowledge-kst) and certain knowledge (obtaining objective truth-ot). the logical corollary to the proposition that epistemic beliefs can be learned, at knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 4 least while attending college, would be the assumption they can also be taught. however, it remains unclear what specific practices in college life provide students with the opportunity to realign their beliefs about knowledge and knowing. numerous courseand program-based studies of domain-specific interventions exist (e.g., kienhues et al., 2008), but these remain limited in replicability, especially with the emergence of studies emphasizing significant differences in epistemic belief development across both disciplines and levels of educational attainment (rosman et al., 2017). perhaps the most promising practice is epistemic reflection, a practice first introduced by baxter magolda (1992) in her longitudinal study of the epistemic beliefs of 101 incoming freshmen. through their reflective writing exercises, baxter magolda argues, the students demonstrated the underlying connection between the development of their personal and academic epistemic beliefs. that said, her study follows students over a significant period (14 years), which invites questions regarding the appropriate timespan for achieving and, by extension, measuring changes in epistemic beliefs. the latter point about measuring is salient to the present study. several researchers have noted persistent challenges in assessing deep-set beliefs as potential constructs are subject to variations in epistemic belief at the meta-level. most studies to date have relied on a variety of validated self-report assessment instruments, with the epistemic belief survey (ebs) used in the present study (wood & kardash, 2002) joining two others, the epistemological questionnaire (schommer, 1990) and the epistemic beliefs inventory (schraw et al., 2002), as the predominant scales (debacker et al., 2008). each of these tools are subject to the increasingly pointed criticisms of student self-report instruments in educational research, but they have been effective in demonstrating changes in epistemic belief over time, at least in the sense that these are articulated in the respective instruments. to both isolate when changes in epistemic beliefs occur and by what means, the present study utilizes a pre/post design with the ebs to assess changes in how psychology students view general ways of knowing after engaging in reflective writing exercises specifically designed to influence those beliefs. the study the purpose of the current study was to explore the effectiveness of a reflective writing task in changing college students’ domain-general epistemic beliefs pertinent to speed (how long will it take to learn: ksp), structure (how complex the subject matter is: kst), construction (how new knowledge is created: kcm), success (what they think one needs to master the subject: ss), and objectivity (how they view the objectivity of the subject matter: ot) of knowledge or knowing. the effectiveness of the intervention was compared across two upper-level psychology classes to examine whether knowledge of research methods plays any role in ascertaining the effectiveness of the intervention. participants’ domain-general epistemic beliefs specific to five dimensions (ksp, kst, kcm, ss, & ot) were captured by an epistemic belief survey consisting of 38 items (wood & kardash, 2002). while the specifics of this survey are provided in the “instrument” section, we would like to briefly identify the variables in the study, knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 5 how they were measured, and the purpose of using multiple analyses. the intervention, along with the preand post-assessments, served as a within-subjects factor, that is, the same participants took the survey before and after the intervention (reflective writing) while class (research methods & cognitive) served as a between-subjects factor, that is, as two different groups. participants’ epistemic scores, specific to five dimensions, served as the critical dependent measures. before calculating the dependent measure for each dimension (i.e., average epistemic score), we conducted a principal component analysis (pca) to identify and isolate those items that were reliable and unique in capturing the assumed dimension. as will be discussed in the “preliminary results” section, this yielded 11 items that uniquely measured only three (kst, ss, kcm) of the original five dimensions. this resulted in three dependent measures, that is, average epistemic scores specific to kst, ss, & kcm dimensions. consequently, we ran (as reported in the “main results”) three individual anovas, one on each dependent measure (specific to kst, ss, & kcm). appendix a lists all the identified 11 items under their respective dimensions. additionally, appendix b lists the original 38 items and the dimension to which each belongs. further, we have reported these analyses under two different headers to underscore the different purposes these two analyses serve: pca under preliminary analyses and anovas under main results. we hypothesized that the epistemic belief scores for all five dimensions (or the number of factors the factor-analytic method yields, which you already know to be three factors) post-reflective task would be lower compared with pre-reflective task. a lower score indicates a more sophisticated view. this prediction is based on king and kitchener’s (1994) findings specific to the reflective judgment task. we also hypothesized the effect of the intervention would be larger for the research methods class. this is because students in research methods classes learn about knowledge and knowing, which may give them an advantage in utilizing the intervention over others. participants for this institutional review board approved study, our initial sample consisted of 53 undergraduate students enrolled in two upper-division psychology courses (cognitive processes: n = 32; research methods: n = 21) at a medium-sized, state comprehensive university located in the midwestern region of the u.s. the two classes are comparable in that mostly second-semester juniors and seniors take these classes. also, the background knowledge and characteristics of these students, though not measured, was comparable. the difference in sample size between the groups has to do with the limited class size of the research methods class. this should not pose an issue statistically as we tested the assumptions of the statistical analyses before interpreting their results. also, there were four students who were a part of both the classes, so their data were excluded from the analyses, resulting in the sample size of 45. although no demographic data were collected, the program-level data suggest that students majoring in psychology identify predominantly as a female and either white or mixed-race. knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 6 instrument we used a condensed version of the epistemic belief survey (ebs) (wood & kardash, 2002) to measure epistemic beliefs about the nature and acquisition of knowledge. it has 38 items (e.g., “you can believe most things you read”) anchored on a 5-point likert-type scale ranging from 1 (strongly-disagree) to 5 (stronglyagree). the ebs has five dimensions: speed of knowledge acquisition (ksp), structure of knowledge (kst), knowledge construction and modification (kcm), meaning of successful students (ss), and the attainability of objective truth (ot). please see appendix b for a complete list of these items. a total score for each dimension is computed by averaging its respective items. the ebs instrument itself is widely viewed as reliable and consistently demonstrates sound psychometric properties (wood & kardash, 2002). in a study validating this instrument, wood & kardash (2002) reported reliable cronbach’s alpha values for ksp, kst, kcm, ss, and ot as .74, .72, .66, .58, and. 54, respectively. procedure a paper-based ebs was administered on the first day of class (baseline measure), after week eight (pre-reflection), and after week nine (post-reflection). students completed the survey during regular class time. no identifiable information was collected. between the second and third administration, students read a review article by jung (2011) that summarized the literature on epistemic beliefs and explained their role in education. afterwards, they wrote a reflective essay in which they identified the nature of their own epistemic beliefs, explained the reason for their identification, and described what could be done to refine their epistemic beliefs. the purpose of this task was to trigger epistemic doubt (bendixen, 2002), leading to dissonance and ultimately a resolution of the dissonance to establish more refined epistemic beliefs. students participated in this study as part of a course assignment for which they received credit. design and analyses we adopted a mixed-methods design with the intervention as the within-subjects factor and class (research methods or cognitive processes) as the betweensubjects factor. to determine which ebs dimensions were relevant to our sample, we used principal component analysis (pca), which yielded three dimensions (reported in the preliminary results section below). therefore, the average epistemic scores specific to each of these three epistemic dimensions alone served as the critical dependent measures (i.e., average epistemic scores pertinent to structure of knowledge-kst, meaning of successful students-ss, and knowledge construction and modification-kcm). we performed three separate 2 x 2 mixedsubjects anovas, one on each identified dimension (reported in the main results subsection). knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 7 results preliminary results given that the epistemic questionnaires such as the epistemic belief survey (ebs) produce sample-specific results (bråten, et al., 2019; ferguson et al., 2013), we scored it using a principal component analysis (pca). pca is a dimensionality reduction technique that allows researchers to determine which items uniquely capture a latent-construct (e.g., structure of knowledge-kst) and no other dimensions. this suggests that the identified items for a given dimension would be reliable in measuring the supposed (sub-)construct (i.e., construct validity). because only uniquely contributing items are identified to extract dimensions, two important consequences emerge: 1) not all items in a dimension are necessarily selected, and 2) not all dimensions (five in this survey) are necessarily captured. the ebs responses from the first day were used to extract latent variables (epistemic dimensions). as recommended by ferguson et al. (2013), we removed items that were either unrelated (factor loading <1.5) or negatively related to other items in that dimension. this resulted in 15 items. an additional four items were removed because they loaded on more than one dimension. an initial principal component analysis with oblique rotation yielded three factors–structure: structure of knowledge-kst, knowledge construction and modification-kcm, and meaning of successful students-ss. eleven items met the kaisser-guttman retention criteria of eigenvalues greater than unity and explained 53.62% of sample-variation. these items were used to calculate average epistemic belief scores (dependent measure) for each dimension in both pre-reflection and post-reflection conditions. please see appendix a for the extracted dimensions and factor loadings related to each dimension. main results three 2 x 2 mixed-subjects anovas, one on each pca-identified belief (kst, ss, & kcm) were conducted. it is a common practice to use quantitative analysis such as anova when the dependent measures are calculated using student responses on a likert scale as a recent review makes clear (willits et al., 2016). the following assumptions were tested each time: a) independence of observations, b) normality, c) homogeneity of variance, d) sphericity, and e) equality of covariances. in each case, all these assumptions were met. a 2 x 2 mixed-subjects anova suggested no significant interaction between intervention and class on the average epistemic score specific to structure of knowledge dimension (p = 0.056). however, there was a significant main effect of intervention, f(1, 43) = 7.65, p = .008, ηp2 = .15. as such, the average scores pertinent to the structure of knowledge belief decreased from pre-reflection (m = 3.56; se = .09) to post-reflection (m = 3.4; se = .11). as stated previously, a lower number indicates a more refined view in each dimension. this means, the intervention led to a refining of epistemic beliefs about the structure of knowledge. knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 8 in other words, students perceived the structure of knowledge as more complex after the intervention. similarly, a 2 x 2 mixed-subjects anova suggested no significant interaction between intervention and class on the average epistemic score specific to successful students dimension (p = .08). however, there was a significant main effect of the intervention for the meaning of successful students-ss belief, f(1, 43) = 6.28, p = .016, ηp2 = .13. as such, the average belief scores about the meaning of successful students-ss decreased—refined from pre-reflection (m = 2.68; se = .12) to post-reflection (m = 2.44; se = .11). in other words, the belief that successful or smart students are inborn decreased after they participated in the intervention. finally, a 2 x 2 mixed-subjects anova on the average epistemic score specific to knowledge construction and modification did not yield any significant interaction or main effects (all fs<1). discussion the possibility of changing general epistemic beliefs through short-term teaching interventions is tantalizing as it opens the door for the identification of other interventions that could form the basis for an emerging body of evidence-based pedagogical practice. as a reminder, however, this study emphasized general, rather than domain-specific, beliefs, but the trend in differentiated results between the two classes (as seen in marginal interaction effects) hints at the potential interplay between the two levels. it certainly seems possible that students engaged in practices regarding the construction of disciplinary-based knowledge, as would be expected in a research methods class, may be predisposed towards greater openness regarding general epistemic beliefs. this hypothesis does not, however, explain why these students registered gains in the structure of knowledge-kst and meaning of successful students-ss beliefs and not the knowledge construction and modification-kcm belief. a partial explanation may lie in the nature of the knowledge construction process associated with the field of psychology. previous experimental studies have emphasized that general epistemic beliefs are not universal and show moderate degrees of sensitivity to disciplinary context (faber et al., 2016; muis et al., 2016). a number of studies of differences in general epistemic beliefs along the biglan (1973) classification of disciplines, for example, found that students majoring in disciplines classified as “hard,” i.e., paradigmatic, often registered as less sophisticated on measures of knowledge structure-kst and objectivity-ot (hofer, 2000; king et al., 1990; paulsen & wells, 1998). one study specifically compared epistemic beliefs in psychology, classified as a “soft” discipline, and the “hard” sciences, noting that psychology students were more likely to recognize complexity (i.e., knowledge structure-kst) and hard science students register stronger beliefs that truth is obtainable (i.e., obtaining objective truth-ot) (hofer, 2000). the patterns revealed in this study (i.e., gains in structure-kst and not in obtaining objectivity-ot) are consistent with these domain-specific findings, but the differences between the two courses of the same domain challenge this hypothesis. knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 9 a future study that assesses similar interventions specifically across research methods courses in a variety of disciplinary contexts might prove to be illuminating. not all disciplines have research methods courses, and they may be positioned at different stages of the curriculum. as noted in the literature review above, previous studies have emphasized that the epistemic beliefs of students change as they move through their undergraduate coursework. that means that another possible explanation for the results of this study could be found in the preand post-test designed throughout the four years of college. remember, the majority of students who participated in this study were already thirdor fourth-year students, and the pre-tests in both classes indicate relatively high degrees of sophistication even prior to the intervention, making it less likely that gains could accrue. by its nature, the study of psychology emphasizes the complexity of learning and, by extension, knowledge, and the signature pedagogy of the field emphasizes the engagement of undergraduate students in the conduct of research or knowledge creation (peden & wilson, 2009). constructs such as growth mindset, as reflected in the successful students-ss subscale, would likely be familiar to many psychology majors. rather than asking why the students in the cognitive class did not register similar gains, we might ask ourselves why the students in the research methods class had more space for gains to register, a question that could be answered by extending the scope of this study to encompass previous coursework in the major. in other contexts, it might be tempting to interpret these findings through the lens of gender, as epistemic beliefs have been shown to be mildly gender-specific in previous studies (wang et al., 2015), but the sample used appeared to be predominantly female, with an insufficient number of males to allow for statistical comparisons. this may have turned out for the best, however, as gender constructs have become the subject of intense scholarly scrutiny, with recent advances in neuropsychology serving largely to debunk many prior assumptions regarding cognitive differences between men and women, now referred to as neurosexism (eliot, 2019). at the same time, however, an increasing number of comparative studies have suggested that epistemic beliefs are sensitive to cultural context, extending from the conditions present in the family home to the broader geopolitical region the student inhabits (bernholt et al., 2021; chan & elliott, 2004). this suggests that future studies should include demographic questions related to cultural background, or perhaps not, as this finding also suggests that the long-standing constructs used in the ebs may need to be critically evaluated for its suitability in measuring epistemic beliefs in cross-cultural contexts. previous studies utilizing the ebs have often incorporated qualitative components to provide deeper insight, a practice which becomes even more salient for capturing likely increasingly diverse constructs of epistemic beliefs. the discerning reader may be slightly disappointed to reach this point in the discussion and realize that this review of potential explanatory factors has not resulted in a clear alternative hypothesis for the findings of this study. what it reveals instead is that the development of epistemic beliefs in college students is a highly complex process, subject to a large and expanding range of confounding variables, and one that does not necessarily fit neatly into a single construct, knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 10 assessment instrument, discipline, or set of students. rather than despair at this result, this study joins others in providing a glimmer of hope that epistemic beliefs can be influenced through teaching interventions, and that realization will hopefully serve to inspire others to try other interventions that might bring these deepseated and often implicit beliefs, held by both faculty and students alike, to light. the sheer complexity of studying epistemic beliefs challenges us to find novel means for enhancing our own beliefs about how knowledge is developed, applied, and fostered in higher education—and perhaps beyond. as bass (2020) himself implores, perhaps we “should balance our well-placed dedication to evidence with professional competency in imagination” (p. 3). conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references barrett, l. (2014). what counts as (non)cognitive? a comment on rowe and healy. behavioral ecology, 25(6), 1293–1294. bass, r. (2020). what’s the problem now? to improve the academy: a journal of educational development, 39(1). baxter magolda, m. b. (1992). knowing and reasoning in college: gender-related patterns in students' intellectual development. jossey-bass. bendixen, l. d. (2002). a process model of epistemic belief change. in b. k. hofer & p. r. pintrich (eds.), personal epistemology: the psychology of beliefs about knowledge and knowing (pp. 191–208). lawrence erlbaum associates. bernholt, a., lindfors, m., & winberg, m. (2021). students’ epistemic beliefs in sweden and germany and their interrelations with classroom characteristics. scandinavian journal of educational research, 65(1), 54–70. biglan, a. (1973). the characteristics of subject matter in different academic areas. journal of applied psychology, 57(3), 195–203. bråten, i., brandmo, c., & kammerer, y. (2019). a validation study of the internetspecific epistemic justification inventory with norwegian preservice teachers. journal of educational computing research, 57(4), 877–900. buehl, m. m., & alexander, p. a. (2006). examining the dual nature of epistemological beliefs. international journal of educational research, 45, 28–42. caviglia-harris, j., & maier, k. (2020). it's not all in their heads: the differing role of cognitive factors and non-cognitive traits in undergraduate knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 11 success. education economics, 28(3), 245–262. chan, k.-w., & elliott, r. g. (2004). epistemological beliefs across cultures: critique and analysis of beliefs structure studies. educational psychology, 24(2), 123–142. doi:10.1080/0144341032000160100 debacker, t. k., crowson, h. m., beesley, a. d., thoma, s. j., & hestevold, n. l. (2008). the challenge of measuring epistemic beliefs: an analysis of three self-report instruments. journal of experimental education, 76(3), 281–312. doi:10.3200/jexe.76.3.281-314 donald, j. g. (1990). university professors’ views of knowledge and validation processes. journal of educational psychology, 82(2), 242–249. duckworth, a. l., peterson, c., matthews, m. d., & kelly, d. r. (2007). grit: perseverance and passion for long-term goals. journal of personality and social psychology, 92(6), 1087–1101. dweck, c. s. (2008). mindset: the new psychology of success. random house digital. eliot, l. (2019). neurosexism: the myth that men and women have different brains. nature, 566(7745), 453–455. faber, c., vargas, p., & benson, l. (2016, october). measuring engineering epistemic beliefs in undergraduate engineering students. 2016 ieee frontiers in education conference (fie), 16505079, 1–7. doi:10.1109/fie.2016.7757461 ferguson, l.e., bråten, i., strømsø, h. i., & anmarkrud, ø. (2013). epistemic beliefs and comprehension in the context of reading multiple documents: examining the role of conflict. international journal of educational research, 62, 100–114. hofer, b. k. (2000). dimensionality and disciplinary differences in personal epistemology. contemporary educational psychology, 25, 378–405. hofer, b. k., & pintrich, p. r. (1997). the development of epistemic theories: beliefs about knowledge and knowing and their relation to learning. review of educational research, 67(1), 88–140. huber, m. t. (2013). the scholarship of teaching and learning in and across disciplines. indiana university press. jung, e. j. (2011). literature review: the importance of epistemology in education: the role of epistemology. department of educational psychology and instructional technology university of georgia. knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 12 https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.518.8409&rep=r ep1&type=pdf. karakolidis, a., pitsia, v., & emvalotis, a. (2019). the case of high motivation and low achievement in science: what is the role of students’ epistemic beliefs? international journal of science education, 41(11), 1457–1474. kautz, t., heckman, j. j., diris, r., ter weel, b., & borghans, l. (2014). fostering and measuring skills: improving cognitive and non-cognitive skills to promote lifetime success. oecd education working papers, no. 110, oecd publishing, paris. https://doi.org/10.1787/5jxsr7vr78f7-en khine, m. s., fraser, b. j., & afari, e. (2020). structural relationships between learning environments and students’ non-cognitive outcomes: secondary analysis of pisa data. learning environments research, 23, 395–412. kienhues, d., bromme, r., & stahl, e. (2008). changing epistemological beliefs: the unexpected impact of a short-term intervention. british journal of educational psychology, 78(4), 545–565. king, p. m., & kitchener, k. s. (1994). developing reflective judgment: understanding and promoting intellectual growth and critical thinking in adolescents and adults. jossey-bass. king, p. m., wood, p. k., & mines, r. a. (1990). critical thinking among college and graduate students. the review of higher education, 13, 167–186. maddi, s. r., matthews, m. d., kelly, d. r., villarreal, b. j., gundersen, k. k., & savino, s. c. (2017). the continuing role of hardiness and grit on performance and retention in west point cadets. military psychology, 29(5), 355–358. manarin, k., adams, c., fendler, r., marsh, h., pohl, e., porath, s., & thomas, a. (2021). examining the focus of sotl literature—teaching and learning? teaching & learning inquiry, 9(1), 349–364. muis, k.r., bendixen, l.d., & haerle, f. (2006). domain-generality and domainspecificity in personal epistemology research: philosophical and empirical reflections in the development of a theoretical framework. educational psychology review, 18, 3–54. muis, k. r., trevors, g., duffy, m., ranellucci, j., & foy, m. j. (2016). testing the tide: examining the nature of students’ epistemic beliefs using a multiple methods approach. the journal of experimental education, 84(2), 264–288. park, j., & lee, l. (2004). analysing cognitive or non-cognitive factors involved in the process of physics problem-solving in an everyday context. international journal of science education, 26(13), 1577–1595. knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 13 paulsen, m. b., & wells, c. t. (1998). domain differences in the epistemological beliefs of college students. research in higher education, 39(4), 365–384. peden, b. f., & wilson vanvoorhis, c. r. (2009). developing habits of the mind, hand, and heart in psychology undergraduates. in r. gurung & n. chick (eds.), exploring signature pedagogies: approaches to teaching disciplinary habits of mind (pp. 161–182). stylus. perry, w. g., jr. (1970). forms of intellectual and ethical development in the college years: a scheme. jossey-bass. rosman, t., mayer, a. k., kerwer, m., & krampen, g. (2017). the differential development of epistemic beliefs in psychology and computer science students: a four-wave longitudinal study. learning and instruction, 49, 166– 177. schommer-aikins, m., duell, o. k., & hutter, r. (2005). epistemic beliefs, mathematical problem-solving beliefs, and academic performance of medical school students. the elementary school journal, 105(3), 289–304. schommer, m. (1990). effects of beliefs about the nature of knowledge on comprehension. journal of educational psychology, 82(3), 498–504. schommer, m. (1992, october 14–17). predictors of epistemic beliefs: comparing adults with only a secondary education to adults with post-secondary education [conference presentation]. fourteenth annual meeting of the midwestern american educational research association, chicago, il, united states. schommer, m. (1993). comparisons of beliefs about the nature of knowledge and learning among postsecondary students. research in higher education, 34(3), 355–370. schraw, g. (2001). current themes and future directions in epistemological research: a commentary. educational psychology review, 13, 451–465. schraw, g., bendixen, l. d., & dunkle, m. e. (2002). development and validation of the epistemic belief inventory (ebi). in b. k. hofer & p. r. pintrich (eds.), personal epistemology: the psychology of beliefs about knowledge and knowing (pp. 261–275). lawrence erlbaum associates. she, h. c., lin, h. s., & huang, l. y. (2019). reflections on and implications of the programme for international student assessment 2015 (pisa 2015) performance of students in taiwan: the role of epistemic beliefs about science in scientific literacy. journal of research in science teaching, 56(10), 1309–1340. knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 14 stankov, l., & lee, j. (2014). quest for the best non-cognitive predictor of academic achievement. educational psychology, 34(1), 1–8. wang, m.-t., degol, j., & ye, f. (2015). math achievement is important, but task values are critical, too: examining the intellectual and motivational factors leading to gender disparities in stem careers. frontiers in psychology, 6, 36, doi:10.3389/fpsyg.2015.00036 wiley, j., griffin, t. d., steffens, b., & britt, m. a. (2020). epistemic beliefs about the value of integrating information across multiple documents in history. learning and instruction, 65, 101266. https://doi.org/10.1016/j.learninstruc.2019.101266 willits, f. k., theodori, g. l., & luloff, a. e. (2016). another look at likert scales. journal of rural social sciences, 31(3), 126–139. wood, p. k., & kardash, c. a. (2002). critical elements in the design and analysis of studies of epistemology. in b. k. hofer & p. r. pintrich (eds.), personal epistemology: the psychology of beliefs about knowledge and knowing (pp. 231–260). lawrence erlbaum associates. yeager, d. s., hanselman, p., walton, g. m., murray, j. s., crosnoe, r., muller, c., tipton, e., schneider, b., hulleman, c. s., hinojosa, c. p., paunesku, d., & dweck, c. s. (2019). a national experiment reveals where a growth mindset improves achievement. nature, 573(7774), 364–369. knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 15 appendix a list of identified items under the corresponding extracted dimensions with factor loadings for each item that resulted from the final principal component analysis variables & items factor loadings kcm kst ss q25. i try my best to combine information across chapters or even across classes. .81 q20. today’s facts may be tomorrow’s fiction. .65 q10. you should evaluate the accuracy of information in textbooks if you are familiar with the topic. .47 knowledge construction & modification eigenvalue = 2.50 q21. i really appreciate instructors who organize their lectures carefully and then stick to their plan. .76 q30. it is annoying to listen to lecturers who cannot seem to make their mind up as to what they really believe. .73 q13. if professors would stick more to the facts and do less theorizing, one could get more out of college. .65 q26.i don’t like movies that don’t have a clear-cut ending. .57 knowledge structure eigenvalue = 1.80 q19. successful students understand things quickly. .8 2 q29. understanding main ideas is easy for good students. .7 2 q35. the really smart students don’t have to work hard to do well in school. .6 1 q14. being a good student generally involves memorizing a lot of facts. .5 2 successful students eigenvalue = 1.60 knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 16 appendix b items from the epistemic belief survey listed under the intended epistemic belief category epistemic belief (latent factor) item # item speed of knowledge acquisition (ksp) 3 if something can be learned, it will be learned immediately. 7 almost all the information you can understand form a textbook you will get during the first reading. 11 you will just get confused, if you try to integrate new ideas in a textbook with knowledge you already have about a topic. 16 working on a difficult problem for an extended period of time only pays off for really smart students. 18 usually, if you are ever going to understand something, it will make sense to you the first time. 24 if i can't understand something quickly, it usually means i will never understand it. 34 most words have one clear meaning. 38 the information we learn in school is certain and unchanging. structure of knowledge (kst) 4 i like information to be presented in a straightforward fashion; i don't like having to read between the lines. 5 it is difficult to learn from textbook unless you start at the beginning and master one section at a time. 12 when i study, i look for specific facts. 13 if professors would stick more to the facts and do less theorizing, one could get more out of college. 21 i really appreciate instructors who organize their lectures carefully and then stick to their plan. 26 i don't like movies that don't have a clear-cut ending. 28 it's waste of time to work on problems that have no possibility of coming out with a clear-cut answer. 30 it is annoying to listen to lecturers who cannot seem to make their mind up as to what they really believe. 31 a good teacher's job is to keep students from wandering from the right track. knowing about knowing journal of effective teaching in higher education, vol. 5, no. 1 17 33 the best thing about science courses is that most problems have only one right answer. 36 when i learn, i prefer to make things, as simple as possible. knowledge construction & modification (kcm) 2 the only thing that is certain is uncertainty itself. 6 forming you own ideas is more important than learning what the textbooks say. 8 a really good way to understand a textbook is to reorganize the information according to your own personal scheme. 10 you should evaluate the accuracy of information in textbooks if you are familiar with the topic. 15 wisdom is not knowing the answers but knowing how to find answers. 20 today's facts may be tomorrow's fiction. 22 the most important part of scientific work is original thinking. 23 even advice from experts should be questioned. 25 i try my best to combine information across chapters or even classes. 32 a sentence has little meaning unless you know the situation in which it was spoken. 37 i find it refreshing to think about issues that experts can't agree on. characteristi cs of successful students (ss) 14 being a good student generally involves memorizing a lot of facts. 17 some people are born good learners; others are just stuck with a limited ability. 19 successful students understand things quickly. 29 understanding main ideas is easy for good students. 35 the really smart students don't have to work hard to do well in school. attainment of objective truth (ot) 1 you can believe most things you read. 9 if scientists try hard enough, they can find the answer to almost every question. 27 scientists can ultimately get to the truth. microsoft word linkennette_jethe_final.docx journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.250 creating an inclusive learning community to better serve minority students phoebe s. lin, framingham state university lynne n. kennette, durham college (canada), lynne.kennette@durhamcollege.ca abstract. as campuses become increasingly diverse, it is important that faculties maintain inclusive classrooms. students of underrepresented ethnic/racial groups are more likely to experience disengagement in an academic setting (nagasawa & wong, 1999), which can lead to underperformance (major et al., 1998). students with lgbtqa+ (lesbian, gay, bisexual, transgender, queer, or asexual) identities are at higher risk of poor mental health and lower academic performance compared to cisgender and heterosexual students (aragon et al., 2014). these detrimental experiences can lead to even more harm in a remote learning environment, where students have fewer opportunities to feel a sense of belonging and connect with their peers and/or instructors. this paper will consider strategies of inclusiveness in the online classroom and in-person learning environment within a social psychology framework to better support underprivileged students to improve academic performance and the overall educational experience. the suggestions and discussions provided apply to both in-person learning as well as remote delivery. keywords: diversity; inclusion; minority; prejudice; classroom some aspects of diversity on campuses are easy to see, but some of this diversity is less visible. creating a feeling of inclusion for all students is important for students to feel safe in order to learn (cast, 2018). additionally, this sense of inclusion may be especially important in a remote learning environment where students may feel more isolated and disconnected to begin with. social psychologists have proposed that all humans possess the need to belong (baumeister & leary, 1995), a cross-cultural and inherent drive that motivates forming social bonds to feel accepted by others. creating a classroom environment that is a safe space that meets the needs of all students is both a best practice and a component of universal design for learning (udl) and has been shown to increase student learning, and, ultimately, student success (cast, 2018). this paper will examine concepts related to classroom environments that may not feel welcoming to all (e.g., microaggressions, stereotype threat), as well as approaches to ensure a more inclusive classroom (e.g., inclusive language, including the perspectives of minority groups) to improve all students’ overall learning experiences. we also include suggestions of what remote teachers can do to increase equity and better engage in outreach to support students in remote learning. we encourage instructors to incorporate these inclusive teaching practices across various modes of instruction (remote learning, hybrid, face-to-face) to embrace diversity and support minority students. the paper analyzes structural issues in the classroom, explores how they affect classroom dynamics and experiences, and proposes solutions primarily within a social psychology and pedagogical framework. much research on prejudice and inclusive teaching creating an inclusive learning community to better serve minority students 2 journal of effective teaching in higher education, vol. 4, no. 3 practices have been conducted by experts trained in social psychology, with additional recommendations from experts in education, so the inclusion of both of these perspectives allows for a more fulsome discussion. we have included several suggestions that are adaptable to courses across disciplines, though some may be more amenable to the social sciences, education, and the humanities. however, we have also included additional resources so that instructors can review other sources of inclusive teaching practices that may be more discipline-specific. criteria for selecting literature presented given that this paper is a critical analysis of contemporary literature in education and the social sciences, we chose to cite scholars who are experts in their field indicated by a substantial record of publication (e.g., shaun harper’s work on racial biases in education, claude steele and joshua aronson’s works on stereotype threat, derald wing sue and kevin nadal’s works on microaggressions) or scholars who are among the first to publish work on a groundbreaking concept that widely influenced their field (e.g., peggy mcintosh on white privilege, anthony greenwald on the implicit association test). for additional scholarship, we chose to cite those who are highly influential in their areas of expertise, evidenced by a high number of citations (e.g., steven aragon, penelope lockwood, anna woodcock) or those who have much knowledge in matters of diversity, equity, and inclusion, also indicated by their publication records (e.g., sharon fries-britt, richard nagasawa). books we have chosen to include (white fragility: why it’s so hard for white people to talk about racism by robin diangelo and me and white supremacy: combat racism, change the world, and become a good ancestor by layla f. saad) were selected based on the authors’ credentials and expertise (a former professor of multicultural education and a law school graduate, respectively), positive critical reception of their work, and their influence in academia (e.g., both books are regularly assigned in faculty book groups that emphasize diversity, inclusion, and anti-racism). for online teaching resources that are tailored toward college instructors, we included resources from institutions with an exemplary record of commitment to diversity and inclusion evidenced by diversity awards (presented to the university of delaware, center for teaching and assessment for learning and the university of southern california, rossier school of education), or institutions that have been awarded numerous grants with the goal of launching and sustaining diversity initiatives (cornell university, center for teaching innovation and the university of michigan, center for research on teaching and learning). finally, in recognizing the importance of universal design to support inclusion (in all aspects of life, but especially education), we also included resources from cast, the center which developed the universal design for learning (udl) framework in the mid-1980s, and has since become the resource for educators regarding udl. we felt it was important to include cast because it promotes the design of safe learning spaces for all students, knowing that an unsafe learning space creates a barrier to learning and reduces the motivation to learn. creating an inclusive learning community to better serve minority students 3 journal of effective teaching in higher education, vol. 4, no. 3 combating microaggressions that lead to inequity in higher education microaggressions are subtle forms of prejudice that can be verbal (insulting comments), behavioral (expressing surprise at a minority student’s academic accomplishments), or environmental (a male-dominated workplace that conveys a message that female employees are not welcome), which some have argued can be more harmful than overt prejudice (sue, 2010). cumulative experiences with microaggressions can lead to poorer mental health, diminished confidence, and feelings of self-doubt (pierce, 1995), which lead to students feeling further disengaged, decreasing motivation to improve performance, choosing a different major, transferring to a different institution, devaluing education, or dropping out of college (fogliati & bussey, 2013; woodcock et al., 2012; yosso et al., 2009). all of these outcomes can be exacerbated in a remote environment, where it is already more challenging for instructors to engage students with the course material and motivate achievement. to facilitate greater rumination about how white privilege and systems of oppression perpetuate inequality, instructors can consider having students complete a white privilege checklist, which includes aspects of belonging (seeing oneself reflected in media representations), assumptions of law-abiding behavior (not being followed in a store), and assumptions of intelligence and ability (perceptions of being hired due to qualifications rather than filling an affirmative action quota) (mcintosh, 1988). we encourage instructors to complete this checklist as well (regardless of their racial identity) to reflect on the topic along with their students. following completion of the scale, instructors can initiate a dialogue asking students how white privilege impacts them both directly and indirectly and what are continued harms of white privilege that operate within a culture of white supremacy. to encourage empathy, instructors can have students complete a microaggressions checklist, which addresses more subtle aspects of prejudice, such as having one’s citizenship status questioned/doubted or being ignored in a professional setting (nadal, 2011) to reflect on how their identities afford them various privileges or, conversely, how their identities may make access to such privileges more challenging. we also recommend that instructors complete the checklist to ruminate over their own experiences with microaggressions to encourage greater awareness of how race/ethnic identity affects how we navigate the world and interactions with others. following either (or both) of these exercises, instructors can provide recommendations on combating microaggressions in social and professional contexts to facilitate more productive intergroup communication and relations. for instance, instructors can encourage more privileged students to refrain from interrupting others when speaking given that men are more likely to interrupt women and white individuals are more likely to interrupt people of color (elsass & graves, 1997; smith-lovin & brody, 1989). we should also remind more privileged students to use their privilege to advocate for others, such as by amplifying the contributions of people of color, given that contributions from those with marginalized identities are more likely to be overlooked (liss et al., 2019). similarly, instructors must be mindful of and follow their own advice, especially by creating an inclusive learning community to better serve minority students 4 journal of effective teaching in higher education, vol. 4, no. 3 reminding themselves that there are tangible power dynamics within the classroom, and, as authority figures, instructors generally hold privileges that students may not (age, education level, job stability, financial status, etc.). research in social psychology has shown that improved communication between group members and across different social groups facilitates cooperation, strengthens group cohesion, increases trust, and encourages individuals to adhere to group norms by setting them at the beginning of the interaction (bornstein et al., 1989; bouas & komorita, 1996; drolet & morris, 2000; pruitt, 1998). all of these outcomes can lead to a more cooperative and supportive learning environment that is more inclusive of diversity, whether pertaining to identity, background, or perspective. a meta-analysis (analyzing 23 data sets from 20 empirical studies) found that various aspects of a student’s social identity can lead to lower grades due to automatic stereotype activation tied to implicit biases (malouff & thorsteinsson, 2016). factors that led to lower grades included racial/ethnic identity (being a student of color) and underachievement in past academic performance. to avoid activating these stereotypes, which can lead to microaggressions related to assumptions of abilities, instructors should consider anonymous grading to remove potential biases associated with students’ identities (this is already a feature of most learning management systems). thus, educators should reach out to those who maintain the learning management system, if such an office exists, to obtain technical support in successfully utilizing these online teaching tools to best serve their students. if such an office does not exist at the institution, instructors could attend online workshops, webinars, or discussion groups to receive support from one another to maximize teaching strategies in encouraging and empowering minority students. whenever possible, instructors should also consistently give positive feedback on student work and in response to student comments during discussions (whether in person or online) to increase self-efficacy and combat the negative effects of prejudice. in combination with anonymous grading, instructors should create and consistently use grading rubrics to assess assignments in order to increase objectivity in evaluating student work. this method, along with anonymous grading, can be implemented in any discipline, and instructors should communicate these methods to students consistently throughout the semester to highlight a commitment to equality in reducing biases in assessment. additionally, students will likely appreciate transparency in the grading process, especially if they are provided with grading rubrics beforehand, given that honest communication increases trust (pornpitakpan, 2004). quinn (2021) found that using a rubric reduced the likelihood of stereotypes affecting instructor perceptions of students’ work, increasing equality and objectivity. with no rubric, a submission from a “black sounding” student name (dashawn) was consistently given a lower grade for the same writing sample as a submission from “white sounding” student name such as connor (quinn, 2021). with the use of a rubric, however, teachers gave much more similar evaluations of both writing samples. interestingly, quinn (2021) also assessed teachers’ implicit racial biases via the implicit association test (iat, greenwald et al., 1998) and found that without rubrics, even those with more favorable attitudes toward african americans exhibited racial bias in grading. creating an inclusive learning community to better serve minority students 5 journal of effective teaching in higher education, vol. 4, no. 3 to motivate students to engage in greater self-reflection of their own potential racial biases, instructors can create an iat assignment in which students read background information pertaining to project implicit, take the iat (https://implicit.harvard.edu/implicit/index.jsp), and then respond to corresponding questions, such as “what is your reaction to your results?”, “how are you interpreting your results?”, and “do you think all human beings hold implicit bias toward outgroups? why or why not?” despite some issues with the iat test, it shows predictive validity such that high implicit bias predicts lower quality crossrace interactions (greenwald et al., 2009). thus, instructors can remind students to ruminate over their iat results and encourage them to be more mindful of their assumptions of others in cross-race interactions, which can discourage microaggressions such as making stereotypical statements or making comments that invalidate minority group members’ identities and experiences (“i don’t see color”). we strongly recommend that instructors also complete the iat and reflect on the questions assigned to the students, so they can be better aware of their own potential biases and engage in more open and honest dialogue with their students in follow-up discussions. racial spotlighting, a particular kind of microaggression, is placing unwelcome attention on a member of a minority group, which creates psychological tension (carter, 2008). examples of spotlighting include asking minority students to act as a spokesperson on behalf of their social group (ex. “what do latinx individuals think about this issue?”) or soliciting comments to obtain “a different perspective.” racial spotlighting can cause harm in several ways. asking one individual to act as a spokesperson conveys a false assumption that the social group is homogeneous with little to no variation. relatedly, specifying that a minority student’s views offer “a different perspective” assumes that the student is an outsider, creating an “us versus them” mentality that can exacerbate ingroup bias and outgroup derogation. thus, instructors should avoid singling out minority students and emphasize that there is much diversity and variation within all social groups. in an online learning community, instructors can consider reading aloud or sharing the work of a minority student anonymously as a teaching and learning tool. sometimes, it may be difficult for students to take the perspective of another group member, especially related to diversity and cultural issues, if they lack interactions and experiences with those of different identities/backgrounds. sharing the voice of a minority student amplifies the voices of those who are underrepresented in society, gives additional perspective to dominant group members, can encourage empathy and perspective taking, yet avoids spotlighting so that attention is focused on the speaker’s voice and perspective while preserving anonymity if they do not wish to be the focus of attention. in a remote environment, where instructors are more heavily utilizing technology as teaching tools, we recommend using polls to further engage the students by giving them the opportunity to express their reactions. using this technique, students will be able to share what they think in response to the course content, and the instructor can choose to share the results instantly by displaying aggregated responses on the screen, whether immediately or at a later time. by amplifying the voices of those who are underrepresented in society and sharing the group’s reactions (especially in real time), students may have a greater appreciation of diverse perspectives, and this technique can increase a sense of creating an inclusive learning community to better serve minority students 6 journal of effective teaching in higher education, vol. 4, no. 3 community in a remote environment. furthermore, allowing students to see one another’s responses in real time may increase a sense of bonding or validation by showing students that their peers are experiencing similar emotions or reactions. when sharing an individual student’s work, we recommend that the instructor provide an opportunity for the speaker to identify themselves if they wish to clarify or elaborate on points made or address questions others may have. it is important to note that the instructor should not pressure the student to reveal their identity if they do not feel comfortable doing so; thus, we suggest that the instructor emphasize the speaker’s autonomy in whether they wish to remain anonymous or identify themselves. in a remote learning environment, instructors have various options to share a student’s work including posting the assignment on the shared learning management site (e.g., blackboard, canvass, moodle, etc.) with identifying information removed, emailing the document to the class list, or reading the work aloud during an online session. with each of these methods, we encourage the instructor to make a statement beforehand (whether written or stated aloud), such as “if you are the author of this piece, please feel free to identify yourself if you would like to do so. if, however, you would prefer to remain anonymous, as an inclusive learning community, we respect your wishes. i encourage you to choose whichever option you are most comfortable with.” while we acknowledge that some disciplines (e.g., psychology or sociology) cover topics that more readily facilitate discussions of race-related issues, we encourage instructors to highlight a theme of equity throughout the semester to make students more mindful of how racial discrimination affects students of color. this may be especially helpful in encouraging white students to further ruminate over race-related issues, given that individuals who are unaffected by social issues are less likely to see them as problematic and, therefore, may be less motivated to engage in social justice activism (diangelo, 2018; saad, 2020). for instance, instructors can mention the racial achievement gap, a documented phenomenon in which white students tend to obtain higher grades and graduation rates compared to students of color, in conjunction with evidence that factors such as socioeconomic status, lack of access to resources, and race-based discrimination cause these discrepancies rather than differences in intrinsic ability (aronson et al., 1998; jacoby & glauberman, 1995; jencks & phillips, 1998; steele, 1997). students can then locate a scholarly article that amplifies the voices of diverse researchers who are experts in education and race-relations (see carnevale & strohl, 2013; harper, 2010; harper & davis, 2012; harper & hurtado, 2007; harper et al., 2009 as recommended readings) and write a reaction paper in response to the reading(s) to raise awareness of inequity in higher education. as a follow-up exercise, instructors can assign students an additional reflection assignment in which they must find a scholarly article that covers a topic relevant to the course written by a researcher from an underrepresented social group. students can then hold a small group discussion with classmates and/or write a paper that highlights the author’s contributions to the field and consider obstacles minorities are more likely to experience in research/academia compared to members of the dominant group. further, to reinforce a commitment to equity, instructors can provide data and statistics from their own institution (if available) creating an inclusive learning community to better serve minority students 7 journal of effective teaching in higher education, vol. 4, no. 3 that indicate race-based discrepancies in educational and/or professional outcomes to motivate greater anti-racism activism within the student body by making the social issue more salient and personally relevant. this exercise can also make faculty more aware of how they and their colleagues need to be part of the solution, and thus we recommend ongoing conversations at both the departmental and institutional level between faculty to address anti-racism in the classroom at the cultural level. microaggressions cause harm in various ways (sue, 2010), and victims of microaggressions are further disadvantaged in that deciding how to respond can be a lose-lose situation (yosso et al., 2009). those who confront perpetrators of microaggressions are often further invalidated in the form of racial gaslighting, which diminishes or discredits their claims and their overall experiences with prejudice (for instance, perpetrators may say the victim is being “too sensitive” or “can’t take a joke”). the other option for victims of microaggressions is to suffer in silence. however, perpetrators often claim that they were unaware that their comments were harmful or offensive. as an educational exercise, instructors could present commonly reported microaggressions (e.g., “did you get into college because of affirmative action?”) and facilitate a class wide discussion in which students analyze why these statements are harmful and problematic. these discussions can lead to students becoming better informed about social issues and increase awareness of injustices in both social and academic settings. in situations of remote teaching and learning, the instructor may consider using an online reporting system so that students can anonymously notify the instructor of prejudicial/discriminatory incidents committed by other students. the instructor can also make a statement encouraging more privileged students to advocate for others by reporting acts of bias that they overhear or witness that were directed toward students of marginalized identities. this could encourage greater anti-racism activism through direct actions and raise awareness of microaggressions that students of colors experience, which white students are less likely to experience. if an incident is reported, the instructor can then reach out to the perpetrator to better educate the student on why the act was harmful, act as a mediator between the target and perpetrator to engage in productive conflict resolution, if needed, and take punitive actions toward the perpetrator when warranted. we recommend that instructors also make statements to students at the beginning of the semester that if the instructor engages in behavior(s) that cause harm to students of marginalized identities, students should bring this up in an open dialogue to facilitate better understanding of how instructors may unintentionally perpetuate biases. this then, could facilitate productive conflict resolution to improve classroom dynamics. we also suggest that instructors state that if students are uncomfortable bringing the matter up directly to the instructor given the power dynamics, they should speak with the department chair, who can act as a mediator to facilitate dialogue between the instructor and student(s). if students would rather not speak to the chair, the instructor can recommend other potential mediators, such as an academic advisor, a student diversity committee/organization, a staff member who works in a division of creating an inclusive learning community to better serve minority students 8 journal of effective teaching in higher education, vol. 4, no. 3 diversity/inclusion, or a bias education response team if these committees/positions exist at the institution. combating stereotype threat the process of stereotype threat occurs when one is reminded of a negative selfrelevant stereotype prior to a cognitive task, which then impairs performance via distraction and anxiety, and can lead to disengagement (steele & aronson, 1995). disengagement involves decreased motivation and the devaluing of a specific domain, such as academic success. there is a great deal of evidence that the performance gap between caucasian students and students of color is not due to differences in intrinsic ability, but rather due to social factors, such as stereotype threat and a lack of equity in opportunities (aronson et al., 2002). a successful method of minimizing or negating the harmful effects of stereotype threat is for instructors to emphasize a malleable theory of intelligence. in other words, instructors can explain to students that abilities can improve with effort and practice (as opposed to a fixed theory, where abilities are inherent and constant). a malleable theory of intelligence can counter the harms of stereotype threat. students who adopt a malleable theory of ability show greater engagement with learning and higher academic achievement (aronson et al., 2002). thus, professors should emphasize the malleable theory of intelligence and consistently provide positive feedback that values effort, dedication, and improvement over inherent ability. using this method can also be a meaningful way for course instructors to develop a mentoring relationship with students in times of remote learning where interactions are more limited. minority students may struggle academically due to increased pressure to succeed and a lack of role models (fries-britt & turner, 2002; mcdonald et al., 2004). a strategy to inspire success is to provide successful role models with which the students can identify. role models provide inspiration by showing that someone similar has overcome challenges and can provide examples of strategies that promote success (lockwood, 2006). indeed, research has indicated that role models are most successful in motivating achievement when the target and role model are matched for sex/gender (marx & roman, 2002) or race/ethnicity (marx et al., 2009). role models can also buffer against stereotype threat. for instance, women can improve math performance by reading about successful women in science, technology, engineering, and math (mcintyre et al., 2005). therefore, professors should highlight successful exemplars from underrepresented groups for students to identify with. doing so can make success more salient, increase motivation, provide inspiration to improve learning outcomes, and improve the overall educational experience for minority students. another advantage of highlighting diverse scholars is to show that the field has benefited from the contributions of non-dominant groups. further, increasing representation whether related to race/ethnicity/culture, gender identity, and orientation conveys a message to the students that the field embraces and values the works of all individuals, not just members of the dominant group. instructors could promote this by choosing a specific textbook that highlights contributions of creating an inclusive learning community to better serve minority students 9 journal of effective teaching in higher education, vol. 4, no. 3 minority groups, sharing a photo of a diverse scholar when covering course content to highlight diversity in the field, and providing autobiographical information about diverse scholars when presenting lectures and during class discussions. using inclusive practices lgbtqa+ students are also susceptible to microaggressions and hostile learning environments. instructors should use gender inclusive language (for example, saying “you all” instead of “you guys”) to embrace students of all gender identities, especially those from underrepresented groups such as women, transgender individuals, and those with a non-binary gender identity. additionally, instructors should take attendance on the first day of classes in a way that avoids spotlighting transgender students and/or those with a non-binary gender identity. transgender students, whether they are in transition or have completed the transition process, often have a name and personal pronouns that differ from the legal/birth name and implied pronouns of the gender assigned at birth. professors should allow students to introduce themselves using their preferred names and ask for their pronouns in writing. we recommend that instructors ask students to state their preferred names aloud so that professors and peers know how to address one another but ask that students give their personal pronouns in writing because stating one’s personal pronouns aloud, especially if meeting a large group of strangers, may induce anxiety. for lgbtqa+ individuals, they perpetually need to make a conscious decision of whether they want to “come out” each time they meet someone (ellis et al., 2019). further, if one asks each student to state pronouns aloud on the first day of class, this forces those who may be questioning their gender identity to make a public declaration of their current status, which may change later (manion, 2018). thus, we recommend that professors state aloud on the first day of class that students should write their personal pronouns on a piece of paper to be collected at the end of class, and then add that if anyone’s pronouns change at any time, they are encouraged to notify the professor in a one-on-one conversation or through written communication to indicate that the instructor supports lgbtqa+ students and understands that one’s identity may change over time. further, this reinforces a norm of respect by demonstrating that the professor will adhere to a student’s pronouns whether they are consistent or change over time. in the format of remote teaching, instructors may wish to encourage students to turn on their web cameras during online sessions but should not pressure students to do so. students who are from a lower socioeconomic status background may not wish to turn on their cameras to give others “access” or a “view” into their living space. further, students with a transgender or non-binary identity may not feel comfortable being visible when they have the option to be heard, but not seen, especially if they are in the beginning or middle of a transition process. therefore, instructors can encourage participation through verbal contributions, using a chat function, and through the use of non-verbal reactions if available (such as using on screen reaction symbols to indicate agreement in zoom). this can increase engagement while being respectful of each individual student’s comfort level. creating an inclusive learning community to better serve minority students 10 journal of effective teaching in higher education, vol. 4, no. 3 another option is to use a virtual background if a student is comfortable being visible to others but does not wish to share the appearance of their living space. doing so shifts the central focus onto the participant rather than their place of residence, but facilitates interactions by maintaining eye contact, showing facial expressions, and highlighting body language. an additional benefit of using a virtual background is that students can also share their interests/hobbies, such as using a background that showcases their favorite sport, novel, musician, etc. which would allow students to connect over shared interests and perceive the class as a social unit. this can be especially helpful in a remote learning environment as opportunities to socialize with classmates in virtual classes are fewer, thus the instructor could also begin remote sessions early to allow students to bond, similar to how students can hold informal conversations with one another in a traditional classroom if they arrive early and are waiting for the class to begin. we encourage faculty to incorporate discussions of diversity and anti-racism at department meetings to facilitate exchange of ideas in promoting inclusive teaching practices and anti-racism activism. some suggestions for fields outside of social sciences include, for instance, in natural science courses (e.g., biology, chemistry, physics), instructors could discuss henrietta lacks (an african american woman whose cells were taken and studied without consent, leading to the discovery of hela cells) or the tuskegee syphilis study (an experiment in which african american men who had been diagnosed with syphilis were not disclosed their diagnosis status, nor were they provided with medical treatment) to highlight racial injustices in access to healthcare and ethical violations tied to racism within medical research. in certain math, economics, or business courses, the instructor could discuss redlining (the process of limiting whether african americans would be approved for a mortgage to purchase a home) and emphasize how this affected accumulated wealth tied to home ownership across generations, exacerbating the racial wealth gap. the instructor could then teach students the application of mathematical formulas and statistics to real world social issues that perpetuate inequality. for additional ideas, we recommend the center for research on teaching and learning (2021), center for teaching and assessment of learning (2021), center for teaching innovation, and usc rossier school of education (2021) for suggestions of additional inclusive teaching practices and resources which are relevant to various disciplines. using diverse perspectives and cooperation to enhance the learning experience the jigsaw classroom is an integrative learning technique in which students are given small portions of information on the same topic, and then work in small groups to teach one another their individual share of the course material (aronson, 1978). through the use of discourse, interdependence, and cooperation to accomplish a shared goal with communally beneficial outcomes, students learn to value one another’s contributions and respect one another regardless of differences in background or identity (aronson, 2002). the jigsaw classroom has indeed been shown to increase empathy, reduce racial prejudice, reduce intergroup hostility, and improve academic performance, especially for students of underrepresented groups (aronson & bridgeman, 1978; walker & crogan, 1998). therefore, creating an inclusive learning community to better serve minority students 11 journal of effective teaching in higher education, vol. 4, no. 3 instructors should assign groups (rather than asking students to select groups themselves) so that students have the opportunity to work with new individuals each time there is group work. this increases exposure to diversity in thought, perspective, and identity and promotes respect for those who are different from themselves. the use of team-based learning can be especially helpful in times of remote learning to facilitate social bonding among students, which can increase support for one another in facing academic challenges and allow students to feel interpersonally connected with one another. the benefits of group work and cooperation have also been demonstrated by implementing a superordinate goal, where individuals must work collaboratively to obtain an outcome that benefits everyone (sherif, 1954; sherif et al., 1961). the robbers cave field study (sherif et al., 1961) documented that ingroup bias and outgroup hostility arose between two groups of adolescent boys despite no group differences in ethnicity, age, or socioeconomic status. the simple categorization into two separate groups was enough to elicit competition, distrust, and outgroup denigration. researchers eventually implemented a superordinate goal, emphasizing that non-cooperation would mean a loss for all while successful cooperation would benefit all members of both groups. by facilitating cooperation, researchers found that the two groups, after obtaining a successful outcome, merged to form one larger group, no longer maintaining an “us versus them” mentality. additionally, the formation into a combined group promoted more positive attitudes across individuals and led to friendship development. though it may be more challenging to implement group work and group discussions in a remote format, online management tools/systems can facilitate this process. for instance, blackboard, brightspace/d2l, and other learning management systems allow the course instructor to create group pages so that students can share work and initiate group discussions, and zoom utilizes breakout rooms in which participants can form small groups to engage in discussions and collaborative efforts. additionally, instructors can encourage students to use google docs and/or google slides to work cooperatively on assignments and projects so that they are able to give one another constructive feedback and see one another’s edits and comments in real time. instructors should also encourage students to provide positive comments on each other’s work whenever possible to emphasize a learning community, foster closer working relationships, and reinforce that the class functions as a social unit to support one another in a remote learning environment. some cognitive psychologists recommend implementing collaborative note-taking during each class session such that two students are assigned as note-takers and simultaneously summarize the course content in a google doc, which is later made available to all students and the instructor (e.g., patson, 2021). alternating pairs of note-takers throughout the semester sets a norm of collaborative learning, signaling that students are responsible for their peers’ learning process in addition to their own. this technique also allows the instructor to review each set of notes to ensure that students adequately understand the course material, allowing the instructor to allocate more time to review any concepts students may be having difficulty with, ensuring greater success and learning outcomes. creating an inclusive learning community to better serve minority students 12 journal of effective teaching in higher education, vol. 4, no. 3 committing ourselves to anti-racism through greater awareness of white supremacy and other systems of oppression we encourage those in higher education (regardless of whether they are students or faculty) to read me and white supremacy (saad, 2020), a book that discusses the pervasiveness of apathy toward racism and how we can better commit ourselves to equity. saad (2020) directly addresses those who hold white privilege by addressing various topics that white individuals may not be aware of (tokenism, tone policing, double standards that benefit white individuals while harming people of color, etc.) and states that her book is an “anti-racism tool structured to help people with white privilege understand and take ownership of their participation in the oppressive system of white supremacy” (saad, 2020, p. 3). it is structured so that readers can read a new chapter each day on a specific topic pertaining to white privilege and/or white supremacy. each chapter also includes question prompts that encourage the reader to maintain a journal to reflect on how the information relates to their own lives and their own privileges, solidifying deeper commitment to antiracism. saad (2020) also includes many suggestions of actionable items of how white individuals can better use their white privilege to advocate for others. her examples explore how to continue our own journey of self-education about racial issues, including amplifying the voices and contributions of people of color, being more mindful in verbal communication to empower and center people of color, decentering oneself in conversations to better support people of color, avoiding cultural appropriation, and speaking about racism to other white individuals in a way that encourages greater mindfulness of the harms of racism and white supremacy. conclusion as enrollment for minority students continues to increase, including both domestic and international students, it is imperative for faculties in higher education to reinforce the message that diversity is valued by maintaining an inclusive learning environment, especially in remote situations where face to face meetings are not possible or are less frequent. faculties need to be sensitive to the various biases that these students are more likely to encounter, provide paths to success for underprivileged students, use language that encompasses individuals of all identities, and facilitate collaboration between students in order to send the message that differences in background are appreciated and valued. implementing these strategies creates a safer and more inclusive learning environment (cast, 2018) which can improve motivation, academic success, and retention, leading to greater professional success when students transition from higher education to long-term careers. creating an inclusive learning community to better serve minority students 13 journal of effective teaching in higher education, vol. 4, no. 3 conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references aragon, s. r., poteat, v. p., espelage, d. l., & koenig, b. w. (2014). the influence of peer victimization on educational outcomes for lgbtq and non-lgbtq high school students. journal of lgbt youth, 11, 1–19. https://doi.org/10.1080/19361653.2014.840761 aronson, e. (1978). the jigsaw classroom. sage. aronson, e. (2002). building empathy, compassion, and achievement in the jigsaw classroom. in j. aronson (ed.), improving academic achievement (pp. 209– 225). academic press. https://doi.org/10.1016/b978-012064455-1/50013-0 aronson, e. & bridgeman, d. (1978). jigsaw groups and the desegregated classroom: in pursuit of common goals. personality and social psychology bulletin, 5, 438–446. https://doi.org/10.1177/014616727900500405 aronson, j., fried, c. b., & good, c. (2002). reducing the effects of stereotype threat on african american college students by shaping theories of intelligence. journal of experimental social psychology, 38, 113–125. https://doi.org/10.1006/jesp.2001.1491 aronson, j., quinn, d., & spencer, s. (1998). stereotype threat and the academic underperformance of women and minorities. in j. swim & c. stangor (eds.), stigma: the target’s perspective. academic press. baumeister, r. f., & leary, m. r. (1995). the need to belong: desire for interpersonal attachments as a fundamental human motivation. psychological bulletin, 117, 497–529. https://doi.org/10.1037/0033-2909.117.3.497 bornstein, g., rapoport, a., kerpel, l., & katz, t. (1989). withinand betweengroup communication intergroup competition for public goods. journal of experimental social psychology, 25, 422–436. https://doi.org/10.1016/00221031(89)90031-0 bouas, k. s., & komorita, s. s. (1996). group discussion and cooperation in social dilemmas. personality and social psychology bulletin, 22, 1144–1150. https://doi.org/10.1177/01461672962211005 carnevale, a. p., & strohl, j. (2013). separate and unequal: how higher education reinforces the intergenerational reproduction of white racial privilege. center on education and the workforce. creating an inclusive learning community to better serve minority students 14 journal of effective teaching in higher education, vol. 4, no. 3 https://vtechworks.lib.vt.edu/bitstream/handle/10919/83041/separateandun equal.pdf?sequence=1&isallowed=y carter, d. j. (2008). on spotlighting and ignoring racial group members in the classroom. in m. pollock (ed.), everyday antiracism: getting real about race in school (pp. 230–234). the new press. cast (2018). universal design for learning guidelines version 2.2. http://udlguidelines.cast.org center for research on teaching and learning (2021). inclusive teaching resources and strategies. center for research on teaching and learning. https://crlt.umich.edu/multicultural-teaching/inclusive-teaching-strategies center for teaching and assessment for learning (2021). diversity and inclusive teaching. center for teaching and assessment for learning. https://ctal.udel.edu/resources-2/inclusive-teaching/ center for teaching innovation (2021). incorporating diversity. center for teaching innovation. https://teaching.cornell.edu/teaching-resources/designing-yourcourse/incorporating-diversity diangelo, r. (2018). white fragility: why it’s so hard for white people to talk about racism. beacon press. drolet, a. l., & morris, m. w. (2000). rapport in conflict resolution: accounting for how face-to-face contact fosters mutual cooperation in mixed-motive conflicts. journal of experimental social psychology, 36, 26–50. https://doi.org/10.1006/jesp.1999.1395 ellis, s. j., riggs, d. w., & peel, e. (2019). lesbian, gay, bisexual, trans, intersex, and queer psychology. cambridge university press. elsass, p. m., & graves, l. m. (1997). demographic diversity in decision-making groups: the experiences of women and people of color. academy of management review, 22, 946–973. https://doi.org/10.2307/259250 fogliati, v. j., & bussey, k. (2013). stereotype threat reduces motivation to improve: effects of stereotype threat and feedback on women’s intentions to improve mathematical ability. psychology of women quarterly, 37, 310–324. https://doi.org/10.1177/0361684313480045 fries-britt, s., & turner, b. (2002). uneven stories: successful black collegian at a black and white campus. the review of higher education, 25, 315–330. https://doi.org/10.1353/rhe.2002.0012 creating an inclusive learning community to better serve minority students 15 journal of effective teaching in higher education, vol. 4, no. 3 greenwald, a. g., mcghee, d. e., & schwartz, j. l. (1998). measuring individual differences in implicit cognition: the implicit association test. journal of personality and social psychology, 74, 1464–1480. https://doi.org/10.1037/0022-3514.74.6.1464 greenwald, a. g., poehlman, t. a., uhlmann, e. l., & banaji, m. r. (2009). understanding and using the implicit association test: iii. meta-analysis of predictive validity. journal of personality and social psychology, 97, 17–41. https://doi.org/10.1037/a0015575 harper, s. r. (2010). an anti-deficit achievement framework for research on students of color in stem. new directions for institutional research, 2010(148), 63–74. https://doi.org/10.1002/ir.362 harper, s. r., & davis iii, c. h. (2012). they (don't) care about education: a counternarrative on black male students' responses to inequitable schooling. educational foundations, 26, 103–120. https://eric.ed.gov/?id=ej968820 harper, s. r., & hurtado, s. (2007). nine themes in campus racial climates and implications for institutional transformation. new directions for student services, 2007(120), 7–24. https://doi.org/10.1002/ss.254 harper, s. r., patton, l. d., & wooden, o. s. (2009). access and equity for african american students in higher education: a critical race historical analysis of policy efforts. the journal of higher education, 80(4), 389–414. https://doi.org/10.1353/jhe.0.0052 jacoby, r., & glauberman, n. (1995). the bell curve debate. random house. jencks, c., & phillips, m. (1998). the black–white test score gap: why it persists and what can be done. the brookings review, 16(2), 24–27. https://doi.org/10.2307/20080778 liss, m., k. richmond, & m. j. erchull (2019). psychology of women and gender. w. w. norton and company. lockwood, p. (2006). someone like me can be successful: do college students need same-gender role models? psychology of women quarterly, 30, 36–46. https://doi.org/10.1111/j.1471-6402.2006.00260.x major, b., spencer, s., schmader, t., wolfe, c., & crocker, j. (1998). coping with negative stereotypes about intellectual performance: the role of psychological disengagement. personality and social psychology bulletin, 24, 34–50. https://doi.org/10.1177/0146167298241003 marx, d. m., ko, s. j., & friedman, r. a. (2009). the “obama effect”: how a salient role model reduces race-based performance differences. journal of creating an inclusive learning community to better serve minority students 16 journal of effective teaching in higher education, vol. 4, no. 3 experimental social psychology, 45, 953–956. https://doi.org/10.1016/j.jesp.2009.03.012 marx, d. m., & roman, j. s. (2002). female role models: protecting women’s math test performance. personality and social psychology bulletin, 28, 1183–1193. https://doi.org/10.1177/01461672022812004 malouff, j. m., & thorsteinsson, e. b. (2016). bias in grading: a meta-analysis of experimental research findings. australian journal of education, 60, 245– 256. https://doi.org/10.1177/0004944116664618 manion, j. (2018, november 17). the performance of transgender inclusion: the pronoun go-round and the new gender binary. public seminar. https://publicseminar.org/essays/the-performance-of-transgender-inclusion/ mcdonald, t. w., toussaint, l. l., & schweiger, j. a. (2004). the influence of social status on token women leaders' expectations about leading male-dominated groups. sex roles, 50, 401–409. https://doi.org/10.1023/b:sers.0000018894.96308.52 mcintosh, p. (1988). white privilege: unpacking the invisible knapsack. peace and freedom, 49(4), 10–12. mcintyre, r. b., lord, c. g., gresky, d. m., ten eyck, l. l., frye, g. j., & bond c. f. (2005). a social impact trend in the effects of role models on alleviating women’s mathematics stereotype threat. current research in social psychology, 10, 116–36. https://doi.org/10.1016/s0022-1031(02)00513-9 nadal, k. l. (2011). the racial and ethnic microaggressions scale (rems): construction, reliability, and validity. journal of counseling psychology, 58, 470–480. https://doi.org/10.1037/a0025193 nagasawa, r. & wong, p. (1999). a theory of minority students’ survival in college. sociological inquiries, 69, 76–90. patson, n. d. (2021, may 12). collaborative note-taking as an alternative to recording online sessions. faculty focus. https://www.facultyfocus.com/articles/online-education/online-assessmentgrading-and-feedback/collaborative-note-taking-as-an-alternative-torecording-online-sessions/ pierce, c. m. (1995). stress analogs of racism and sexism: terrorism, torture, and disaster. in c. v. willie, p. p. rieker, b. m. kramer, & b. s. brown (eds.), mental health, racism, and sexism (pp. 277–293). university of pittsburgh press. creating an inclusive learning community to better serve minority students 17 journal of effective teaching in higher education, vol. 4, no. 3 pornpitakpan, c. (2004). the persuasiveness of source credibility: a critical review of five decades’ evidence. journal of applied social psychology, 34, 243–281. https://doi.org/10.1111/j.1559-1816.2004.tb02547.x project implicit. (1998). project implicit social attitudes. project implicit. https://implicit.harvard.edu/implicit/index.jsp pruitt, d. g. (1998). social conflict. in d. gilbert, s. t. fiske, & g. lindzey (eds.), handbook of social psychology (4th ed., 470–503). mcgraw-hill. quinn, d. m. (2021). how to reduce racial bias in grading. education next, 21(1). https://www.educationnext.org/how-to-reduce-racial-bias-in-gradingresearch/ saad, l. f. (2020). me and white supremacy: combat racism, change the world, and become a good ancestor. sourcebooks. sherif, m. (1954). experimental study of positive and negative intergroup attitudes between experimentally produced groups: robbers cave study. university of oklahoma. sherif, m., harvey, o.j., white, b. j., hood, w., & sherif, c. w. (1961). intergroup conflict and cooperation: the robbers cave experiment. the university book exchange. smith-lovin, l., & brody, c. (1989). interruptions in group discussions: the effects of gender and group composition. american sociological review, 54(3), 424– 435. https://doi.org/10.2307/2095614 steele, c. m. (1997). a threat in the air: how stereotypes shape intellectual identity and performance. american psychologist, 52, 613–629. https://doi.org/10.1037/0003-066x.52.6.613 steele, c. m., & aronson, j. (1995). stereotype threat and the intellectual test performance of african americans. journal of personality and social psychology, 69, 797–811. https://doi.org/10.1037/0022-3514.69.5.797 sue, d. w. (2010). microaggressions in everyday life: race, gender, and sexual orientation. wiley & sons. usc rossier school of education. (2021, january 15). tools for inclusive teaching. usc rossier school of education. https://rossier.usc.edu/tools-for-inclusiveteaching/ walker, i. & crogan, m. (1998). academic performance, prejudice, and the jigsaw classroom: new pieces to the puzzle. journal of community and applied creating an inclusive learning community to better serve minority students 18 journal of effective teaching in higher education, vol. 4, no. 3 social psychology, 8(6), 381–393. https://doi.org/10.1002/(sici)10991298(199811/12)8:6%3c381::aid-casp457%3e3.0.co;2-6 woodcock, a., hernandez, p. r., estrada, m., & schultz, p. w. (2012). the consequences of chronic stereotype threat: domain disidentification and abandonment. journal of personality and social psychology, 103, 635–646. https://doi.org/10.1037/a0029120 yosso, t. j., smith, w. a., ceja, m., & solórzano, d. g. (2009). critical race theory, racial microaggressions, and campus racial climate for latina/o undergraduates. harvard educational review, 79(4), 659–691. https://doi.org/10.17763/haer.79.4.m6867014157m707l microsoft word babcockgeorgiou_final version.docx journal of effective teaching in higher education, vol. 2, no. 2 comparison of two online courseware instructional methods using propensity score matching renée l. babcock, central michigan university, babco1rl@cmich.edu jean georgiou, new jersey city university abstract. this study used propensity score matching to compare two online courseware systems intended for use as supplements to either face-to-face or online courses. one courseware system focused on adaptive features available to students while completing assignments; whereas the other system was an interactive courseware that provided embedded media supplements and assessments. participants were students enrolled in a psychology course taught by an instructor who used the adaptive courseware in fall 2013, switching to the interactive courseware in fall 2015 and 2016. propensity score matching was used to match students in 2015 with students in 2013. a separate matching process was done for the 2016 versus 2013 students. the use of propensity score matching was successful in allowing comparison across groups. however, the only consistent finding across the two matched samples was that students appeared to spend less time in the interactive courseware than the adaptive courseware. the most likely reason for the difference is a function of the requirements of the two courseware systems. keywords: propensity score matching, digital learning systems, adaptive courseware, interactive courseware a common assumption about millennials is that they thrive in a digital world that allows them to instantly interact with endless amounts of material at their fingertips. this assumption is not unfounded: pew research center found that 97% of millennials (ages 22-37) use the internet regularly (jiang, 2018). capitalizing on that information, the use of online learning materials to improve student learning has been increasingly adopted by college instructors, as publishers compete to offer digital supplements with their textbooks. pearson publishers, mcgraw-hill education, wiley, cengage, and others offer access to digital products intended to draw students more deeply into the material and offer various strategies to do so. such web-based systems can enhance instructor-student interactions by providing additional resources for the student to explore in-depth those topics that can typically be explored only at a broader level in classroom settings. understanding whether these courseware systems are effective in learning is important for instructors in decisions on the amount and type of material to assign. that is, knowing whether the digital learning systems are efficacious in knowledge acquisition is important for instructors as they choose whether or not to adopt them for use in the classroom. one method of determining efficacy is to use propensity score matching when comparing systems. as noted, there are a range of types of programs available, with two common strategies being adaptive learning and interactive programs that provide multiple types of material all within one source. the current study used propensity score matching to compare two such programs offered by pearson publishers. comparison of two online courseware instructional methods 103 journal of effective teaching in higher education, vol. 2, no. 2 online adaptive courseware the core learning design element of this courseware is its adaptive function that includes learning activities and features designed to measure learners’ strengths and weaknesses on topics and guide them to study the topics that require practice. that is, this type of courseware provides additional material and quizzing on topics that the student has not yet mastered and limits the presentation of material that the student has already mastered. the goal of such systems is to optimize a student’s study time and effort. the online adaptive courseware was designed based on research in learning principles. the key adaptive features include scaffolding, feedback, and a reduction in cognitive load. one of the components of an online adaptive courseware is an instructional approach known as scaffolding. often useful in assisting in the construct of appropriate levels of cognitive load, scaffolding offers assistance or support to the learner as needed (bunch, 2009; sharma & hannafin; 2007; van merrienboer, kirschner, & kester; 2003). however, once the goal is achieved, or the learner no longer needs the assistance, the support disappears (bunch, 2009). while utilizing different scaffolding approaches, several researchers noted positive findings based on their achievement of complex cognitive skills across a variety of domains reducing the overall cognitive load (bunch, 2009; gerjets, scheiter, & catrambone; 2004; ong & tasir, 2015; xie et al., 2018; yaman, nerdel, & bayrhuber; 2008). the adaptive courseware here provides students with scaffolding or technologybased support on specific learning tasks, such as concept mastery through practice quizzing. the adaptive features embedded in the courseware were based on recommendations from research on scaffolding in educational technology (bunch, 2009; sharma & hannafin, 2007). in this online adaptive courseware, students complete a pretest at the beginning of each unit to determine what they have already mastered and what they have not yet mastered. students are then provided with feedback that suggests where they most need to focus their attention when reviewing the course material. suggestions might include reviewing sections of the text, viewing a video, completing a simulation, or testing on concepts. after students have completed the suggestions, they complete a post-test to determine if they have mastered the material. after the post-test, students again receive feedback on what they may want to continue studying. research suggests that feedback from the preand post-tests may boost students’ long-term retention and confidence (hattie, 2009, 2012). one of the assessments that reddy, labutov, and joachims (2016) utilized in their study provided learners with an opportunity to improve their skills over time by requiring a prerequisite. when a learner successfully completed a module and was assessed with a pass, he/she was then able to move forward, hence receiving positive feedback. in addition to using pre and post-tests, ong and tasir (2015) provided those in their study with a cognitive load measurement for self-reporting. this selfreporting scale measured the users invested mental effort. the scale ranged from “very, very low mental effort” to “very, very high mental effort” (p. 508). while this instrument may have been chosen for its reliability, simplicity, and practicality, it comparison of two online courseware instructional methods 104 journal of effective teaching in higher education, vol. 2, no. 2 helped the learner as well as the researchers receive valuable feedback regarding the learner’s overall invested mental effort while their cognitive load was being measured. in cognitive psychology, cognitive load refers to the total amount of mental effort being used in working memory. this includes extraneous cognitive load (i.e., mental effort spent on distracting elements that are not relevant to the learning). research shows that if one can reduce extraneous cognitive load for students when they are reading or studying, one can improve the effectiveness of the students’ ability to process the important information and move it from working memory to long term memory (pociask, dizazzo-miller, & samuel, 2013; miller, 1956; sweller, 1988). ong and tasir (2015) discuss the importance of incorporating certain aspects of cognitive psychology and cognitive load theory to reduce the overall cognitive load and the beneficial effects it has on “training time, improved performance on tasks using learned knowledge/skills, and improved performance on other tasks where the learned information can be applied” (p. 503). when properly accomplished the learner’s cognitive resources are released perhaps freeing them for another learning experience. online interactive courseware the online interactive courseware was created to provide a more engaging experience for the students. as students read through the text, they encounter several opportunities to deepen their learning by watching videos, manipulating points on graphs, and taking brief quizzes. a benefit of this digital product was that everything was contained within a single source: the e-text, interactive charts/graphs, videos, and assessments. unlike the adaptive learning system described above, students do not have to leave the e-text to study as everything is contained with the e-text. additionally, this courseware does not change or adapt based on a student’s performance. the interactive courseware was designed utilizing research-based design principles intended to help students learn. the key learning design principles included reduced cognitive load, multi-media presentation of content, and embedded assessments. similar to the adaptive courseware, the interactive courseware attempts to lessen extraneous cognitive load by clearly organizing materials into topics and subtopics. each chapter is divided into modules that cover a small amount of material. at the close of each subtopic, students complete a brief (3-5 question) quiz, thus minimizing the amount of material that needs to be mastered at any given point. overall, this interactive courseware provides students with a centralized learning environment with easy access to assignment instructions and other resources needed to complete any assignment. mayer and moreno (2003) define meaningful learning as “deep understanding of the material, which includes attending to important aspects of the presented material, mentally organizing it into a coherent cognitive structure, and integrating it with relevant existing knowledge” (p. 43). when material is delivered through multimedia technology the benefits may be as limitless as the technology itself. comparison of two online courseware instructional methods 105 journal of effective teaching in higher education, vol. 2, no. 2 this may be due in part to the similarities of multimedia and a person’s innate way of learning or their cognitive processing as it is often referred to (jereb & smitek, 2006; mayer, 2002; ong & tasir, 2015; xie et al., 2018). xie et al. (2018) were able to show positive results using the coordination of visual and auditory cueing in multimedia learning. their results indicated that by incorporating dual modality cues (visual and auditory) to the online lessons, the learners were able to achieve an increase in overall retention span with online learning in addition to a more time efficient learning process (xie et al., 2018). the online interactive courseware utilized in this study attempts to follow research-based multimedia learning principles so that students can apply new learning and assessment strategies in ways that would not be possible with a printed textbook (mayer & moreno, 2003). specifically, the interactive courseware contains narrative text that is combined with interactive elements. interactive media provides many more options for how to present information and ideas than static text, which in turn leads to opportunities to more clearly present those ideas and information. often when working with interactive media, students can choose to pause what they are reading so that they have the opportunity to stop and process information, which is intended to help them connect the media to the text and build a richer knowledge base (craik & tulving, 1975; mayer, 2002; virk, clark, & sengupta, 2015). the interactive courseware contains embedded assessments, the purpose of which is to allow students to check their understanding and receive immediate feedback. jereb and smitek (2006) stress the importance of using self-assessment instruments and recommend creating three assessments: one to be used in the beginning, one for the middle, and one for the end of the learning material so that the learner can gain the most from their multimedia learning experience (jereb & smitek). reddy et al. (2016) utilized several embedding techniques in their study, one of which is pass/fail assessments that allow the learner to improve by working on modules over a period. additional research suggests that such embedded assessments and the feedback they provide can help build long-term retention and increase students’ confidence and motivation (hattie, 2009, 2012). the spacing of assessment opportunities is aligned with research findings on how to optimize memory (e.g., clark & bjork, 2014). in general, both online courseware systems are similar in their learning design principles. the main difference between them is that one utilizes an adaptive learning approach whereas the other utilizes an interactive learning approach. the objective of this study is to use propensity score matching to examine whether there are differences in students’ learning outcomes and behaviors in an introductory psychology course when utilizing the two types of online courseware. thus, this study investigates whether students do better if they prepare using interactive tools or if they have adaptive features to help them with their assignments. comparison of two online courseware instructional methods 106 journal of effective teaching in higher education, vol. 2, no. 2 methods participants this study took place at a public university in the midwest. the instructor in the study taught an introductory psychology course during fall semesters. in fall 2013, the instructor used the online adaptive courseware with 200 students. then in fall 2015, the instructor switched to the interactive courseware with 174 students and continued to use the interactive courseware in fall 2016, with 191 students. measures the two online supplemental courseware systems described above were utilized in the study. for each, there are data that come from the online courseware system as well as data from performance in the course. system log data for the interactive courseware and the adaptive courseware were extracted. these data came from students’ use of the courseware and included variables on the average percent correct that the students achieved in the assignments within the courseware, the total time spent in the courseware, and the total items attempted in the courseware. these system log data variables were the outcomes that were examined after matching. the system log data were merged with the exam grades the students obtained in the course. these data were then merged with data on students’ baseline characteristics that are described below and that were used in the matching. in addition to the system log data variables that were examined as outcomes, the average grade across all exams was also examined as an outcome. to account for differences between students who were enrolled in the different semesters, this study used propensity score matching. propensity score methodology is a recommended method when the goal is to estimate causal effects using observational (nonrandomized) data because it mimics some of the particular characteristics of a randomized study (rosenbaum & rubin, 1984), which is aligned to the research design of this study. the matching algorithm used was one-to-one matching with replacement, that is, each student in the interactive courseware class was matched to one adaptive courseware student, where the adaptive courseware students could be matched more than once, if they were the closest match to the interactive courseware student. unlike normal matching, in which participants are typically matched on a single characteristic, propensity score matching utilizes a predicted probability of group membership based on observed predictors. thus, propensity scores are used to reduce selection bias by equating groups when randomization isn’t possible. the baseline characteristics that were being matched on include act composite score, gender, minority, freshman standing, and cumulative gpa at the beginning of the semester. with act composite and cumulative gpa, differences in prior achievement (an important covariate to consider) were addressed in the study. comparison of two online courseware instructional methods 107 journal of effective teaching in higher education, vol. 2, no. 2 procedure students in the semester utilizing the adaptive courseware were required to complete pre-tests for each chapter of the book. the pre-tests were used to provide an individualized study plan for each student. students were encouraged, but not required to complete all of the assignments suggested within their unique plan. at the end of each chapter, students were then required to complete a post-test for the chapter. the post-tests were presented with a mastery approach such that the highest score achieved on the post-test was the score recorded for their grade. students in the semesters utilizing the interactive courseware were required to complete brief quizzes at the end of each module of a chapter and at the end of each chapter. scoring of the quizzes encouraged students to interact with the material prior to responding to each question. that is, if the student chose the correct response to the question on the first attempt, three points were earned. if the student needed two attempts, then two points were earned; and, if the student needed three attempts, only one point was earned. thus, students were encouraged to look back through the material prior to responding to each quiz question. in all three semesters, students were required to complete either five or six high-stakes exams in the classroom. results this study compared students using the interactive courseware to the students using the adaptive courseware using propensity score matching. two comparisons were made. first the fall 2015 interactive courseware class was compared to the fall 2013 adaptive courseware class. the second comparison was between the fall 2016 interactive class and the fall 2013 adaptive class. figures 1a to 2c show the descriptive statistics of the samples before and after matching. overall, for the fall 2013 (adaptive courseware) and fall 2015 (interactive courseware) samples, the descriptive statistics for act composite, cumulative gpa, female, non-white, and freshman showed that the one-to-one matching did a good job and the differences between the two groups were smaller after matching, except for only one demographic characteristic—non-white. comparison of two online courseware instructional methods 108 journal of effective teaching in higher education, vol. 2, no. 2 note: after matching, the outliers are reduced in the adaptive courseware group (fall 2013) and the mean is closer to the interactive courseware (fall 2015) mean. figure 1a: act composite descriptive statistics for the adaptive courseware group (fall 2013) and the interactive courseware group (fall 2015) note: after matching, the interquartile range and mean in the adaptive courseware group (fall 2013) are closer to the interactive courseware group (fall 2015). figure 1b. cumulative gpa descriptive statistics for the adaptive courseware group (fall 2013) and the interactive courseware group (fall 2015) 0 4 8 12 16 20 24 28 32 36 fall 2015 befor matching [mean=22.3, n=174] fall 2013 before matching [mean=21.7, n=200] fall 2015 after matching [mean=22.3, n=174] fall 2013 after matching [mean=22.7, n=174] act composite 0 0.5 1 1.5 2 2.5 3 3.5 4 fall 2015 before matching [mean=3.00, n=174] fall 2013 before matching [mean=2.82, n=200] fall 2015 after matching [mean=3.00, n=174] fall 2013 after matching [mean=2.96, n=174] cumulative gpa comparison of two online courseware instructional methods 109 journal of effective teaching in higher education, vol. 2, no. 2 note: the percentages between the two groups are closer after matching except for non-white. figure 1c: demographic descriptive statistics for the adaptive courseware group (fall 2013) and the interactive courseware group (fall 2015) 0% 20% 40% 60% 80% 100% female (before matching) non-white (before matching) freshman (before matching) female (after matching) non-white (after matching) freshman (after matching) demographics fall 2015 fall 2013 0 4 8 12 16 20 24 28 32 36 fall 2016 before matching [mean=22.6, n=191] fall 2013 before matching [mean=21.7, n=200] fall 2016 after matching [mean=22.6, n=191] fall 2013 after matching [mean=22.4, n=191] act composite comparison of two online courseware instructional methods 110 journal of effective teaching in higher education, vol. 2, no. 2 note: after matching, the adaptive courseware group (fall 2013) mean is closer to the interactive courseware group (fall 2016) mean. figure 2a: act composite descriptive statistics for the adaptive courseware group (fall 2013) and the interactive courseware group (fall 2016) note: the distributions of the cumulative gpa between the two groups are almost identical before matching. hence, it is not surprising that, after matching and removing some of the unmatched students in the adaptive courseware group (fall 2013), the distribution is a little more disperse, though there is still a high degree of overlap after matching. thus, balance is not an issue here. figure 2b: cumulative gpa descriptive statistics for the adaptive courseware group (fall 2013) and the interactive courseware group (fall 2016) 0 0.5 1 1.5 2 2.5 3 3.5 4 fall 2016 before matching [mean=2.81, n=191] fall 2013 before matching [mean=2.82, n=200] fall 2016 after matching [mean=2.81, n=191] fall 2013 after matching [mean=2.67, n=191] cumulative gpa comparison of two online courseware instructional methods 111 journal of effective teaching in higher education, vol. 2, no. 2 note: the percentages between the two groups are closer after matching except for female. figure 2c: demographic descriptive statistics for the adaptive courseware group (fall 2013) and the interactive courseware group (fall 2016) similarly, for the fall 2013 (adaptive courseware) and fall 2016 (interactive courseware) samples, the descriptive statistics across act composite, cumulative gpa, female, non-white, and freshman showed that the one-to-one matching did a decent job and there is balance (i.e. good amount of overlap) between the two groups after matching, except for one demographic characteristic—female. the descriptive statistics presented before and after matching only provide a visual examination of the differences between the two groups. such visual examination involves multiple comparisons and considers each characteristic equally important. for a more rigorous analysis of balance between the two groups, rubin’s (2001) criteria were also used: 1. the difference in the mean propensity score in the two groups should be near zero. that is the difference in the mean of the propensity score should be less than half a standard deviation. 2. the ratio of the variance of the propensity score in the two groups should be near one, where less than 0.5 or larger than 2 are considered extremes. 3. the ratio of the variances of the (continuous) covariates after adjusting for the propensity scores should be close to 1. ratios between 0.80 and 1.25 are acceptable. ratios less than 0.50 or greater than 2.0 are considered extremes. 0% 20% 40% 60% 80% 100% female (before matching) non-white (before matching) freshman (before matching) female (after matching) non-white (after matching) freshman (after matching) demographics fall 2016 fall 2013 comparison of two online courseware instructional methods 112 journal of effective teaching in higher education, vol. 2, no. 2 when rubin’s (2001) balance diagnostics was performed across both the fall 2013 versus fall 2015 and fall 2013 versus fall 2016 samples, there were eight diagnostics comparisons performed in total, with seven of the eight comparisons meeting the acceptable range, and only one comparison being out of the acceptable range (that is, out of the acceptable range of 0.8 to 1.25 for the ratio of variances). however, it is still not in the extremes (which is less than 0.5 or greater than 2). therefore, we can conclude that the matched samples were relatively balanced after propensity score matching and can proceed with the analysis. after matching, paired t-tests were conducted for the matched pairs separately for the fall 2015 (versus fall 2013) sample and the fall 2016 (versus fall 2013) sample. table 1 shows the results, which were mixed. in most cases significant findings for fall 2015 were not replicated in fall 2016. for example, students received a significantly lower exam average score in fall 2015 compared to fall 2013 whereas the difference in exam average score between the fall 2016 and the fall 2013 samples was not significant. it could be that the instructor was learning and adjusting to a new teaching format, which could account for the significantly lower exam average in fall 2015. table 1a paired t-test results after one-to-one propensity score matching: fall 2015 and fall 2013 matched sample outcome paired mean difference sample size p-value exam average -10.53 174 <.0001 system log data variables average percent correct 7.97 171 <.0001 total time (hours) -6.55 174 <.0001 total items attempted -114.3 174 .0415 table 1b paired t-test results after one-to-one propensity score matching: fall 2016 and fall 2013 matched sample outcome paired mean difference sample size p-value exam average -3.65 191 .0772 system log data variables average percent correct -0.11 186 .9163 total time (hours) -4.73 191 <.0001 total items attempted 119.0 191 .0748 notes for tables 1a and 1b: 1. the system log data variable, average percent correct, for fall 2013 (adaptive courseware) was calculated based on the scores that each student comparison of two online courseware instructional methods 113 journal of effective teaching in higher education, vol. 2, no. 2 received for post-tests averaged across all the post-test assessments that the student took. 2. similarly, the average percent correct for fall 2015 and fall 2016 (interactive courseware) was calculated based on the scores that each student received for assessments average across all the assessments that the student took. 3. a negative difference indicates that overall, the fall 2013 (adaptive courseware) class had higher outcome values, while a positive difference indicates that overall, the fall 2015 or fall 2016 (interactive courseware) class is had higher outcome values. the only consistent finding across the two matched samples is that students appeared to spend less time in the interactive courseware than the adaptive courseware. there are at least two possible reasons for this result. the first is that in the adaptive program, once students complete the pre-test, they use links to other material that could assist in their learning. doing so would ultimately result in longer amounts of time spent within the courseware. another possible reason is that the learning design principle of the interactive courseware emphasized “read a little, do a little,” thus resulting in shortened times within courseware. a limitation of this analysis is that the data we have on time spent did not differentiate between the time when students were actively logged into the courseware versus the time when they were idle while logged in. hence, having a significant finding on time spent might not be indicative of time actively engaged with the courseware. discussion educational technology has opened up new ways for students to learn in class that are not possible with a print text. assessing whether supplemental courseware systems are effective in improving learning is challenging. one of the goals of the current study was to use propensity score matching to compare two online courseware systems: one in which adaptive features were used to optimize studying and a second in which interactive features were embedded within an etext with the intent on making the material more engaging. the use of propensity score matching enables more accurate comparisons to be made. a second goal of this research was to determine if students do better when they prepare using interactive tools or if they do better when using adaptive features to help them with their assignments. the current study did not seem to show clear cut evidence that one approach is better than the other. some of the mixed results could be due to the fact that the instructor was using the interactive courseware for the first time in fall 2015 and so, was still adjusting to it. another possible explanation is that students’ individual learning styles influenced how they interacted with the courseware; and, in turn, affected their learning and achievement. though differences in the two systems were not apparent, it is important to note that either or both of the courseware systems might be effective in improving learning over and above that which normally takes place within traditional course instruction. that is, the current study did not include a control group that did not use any digital learning software. anecdotally, the instructor of the course noted that overall student performance improved with the use of either system compared comparison of two online courseware instructional methods 114 journal of effective teaching in higher education, vol. 2, no. 2 to when no system was used (e.g., overall failure rates were reduced from 10-12% to fewer than 7%). there are several important messages that can be gleaned from this study. first, and perhaps foremost, we are reminded that not all students learn in the same manner. there is no ideal way of teaching whether in the classroom or on a multimedia course with specific courseware. bikowski and casal (2018) suggested that digital materials are most effective when they are customized, interactive, and useable for both faculty and students. though one of the courseware products studied in this project was considered “interactive”, bikowski and casal (2018) were more likely referring to the interactive nature of e-texts in that they can be used to present videos, demonstrations, and simulations. it remains possible that both of the courseware programs improve learning. though neither online courseware system seemed better than the other, the importance of using propensity score matching to examine or compare efficacy of a program should not be overlooked. the educational system lends itself to the study of non-randomized groups and propensity scoring enables comparison of those groups. this is especially true if the two groups do not have much overlap. there are limitations to the current study that could be addressed in future research. one limitation is that there was not a control group that did not utilize courseware. because of this, interpretation of some of the results needed to rely on anecdotal evidence. future research is needed to confirm that any digital learning system is better than none at all. another limitation is that because the two courseware systems were designed by publishers for pedagogical rather than empirical use, it is impossible to isolate and control for variables that differ in the adaptive and interactive courseware systems. this could be improved with further study regarding student engagement with each of the tools. furthermore, the use of qualitative interviews with the students and professor would help answer some of the questions regarding why the students were online more in the adaptive courseware, and how the instructor's approach changed in the fall 2015 class. finally, it would be useful to know which type of system fits best for different learning styles. though it is not practical to think that each student could select her/his own system, different learning institutions may determine that the majority of their students would benefit from a particular type of courseware. such information may also be informative for textbook publishers as they continue to enhance their programs. technology is taking teaching and learning to new heights, once never imagined. the tools for teaching must be enhanced to teach the learners of today. as technology grows so must the teaching forum. technology is not only opening doors, but is also giving educators an opportunity to enhance the learning experience. what is done with technology may shape how the educational arena of tomorrow is viewed. the findings of this study are valuable to the educator who wishes to adapt some type of courseware but has been hesitant to do so or has not known where to start. the authors of the current study have anecdotal evidence that regardless of which system one uses, overall performance in the class will improve over using no courseware system at all. additionally, students independently commented that they found the systems beneficial to their learning. comparison of two online courseware instructional methods 115 journal of effective teaching in higher education, vol. 2, no. 2 perhaps the courseware chosen is less important than simply providing students with an additional study tool. the question may no longer be whether or not to use online courseware, but rather which courseware to use. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references bikowski, d. & casal, e. (2018). interactive digital textbooks and engagement: a learning strategies framework, pennsylvania state university language learning & technology 22 (1), 119–136. bunch, j.m. (2009). an approach to reducing cognitive load in the teaching of introductory database concepts. journal of information systems education, 20(3), 269-275. clark, c. m. & bjork, r. a. (2014). when and why introducing difficulties and errors can enhance instruction. in v. a. benassi, c. e. overson, & c. m. hakala (eds.). applying science of learning in education: infusing psychological science into the curriculum. retrieved from the society for the teaching of psychology website: http://teachpsych.org/ebooks/asle2014/index.php craik, f. i. m., & tulving, e. (1975). depth of processing and the retention of words in episodic memory. journal of experimental psychology: general, 104(3), 268-294. ellman, m.s., schwartz, m.l. (2016). online learning tools as supplements for basic and clinical science education, journal of medical education and curricular development, 3, 109-114. gerjets, p., scheiter, k., & catrambone, r. (2004). designing instructional examples to reduce intrinsic cognitive load: molar versus modular presentation of solution procedures. instructional science, 32(1-2), 33-58. hattie, j. (2009). visible learning: a synthesis of over 800 meta-analyses relating to achievement. new york, ny: routledge. hattie. j. (2012). visible learning for teachers: maximizing impact on learning. new york, ny: routledge. jereb, e., & šmitek, b. (2006). applying multimedia instruction in e learning. innovations in education and teaching international, 43(1), 15–27. https://draweb.njcu.edu:2103/10.1080/14703290500467335 jiang, j. (2018). millennials stand out for their technology use. pew research center, h t t p : / / w w w . p e w r e s e a r c h . o r g / f a c tt a n k / 2 0 1 8 / 0 5 / 0 2 / mi l l e n n i a l ss t a n d o u t f o r t h e i r t e c h n o l o g y u s e b u t o l d e r g e n e r a t i o n sa l s o e mb r a c ed i g i t a l l i f e / comparison of two online courseware instructional methods 116 journal of effective teaching in higher education, vol. 2, no. 2 mayer, r. e. (2002). rote versus meaningful learning. theory into practice, 41(4), 226-232. mayer, r. e., & moreno, r. (2003). nine ways to reduce cognitive load in multimedia learning. educational psychologist, 38(1), 43-52. miller, g. a. (1956). the magical number seven, plus or minus two: some limits on our capacity for processing information. psychological review, 63(2), 81-97. ong, c. p. & tasir, z. (2015). self-instructional module based on cognitive load theory: a study on information retention among trainee teachers. educational technology research and development, 63(4), 499-515. pociask, f.d., dizazzo-miller, r., & samuel, p.s. (2013). reducing cognitive load while teaching complex instruction to occupational therapy students. the american journal of occupational therapy, 67(5), 92-98. reddy, s., labutov, i. & joachims, t. (2016). latent skill embedding for personalized lesson sequence recommendation. under review by the acm sigkdd conference on knowledge discovery and data mining rosenbaum, p. r., & rubin, d. b. (1984). reducing bias in observational studies using sub-classification on the propensity score. journal of the american statistical association, 79(387), 516-524. rubin, d. b. (2001). using propensity scores to help design observational studies: application to the tobacco litigation. health services and outcomes research methodology, 2(3-4), 169-188. sharma, p., & hannafin, m. j. (2007). scaffolding in technology--enhanced learning environments. interactive learning environments, 15(1), 27-46. sweller, j. (1988). cognitive load during problem solving: effects on learning. cognitive science, 12(2), 257-285. van merrienboer, j. j. g., kirschner, p.a., & kester, l. (2003). taking the load off a learner’s mind: instructional design for complex learning. educational psychologist, 38(1), 5-13. virk, s., clark, d. & sengupta, p. (2015). digital games as multirepresentational environments for science learning: implications for theory, research, and design. educational psychology, 50(4), 284-312. xie, h., mayer, r. e., wang, f., & zhou, z. (2018, july 2). coordinating visual and auditory cueing in multimedia learning. journal of educational psychology. advance online publication. http://dx.doi.org/10.1037/edu0000285 yaman, m., nerdel, c., & bayrhuber, h. (2008). the effects of instructional support and learner interests when learning using computer simulations. computers & education, 51(4), 1784-1794. microsoft word anderson et al_final.docx journal of effective teaching in higher education, vol. 5, no. 1 https://doi.org/10.36021/jethe.v5i1.252 assessing a modified jigsaw technique with theoretical triangulation bill anderson, illinois state university, jander2@ilstu.edu rachel rymer, metrohealth jennifer versaskas, aurora university abigail bueter, adler university mahalia masood, illinois state university abstract. the jigsaw technique has been successfully used in classrooms for decades, though less in higher education. groups are formed with each student having a precise piece of information necessary to complete the assignment. next, expert groups of students with the same material meet to sharpen their understanding before reforming their initial group to share their new insights. this study sought to assess a modified jigsaw format where students received their material two days before the class, began with the expert groups, and closed with mixed groups where students taught their material and completed the assignment. a random 30% sample of assignments (n = 110) from three classes was analyzed using a pattern-matching technique (yin, 2013) and coded utilizing the six levels of bloom’s revised taxonomy (anderson & krathwohl, 2001) and six significant learning types (fink, 2013) as predetermined codes. one-way anova found no significant difference between classes, f (2, 540) = 1.244, p. = 0.289, when coded with the taxonomy. results were positive, indicating thinking well above rote learning (m = 2.9, sd = 1.233) with most (39.9%) students effectively comparing material. coding was similarly positive for learning types (fink, 2013), again, with no significant difference between classes, f (2, 501) = 3.036, p. = 0.084. integration, making connections between varied information, was the primary type of learning (31.9%) used. evidence of students learning about themselves was also noted (31.3%). the modified jigsaw was a well-received addition to the class and effective in teaching this material. keywords: jigsaw technique; collaborative learning; cooperative learning; critical thinking collaborative learning is an overarching term for a many educational approaches that require group efforts by students, promote critical thinking, and emphasize learning and personal responsibility (laal & ghodsi, 2012). cooperative learning is a specific kind of collaborative learning that involves purposeful interaction to enable students to learn together by teaching each other (nusrath et al., 2019; yoshida, 2018) and to produce a shared result (turkman & buyukaltay, 2015). although all learners are responsible for the success of the group (maden, 2011; timayi et al., 2015), turkman and buyukaltay (2015) assert that group work only becomes actual cooperative learning when students make intentional effort to bring the learning of all group members to the topmost level. assessing a modified jigsaw technique 54 journal of effective teaching in higher education, vol. 5, no. 1 one of the most effective and commonly used forms of cooperative learning is the jigsaw technique (costouros, 2020; lalit & piplani, 2020; yoshida, 2018) that purposefully breaks down a larger concept into smaller, but critical, pieces of information. studies have determined that jigsaw is highly effective because it engages students twice (in their original groups and in their expert groups), encourages learning from one another, increases self-esteem, makes students responsible for teaching in their groups, improves social/relationship skills, increases student engagement, short-term and long-term retention of the subject matter. (yoshida, 2018, p. 209) this study involved modifying the traditional jigsaw and evaluating success based on the level of student thinking, or processing, and the type(s) of learning necessary to complete the assignment. literature review the jigsaw classroom was initially implemented in 1971 to specifically address the tension of the recent desegregation of american schools by lessening competition between students and motivate learners to see each other as resources (aronson et al., 1978). aronson et al. (1978) explained, “it would be valuable if the basic [classroom] process could be changed so that children could learn to like and trust each other not as an extracurricular activity but in the course of learning…” (p. 23). initially, the instructor introduces the assignment and the overall topic. next, home groups are formed, and each member is assigned an essential part of the information to further develop. students then briefly discuss the assignment and familiarize themselves with their material. “the home groups then break apart, like pieces of a jigsaw puzzle, and the students move into jigsaw groups consisting of members from the other home groups who have been assigned the same portion of the material” (maden, 2011, p. 913). these expert groups primarily involve sharing insights about the specific subtopic and addressing related questions (aronson, 2021; aronson & patnoe, 2011). together they seek to develop mastery of their assigned material (barkley, 2010; costouros, 2020), with each student contributing from their perspective and skills. next, the original home groups are reassembled with students taking responsibility for selecting what needs to be taught and how to teach it (costouros, 2020), then sharing their new knowledge with their team members (aronson, 1971, 2021). by making each student part of the solution, the jigsaw technique blurs the distinction between students who know and students who do not yet know, requires all students to make active responses, and moves away from the experience of learning as a solitary activity that is detached from the social context. (maden, 2011, p. 915) assessing a modified jigsaw technique 55 journal of effective teaching in higher education, vol. 5, no. 1 “once each member of the home group has taught their piece of the puzzle to the other members of the group, the whole puzzle is formed” (costouros, 2020, p. 156). afterwards, students will then take the quiz or complete the assignment. although, until recently, the jigsaw technique had been minimally used in higher education (crone & portillo, 2013; weidman & bishop, 2009), studies have indicated that the technique has proven to be quite adaptable to college classrooms (costouros, 2020; dhull & verma, 2019; turkmen & buyukaltay, 2015). as a result, current applications in the college classroom are not only increasing but continuing to evolve (e.g., lalit & piplani, 2019; nusrath et al., 2019; yoshida, 2018). the original design (aronson et al., 1978) previously described remains the most often used, but several variations are in common use as well. described briefly, these variations include the following: • jigsaw ii assigns all material to all students, tests learning after the expert groups, and rewards the team with the highest score (doymus et al., 2010; yoshida, 2018). • jigsaw iii is similar to jigsaw ii but with the competition between groups removed. also, there is intentional consideration of language proficiencies in bilingual classrooms (doymus et al., 2010; hedeen, 2003). • jigsaw iv is like ii and iii but reteaches material as necessary after the quiz on expert group material (timayi et al., 2015; turkmen & buyukaltay, 2015). • a reverse jigsaw focuses on perception and on the understanding of a range of interpretations, with each expert group then teaching the class as a whole (hedeen, 2003; samuel, 2018). • the subject jigsaw involves developing problem solving in the subject matter and utilizes a second home group mixing both material and students (haviz & lufri, 2019). • communication jigsaws opens access of all information to all students and seeks to extend interaction and to expand connections between students (yoshida, 2018). all the variations listed incorporate the structured interaction of the jigsaw and share the basic parts of the strategy (doymus et al., 2010). all variations include students taking the responsibility for their own and other students’ learning and groups cooperating toward a common educational goal. current research with jigsaw learning is largely positive, finding more active participation (lalit & piplani, 2020) and evidence of improved academic performance (dewati et al., 2019; nusrath et al., 2019; turkmen, & buyukaltay, 2015). also, and in agreement with aronson’s (1978) original work, studies have reported the jigsaw method can increase college students’ oral communication skills and their confidence in themselves as students and scholars (crone & portillo, 2013; lalit & piplani, 2019; nusrath et al., 2019). but perhaps of greatest relevance here are reports of higher order thinking and analysis (dewati et al., assessing a modified jigsaw technique 56 journal of effective teaching in higher education, vol. 5, no. 1 2019; nusrath et al., 2019) and findings of more frequent usage of critical thinking (yatimah et al., 2019). there are, of course, recognized difficulties associated with cooperative learning, primarily as some students simply do not enjoy group work. many perceive it as more difficult than working alone, and group work typically does require more concentrated effort (weimer, 2017). common negative observations also include ineffective communication (weimer, 2020) and lack of preparation (weimer, 2017). the latter has also been identified as problematic using the jigsaw technique (benton, 2016; costouros, 2020; nusrath et al., 2019). although most studies specific to jigsaw work have reported positive student responses (calkins & rivnay, 2021; lalit & piplani, 2020; maden, 2011), others found no significant difference in student satisfaction between lecture based and jigsaw learning (costouros, 2020; nusrath et al., 2019). indeed, students who do not enjoy the method may not put forth their best effort (costouros, 2020; nusrath et al., 2019), and this is likely detrimental to the jigsaw process for all students. still, “higher student satisfaction is not necessarily indicative of learning” (costouros, 2020, p. 159). modified jigsaw the jigsaw assignment considered here was to compare common parenting practices of americans of african, asian, european, indigenous, latinx, and middle eastern descent. this topic, and others like it, are generally considered in distinct textbook chapters separated by ethnicity. however, the goal here was not only to learn unique distinctions but to purposefully consider similarities as well. the material divided easily and intuitively with each part critical to the final jigsaw assignment. also, with classroom discussions acutely addressing race or racism being difficult for some (kaplowitz et al., 2019), the jigsaw technique’s history of specific and intentional awareness of racial tension in education (aronson et al., 1978) was considered relevant and valued. interestingly, this element was still noted 25 years later (e.g., crone & portillo, 2013) and contained the added possibility that students could learn to be more tolerant, understanding, and accepting of others and their differences (costouros, 2020). therefore, the jigsaw was an immediate and logical choice when incorporating multiple subtopics with a strong, common theme and attempting to positively effect racial understanding (e.g., aronson & patnoe, 2011). the actual jigsaw assignment initially followed the progression of the subject jigsaw technique (doymus et al., 2010; haviz & lufri, 2019). the general topic and assignment were introduced in class, and students received a packet of information on one of the six groups. because, as previously noted, preparation is critical, students were given two days to prepare for the actual in-class jigsaw. following recommendations to assess discussion readiness prior to the jigsaw (e.g., barkley, 2010; benton, 2016), students were required to submit, prior to class, a brief review of their material, 3-4 bulleted statements of information they found unfamiliar or significant and why, and 2 pertinent, upper-level questions prepared for their first group. assessing a modified jigsaw technique 57 journal of effective teaching in higher education, vol. 5, no. 1 in class, the expert groups (students with the same topic) met first. this change was based on a general characteristic of the subject-jigsaw model (haviz & lufri, 2019) and the successful use of this variation by crone and portillo (2013). students initially discussed their questions, specifically noted similarities and differences in parenting styles and priorities, and determined information to teach in the groups to follow. thirty minutes was allowed for this. next, they moved to jigsaw groups consisting of four to six experts, depending on class size and attendance; taught their material to the group; and were questioned by others. following this, the conversation turned to determining similarities between groups and points that were unique to one, or some, of the groups. thirty-minutes was also allowed here. following this home group, a 15-minute class debriefing was used. after a quick review of the concept of stereotypes, the class was asked if any prevailing stereotypes were challenged by the material or in their conversation. students were then given three days to complete the final assignment that consisted of a brief description of their expert group, a description of the jigsaw group using all materials, a detailed conclusion based on comparing/contrasting, and questions they were left with. this differs from the traditional final group assignment or quiz but still involves accountability and cooperating toward a common educational goal (e.g., yoshida, 2018). purpose of this study this study first sought to evaluate student satisfaction with the activity and to assess the efficacy of this jigsaw variation to establish higher levels of critical thinking (anderson & krathwohl, 2001) and to determine what type of learning (fink, 2013) was most prevalent. because satisfaction with the jigsaw technique may be predictive of the success of the modified jigsaw here (costouros, 2020; nusrath et al., 2019), it will be considered first, using confidential student feedback provided through the individual development and educational assessment (idea, 2021a). were they able to learn from others and did they perceive it as truly cooperative and meaningful to their learning? it is likely that if student satisfaction with the jigsaw, and cooperative learning in general, is low, the student engagement will be low as well, and vice versa. next, did the assignment necessitate the higher levels of critical thinking and reasoning necessary for meaningful learning (e.g., anderson & krathwohl, 2001, p. 65)? it was hoped that this jigsaw experience would require less simple reporting and greater instances of comparing, contrasting, and inferring as described by anderson & krathwohl’s (2001) revision of bloom’s taxonomy of learning. this is a well-established hierarchical continuum with each progressively higher level indicating increasing cognitive complexity and processing. briefly described from least complex to most complex, the levels can be described as follows: 1. remembering: exhibiting memory of previously learned information 2. understanding: demonstrating previous learning by connecting assessing a modified jigsaw technique 58 journal of effective teaching in higher education, vol. 5, no. 1 3. applying: using existing knowledge, or creating new skills, to solve problems 4. analyzing: breaking down existing knowledge and examining specific parts 5. evaluating: presenting informed judgments and justifying a position 6. creating: elaboration used to put information together in new ways it was expected that the coding would reveal an average level of reasoning used by students to complete the assignment significantly beyond anderson and krathwohl’s (2001) remembering level. though remembering is indeed crucial for meaningful and complex learning to occur, it is viewed as insufficient for completing the assignment here. accurately noting both similarities and differences would certainly require comparing from the understanding level. however, it was hoped that the median response will be at the analyzing level identified by discrimination and insight (anderson & krathwohl, 2001). lastly, fink’s (2013) model of significant learning was selected to determine the types of learning that were most prevalent. although fink (2013) acknowledged bloom’s taxonomy as having great value, adding “any model that commands this kind of respect half a century later is extraordinary” (p. 34), he was more interested in important types of learning in higher education that are not readily described by the taxonomy. his model goes, “beyond the cognitive domain of bloom’s taxonomy and beyond cognitive learning itself to determine what kinds of learning are resulting from the assignment” (fink, 2013, p. 34). a critical distinction here is that fink’s (2013) types of learning are interdependent as opposed to linear and hierarchical like the taxonomy. therefore, “achieving any one kind of learning simultaneously enhances the possibility of achieving the other kinds of learning as well” (fink, 2013, p. 37). types include the following: 1. foundational knowledge concerns valid remembering and understanding of information and ideas. 2. application involves using and developing skills and when to appropriately apply them. 3. integration is noticing, identifying, and understanding the connections between things 4. human dimension describes learning about self and others to increase effective interaction. 5. caring focuses on developing new interests and feelings. 6. learning how to learn results in a deeper understating of learning itself and becoming a more effective learner. to best accomplish the level of comparing, contrasting, and thinking mentioned earlier, this assignment was designed as integration learning, requiring the perceiving of similarities and differences, and connecting, “one body of knowledge assessing a modified jigsaw technique 59 journal of effective teaching in higher education, vol. 5, no. 1 with other ideas and bodies of knowledge” (fink, 2013, p. 86). certainly, integration bears an intuitive and clear connection to jigsaw learning. not only to the structure but also to the goal of fostering critical thinking that can help bring about a deeper comprehension of material (ghaith & el-malak, 2004). methods participants were female students majoring in human development and family science. most were juniors (54.6%), followed by 40% seniors and 5.4% sophomores. most were european american (78.2%), 9.1% were latinx, 6.4% were african american, 3.7% were of middle eastern descent, and 2.6% were asian american. the jigsaw assignment was applied to three consecutive semesters in an undergraduate multicultural studies class, n= 37, n = 38, and n = 43, and was completed by 110 of 128 students. a random sample of 30% (n = 34) of the assignments was selected for analysis. case study and theoretical triangulation a case study approach was utilized as it is suitable for examining educational processes (hamilton & corbett-whittier, 2013) and understanding the meaning for students involved (merriam, 1998). case study is defined as an in-depth study of a particular instance (hyde, 2000; merriam, 1998) and is therefore “not a methodological choice but a choice of what is to be studied” (stake, 2005, p. 443). because case studies in education commonly focus on a single assignment or course and results are sometimes difficult to perceive objectively (bishop-clark & dietz-uhler, 2012; hamilton & corbett-whittier, 2013), triangulation is more necessary in case-study research than other designs (divan et al., 2017) and involves using multiple perspectives in the study of the same phenomenon (hopper & hoque, 2000; van drie & dekker, 2013; stake, 2005). this multimethodmultimeasure process ensures that the findings are not dependent on any single measure or method of measurement (krathwohl, 2009), provide a more complete perspective (van drie & dekker, 2013), and improve an educator’s understanding of the findings (bishop-clark & dietz-uhler, 2012). krathwohl’s (2009) multimethod requirement will include student assignments from three classes, replicated across time, to examine the same learning outcome and independent student feedback from each class, “a redundancy of data gathering” (stake, 2005, p. 454). concerning the latter, all three course sections were evaluated using the individual development and educational assessment instrument (idea, 2021a) designed to assess progress on teaching objectives with results reported on a 5-point scale as follows: 1 = no apparent progress; 2 = slight progress; 3 = moderate progress; 4 = substantial progress; 5 = exceptional progress. the three, out of 32, idea objectives most relevant for considering student satisfaction with the jigsaw technique are grouped within the instrument under the title, fostering student collaboration, and include the following (idea, 2021b): • #5: formed teams or discussion groups to facilitate learning assessing a modified jigsaw technique 60 journal of effective teaching in higher education, vol. 5, no. 1 • #16: asked students to share ideas and experiences with others whose backgrounds and viewpoints differ from their own • #18: asked students to help each other understand ideas or concepts krathwohl’s (2009) multimeasures will include theoretical triangulation, the use of multiple theoretical perspectives simultaneously to study the same aspect of the research (hopper & hoque, 2006), to validate findings by showing that independent measures agree (van drie & dekker, 2013; yin, 2003). although the role of theory in the scholarship of teaching and learning (sotl) research has received little attention (divan et al., 2017), there is evidence that case studies in education can indeed be supported by educational theory (anderson, 2019; yin, 2014). for this study, two established educational theories were utilized to understand student learning outcomes by identifying levels of cognitive processing (anderson & krathwohl, 2001) and types of learning (fink, 2013). this was accomplished using pattern-matching analysis (yin, 2003, 2014), designed to compare a predicted pattern with the empirical findings of the study (e.g., anderson, 2019) and recommended for hypothesis testing in case studies (pearse, 2019). although yin (2003) originally prescribed generating an initial theory from existing research and using an opposing alternative theory for comparison, he later explicitly recommended that analyzing data with the theoretical model(s) that led to the study (e.g., fink, 2014) could produce dependable analysis and results. the technique used here reflects this later view by utilizing both previously mentioned theories as predetermined codes (pearse, 2019). a one-way anova was used to determine difference between the three classes and the acceptability of combining the classes for analysis. an average for each objective was reported. coding first, two independent coders utilized the propositions of anderson and krathwohl (2001) to identify levels of cognitive processing. descriptors for each level of the taxonomy were selected from the revised taxonomy (anderson & krathwohl, 2001) in a pilot study of six assignments. the most relevant are listed in table 1 below. table 1 student essay coding scheme for bloom’s revised taxonomy next, independent coders utilized the propositions of fink (2013) to identify types of learning. brief definitions of each type were developed and selected in a pilot study as well. these are listed in table 2. code taxonomy level example descriptors 1 remembering telling, recalling, recognizing 2 understanding comparing, contrasting, interpreting 3 applying implementing, utilizing, making use of 4 analyzing attributing, identifying motives, distinguishing 5 evaluating interpreting, prioritizing, critiquing, supporting 6 creating elaborating, predicting, imagining, proposing assessing a modified jigsaw technique 61 journal of effective teaching in higher education, vol. 5, no. 1 table 2 student essay coding scheme for types of learning code learning type example descriptors 1 foundational basic facts and understanding major ideas 2 application applying or learning a new skill or way of learning 3 integration connecting new ideas, experiences, or information 4 human dimension learning about self and others; societal implications 5 caring developing interest or seeing value in the subject 6 learning to learn understanding the process of learning; motivated at completion, a kappa score of intercoder reliability was determined for both using the intra-class correlation coefficient function of spss v.26. a score of >0.61 was considered substantial (mchugh, 2012). a one-way anova was used to determine difference between the three classes. results student satisfaction all students responding voluntarily with the idea (idea, 2021b) were quite positive concerning group work, rating each objective as a 4 or 5. a one-way anova indicated a statistically significant difference between the three classes, f (2, 6) = 9.000, p. = 0.16. post hoc analysis using a scheffé test indicated the difference was in the pairwise comparison of class 1 and class 3, with no significant difference between 1 and 2 nor 2 and 3. the effect is possibly attributable to unequal student response sizes. classes were considered together and an average for each is stated below and all indicate noticeable student satisfaction. • #5 formed teams or discussion groups to facilitate learning (m = 4.8/5.0, sd = .1000) • #16 asked students to share ideas and experiences with others whose backgrounds and viewpoints differ from their own (m = 4.9/5.0, sd = .000) • #18 asked students to help each other understand ideas or concepts (m = 4.7/5.0, sd = .046) level of thinking coding results using the six levels of bloom’s revised taxonomy (anderson & krathwohl, 2001) yielded a kappa score of interrater reliability of 0.701. means were very similar across all three classes: class 1 (m = 2.88, sd = 1.219), class 2 (m = 2.80, sd = 1.248), and class 3 (m = 3.01, sd =1.242), and a one-way anova indicated no statistically significant difference between the three, f (2, 540) = 1.244, p. = 0.289. considering the three classes together yielded m = 2.9 (sd = 1.233), indicating the applying level. however, this only accounted for 19.2% of the assessing a modified jigsaw technique 62 journal of effective teaching in higher education, vol. 5, no. 1 responses. considering a mode of 2 and examining the resulting rank order comparisons indicated the greatest number of responses (60.4%) at the understanding level (39.9%) and the analyzing level (20.5). other results included remembering at 8.4%, evaluating at 9.7%, and creating at 2.2%. type of learning coding with fink’s (2013) six types of learning produced a kappa score of interrater reliability of 0.677. means were again very similar across all three classes: class 1 (m = 2.99, sd = 0.991), class 2 (m = 3.03, sd = 1.237), and class 3 (m = 3.24, sd =1.080). a one-way anova indicated a no statistical difference between the three, f (2, 501) = 3.036, p. = 0.084. considering the classes together yielded m = 3.09 (sd = 1.109) and a mode of 3, indicating the greatest response at integration (31.9%). however, human dimension (31.3%) produced a near identical measure. percentages for other types of learning were much less common and included application at 19.2%, functional at 9.7%, caring at 7.1%, and learning to learn at 6%. discussion and limitations in agreement with maden (2011) and, more recently, lalit and piplani (2020) most students in all three classes appeared to be satisfied with jigsaw group work. each idea (idea, 2021a) objective relevant to group work was rated favorably by the instrument, and all three were listed as a strength to retain, defined as using the technique more often than classes of similar size and comparable student motivation. although written for general classroom group work, objective #16: “asked students to help each other understand ideas or concepts” (m = 4.9, sd = 0.0) was particularly interesting as it essentially described students teaching each other (e.g., barkley, 2010; costouros, 2020; nusrath et al., 2019; yoshida, 2018). this is certainly a critical element without which, according to benton (2016), “the [jigsaw] process falls apart” (p. 42). as discontent or frustration with the technique could have had a negative influence on other group members (benton, 2016; costouros, 2020; nusrath et al., 2019), it is intuitively likely that high student satisfaction with the technique influenced the positive coding outcomes discussed next. although pattern-matching analysis (yin, 2014) with the revised taxonomy (anderson & krathwohl, 2001) did not achieve a mean at the anticipated analyzing level, (4.0/6.0), it did show cognitive processing well above the remembering level (1.0/6.0) of rote learning and appropriate to the assignment requirements. results indicated an average response at the applying level (m = 2.9, sd =1.233), indicating students making apt use of existing knowledge and available skills to prepare for the jigsaw and complete the assignment. however, applying did not account for most responses. almost two-thirds (60.4%) exhibited a combination of comparing and contrasting at the understanding level (2.0/6.0) and attributing and distinguishing at the analyzing level (4.0/6.0). according to anderson and krathwohl (2001), this is not unusual because “most authentic academic assessing a modified jigsaw technique 63 journal of effective teaching in higher education, vol. 5, no. 1 tasks require the coordinated use of several cognitive processes” (p. 89) to create meaningful knowledge, and the two here aligned well with the assignment requirements. most of the statements coded at the understanding level reflected statements of comparing multiple sources to build a better appreciation of all. these included the following examples: we found that all parents wanted their children to succeed and get the best education possible. however, we also found that children of native american and latinx culture may not have the same opportunities...this is definitely contrary to existing stereotypes. many of these populations have immigrated to america and have been disadvantaged most of their time here. analyzing responses were somewhat more diverse in their application. all related to breaking information into parts for closer inspection, but most statements coded as analyzing here involved “determining a point of view, bias, values, or intent underlying presented material” (anderson & krathwohl, 2001, p. 68) or students making attributions based on these. many were indicative of statements focusing on specific critical points of information with students attempting to identify and make sense of causes or motives, a very specific form of analysis. examples included the following: i have heard the term “tiger moms” in reference to asian american mothers. it is a stereotype that the parents push their children to extremes to be successful by any means. when discussing the article, that stereotype proved false. what mainstream society might view as pushing their children too hard should be viewed as fostering a love of learning and expecting their children to work hard. overall, parents in all these cultures raise their children similarly when it comes to education and family. parents want to see their children succeed and to be the best individuals they can be… it left me thinking how much influence socio-economic status has on children’s futures. another aspect of analyzing noted here was distinguishing. this can be seen in the examples below as attempts to discern relevant from irrelevant parts but more often to distinguish fact from fiction: one of the more important factors that was discussed was that the latina/o community come from many different countries making the group actually very diverse, though they share many of the same values. i feel people often assume the latinx community is all from the same country with the same traditions because they speak spanish. i was aware of a few stereotypes about the [asian american] culture, such as “all asians are good at math” or the idea that parenting is more of a assessing a modified jigsaw technique 64 journal of effective teaching in higher education, vol. 5, no. 1 dictatorship. fortunately, the material i was given not only challenged these stereotypes but provided me with a whole new perspective... as indicated by the mean, student thinking here is less related to retention and more related to creating new understanding and transferring learning to other areas. this meets anderson and krathwohl's (2001) definition of meaningful learning and “provides students with the knowledge and cognitive processes they need for successful problem solving” (p. 65). results here also specifically confirmed anderson and krathwohl’s (2001) observation that “learning to analyze may be viewed as an end in itself, [but] it is probably more defensible educationally to consider analysis as an extension of understanding” (p. 79). applying pattern-matching analysis (yin, 2014) with fink’s (2013) significant learning experiences found positive results as well. as predicted, an average response of integration (m = 3.09, sd = 1.109) was found. findings here demonstrated that the cognitive process of understanding and learning to integrate are perhaps relatable, though not identical, in this collaborative setting. integration certainly involves seeking connections, but it is not limited to the assigned material. for instance, the example below connects information from the jigsaw with ideas introduced weeks earlier in the class: the emphasis that african americans place on balancing individual needs and the family as well as respecting elders is also seen in the latinx and native american culture. we discussed how the balance between the individual and family needs can be the outcome of parents trying to keep their family subsystems and culture flourishing. with the integration of a variety of cultures, we thought that emphasizing the importance of their culture was a way to preserve their history and legacy. a culture’s uniqueness gives people the guidance of knowing who they are and where they come from. this reminded us the discussion on cultural competence during the first week of class. likewise, integrating can facilitate an interdisciplinary awareness and learning (fink, 2013). the statement below indicates a possible connection to coursework in human development: …puerto rican mothers stressed interdependence and the european american mothers pushed for independence. after learning about this, i began to think about how humans innately need others to survive and that we need interdependence for relationships to be considered relationships. i believe the way latina/o culture teaches children is more developmentally appropriate… integrative learning can also assist students in learning to link different views and methods of analysis and begin to develop a more inclusive perspective of a problem (fink, 2013). this is comparable to what fink (2013) described when stating, the theme that occurs repeatedly in the discussion of these ventures is the desirability of breaking down walls and overcoming the isolation of students assessing a modified jigsaw technique 65 journal of effective teaching in higher education, vol. 5, no. 1 and subjects from each other that is they are seeking to create hitherto absent connections and integration among different people and different ideas. (p. 50) interestingly, coding with fink (2013) again resulted in two distinct but related elements, integration (31.9%) and human dimension (31.3%), again accounting for approximately two thirds (62.2%) of the responses. examples of learning about others included this observation: in my [expert] group, we all agreed that native american parenting is similar to typical american parenting, but better. we said this because these parents have goals for their kids such as having them become more independent, having a good work ethic, and having them care for others. the goals are the same for native american parents as well, but they seem to follow through with them. learning about the self can be seen below. i had always heard that many latina/o students received more financial aid than most others. but they have access to less aid for higher education…i’m a little embarrassed but i can now correct my thinking. …it made me realize how we tend to focus on every existing difference, when we share a variety of similarities. like the results with the taxonomy, these two types of learning are also relatable. fink (2013) stated confidently that integrative learning can aid students in building connections between what they are learning in class and their daily lives, the human dimension. lastly, it should be noted again that all participants here were female. although our family and consumer sciences department always includes a smaller percentage of male students, all female classes are typical in the human development and family science major. therefore, care should be taken in generalizing these results to male students or more diverse classes. future directions costouros (2020) noted that the jigsaw technique could vary with the instructor. certainly, various jigsaw models have a range of strengths, applications, and goals. all involve a search to better understand an existing identified whole by considering the pieces, as with the parenting practices in america used here. however, it is also conceivable that this modified form could have usefulness beyond analyzing distinct elements of a single previously identified concept or topic. considering that integrative learning increases the possibility of other types of learning (fink, 2013), it seems likely that broader, more open-ended topics could be explored in the classroom with this jigsaw as well. for instance, contemporary and historical views of the voting rights act could be compared and contrasted. similarly, varied assessing a modified jigsaw technique 66 journal of effective teaching in higher education, vol. 5, no. 1 arguments concerning climate change could be explored using credible news articles. perhaps this more investigative open-ended search could be used with such important current topics to identify unanswered questions or separate fact from opinion. although there may be no specific completed puzzle as a goal, practicing analysis and problem-solving has positive educational value (anderson & krathwohl, 2001). also, though theory in teaching and learning design and sotl research has been somewhat overlooked (divan et al., 2017), educational theory played a significant and necessary role in design and analysis here (e.g., merriam, 1998). more specifically, yin’s (2014) observation that theory used in design could play an authentic role in analysis was confirmed in coding and triangulation. perhaps it is time for theory, theorizing, and theory testing to play a greater role in sotl work. conclusion collaborative learning has consistently been shown to encourage critical thinking (dewati et al., 2019; nusrath et al., 2019) to enable students to teach others (crone & portillo, 2013) and to result in meaningful learning experiences. likewise, the jigsaw technique (aronson, 2021; aronson & patnoe, 2011) has long proven useful in fostering higher order thinking and processing (crone & portillo, 2013; nusrath et al., 2019) across multiple variations. that was the case here as well as the analysis revealed two primary levels of processing, understanding, and analyzing, both significantly beyond rote learning and strongly related to each other. although the comparing aspects of each were perhaps an inevitable part of the assignment and therefore not unexpected. the same could be said for the integration type of learning (fink, 2013) as considering seemingly dissimilar information to create new connections was also a part of the assignment. however, other findings, also positive, were unexpected as both thinking and learning went deeper as aspects of analyzing began to challenge known stereotypes held by some students. not only did this confirm anderson and krathwohl’s (2001) probable link between processing at understanding and analyzing, for some this became an opportunity to learn and to change their mind on some previously held beliefs. it seems possible that these new insights could also be understood as learning about self and others, or fink’s (2013) human dimension. like understanding and analyzing above, results here appear in agreement with fink’s (2013) observation that significant learning “is not hierarchical but rather relational and even interactive” (p. 37). these results indicated that the changes made here maintained the educational value of the jigsaw technique. this modified form was indeed successful in producing consistent, worthwhile learning and more critical thinking for most students. maden (2011) and costouros (2020) have described the jigsaw as having potentially limitless variations. findings here certainly serve to confirm such flexibility. assessing a modified jigsaw technique 67 journal of effective teaching in higher education, vol. 5, no. 1 conflict of interest statement the authors declare that there are no conflicts of interest regarding the publication of this article. references anderson, b. (2019). teaching developmental theory with interrupted video case studies: multi-measure evidence. journal of scholarship of teaching and learning, 19(5), 123–136. doi:10.14434/josotl.v19i5.25385 anderson, l. w., & krathwohl, d. r. (2001). a taxonomy for learning, teaching, and assessing: a revision of blooms taxonomy of educational objectives. longman. aronson, e. (2021). jigsaw in 10 easy steps. http://jigsaw.org/#steps aronson, e., blaney, n., stephan, c., sikes, j., & snapp, m. (1978). the jigsaw classroom. sage publications. aronson, e., & patnoe, s. (2011). cooperation in the classroom: the jigsaw method. pinter and martin. barkley, e. f. (2010). student engagement techniques. jossey-bass. benton, r. (2016). put students in charge: a variation on the jigsaw discussion. college teaching, 64(1), 40–45. https://doi.org/10.1080/87567555.2015.1069725 bishop-clark, c., & dietz-uhler, b. (2012). engaging in the scholarship of teaching and learning: a guide to the process, and how to develop a project from start to finish. stylus. calkins, s. c., & rivnay, j. (2021). the jigsaw design challenge. journal of effective teaching in higher education, 4(3), 19–35. https://doi.org/10.36021/jethe.v4i3.249 costouros, t. (2020). jigsaw cooperative learning versus traditional lectures: impact on student grades and learning experience. teaching & learning inquiry, 8(1), 154–172. doi:10.20343/teachlearninqu.8.1.11 crone, t. s., & portillo, m. c. (2013). jigsaw variations and attitudes about and the self in cognitive psychology. teaching of psychology, 40(3), 246–251. doi: 10.1177/0098628313487451 dewati, k. t. p. s., basori, b., & efendi, a. (2019). comparison of application of jigsaw iv and reverse jigsaw methods based on students’ participation and achievement. journal of informatics and vocational education, 2(2), 75–81. doi:10.20961/joive.v2i2.35750 assessing a modified jigsaw technique 68 journal of effective teaching in higher education, vol. 5, no. 1 divan, a., ludwig, l., matthews, k., motley, p., & tomljenovic-berube, a. (2017). survey of research approaches utilized in the scholarship of teaching and learning publications. teaching & learning inquiry, 5(2). https://doi.org/10.20343/teachlearninqu.5.2.3 doymus, k., karacop, a., & simsek, u. (2010). effects of jigsaw and animation techniques on students’ understanding of concepts and subjects in electrochemistry. educational technology research and development, 58, 671–691. https://doi.org/10.1007/s11423-010-9157-2 dhull, o., & verma, g. (2019). jigsaw teaching technique and teaching science. international journal of research and analytical reviews, 6(2), 809–815. fink, l. d. (2013). creating significant learning experiences. jossey-bass. ghaith, g., & el-malak, m.a. (2004). effect of jigsaw ii on literal and higher order efl reading comprehension. educational research and evaluation, 10, 105– 115. hamilton, l., & corbett-whittier, c. (2013). using case study in education research. sage. haviz, m., & lufri, l. (2019). implementing of subject jigsaw learning model and its impact on students’ achievement in embryology course. jurnal pendidikan biologi indonesia, 5(3), 435–442. https://doi.org/10.22219/jpbi.v5i3.9864 hedeen, t. (2003). the reverse jigsaw: a process of cooperative learning and discussion. teaching sociology, 31, 325–332. doi: 10.2307/3211330 hopper. t., & hoque, z. (2006). triangulation approaches to accounting research. in z. hogue (ed.), methodological issues in accounting research: theories and methods (pp. 477–486). spiramus press. hyde, k. f. (2000). recognizing deductive processes in qualitative research. qualitative market research: an international journal, 3, 279–292. doi: 10.1108/13522750010322089 idea. (2021a). about idea. https://www.ideaedu.org/about-idea/ idea. (2021b). idea notes of instruction. https://www.ideaedu.org/idea-notes-oninstruction/ kaplowitz, d. r., griffin, s. r., & seyka, s. (2019). race dialogues: a facilitator's guide to tackling the elephant in the classroom. teacher’s college press. krathwohl, d.r. (2009) methods of educational and social science research: the logic of methods (3rd ed.). waveland press. assessing a modified jigsaw technique 69 journal of effective teaching in higher education, vol. 5, no. 1 laal, m., & ghodsi, s.m. (2012). benefits of collaborative learning. social and behavioral sciences, 31, 486–490. doi: 10.1016/j.sbspro.2011.12.091 lalit, m., & piplani, s. (2019). active learning technology—jigsaw method: an innovative method of learning anatomy. journal of the anatomical society of india, 68(2), 147–152. doi: 10.4103/jasi.jasi_57_19 maden, s. (2011). effect of jigsaw i technique on achievement in written expression skill. educational sciences: theory & practice, 11(2), 911–917. mchugh, m. l. (2012). interrater reliability: the kappa statistic. biochemia medica, 22(3), 276–282. https://pubmed.ncbi.nlm.nih.gov/23092060/ merriam, s. b. (1998). qualitative research and case study applications in education. josey-bass. nusrath, a., dhananjaya, s. y., dyavegowda, n., arasegowda, r., ningappa, a., & begum, r. (2019). jigsaw classroom: is it an effective method of teaching and learning? student’s opinions and experience, 13(2): jc01–jc04. doi: 10.7860/jcdr/2019/39613.12540 pearse, n. (2019). an illustration of deductive pattern-matching procedure in qualitative leadership research. the electronic journal of business research methods, 17(3), 143–154. doi:10. 34190/jbrm.17.3.004 samuel, i. r. (2018). effects of jigsaw iv, group investigation and reversed jigsaw cooperative instructional strategies on basic science students’ achievement and retention. international journal of innovative education research, 6(2), 54–62. stake, r. e. (2005) qualitative case studies. in n. k. denzin & y. s. lincoln (eds.), the sage handbook of qualitative research (pp. 443–466). sage. timayi, j., bolaji, c., & kajuru, y. (2015). effects of jigsaw iv cooperative learning strategy (j4cls) on academic performance of secondary school students in geometry. international journal of mathematics trends and technology, 28, 12–18: https://www.ijmttjournal.org/archive/ijmtt-v28p504 turkman, h., & buyukaltay, d. (2015). which one is better? jigsaw ii versus jigsaw iv on the subject of the building blocks of matter and atom. journal of education in science, environment and health, 1(2), 88–94. doi:10.21891/jeseh.43349 van drie, j., & dekker, r. (2013). theoretical triangulation as an approach for revealing the complexity of a classroom discussion. british educational research journal, 39(2), 338–360. https://doi.org/10.1080/01411926.2011.652069 assessing a modified jigsaw technique 70 journal of effective teaching in higher education, vol. 5, no. 1 weidman, r., & bishop, m. j. (2009). using the jigsaw model to facilitate cooperative learning in an online course. the quarterly review of distance education, 10(1), 51–64. https://www.learntechlib.org/p/103626/ weimer, m. (2017, july). my students don’t like group work. the teaching professor. https://www.teachingprofessor.com/topics/for-those-whoteach/my-students-dont-like-group-work/ weimer, m. (2020, march). student attitudes about group work. the teaching professor. https://www.teachingprofessor.com/topics/for-those-whoteach/student-attitudes-about-group-work/ yatimah, d., solihin, s., adman, a., & syah, r. (2019). jigsaw learning model based on cooperative instructional strategies to improve academic discussion and adult education on environmental concepts. journal of physics: conference series, 1402(3), 1–4. doi:10.1088/1742-6596/1402/3/033039 yin, r. k. (2003). case study research: design and methods (2nd ed.). sage. yin, r. k. (2014). case study research: design and methods (5th ed.). sage. yoshida, m. (2018). communication jigsaw: a teaching method that promotes scholarly communication. international journal of emerging technologies in learning, 13 (10), 208–224. doi.org/10.3991/ijet.v13i10.8850 microsoft word rimarodriguez_jethe_interdisciplinary_final.docx journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.270 what did we learn? assessment of an interdisciplinary student project brandi rima, bronx community college cuny, brandi.rima@bcc.cuny.edu crystal rodriguez, bronx community college cuny abstract. assessment is critical to effectively implementing interdisciplinary pedagogy in higher education. we developed an interdisciplinary project for an introduction to psychology and introduction to criminal justice learning community at an urban community college. the semester-long project involved completing a series of lessons and assignments in both courses. the study objective was to evaluate the extent to which the interdisciplinary project supported students’ learning, with attention to their academic skill development, progress on course learning outcomes, and academic progress in the psychology course. data procedures included summative assessment of final paper submissions and collection of final paper grades and final grades in the psychology course across two cohorts of learning community students. findings suggest the interdisciplinary project positively supported students’ academic skill development, progress on course learning outcomes, and overall success in the psychology course. these findings demonstrate the value of interdisciplinary teaching and assessment. we offer the interdisciplinary project and assessment procedures presented here as models for higher education faculty interested in interdisciplinary pedagogical approaches. keywords: assessment; interdisciplinary pedagogy; learning communities; psychology; criminal justice assessment is central to effective pedagogy and student success in higher education. benefits of integrating assessment into teaching include gaining a better understanding of how coursework supports student progress, how to enhance the feedback process, and which modifications to teaching strategies, lessons, and assignments are successful (kesianye, 2015). in effort to effectively adopt assessment as part of our collaborative teaching work, we conducted an assessment study of an interdisciplinary class project entitled the kalief browder project. we created the project as part of a learning community curriculum for first year criminal justice students. the objectives of this paper are to explain the interdisciplinary project and report on the assessment study procedures and findings. faculty interested in collaborative, interdisciplinary pedagogy can use the student project and assessment procedures we describe here as models. interdisciplinary pedagogy interdisciplinary teaching fully integrates two or more disciplines in academic study (jones, 2009). the benefits of interdisciplinary instruction are multifaceted, including but not limited to helping students to develop college and career success skills, evaluate course content from diverse perspectives, perceive the connections among academic disciplines, network across college disciplines and departments, and enhance their communication and analytical skills (jones, 2009; kleinberg, assessment of an interdisciplinary project 54 journal of effective teaching in higher education, vol. 4, no. 3 2008). importantly, students value interdisciplinary learning as they enjoy their peer interactions and learning the intersections among disciplines (knapp & merges, 2017). merging the study of real-life events and “making meaning” into interdisciplinary teaching provides even more advantages. examination of real-life events in interdisciplinary study can foster community among students and improve their understanding of academic and career opportunities (knapp & merges, 2017). making meaning or reflective pedagogy is the practice of making connections among course content and students’ own life experiences (desjarlais & smith, 2011). faculty create making meaning in the classroom by purposefully teaching content that speaks to students’ everyday lives (guthrie & mccracken, 2010). this approach creates opportunities for students to identify, analyze, and connect to the course material. assessment assessment of interdisciplinary teaching and learning is important to measure student progress and success from these pedagogical practices. because of the need for collaboration and creativity, interdisciplinary teaching can involve a fair amount of trial and error. assessing student progress through their work on interdisciplinary initiatives enhances these pedagogical efforts. the assessment process typically begins with setting explicit goals for students’ learning outcomes. with interdisciplinary teaching, the process includes an additional step with faculty agreeing upon the goals. next, faculty develop a plan for how students can achieve the goals (hutchings, 2019). vehicles for assessment of student learning outcomes include pre and post surveys, rubrics, classroom-based performance assessments, capstone projects and exams (hutchings, 2019; jankowski et al., 2018; wenninger, 2018). however, for new pedagogical approaches, the assessment of interdisciplinary teaching is best assessed using assignments such as projects, essays, and discussions (mueller et al., 2014). assessment should be designed for students to continue learning and develop skills in preparation for their future careers (boud & soler, 2016). the next step is to develop an objective rubric that can measure the students’ learning through their work on projects, essays, and/or discussions. assessment procedures can be conducted throughout (i.e., formative assessment) and/or at the completion of an instructional period (i.e., summative assessment; dixson & worrell, 2016). we adopted a summative assessment approach in the present study to give insight into overall student achievement and learning through the interdisciplinary project. additional resources and restructuring of the traditional college model allow for fully integrating interdisciplinary teaching into standard curricula. interdisciplinary instruction can necessitate cross-discipline collaboration; integration of syllabi, lessons, and coursework across courses; development of new course materials that effectively merge content and align with the learning objectives of multiple courses; and coordinated assessment efforts. colleges and universities must equip their faculty with the training, compensation, time, and course schedules and programming that best serve extensive cross-discipline and interdepartmental assessment of an interdisciplinary project 55 journal of effective teaching in higher education, vol. 4, no. 3 collaboration. learning community programming is one approach to providing this support. the learning community model the learning community model involves a peer group of students completing two or more courses together (kuh, 2008; visher et al., 2011; weiss et al., 2014). a fundamental characteristic of learning communities is extensive faculty collaboration to integrate courses across disciplines in both structure and content. learning communities, therefore, provide avenues for interdisciplinary pedagogy and coursework (weiss et al., 2015). other common characteristics of learning community programming include blocked scheduling of courses, reduced class sizes, special advisement and mentorship opportunities for students, and supplemental training and compensation (e.g., reduced course loads) for faculty. research supports learning communities as a high-impact teaching practice that is especially effective for first-year college students (fink & inkelas, 2015; kuh, 2016; rima et al., 2018). as faculty at an urban community college in the bronx, new york, we developed and taught an introduction to psychology and introduction to criminal justice learning community for first-year criminal justice students. goals of this learning community included providing supplemental support and guidance in the first year of college, building community among students sharing a major, and facilitating completion of degree requirements. we employed the learning community model as a vehicle for instituting a more comprehensive interdisciplinary teaching approach. this is important for students seeking careers in criminal justice as the field is exceptionally interdisciplinary and often requires expertise in other areas including but not limited to psychology, sociology, political science, communication, computer science, language, and mathematics. an interdisciplinary educational experience in the first year therefore strengthens student readiness for advanced study and careers in criminal justice. we designed the kalief browder project as part of our effort to provide an interdisciplinary curriculum for criminal justice students. the kalief browder interdisciplinary project we developed the kalief browder project to meaningfully integrate the psychology and criminal justice course content. the project guides students to apply their knowledge from both courses to critically examine a former student’s experiences in the criminal justice system. kalief browder, a black man from a low-income neighborhood in new york city, was arrested for allegedly stealing a backpack at the age of 16 and subsequently spent three years awaiting trial in rikers island jail (gonnerman, 2014; 2015a, 2015b, 2015c). for approximately two of these three years, browder was held in solitary confinement (gonnerman, 2014). throughout this time, browder maintained his innocence and refused to accept a plea bargain. the prosecutor dropped the case three years after browder’s arrest. following his release, browder attended bronx community college in the bronx, new york, and excelled in his studies. however, he experienced psychological distress such as depression, anxiety, and paranoia (gonnerman, 2014; 2015a, 2015b, 2015c) assessment of an interdisciplinary project 56 journal of effective teaching in higher education, vol. 4, no. 3 during his time in rikers island and upon release back into his community. tragically, he died by suicide two years after release from jail. kalief browder and his experiences with the criminal justice system were the inspiration for the learning community’s interdisciplinary project. the kalief browder project integrates content across the two learning community courses: introduction to psychology and introduction to criminal justice. throughout the semester, students participated in a series of lessons and complete lowand high-stakes assignments for the project in both courses. project activities included discussions and readings about browder and his case (gonnerman, 2014; 2015a, 2015b, 2015c) and viewings of the netflix documentary time: the kalief browder story (carter & weinstein, 2017) in both courses. lessons and assignments in both courses prepare students to complete the final paper of the kalief browder project. in the introduction to criminal justice course, students (1) learn about the issues and disparities with the criminal legal system through weekly discussions applying browder’s case to course content, (2) complete a writing assignment to identify relevant issues within police, courts, and corrections, and (3) practice developing solutions to reform the criminal justice system. in the introduction to psychology course, students worked on a final paper assignment of the kalief browder project, which involved scaffolded writing assignments. the semester-long project allowed students to improve their understanding of the content across the criminal justice and psychology disciplines, critically analyze issues and experiences through different lenses, and develop solutions based on their knowledge of newly acquired content (mueller et. al, 2014). importantly, the project has meaning for our students as, like kalief browder, many of them are also young persons of color living in the bronx, an urban, low-income neighborhood. the project creates opportunities for students to discover shared experiences with kalief and to also view their own lives through a new lens that is informed by the course content. for a full description of the interdisciplinary project see rima and rodriguez (2021). assessment of the capstone assignment of the project, a final paper in the psychology course, is the focus of the present study. a central aim of the paper assignment was to give students an opportunity to reflect upon kalief browder’s experiences through an interdisciplinary lens. students were tasked with using their psychology knowledge to identify and meaningfully reflect upon criminal justice issues relevant to browder’s case and develop their own informed solutions. lessons in both courses and other project activities prepared students for this capstone assignment. instruction in the criminal justice course provided students with knowledge and vocabulary needed to understand how the criminal justice system functions, including its shortcomings. preparation for writing the final paper in psychology included two scaffolded writing assignments: a paper outline and rough draft. the psychology professor provided constructive feedback on each scaffolded assignment. the paper assignment aligned with two of the six learning outcomes the college’s psychology faculty designated for the introduction to psychology course: assessment of an interdisciplinary project 57 journal of effective teaching in higher education, vol. 4, no. 3 learning outcome 1: students will demonstrate understanding of psychological theories, principles, and concepts. learning outcome 2: students will produce well-reasoned written and/or oral arguments applying psychological concepts and principles to demonstrate an understanding of social and cultural phenomena. study purpose & hypotheses the objective of this study was to evaluate student learning at the completion of the kalief browder project. more specifically, we aimed to evaluate the extent to which the project supported students’ learning, with attention to their academic skill development and progress on the psychology course’s learning outcome 1 and learning outcome 2. to achieve this objective, we assessed final paper submissions across two cohorts of learning community students (spring 2019 and fall 2019). we also report on students’ final paper grades as a second indicator of academic progress on the kalief browder project. additionally, we aimed to explore whether participation in the learning community improved academic performance in introduction to psychology overall. we report on final grades in introduction to psychology comparing the learning community students to other first-year criminal justice students in non-learning community sections of the course. we expected that students in the learning community would earn higher final grades in introduction to psychology than non-learning community students. see below for the null and alterative hypotheses. null hypothesis: there is no difference among groups and their academic performance. alternative hypothesis: there is a difference among groups and their academic performance. study context and method the study took place at bronx community college (bcc) of the city university of new york (cuny), which serves a mostly minority and socioeconomically underprivileged student population. student enrollment at the college is approximately 10,519 students with 64% attending full time and 36% part time. slightly more than half (55%) identify as first-generation college students. seventy-one (71%) report an annual household income of less than $30,000 (bcc’s office of institutional research). bcc’s criminal justice program partners with john jay college of the city university of new york (cuny) to facilitate community college students’ transition to a fouryear college to complete their bachelor’s degree. the criminal justice program is bcc’s second largest academic program with approximately 1,000 students enrolled. special programming and support benefits criminal justice students as they typically have lower gpas, decreased program persistence, and lower assessment of an interdisciplinary project 58 journal of effective teaching in higher education, vol. 4, no. 3 graduation rates compared to the general student population (bronx community college, office of institutional research). we created the introduction to psychology and introduction to criminal justice learning community to offer needed supplemental support to criminal justice students. introduction to psychology is a core requirement of the criminal justice degree. the course has a standard curriculum and a cumulative final examination. successful completion of the psychology course can be challenging for criminal justice majors. for instance, in spring and fall 2018 only about 63% of criminal justice students enrolled in introduction to psychology earned passing grades in the course (bronx community college, office of institutional research). we aimed to use an interdisciplinary teaching approach to improve criminal justice students’ success in the psychology course. students in the spring and fall 2019 introduction to psychology and introduction to criminal justice cohorts submitted the final papers included in this assessment study. as per college policy, learning communities enroll a maximum of 25 students. additionally, learning community clusters often do not meet maximum enrollments for various reasons, including fewer students meeting the requirements for both courses and scheduling and registration challenges. this applies to the introduction to psychology and introduction to criminal justice learning community as it was designed specifically for first-year criminal justice students. across both semesters, 36 students were enrolled in the learning community. of these students, 31 (13 and 18 in spring and fall 2019, respectively) submitted a final paper for the kalief browder project. all 31 submitted papers were included in this study. importantly, because of this study’s small sample size, our findings are more suited to inform future pedagogical innovation and research than to draw definitive conclusions. all study procedures were approved by the college’s institutional review board (irb). assessment measures & procedures we selected an assessment rubric that best meets the standards of the interdisciplinary project created for learning community students. as per the recommendations of the association of american colleges and universities (aac&u), we revised and adapted their written communication value rubric (2009) for this study. all revisions best aligned the assessment rubric with three primary areas of interest: academic skill development, learning outcome 1, and learning outcome 2. we made all revisions in consultation with bronx community college’s assessment office. the finalized rubric included four assessment categories: 1) content development, 2) meaningful reflection, 3) context & purpose, and 4) control of syntax and mechanics. the content development category measured the extent to which students’ written arguments demonstrated mastery of psychology content, which assessed progress on learning outcome 1. we used the meaningful reflection category, which measured the students’ application of psychological principles to demonstrate an understanding of the social and cultural phenomena to measure assessment of an interdisciplinary project 59 journal of effective teaching in higher education, vol. 4, no. 3 progress on learning outcome 2. the remaining two categories measured aspects of academic skill development. the context and purpose category assessed students’ ability to demonstrate a thorough understanding of the purpose of the assignment and responsiveness to the assignment tasks. the control and syntax category measured students’ ability to use advanced language and skillfully communicate meaning through writings. the rubric has five ratings from below benchmark (0), meeting the benchmark (1–3), and exceeding the benchmark (4). see appendix a for the assessment rubric. we also obtained secondary data from college records to assess students’ academic progress. these data included final paper grades and final grades in the psychology course. additionally, we obtained the final grades of first-year criminal justice students enrolled in all other sections of introduction to psychology during the spring and fall 2019 semesters. data analyses to assess students’ final papers, we applied norming (also called calibration) procedures. norming is a process of establishing consistency across faculty raters in their assessment of student work. we followed the rhode island department of education (2020) and washington state university’s office of assessment for curricular effectiveness’ (2020) guidelines for norming, which include conducting practice rounds, time for raters to independently assess student work, and discussion of scores to reach consensus on scoring. in the present study, two raters conducted two practice norming rounds to establish scoring consistency before commencing study assessment procedures. for both practice rounds, raters scored paper submissions from a learning community cluster not included in this study. significant modifications (e.g., writing lessons and scaffolded assignments) to the kalief browder project beginning in spring 2019 deemed papers from previous semesters ineligible for inclusion in the present study but appropriate for practice rounds. following wang’s (2009) recommendations to maintain anonymity in all assessment procedures, we randomly selected paper submissions for inclusion in practice rounds and removed student identifiers. we labeled papers by number for organizational purposes. these steps ensured anonymous assessment of the sample papers. for both practice rounds, raters scored papers selected at random from the fall 2018 cohort. in the first practice round, raters independently assessed three papers in the categories of content development, meaningful reflection, context & purpose, and control & syntax. the raters agreed on 41% of the assessment ratings across all four assessment categories. after discussing reasons for discrepancies and reaching agreement, raters conducted the second round of assessment with three additional papers. for round 2, raters reached 75% agreement across all four assessment categories establishing acceptable inter-rater agreement (stemler, 2004). assessment of an interdisciplinary project 60 journal of effective teaching in higher education, vol. 4, no. 3 following the second practice round, raters commenced study assessment procedures. like the two practice rounds, we applied strategies wang (2009) recommended to ensure anonymity in assessment procedures. we removed all student identifiers and assigned each paper a random number. raters independently assessed the papers in groups of three to four. raters met to compare scores, discuss discrepancies, and reach agreement when needed after independently scoring each group (rhode island department of education, 2020; washington state university’s office of assessment for curricular effectiveness, 2020). to compare final grade data of students enrolled in the learning community to those of criminal justice students in other sections of introduction to psychology during the spring and fall 2019 semesters, we calculated percentages and conducted a one-way analysis of variance (anova) test. results the results are organized in three overall sections: academic skill development (context & purpose and control & syntax rubric categories), learning outcome 1 (content development rubric category), and learning outcome 2 (meaningful reflection rubric category). for all assessment categories, results indicate the percentage of papers exceeding (rubric rating of 4), meeting (rubric rating of 1–3) or falling below (rubric rating of 0) benchmark. academic skill development students’ academic skill development was assessed using the context & purpose and control & syntax categories. for context & purpose, 100% (n = 31) of the students met the benchmark. more specifically, the majority (80%; n = 25) of students met, 19% (n = 6) exceeded, and 0% fell below the benchmark. most students, therefore, demonstrated a thorough understanding of the purpose of the assignment and were responsive to the assignment tasks. assessment ratings for the control & syntax category showed that 100% (n = 31) met the benchmark. the majority of students demonstrated their ability to use advanced language and skillfully communicate meaning through writing. learning outcome 1 learning outcome 1 required students to demonstrate understanding of psychological theories, principles, and concepts. specifically, students should demonstrate learning in at least one of the following areas: human development, social interaction, psychopathology, cognitive processes, and biological bases of behavior. we assessed learning outcome 1 using the content development category. all students (100%; n = 31) met the benchmark in the content development category demonstrating their understanding of foundational course concepts in the final paper. assessment of an interdisciplinary project 61 journal of effective teaching in higher education, vol. 4, no. 3 learning outcome 2 learning outcome 2 aimed for students to produce well-reasoned written and/or oral arguments applying psychological concepts and principles to demonstrate an understanding of social and cultural phenomena. the meaningful reflection category assessed students’ progress on this learning outcome. most (74%; n = 23) students met the benchmark for meaningful reflection. a few (26%; n = 8) did not meet the benchmark. most students applied concepts and principles to demonstrate understanding of social and cultural phenomena. academic performance we also explored the learning community students’ overall performance on the kalief browder project as well as in the introduction to psychology course. student work on the final paper assignment is an indicator of overall project performance as it is the capstone assignment and requires application of knowledge and skills gained from all project activities. we collected final paper grades as a measure of project performance. of the 31 final papers submitted for the kalief browder project most (58%; n = 18) were assigned above average (a or b) grades, 32% (n = 10) were assigned an average grade (c), and 10% (n = 3) were assigned a below average grade (d or f). ninety percent (n = 28) of the students who completed the final paper assignment demonstrated average or above average performance on the final paper assignment of the project. to evaluate the students’ academic performance in introduction to psychology, we compared their final grades to those of other criminal justice majors in non-learning community sections of introduction to psychology in spring and fall 2019. the college’s office of institutional research provided all final grade data. for this comparison, we included final grade data for students enrolled in the learning community n = 43), excluding seven students that withdrew from the learning community (officially or unofficially). comparing final grades in introduction to psychology, the learning community first-year students outperformed other criminal justice majors. for example, 86% (n = 31) of the learning community students compared to 53% (n = 172) earned above average (a or b) grades. about 8% (n = 3) compared to 29% (n = 95) earned average (c) grades and 5% (n = 2) compared to 17% (n = 57) earned below average (d or f) grades. students in the learning community were more likely than their peers to earn above average grades in introduction to psychology. a one-way anova test indicated a statistically significant difference between group means, f(1, 178) = 5.86, p = .017. the null hypothesis was rejected. discussion the primary objective of this study was to conduct a summative assessment of a student project, the kalief browder project. the project was part of an interdisciplinary learning community curriculum for first-year criminal justice students. the project included a series of lessons and assignments students completed throughout the semester and culminated in a capstone paper assessment of an interdisciplinary project 62 journal of effective teaching in higher education, vol. 4, no. 3 assignment. this study involved assessment of final paper submissions across two cohorts of students in an introduction to psychology and introduction to criminal justice learning community. we also evaluated students’ academic progress in the psychology course, reporting on their final paper grades as well as final grades in the psychology course. study results supported the interdisciplinary project as an effective strategy for facilitating students’ academic skill development, mastery of course learning outcomes, and academic progress in the psychology course. findings demonstrate that the learning community curriculum supported students’ academic skill development and success in the psychology course. we adopted and revised the association of american colleges and universities’ (aac&u) written communication value rubric (2010) rubric to conduct assessment procedures. the rubric guided assessment of student work across in the areas of context & purpose, control & syntax, content development, and meaningful reflection. these rubric categories assessed student achievement in three main areas: 1) academic skill development (context & purpose and control & syntax), 2) learning outcome 1 (demonstrate understanding of psychological theories, principles, and concepts; content development), and 3) learning outcome 2 (produce well-reasoned written and/or oral arguments applying psychological concepts and principles to demonstrate an understanding of social and cultural phenomena; meaningful reflection). overall, students successfully demonstrated their academic skill development and mastery of both learning outcomes through their work on the project’s capstone assignment. we found that students demonstrated their academic skill set through their work on the paper. students tended to understand the purpose of the interdisciplinary project, completed the required assignment tasks correctly, and used advanced language to communicate their ideas in writing. the learning community structure, scaffolded nature, and semester-long duration of the project likely fostered students’ ability to successfully demonstrate these academic skills in the capstone assignment of their project. the learning community model gave students access to increased support, guidance, and feedback on project progress than is typically available in stand-alone courses. completing the project in two courses also provided more opportunities for clarification and reinforcement regarding project requirements. scaffolding assignments throughout the semester allowed for an effective feedback and revision process. findings also suggest that the interdisciplinary project supported students’ mastery of course learning outcomes. most students demonstrated understanding of foundational psychology theories, concepts, and principles (learning outcome 1). even more, students successfully applied psychological concepts to real life, social, and cultural situations through their writing (learning outcome 2). many students meaningfully analyzed how psychological concepts relate to kalief browder’s experiences with the criminal justice system and extended these connections to their social and cultural lives. thus, the learning community students were equipped to meaningfully apply their learning, view content through different perceptual lenses, and handle the complexity of the interdisciplinary project, which assessment of an interdisciplinary project 63 journal of effective teaching in higher education, vol. 4, no. 3 are all skills fostered through interdisciplinary pedagogy (jones, 2009; kleinberg, 2008; latucca, 2001; spelt et al., 2009). finally, our analysis of student grades on the project’s capstone assignment and final grades in the psychology course demonstrated their academic progress. most students earned average or above average grades on the capstone paper assignment suggesting their successful completion of the kalief browder project. these results supported the proposed alternative hypothesis. learning community students were more likely than criminal justice students enrolled in other sections of introduction to psychology in spring and fall 2019 to earn above average final grades in the course. combined with our assessment results, evaluation of student grade data suggests that interdisciplinary teaching and learning supported students’ academic success in the psychology course. these findings are important because passing the psychology course is required for the criminal justice major. interdisciplinary teaching in both learning community courses supported students’ success in the psychology course, including their progress on learning outcomes, satisfactory performance on the interdisciplinary project’s capstone assignment, and successful completion of the psychology course. effectively expressing understanding of course concepts and meaningfully applying gained knowledge requires skills fostered through interdisciplinary pedagogy, including communication and analytical skills and the ability to evaluate course content from diverse perspectives (jones, 2009; kleinberg, 2008; latucca, 2001). we offered the introduction to psychology and introduction to criminal justice learning community at bronx community college (bcc) in part because the psychology course is a requirement for the criminal justice degree and also challenging to master. importantly, students in the learning community demonstrated significant progress on the psychology course learning outcomes. using students’ interest in criminal justice and integrating criminal justice and psychology likely made psychology more fun and engaging for students, which supported their learning and success in the course (costanzo & costanzo, 2013). recommendations the findings of this study reiterate the value of interdisciplinary teaching in higher education. we offer recommendations for administrators and faculty to successfully implement interdisciplinary learning opportunities. recommendations for administrators multifaceted institutional support is key. coordinated course scheduling, registration, and advisement efforts can facilitate interdisciplinary teaching, especially for those adopting team teaching across disciplines or the learning community model (weiss et al., 2015). because assessment is fundamental to the interdisciplinary teaching process, institutional support should also include funding, training, and mentorship for assessment. assessment of an interdisciplinary project 64 journal of effective teaching in higher education, vol. 4, no. 3 adequate resources and training are needed to support interdisciplinary innovation (larson et al., 2011). interdisciplinary teaching requires additional effort to foster collaborations across disciplines and departments, learn and test strategies for merging disciplines and courses, and create new course materials and assessment vehicles. for example, monetary compensation, course load reductions, mentorship, and professional development programs are some strategies for advancing and institutionalizing interdisciplinary pedagogy. administrators can work to develop a culture of collaboration among faculty through strategies like the above to support continued cross-discipline and interdisciplinary teaching. recommendations for faculty we also recommend offering support for teaching collaborations. interdisciplinary teaching requires partnerships with faculty outside of our own disciplines and departments. forming these relationships can be challenging in the traditional college environment, which typically houses faculty by discipline, provides few opportunities for cross-departmental networking, and can create competition among academic departments. programs designed to bring diverse faculty together for networking and building teaching collaborations can facilitate interdisciplinary teaching. the faculty members teaching in this learning community both received a reduced course load to coordinate and integrate their courses as well as the support of a learning community faculty coordinator. faculty also met monthly to align their content and discuss student progress. as hutchings (2019) recommends, faculty can join assessment councils at their institutions and attend assessment conferences to connect with other faculty. our psychology faculty member was also on the college’s assessment committee. furthermore, merging interdisciplinary teaching with other high-impact practices will best promote student learning. for example, making meaning or reflective teaching allows students the opportunity to personally connect with the learning (desjarlais & smith, 2011). also, analysis of real-life events can strengthen interdisciplinary learning (mueller et al., 2014; spelt et al., 2004). students can better understand the value of adopting an interdisciplinary approach when examining how content connects to their own lives and world around them, which may foster greater investment in learning. we believe the kalief browder project was an effective interdisciplinary pedagogical tool in part because it involved learning about a fellow student. the learning community students could likely identify with browder as they occupied the same spaces and shared common experiences and circumstances, which we believe motivated their learning and encouraged deeper reflection and analysis. limitations & future directions this study provides a useful template for assessment of interdisciplinary projects and supports the value of interdisciplinary pedagogy for student academic success; however, there are some methodological limitations. first, due to the nature of the study, the sample size is relatively small. we focused on student work collected from learning community cohorts, which enroll a limited number of students. assessment of an interdisciplinary project 65 journal of effective teaching in higher education, vol. 4, no. 3 second, results may not be generalizable to other student populations as the study took place at one urban community college. other institutions should make modifications to our model for their student populations. third, there is potential for self-selection bias as students opted to participate in the learning community. the learning community model may be more attractive to certain types of students. based on our experiences conducting the present assessment study, we offer suggestions regarding assessment of interdisciplinary pedagogy and future research. interdisciplinary work should include both formative and summative assessment (dixson & worrell, 2016). this paper provides a model of summative assessment (i.e., evaluating student learning at the completion of instruction) of interdisciplinary learning. including formative assessment (i.e., assessment of student learning throughout the instructional process) would provide a richer understanding of student learning and allow for needed modification during the learning process (mueller et al., 2014). second, assessment should be ongoing, flexible, and open to regular revision to assessment procedures, curricula, and coursework (hutchings, 2019). this is especially true for interdisciplinary teaching as it requires coordination across disciplines and courses. assessment must accommodate each individual discipline as well as new objectives that arise from merging disciplines. the present study focused only on assessment of learning in the psychology course. future studies should explore best strategies for assessing overall objectives of interdisciplinary teaching as well as individual course learning outcomes. finally, we recommend making assessment procedures more transparent to students. notably, the learning community students did not have access to the assessment rubric for the capstone assignment of the kalief browder project. sharing the assessment rubric with students prior to assignment completion may have better guided students’ fulfillment of assignment expectations (mueller et al., 2014). adding this step into the assessment process may help more students meet or exceed assessment benchmarks. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references association of american colleges and universities (aac&u). (2009). inquiry and analysis value rubric. https://www.aacu.org/value/rubrics/inquiry-analysis boud, d., & soler, r. (2016). sustainable assessment revisited. assessment & evaluation in higher education, 41(3), 400–413. https://doi.org/10.1080/02602938.2015.1018133 carter, s., & weinstein, h. (executive producers). (2017). time: the kalief browder story [tv series]. the cinemart; the weinstein company. assessment of an interdisciplinary project 66 journal of effective teaching in higher education, vol. 4, no. 3 costanzo, m. l., & costanzo, m.a. (2013). teaching clinical (and nonclinical) psychology through applications to the legal system: violence risk assessment and the insanity defense. teaching of psychology, 40(3), 252– 256. https://doi.org/10.1177/0098628313487452 desjarlais, m., & smith, p. (2011). a comparative analysis of reflection and self-assessment. international journal of process education, 3(1), 3–18. https://www.ijpe.online/2011/reflection.pdf dixson, d. d., & worrell, f. c. (2016). formative and summative assessment in the classroom. theory into practice, 55(2), 153–159. http://doi.org/10.1080/00405841.2016.1148989 fink, j. e., & inkelas, k. k. (2015). a history of learning communities within american higher education. new directions for student success, 149, 5–15. https://doi.org/10.1002/ss.20113 gonnerman, j. (2014, october 6). before the law. the new yorker, 26–32. https://www.newyorker.com/magazine/2014/10/06/before-the-law gonnerman, j. (2015a, april 15). kalief browder and a change at rikers. the new yorker. https://www.newyorker.com/news/news-desk/kalief-browder-and-achange-at-rikers gonnerman, j. (2015b, april 23). exclusive video: violence inside rikers. the new yorker. https://www.newyorker.com/news/news-desk/exclusive-videoviolence-inside-rikers gonnerman, j. (2015c, june 7). kalief browder, 1993–2015. the new yorker. https://www.newyorker.com/news/news-desk/kalief-browder-1993-2015 guthrie, k. l., & mccracken, h. (2010). reflective pedagogy: making meaning in experiential based online courses. the journal of educators online, 7(2), 121. https://files.eric.ed.gov/fulltext/ej904075.pdf hutchings, p. (2019, january). what new faculty need to know about assessment. national institute for learning outcomes assessment. https://www.learningoutcomesassessment.org/wpcontent/uploads/2019/08/assessment-brief-faculty.pdf jankowski, n. a., timmer, j. d., kinzie, j., & kuh, g. d. (2018, january). assessment that matters: trending toward practices that document authentic student learning. university of illinois and indiana university, national institute for learning outcomes assessment (niloa). https://eric.ed.gov/?id = ed590514 jones, c. (2009). interdisciplinary approach: advantages, disadvantages, and the assessment of an interdisciplinary project 67 journal of effective teaching in higher education, vol. 4, no. 3 future benefits of interdisciplinary studies. essai, 7, 1–6. http://dc.cod.edu/essai/vol7/iss1/26 knapp, s., & merges, r. (2017). an evaluation of three interdisciplinary social science events outside of the college classroom. college teaching, 65, 137– 141. http://dx.doi.org/10.1080/87567555.2016.1244655 kuh, g. d. (2008). high-impact educational practices: what are they, who has access to them, and why they matter. association of american colleges and universities. https://provost.tufts.edu/celt/files/high-impact-edpractices1.pdf kuh, g. d. (2016). some thoughts about doing the right thing in uncertain times. journal of college and character, 17, 213–222. https://doi.org/10.1080/2194587x.2016.1230761 kesianye, s. k. (2015). the three perspectives of integrating assessment and instruction in the learning of school mathematics. journal of education and practice, 6(19), 212–214. https://files.eric.ed.gov/fulltext/ej1079550.pdf kleinberg, e. (2008). interdisciplinary studies at a crossroads. association of american colleges and universities. https://www.aacu.org/publicationsresearch/periodicals/interdisciplinary-studies-crossroads larson, e. l., landers, t. f., & beggs, m. d. (2011). building interdisciplinary research models: a didactic course to prepare interdisciplinary scholars and faculty. clinical translational science journal, 4(1), 38–41. doi: 10.1111/j.1752-8062.2010.00258.x latucca, l. r. (2001). creating interdisciplinarity: interdisciplinary research and teaching among college and university faculty. vanderbilt university press. https://www.learningoutcomesassessment.org/wpcontent/uploads/2019/08/assessment-brief-faculty.pdf mueller, a., juris, s. j., willermet, c., drake, e., upadhaya, s., & chhetri, p. (2014). assessing interdisciplinary learning and student activism in a water issue course. journal of the scholarship of teaching and learning, 14(2), 111–132. rhode island department of education. calibration protocol for scoring student work: a part of the assessment toolkit. https://www.ride.ri.gov/portals/0/uploads/documents/teachers-andadministrators-excellent-educators/educator-evaluation/onlinemodules/calibration_protocol_for_scoring_student_work.pdf rima, b., rodriguez, c. c., & depaola, t. (2018). supplementing the “college assessment of an interdisciplinary project 68 journal of effective teaching in higher education, vol. 4, no. 3 experience” with first year program-based learning communities. the community college journal of research and practice, 43, 840–853. https://doi.org/10.1080/10668926.2018.1542637 rima, b., & rodriguez, c. c. (2021). bringing meaning to learning: an interdisciplinary project for first-year community college students. teaching of psychology, 48(3), 204–208. https://doi.org/10.1177/0098628320979878 spelt, e. j. h., biemans, h. j. a., tobi, h., luning, p. a., & mulder, m. (2009). teaching and learning in interdisciplinary higher education: a systematic review. educational psychology review, 21, 365–378. https://doi.org/ 10.1007/s10648-009-9113-z stemler, s. e. (2004). a comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. practical assessment, research, and evaluation, 9(4), 1–11. https://doi.org/10.7275/96jp-xz07 visher, m. g., teres, j., & richman, p. (2011, july). breaking new ground: an impact study of career-focused learning communities at kingsborough community college. national center for postsecondary research. https://files.eric.ed.gov/fulltext/ed522631.pdf wang, p. (2009). the inter-rater reliability in scoring composition. ccse, 2(3), 39– 43. https://files.eric.ed.gov/fulltext/ej1083076.pdf washington state university’s office for curricular effectiveness. (2020). quick guide to norming on student work for program-level assessment. https://ace.wsu.edu/documents/2015/03/rubrics-norming.pdf/ wenninger, h. (2018). student assessment of venture creation courses in entrepreneurship higher education—an interdisciplinary literature review and practical case analysis. entrepreneurship education and pedagogy, 2(1), 58– 81. https://doi.org/10.1177/2515127418816277 weiss, m. j., mayer, a., cullinan, d., ratledge, a., sommo, c., & diamond, j. (2014). a random assignment evaluation of learning communities at kingsborough community college: seven years later. mdrc. https://www.mdrc.org/sites/default/files/a_random_assignment_evaluation _of_learning_communities_kcc.pdf weiss, m. j., visher, m. g., weissman, e., & wathington, h. (2015). the impact of learning communities for students in developmental education: a synthesis of findings from randomized trials at six community colleges. educational evaluation and policy analysis, 37, 520–541. https://doi.org/10.3102%2f0162373714563307 assessment of an interdisciplinary project 69 journal of effective teaching in higher education, vol. 4, no. 3 appendix a revised written communication value rubric from the association of american colleges & universities (aac&u) “definition: written communication is the development and expression of ideas in writing. written communication involves learning to work in many genres and styles. it can involve working with many different writing technologies, and mixing texts, data and images. written communication abilities develop through iterative experiences across the curriculum. evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance.” capstone 4 milestones 3 2 benchmark 1 below benchmark 0 context of and purpose of writing includes consideration of audience, purpose and the circumstances surrounding the writing task(s). demonstrates thorough understanding of purpose that is responsive to the assigned tasks and focuses on all elements of the work. demonstrates adequate consideration of purpose that is responsive to the assigned tasks and focuses on all elements of the work. demonstrate s awareness of purpose and assigned tasks. demonstrates minimal attention to purpose and assigned tasks. does not meet the benchmark for context and purpose of writing. content development uses appropriate, relevant, and compelling content to illustrate mastery of the subject, conveying the writer’s understanding, and shaping the whole work. uses appropriate, relevant, and compelling content to explore ideas within the context of the discipline and shape the whole work. uses appropriate and relevant content to develop and explore ideas through most of the work. uses appropriate and relevant content to develop simple ideas in some parts of the work. does not meet the benchmark for content development. meaningful reflection incorporates detailed attention to applying psychology principles to uses consistent application of psychological processes to demonstrate follows expectations to apply psychological processes to demonstrate attempts to apply psychological processes to demonstrate understandin does not meet the benchmark for meaningful reflection. assessment of an interdisciplinary project 70 journal of effective teaching in higher education, vol. 4, no. 3 capstone 4 milestones 3 2 benchmark 1 below benchmark 0 demonstrate understanding of social and cultural phenomena. understanding of social and cultural phenomena. understandin g of social and cultural phenomena. g of social and cultural phenomena. control of syntax and mechanics uses advanced language that skillfully communicates meaning to readers with clarity and fluency and is virtually errorfree. uses straightforwar d language that generally conveys meaning to readers. the writing in the portfolio has few errors. uses language that generally conveys meaning to readers with clarity, although writing might include some errors. uses language that sometimes impedes meaning because of errors in usage. does not meet the benchmark of writing mechanics. microsoft word elhage_final.docx journal of effective teaching in higher education, vol. 5, no. 2 https://doi.org/10.36021/jethe.v5i2.268 alumni perceptions of the educational benefits of their graduate degrees from a pbi rasha elhage, chicago state university, relhage@csu.edu olanipekun laosebikan, chicago state university kimberly black-parker, chicago state university abstract. much of the research on the benefits of graduate education for minority students has focused on historically black colleges and universities. little has been done on predominantly black institutions. this case study attempts to shed light on alumni perceptions of the benefits of their graduate education at a predominantly black institution (pbi), chicago state university. the researchers surveyed the accessible population of chicago state university alumni who completed requirements for their master’s and doctoral degrees between the academic years 2008 and 2018. results indicate that perceived benefits fell into two categories: 1) personal benefits that accrued to the graduate and 2) enabling benefits that encouraged the completion of graduate study. personal benefits included personal fulfillment, an enhanced knowledge of a profession or discipline, and advancement in a career including promotions and salary raises, while some enabling benefits included perceived program quality, the financial accessibility of graduate study at a pbi, and a supportive learning environment. findings from this study of alumni can lend insights to effective approaches to teaching graduate students from diverse backgrounds. keywords: predominantly black institutions (pbis); minority serving institutions; african american graduate students; graduate education; educational benefits introduction/rationale graduate education in the united states in recent years has reached never before seen heights. enrollments in master’s, doctoral, and professional degrees have risen dramatically among all racial and ethnic groups including african american students. from 2000 to 2016, african american enrollment in graduate school grew by more than 100% from 181,000 to 362,000 (de brey et al., 2019, p. 130). minority serving institutions (msis), including predominantly black institutions (pbis), have increasingly taken on larger roles in providing graduate education to students. pbis in particular represent a growing share of the minority student, and more specifically african american student, enrollment in higher education. pbis currently make up 3% of all postsecondary institutions yet enroll 9% of all african american college students in higher education (jones, 2019). though they have a significant impact in producing successful outcomes for minority students, pbis have yet to receive much scholarly attention. they are a particularly significant locus of study in understanding educational access among african americans because of their increasingly urban locations and the relative size of these urban populations (parker et al., 2019). alumni perceptions of the educational benefits 94 pbis were first recognized by the u.s. congress with the passage of the higher education opportunity act of 2008. by definition, pbis must serve a student body that is 50% low income or first generation and 40% african american and have an average educational and general expenditure that is low relative to those of institutions of higher education that offer similar instruction. (u.s. department of education [usde], 2014). pbis have a specific and unique focus on individuals of african descent who experience poverty, are the first in their families to attend college and attend under-resourced institutions. the vast majority of the 156 pbis in the united states are two-year colleges located in the south, midwest, and east. chicago state university (csu), the focus of this study, is one of the relatively few that offer graduate level education. though increasing in number, pbi graduate programs are relatively new. currently, only five pbis offer graduate and professional level programs. the growth of graduate education at these institutions was strongly influenced by the passage of the higher education opportunity act (heoa) in 2008, which included a provision for the development of master’s degree programs at pbis. the heoa created the pbi master’s program grant which allowed pbis to increase support for or establish new programs at the master’s level for african american students in some of the following fields: mathematics, information technology, physical or natural sciences, computer science, and other related scientific fields (hegji, 2017). while sharing some similarities with other msis, but in particular hbcus, pbis have significant differences in history, mission, scope, location, and focus which make a study of graduate outcomes at these institutions a significant contribution to the scholarly literature. an understanding of what african americans desire and value from their graduate education beyond the obvious goal of attaining a credential is instrumental for effective teaching and learning that responds to their needs. also, a better understanding of key factors in graduate education for institutions serving student populations that are majority first-generation, low-income, and african american is needed to improve educational access and social and economic mobility. literature review much of the research on the benefits of higher education for higher education graduates has focused on undergraduate education. here, the greater share of attention has been on the economic and non-economic or social benefits of higher education to the individual. individuals with a bachelor’s degree or higher are more likely to achieve a host of positive economic outcomes: greater salaries, greater lifetime earnings, employment, job safety, and successful promotions within their careers (ma et al., 2016; trostel, 2015). socially, in comparison to their noncollege peers, college graduates have longer life expectancies, are more likely to be married, be in better health, report higher rates of happiness, and are less likely to be jailed or incarcerated (ma et al., 2016). research suggests, however, that these benefits vary in significance to groups across racial/ethnic, gender, and socioeconomic status (ses) lines. alumni perceptions of the educational benefits 95 challenging conventional thinking, brand and xie’s (2010) study highlights that college education and the benefits thought to accrue with it are not driven by positive selection, where “individuals who are most likely to select into college also benefit most from college” (p. 273). their findings instead show negative selection: those who are least likely to attend college benefit the most from it. as a disadvantaged group in college enrollment, women benefited more from college education than men. similarly, perna (2005) found that degree attainment for women, individuals from low ses backgrounds, and blacks and hispanics resulted in greater economic and non-economic payoffs. women with degrees were more likely to have higher incomes, health insurance, and a lower likelihood of being on public assistance rolls. in a non-economic context, they were more likely to not smoke and to vote regularly. similar results were found when race/ethnicity were taken into account. blacks with some postsecondary education were more likely to receive a greater benefit in health insurance coverage, and perceived employment benefits. according to perna (2005), differences in college enrollment have more to do with the perception of difference rather than the reality. a lack of knowledge or inaccurate knowledge on the part of individuals from low-income backgrounds, blacks, and hispanics are thought to lead to their underestimation of the benefits of college education. the implications of these findings are considerable. one clear need is for greater institutional level information on the benefits to degree attainment by race, gender, and ses. research on msis is proving to be beneficial in addressing this concern. scholarly attention is growing around msis and their impact on student learning. research highlights the nurturing environments of msis, and hbcus in particular, as significant in increasing levels of student learning and engagement. studies show that many msis provide black students with a welcoming environment inclusive of racial and cultural diversity, greater levels of student engagement both in and out of the classroom, and impactful faculty interactions (boland et al., 2019; espinosa et al., 2017; flores & park, 2013; gasman et al., 2017). the significance of a positive campus climate to student retention and overall success cannot be overstated. galotti et al.’s (2016) study of student perceptions of the academic climate of liberal arts institutions they attend found a strong positive correlation with student retention. students who are comfortable in their institutions are likely to have the perception that “they can speak freely, approach instructors, and feel included as part of a learning community” (p. 42). campus environment and climate have proven to be significant in predicting student engagement for black students specifically. as jett (2011) identified, minority students are more likely to persist and report greater learning outcomes when engaged. nelson laird et al. (2007) found that high achieving black students have greater levels of engagement on campus in comparison to their white peers at predominantly white institutions (pwi). however, this often does not translate into increased learning outcomes at pwis due to feelings of alienation and isolation (shappie & debb, 2017). many black students at hbcus find their academic environment to be just as rigorous as at a pwi (seifert et al., 2006) and yet are able to excel as a result of the positive racial environments and protective settings that hbcus offer (shappie & debb, 2017). these supports include positive and alumni perceptions of the educational benefits 96 meaningful relationships with peers and faculty. research conducted by seifert, drummond, and pascarella (2006) indicates that students at hbcus interact with peers both in and out of the classrooms more often than counterparts at both research and regional institutions. due to the historical nature of their mission, faculty at hbcus have shown a greater propensity for developing culturally affirming pedagogical practices that validate the racial and ethnic identities of students who attend these institutions (williams et al., 2021). less clear are the practices of faculty at other msis including pbis and hsis who lack institutional missions and/or historical rationales for serving minority students (hubbard & stage, 2009). boland’s (2018) content analysis of msis’ title iii and v programs based on project abstracts indicates that one of the most espoused outcomes of msis is completion (graduation). msis, according to the american council on education (espinosa et al., 2017), matriculate students at higher rates than the federal graduation rate. using data from a 2007 cohort of the national student clearinghouse, researchers found that up to 66% of full-time hbcu students, 52% of pbi students, and 40% of hispanic serving institutions (hsis) students graduated, all of which are well above the federal graduation rate of 34%. additionally, msis are believed to affirm the values and backgrounds of their students. they foster community and provide conducive environments for the success of their students, including financial access, faculty and peer mentoring, and enrichment programs (conrad & gasman, 2015). there is compelling evidence at the undergraduate level that attending a msi has positive economic benefits related to occupational status as well as income mobility. strayhorn (2016) found that hbcu graduates, as compared to their non-hbcu peers, occupied higher status jobs, earned comparable salaries, and reported very little difference in job satisfaction. espinosa et al. (2018) report found that incomemobility rates for graduates from msis were two to three times higher than for those from non-minority serving institutions. the report studied over half of the 700 msis in the u.s. more than 20% of the students sampled from these institutions were from families in the lowest income quartile. in addition, about 50% of the students in this sample were first generation college students. the average rate of mobility for students from all u.s. higher education institutions was 1.9%, but most msis exceed this statistic. among msis, hispanic serving institutions had the highest rate of income mobility at 4.3%; pbis had the second highest rate of mobility at 3.5% followed by asian-american and native american/pacific-islander serving institutions at 3.3% and hbcus at 2.8%. the return on investment or roi for msis is higher than for non-msis. msi students were found to have higher earnings when compared to their non-msi peers (boland et al., 2019). this finding held true 10 years beyond graduation and also across demographic differences including for both male and female students. unfortunately, there has been very little research on roi or benefits of graduate education at msis. what research does exist is largely focused on hbcus. the hbcu impact in graduate education is particularly notable in specific fields, like stem. according to the national science foundation (2015), hbcu graduate programs enrolled 13.3% of all black graduate students in science and engineering alumni perceptions of the educational benefits 97 programs, 39% of all black students in agricultural sciences, and 32% of all black students in biology. research also shows that the benefits that hbcus provide to graduate students are similar in nature to those provided to undergraduate students. these include greater levels of faculty interactions and a welcoming environment inclusive of racial and cultural diversity, while ensuring positive academic outcomes including less time to degree completion and similar graduation rates as non-hbcus (anderson & hrabowski, 1977; fountaine, 2012; hall & clossen, 2005; palmer et al., 2012). at least one study indicated that there were health benefits that accrued later in life that are attributable to attending an msi (colen et al., 2021). african american graduate students in particular have emphasized the affirming, caring, and nurturing environments of hbcus in influencing their personal and professional development (palmer et al., 2016). theoretical framework there is a growing recognition of the value of msis in providing educational opportunities to underrepresented and low-income students. however, there have been very few efforts to theorize the impact of these institutions in leading to successful outcomes for students of color. our study draws from arroyo and gasman’s (2014) hbcu and institution-based theoretical framework for analyzing black college student success. their hbcu-based framework provides an important lens through which to view black college student success at hbcus and, as this paper attempts to show, success at other msis including pbis. arroyo and gasman theorize that hbcus offer greater access and affordability, supportive environments, and iterative processes and outcomes including achievement, identity formation, and values cultivation. the initial entry point to arroyo and gasman’s framework is “relative institutional accessibility and affordability” (p.66). hbcus, as the authors note, “welcome a diverse applicant population, including students from a range of experiences and backgrounds, through relatively accessible tuition and admissions policies” (p. 66). the next element described in the framework is a supportive environment that is central to black student success. elements of this supportive environment include “opportunities for friendship with peers, faculty, staff and counselors beyond the classroom, engagement in extracurricular campus life including leadership development and a climate of academic development so that an individual can achieve feelings of progress” (p. 64). the last pillar of the theory is a reciprocal process within the learning environment that produces success; these processes include 1) a unique approach to identity formation, 2) values cultivation, and 3) achievement. the outcome of the reciprocal processes is a holistic form of success: graduation, career attainment, and civic contributions. methodology this study presents an answer to a single research question: what are alumni perceptions of the educational benefits of their graduate degrees from a pbi? in asking this central question about educational benefits of a graduate degree from a pbi, insights into the components of what works for successful teaching and learning of students of color can be attained (hutchings, 2000). alumni perceptions of the educational benefits 98 this study was completed as a case study of an urban pbi, chicago state university (csu). csu was founded as a teacher training school in blue island, illinois, on september 2, 1867. today, the university is a fully accredited public, urban institution located on the south side of chicago. csu serves a predominantly black population (70%) (chicago state university, 2018). the university’s five colleges— health sciences, arts and sciences, business, education, and pharmacy—offer 36 undergraduate and 25 graduate and professional degree-granting programs (chicago state university, 2018). csu also offers an interdisciplinary honors college for students in all areas of study and has a division of continuing education and nontraditional degree programs that offers extension courses, distance learning, and not-for-credit programs to the entire chicago community. csu is a mission-driven institution whose aim is to “transform students’ lives by innovative teaching, research, and community partnerships through excellence in ethical leadership, cultural enhancement, economic development, and justice” (chicago state university, 2022, para. 4). the institutional character of csu aims at supporting the educational, social, and economic mobility of racial minorities and their advancement in society. a survey with a mixed-method design was utilized for this study, combining quantitative and qualitative data. johnson et al. (2007) define mixed methods design as “the type of research in which a researcher combines elements of qualitative and quantitative research approaches…for the broad purposes of breadth and depth of understanding and corroboration” (p. 123). watkins and gioia (2015) further elaborate that the “overall rationale for mixing methods is a better understanding of the inherent complexities of human phenomena” (p. ix). the survey was administered to the accessible population of csu alumni who completed requirements for their master’s, professional, and doctoral degrees between the academic years 2008 and 2018. the identification of potential participants was accomplished by a review of data facilitated by the csu office of alumni affairs. a request was submitted to these offices to retrieve data, specifically last known email addresses, for csu alumni who completed a graduate degree during the aforementioned time period. there were 3,700 alumni who successfully completed and earned a graduate, professional degree, or certificate from csu from 2008 to 2018 (csu, 2020). however, 700 of them did not have a valid email on record; hence, the survey was sent to 3,000 participants. inclusion criteria was defined as any csu student who successfully matriculated from a csu graduate program (23 master’s degrees, 1 pharm.d, 1 ed.d, 9 graduate certificates) between 2008 and 2018. the researchers of this study developed a survey inspired by the bachelor and beyond graduate survey 2017 of the institute for educational statistics (ies). permission was obtained from ies to reformat the survey. the survey has relevant questions that relate to the central question of this study. survey development was a collaborative and iterative process after which the final draft was reviewed and piloted to establish face validity. the pilot survey was sent to recent graduates after 2018 from various graduate programs and a number of faculty who shared their alumni perceptions of the educational benefits 99 feedback about the survey’s content and format. final changes were made before fielding it. data for this study was gathered using qualtrics software. this system allowed for the dissemination and collection of survey responses. the researchers emailed the survey link to all potential respondents who held a valid email address, after which two reminders were sent. collected data was used to conduct a quantitative and qualitative analysis. quantitative analysis the study employed descriptive statistics to report percentages for the survey’s discrete categories. qualitative analysis the study employed thematic analysis following a six-step process: familiarization, coding, generating themes, reviewing themes, defining and naming themes, and writing up (clarke & braun, 2013). terry et al. (2017) define thematic analysis as “a method for identifying, analyzing, and interpreting patterns of meaning (‘themes’) within qualitative data” (p. 287). the two most significant themes that were identified were 1) faculty relationships and 2) inclusive environment, which were in line with the quantitative results. participants’ statements about faculty relationships and inclusive environment affirmed students’ perceived benefits in personal motivations, inclusive environment, and overall satisfaction as shown in the results section below. results the total number of respondents who completed the survey was 261, a response rate of 8.7%. survey researchers have witnessed a gradual decrease in survey participation over time (brick & williams, 2013; national research council, 2013). particularly, alumni surveys often have lower response rates than other types of surveys because of bad contact information and other reasons (lambert & miller, 2014). furthermore, the response rate for web surveys is estimated to be 11% lower than other survey modes (fan & yan, 2010). while many researchers assume that the best way to obtain unbiased estimates/results is to achieve a high response rate, others are questioning the widely held assumption that low response rates provide biased results (curtin et al., 2000; groves, 2006; keeter et al., 2000; massey & tourangeau, 2013; peytchev, 2013). research conducted by fosnacht et al. (2017) suggests that their study “did not find that a 5% response rate or even a 75% response rate provides unbiased population estimates under all circumstances …” (p. 262). considering the above mentioned factors, the results of this study are regarded as valid for answering the research question. the following descriptive statistics help to illustrate the various characteristics of the survey respondents. alumni perceptions of the educational benefits 100 demographics of respondents gender sixty-seven percent of respondents were females, 30% were males, and 3% preferred not to answer. race sixty-four percent of respondents were african american; 17% were white; 9% asian; 4.6% hispanic; 1% biracial or multiracial; 1% some other race, ethnicity, or origin; and 3.4% preferred not to respond. a further breakdown of the results by race and gender shows that 47% of respondents were black or african american females, 15.7% were black african american males, 10% were white females, 6.5% were white males, 3% were asian females, 5.7% were asian males, 3% were hispanic females, and 1.5% were hispanic males first generation graduate student fifty-two percent of respondents reported being the first in their family to obtain a graduate degree. a further breakdown of the results by race shows that 52% of the african american respondents were the first in their family to obtain a graduate degree, 39.5% of the white respondents were the first in their family to obtain a graduate degree, and 56.5% of the asian respondents were the first in their family to obtain a graduate degree. degrees obtained seventy-nine percent of respondents received a master's degree from csu, 17% received a doctoral degree, and 4% received a graduate certificate. a further breakdown by race and gender shows that 6.5% of total respondents who received a doctoral degree were african american females, 2.7% were african american males, 1.1% were white females, 3% were white males, and 1.1% were asian (males and females). when looking at the respondents who received a master’s degree, 41.3% were african american females, 13.4% were african american males, 8.4% were white females, 3% were white males, and 7.7% were asian (males and females). returning students thirty-one percent of respondents were returning students who completed their bachelor's degree from csu and returned for their graduate degree, while 69% attended other institutions. a further breakdown of the results by race and gender of returning students show that 19.9% were african american females, 6.1% were african american males, 0.4% were white females, 6.5% were white males 1.1 % were hispanic (males and females) and 0.4% were asian (males and females). alumni perceptions of the educational benefits 101 personal motivations and benefits factors in choosing field of graduate studies when asked about the factors for choosing their field of graduate studies, respondents could report multiple answers. forty-eight percent identified that it was required for a career path; 41% checked the potential to increase earnings; 33% checked that it gave them the ability to contribute to their community via the chosen field; and 30% checked that they chose their field in order to change careers to best fit their personal aspirations. the response rate was lower for factors such as respondent's own aptitude in the field (25%), ability to balance work and family (16%), and increasing representation of gender/racial/ethnic group in a designated profession (16%). main reasons for enrollment respondents' answers show that one of the main reasons students enroll in the graduate program at csu is its affordability (72% checked "best affordable"). other reasons include csu's status as a minority institution (17%), access to professional employment opportunities (17%), and the high ranking of the program (17%). of the total survey respondents, those who checked “best affordable option available at the time” were 36.4% african american, 13.4% white, 5.7%, and 3.4% asian and hispanic. some participants’ statements revealed that their main reason for enrollment was csu’s status as a pbi, while others referred to csu’s reputation and quality of programs. • “after going to schools that were predominantly caucasian or almost all caucasian... i wanted to be with people of color…” • “csu changed my life. i initially attended a pwi. i was lost. at csu i felt valued by my professors. they encouraged me so much. i wish i would have initially attended csu. i might have a doctorate by now.” • “i chose to attend chicago state university graduate program mostly because i have heard many great things about the program from my previous coworkers. in addition, i knew that i would be challenged and receive a great learning experience.” benefits after obtaining a graduate degree respondents’ answers show that the greatest benefit after obtaining their graduate degree was personal fulfillment (64%), followed by advanced knowledge in the discipline (54%) and getting a new job (54%). moreover, 42.5% reported improved leadership skills, 38.7% reported that they obtained a salary increase, 36.8% gained more credibility, 30.3% changed their career path, 29% enhanced their prestige, 24.5% felt increased job security, and 13.4% received a promotion. alumni perceptions of the educational benefits 102 participants expressed additional personal benefits after obtaining their graduate degree from csu such as feeling that their education was competitive with other reputable universities. other statements revealed personal benefits of receiving high quality education. • “my education at csu is integral to the educator i am today. i credit my professors and the challenging curriculum for providing me with instruction that i feel rivals one from any ivy league institution.” • “upon entering my field, i was more prepared than most of my colleagues who had graduated from the same field.” • “my experience in csu’s msw program was nothing short of fruitful. the professors were truly top tier. i’m currently obtaining a second masters from (name withheld) and csu professors are undoubtedly better.” • “since i completed my certificate in 2012, i have been employed with a school corporation. i'm very satisfied.” • “each professor that taught me had a great influence on my career path. i am eternally grateful that i attended csu for my post bachelorette career because i gained an appreciation for new knowledge, different lenses to look at various topics.” finances how graduate studies were financed as noted in the previous section, 72% of the respondents selected csu because it was the most affordable option available to them for graduate study. participants stated, “csu provides a service to the community through reasonably priced postsecondary education” and “csu is affordable”. the majority of respondents (75.9%) financed their graduate education through loans (federal or private). thirty-six percent financed their graduate education through personal/family earnings or savings; 29% through a scholarship, grant, assistantship, or fellowship; and 13.4% through employer reimbursement/ assistance. a further examination of the results showed that 37% of respondents financed their graduate education via loans only; 3% via personal/family earnings or savings only; 3% via scholarship, grant, assistantship, or fellowship only; and 3% via employer reimbursement/assistance only. how much was borrowed the results showed that respondents who borrowed $100,000 or more in loans were the largest group at 19% of total participants, followed by 18% who borrowed $10,000-$29,999, 14% who borrowed $30,000-$49,999, 10.3% who borrowed $50,000-$69,999, 6.7% who borrowed $70,000-$99,999, and only 3.8% who borrowed less than $10,000. participants who reported not financing their graduate education through loans were 29%. alumni perceptions of the educational benefits 103 how much is still owed the results show that 43% of respondents still owe all of the amount borrowed for their student loans, 25% still owe some, and only 8.4% owe none. table 3 reports participants’ results related to their finances. inclusive environment socioeconomic background the results showed that 88% of respondents agreed or strongly agreed that their graduate program provided an inclusive and respectful environment of their socioeconomic background; 9% neither agreed nor disagreed; and 3% disagreed or strongly disagreed that their graduate program provided an inclusive and respectful environment of their socioeconomic background. race/ethnicity the results showed that 87% of respondents agreed or strongly agreed that their graduate program provided an inclusive and respectful environment of their race/ethnicity; 11% neither agreed nor disagreed; and 2% disagreed or strongly disagreed that their graduate program provided an inclusive and respectful environment of their race/ethnicity. gender identity or sexual orientation the results showed that 84% of respondents agreed or strongly agreed that their graduate program provided an inclusive and respectful environment of their gender identity or sexual orientation; 14% neither agreed nor disagreed; and 2% disagreed or strongly disagreed that their graduate program provided an inclusive and respectful environment of their gender identity or sexual orientation. religious/spiritual beliefs the results showed that 73% of respondents agreed or strongly agreed that their graduate program provided an inclusive and respectful environment of their religious/spiritual beliefs, 24% neither agreed nor disagreed, and 3% disagreed or strongly disagreed that their graduate program provided an inclusive and respectful environment of their religious/spiritual beliefs. political orientation the results showed that 68% of respondents agreed or strongly agreed that their graduate program provided an inclusive and respectful environment of their political orientation, 29% neither agreed nor disagreed, and 3% disagreed or strongly disagreed that their graduate program provided an inclusive and respectful environment of their political orientation. alumni perceptions of the educational benefits 104 participants’ comments supported the benefits of the inclusive environment perceived at csu. faculty relationships were the factor that was most significant in supporting this inclusive environment. sense of belonging and community were also reported. • “i loved being a graduate student at csu. i had amazing professors who supported my endeavors and i learned so much from them.” • “my experience as a graduate student was overwhelmingly positive. the education department was extremely supportive and always available. the staff was very understanding and flexible with regard to school and family.” • “great instructors, small class sizes created a great learning environment.” • “it felt good to be part of an environment where people got second chances and people are rooting for you to succeed.” • “the support, sense of community and belonging in my program contributed to my success”. satisfaction with choice of major the results showed that 88% of respondents were extremely satisfied or satisfied with their choice of majors or fields of graduate studies, while 9% were neither satisfied nor dissatisfied, and 3% were dissatisfied or extremely dissatisfied. with the quality of the curriculum the results showed that 88% of respondents agreed or strongly agreed that they felt satisfied with the quality of their program’s curriculum, while 7% neither agreed nor disagreed and 5% disagreed or extremely disagreed. with the quality of graduate education the results showed that 88.5% of respondents were extremely satisfied or satisfied with the quality of the graduate education they received, while 5.75% were neither satisfied nor dissatisfied and 5.75% were dissatisfied or extremely dissatisfied. participants' comments concurred with the quantitative data as participants expressed their satisfaction with the quality of their graduate education. • “my experience as a graduate student was overwhelmingly positive…the staff was very understanding and flexible with regard to school and family...the support and flexibility of the csu staff helped me to complete my graduate studies on time and without interruption. the staff and students were more like family than colleagues. ” • “my experience was positive. it was a very intense program but the faculty was helpful and encouraging. the experience was life changing and csu will never be forgotten.” • “i loved the fact that i was being taught by professors that looked like me.” alumni perceptions of the educational benefits 105 • “wonderful experience, great instruction, hands-on involvement with faculty and students.” with the cost when participants were asked if the graduate education they received was worth its financial cost, 80% answered yes, while 20% answered no. discussion this project set forth to answer the question, “what are alumni’s perceptions of the educational benefits of their graduate degree at a pbi?” the results suggest that the perceived benefits fell into two categories: 1) personal benefits that accrued to the graduate and 2) enabling benefits that encouraged the completion of graduate study. the content and range of the personal benefits revealed in this study confirm arroyo and gasman’s (2014) model of black student success which defines the grand outcome of hbcu education as “holistic success” (p. 71)—a triumvirate matrix of gradation, career success and civic contribution. the enabling benefits of a pbi that were instrumental to the completion of a graduate program include perceived program quality, the financial accessibility of graduate study at a pbi, and a supportive learning environment with supportive faculty. though measured at the end of the process in the case of graduates, these enabling benefits confirmed arroyo and gasman’s model in terms of the importance of the entry points of hbcus and pbis (“relative institutional accessibility and affordability”) and the supportive learning environment, which are enabling conditions for successful completion of graduate programs. personal benefits the primary personal benefit of the graduate degree completed at a pbi described by graduates was personal fulfillment. graduates most consistently reported that completing graduate school was personally fulfilling and satisfying and a third suggested that the degree conferred prestige. this finding lends tentative support to cokley’s (2002) claim that african american students attending hbcus had a significantly higher academic self-concept than students at predominantly white institutions. arroyo and gasman (2014) highlighted the relationship between achievement and identity formation among students at hbcus, and harmon (2012) calls attention to a key benefit of msis in providing students with a means to develop their identities leading to self-worth. the psychosocial benefits from matriculating at a pbi appear to be among the most durable and important benefits of graduates. hardre and hackett (2015) determined that one of the most important reasons graduate students cited for attending graduate school was “to help me achieve my professional goals,’’ which was confirmed in this study. scott and sharp (2019) found that aspects of “cultural beliefs,” which include spirituality, communal responsibility and self-belief, were essential critical factors for educational success at the graduate level for african american males. pbis fostered alumni perceptions of the educational benefits 106 those protective factors associated with positive mental health, including selfesteem, identity, and spirituality (mushonga & henneberger, 2020). this finding suggests that for graduate programs to successfully retain and educate african americans, there should be attention given to providing the psychosocial factors associated with personal fulfillment from advanced study in addition to meeting the traditional concerns of providing a quality curriculum (king et al., 2019) learning understood as mastery of content is insufficient on its own to completely meet the most noted benefit of a graduate degree to pbi graduates. programs should strive to deliver content in a manner that results in the personal fulfillment of the student. the second and third most often cited perceived personal benefits were the enhanced knowledge of one’s discipline that was afforded by advanced studies and that the graduate degree enabled the respondents to advance in their careers. enhanced knowledge is an expected benefit, though not the single most significant one for the respondents. enhanced knowledge supports the goal of career advancement. the pbi, because of its accessibility, allowed respondents to obtain a new job, pursue their chosen career path, obtain a salary increase, experience greater job security, and receive a promotion. this result confirms a finding by hardre and hackett (2015) who found that competitive advancement in one's field was a major reason for attending graduate school. these factors associated with career advancement result in social mobility for minority respondents and confirm hardy et al.’s (2019) findings that although hbcus are comparatively under resourced, “hbcus significantly outperform pwis in providing upward social mobility to students” (p. 474). graduate programs serving underrepresented students should strive to support the specific goal of social mobility through career advancement. another significant reported personal benefit of graduate school was personal development in the form of enhanced leadership skills. over two-fifths (42.5%) of the respondents reported improved leadership skills as a benefit of their degree. this result corroborates arroyo and gasman’s (2014) framework model which indicates that hbcus place a “distinctive emphasis on formation of student identity,” specifically in the area of leadership. the final personal benefit was the ability of the respondents to contribute to their communities through the practice of their careers. one-third (33%) of the respondents indicated that they selected their field of study because it gave them the ability to contribute to their communities via their career. this result indicates the degree to which individuals attending pbis have an innate desire to serve and build up their community through their life’s work. scott and sharp (2019) noted in their study of african american men who possessed a graduate degree that some felt the weight of a “responsibility to be significant contributors within their community” (p. 52). this finding supports boland’s (2018) assertion that goals “including community and culture are often seen as the secret to how [msis] have been able to graduate a higher number of students of color. infusing programs that prioritize student success through connections to community and incorporation of culture can lead towards increased quantifiable outcomes” (p. 14). alumni perceptions of the educational benefits 107 other enabling benefits each respondent reported receiving several personal benefits from completing their graduate studies at a pbi. respondents also reported various benefits that served the purpose of enabling them to successfully complete graduate level work. these benefits support arroyo and gasman’s (2014) discussion of the virtues of the accessibility of hbcus for students of color. baum and steele (2017) found that african american college graduates were proportionally more likely than other ethnic groups to seek master’s education, so simple accessibility to graduate education is a powerful enabling force provided by pbis and hbcus. the first enabling benefit of a graduate education from a pbi is that graduates were highly satisfied with the quality of their programs. a large majority (88%) of the respondents agreed that they felt satisfied with the quality of their program’s curriculum, and 88.5% reported being satisfied with the quality of their graduate education. the second enabling benefit of earning a graduate degree at a pbi is its financial accessibility and its financial value. cossa and barker (2021) and pyne and grodsky (2020) found that finances (debt) were the most influential factor for african americans to prevent enrollment in a graduate program. three-quarters (75.9%) of the respondents financed their graduate education through loans, and nearly as many (72%) indicated that csu was the most affordable option that they had to attend graduate school. this finding is consistent with that of miller and orsillo (2020) which indicates that 79% of african americans complete graduate school with federal debt, and the median graduate debt of african american borrowers is $51,250. nearly one-fifth (19%) of the respondents borrowed $100,000 or more to finance their education while nearly as many (18%) borrowed significantly less, between $10,000–$29,999. only 3.8% borrowed less than $10,000, and the remaining borrowed between $30,000 and $99,999. the vast majority (96.2%) of the respondents borrowed $10,000 or more to finance their education, but despite the costs and debt burden, 80% stated that they believed the education that they received was worth the cost. pyne and grodsky (2020) indicate that african americans benefit more financially from graduate education than other racial groups: “compared to african american bachelor’s degree graduates, african american master’s degree graduates earned about 29% more per year on average, about a third more than the relative premium for white master’s degree-holders” (p. 32). this finding suggests that the financial accessibility of pbis contributes towards the economic mobility of african americans. the implication of this finding reinforces the need of institutions of higher education to keep the costs of attending graduate school affordable as both a recruitment and retention strategy. the final enabling benefit of a graduate education at a pbi is the presence of a supportive learning environment and supportive faculty. respondents reported that they benefited from an inclusive and intimate learning environment that supported success. faculty were specifically mentioned as a key factor supporting the alumni perceptions of the educational benefits 108 respondents to complete their degrees. of the respondents, 88% indicated that their program provided an inclusive and respectful environment of their socioeconomic background; nearly as many (87%) reported their graduate program provided an inclusive and respectful environment of their race/ethnicity; and 84% of respondents agreed that their graduate program provided an inclusive and respectful environment of their gender identity or sexual orientation. one respondent commented, “the support, sense of community and belonging in my program contributed to my success.” many of the comments on open-ended questions on the survey indicated that the support of the faculty was a key factor in the positive learning environment. cokley (2002) indicated that for hbcu students, the most important factor for success was the quality of student-faculty interactions. this also confirms the finding of siming et al. (2015) indicating the importance of student and faculty relationships in overall satisfaction. respondents' comments indicated an appreciation of instructors and intimate class sizes. respondents stated: • “great instructors, small class sizes created a great learning environment,” • “i love faculty, environment, class size, and flexibility,” and • “great instructors, small class sizes created a great learning environment.” other responses specifically noted the impact of a caring faculty: • “i love being a graduate student at csu. i had amazing professors who supported my endeavors and i learned so much from them,” and • “my professors really catered to my academic needs.” another respondent perhaps best summed up the experience at a pbi by stating: • “it felt good to be part of an environment where people got second chances and people are rooting for you to succeed.” in their study of regional institutions which include an hbcu, orphan and broom (2021) found that faculty who identified with their institution’s mission were consequently able to see themselves in their students and exhibited a “normative and affective commitment” to the institution's mission and their love for their students (p. 188). the results of this project underscore the importance of the supportive learning environment as a benefit and supports arroyo and gasman’s (2014) supposition that a “supportive environment is theorized to form the foundation of hbcus’ contributions to black student success” (p. 64). defreitas and bravo (2012) also determined that the involvement with faculty was a factor of success for african american students. defreitas and bravo (2012) affirm the link between student academic achievement and mentoring and involvement with faculty. csu faculty create a rich learning environment which nurtures student success through their commitment to csu’s stated mission, their response to the societal need for upward mobility among members of the predominantly black university community, and the deep, affective connections they have with the institution and their students. the confluence of these factors as a component of alumni perceptions of the educational benefits 109 successful teaching and learning can inform broader educational practices at other institutions. when discussing the findings, it is important to acknowledge that the respondents/participants to this study represent a special population—one that is favorably disposed to the university “since their graduation marks them as having been successful” (johnson-bailey et al., 2009, p. 178). hadre and hackett (2015) describe a pattern in graduate students reported satisfaction across their programs where “the highest scores (indicating most positive perceptions) being at entrance to graduate programs, then a drop (of varying degrees) occurring at mid-point, and this being followed by somewhat higher scores at exit (indicating somewhat more positive perceptions) near completion of the degree” (p. 465). the respondents in this study are students who completed their degrees, and a majority of them went on to pursue careers of their choosing. this study was cross-sectional and therefore only captures sentiment in a snapshot-in-time; sentiments regarding graduate education may shift and change over the span of a career. recommendations and implications for practice an essential concern for msis, as well as other institutions of higher education, is ensuring the “success” of its students, which is generally operationalized as retention and graduation. understanding the nature and extent of the self-reported benefits that students of color at the graduate level received from pbis can create spaces for the improvement of instruction and the delivery of academic programs to this group. there are several recommendations that arise from an analysis of these findings of a study of alumni which can improve educational practice; these recommendations are centered on the creation of educational experiences and environments that are conducive to academic success for graduate students of color. 1. create educational experiences and curricula that support personal fulfillment as a component of academic knowledge acquisition. much of the learning in higher education is driven by the creation of learning outcomes related to the mastery of the theory and practice of a field or discipline. these outcomes generally address the attainment of cognitive and dispositional goals relevant to the field; however, they are often mute about whether these goals ultimately result in a sense of personal fulfillment in the student. 2. create educational experiences that support the attainment of enhanced knowledge that is perceived as such by students of color. it is integral that students of color not only receive knowledge of a discipline or a field but also that they perceive themselves as possessing enhanced knowledge. evidence of the attainment of knowledge is expressed through grades; however, attaining a grade in a class is not the same as the subjective experience of feeling knowledgeable about a subject. instruction should result in mastery of content as well as self-confidence in the attainment of that content. alumni perceptions of the educational benefits 110 3. create an educational environment that supports the development of a positive academic self-concept among students of color. students need to see themselves as scholars who can contribute intellectually to their fields within the broader academic enterprise. there are certain graduate university environments and cultures (martin et al., 2015) that notoriously undermine the academic self-concept of students; these cultures include elements of belittlement, punishment and exclusion (goodboy et al., 2015) and exhibit a high degree of instructional dissent which are particularly detrimental to students of color. 4. create an educational environment that supports the development of a positive professional identity as a person of color within a field of study particularly in fields where people of color are severely underrepresented. graduate students of color need to envision themselves as full and equal members of their professions; creating this vision themselves can be challenging when students don’t see diversity in their fields in real life. 5. connect academic learning concretely to students’ future career and work. as this study reveals, many students pursue a graduate degree for the purpose of finding a new job, maintaining job security, or gain advancement in an existing career. these connections are frequently made in professional fields of study; however, in academic fields, explicitly showing connections between academic knowledge and the practical knowledge of the working world is integral to the successful matriculation of students of color. 6. create learning experiences that help students connect their field of study to the improvement of their communities. many fields of study have some espoused commitment to improving the human condition or human society as a whole. learning experiences should be designed to assist students in understanding how their field of study can specifically improve their communities of origin not just society as a whole. 7. make educational programs affordable. the affordability of a graduate educational program is a broader concern than teaching and learning, but ultimately affordability is as much an academic issue as it is an economic one. faculty and administrators need to become very mindful of the cumulative impact of hidden and extra costs for a degree in conjunction with the high costs of a basic education. a common example of hidden costs are programs that require students to complete lengthy or time-consuming unpaid internships or practica, which can pose special burdens on graduate students; time spent in unpaid internships can preclude them from accepting or maintaining their paid employment. 8. provide a supportive learning environment and faculty. finally, college faculty and administrators need to provide an overall supportive learning environment. faculty, in particular, need to connect with students at a personal and human level to create an environment where learning can happen. alumni perceptions of the educational benefits 111 conclusions the results indicate that there are a myriad of mutually reinforcing benefits of attending a pbi for graduate work including the presence of high quality, affordable programs; a supportive learning environment; and personal benefits, such as personal fulfillment, giving back to the community, developing leadership skills, gaining career advancement, and obtaining advanced knowledge in a discipline. the dearth of studies on efficiency and success of graduate education obtained from msis, particularly pbis, leaves many broad areas open for future study. like hbcus, pbis produce a disproportionate number of graduate and undergraduate african american degree holders. studies of pbi alumni, such as this one, that reflect on the educational experiences that contributed to alumni educational success at the graduate level can inform the scholarship on teaching and learning in a powerful way. investigating enabling conditions that enhance the success rate of people of color in their graduate studies at pbis can serve as lessons learned for predominantly white institutions (pwis) in identifying and understanding the perceptions and motivations of minority students to pursue and complete graduate studies and increasing and enhancing opportunities for these groups. as espinosa and mitchell (2020) suggest, the understanding of educating minority students at msis “should be at the center of any agenda to educate a diverse citizenry” (p. 27). these implications are directed not only to universities but also to policy makers attempting to disrupt educational inequalities in higher education. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references anderson, e. f., & hrabowski, f. a. (1977). graduate school success of black students from white colleges and black colleges. the journal of higher education, 48(3), 294–303. arroyo, a. t., & gasman, m. (2014). an hbcu-based educational approach for black college student success: toward a framework with implications for all institutions. american journal of education, 121(1), 57–85. baum, s., & steele, p. (2017). who goes to graduate school and who succeeds? accesslex institute research paper, (17-01). http://dx.doi.org/10.2139/ssrn.2898458 boland, w. c. (2018). the higher education act and minority serving institutions: towards a typology of title iii and v funded programs. education sciences, 8(1), 33. alumni perceptions of the educational benefits 112 boland, w. c., gasman, m., samayoa, a. c., & bennett, d. (2019). the effect of enrolling in minority serving institutions on earnings compared to nonminority serving institutions: a college scorecard analysis. research in higher education, 1(30), 121–150. brand, j. e., & xie, y. (2010). who benefits most from college? evidence for negative selection in heterogeneous economic returns to higher education. american sociological review, 75(2), 273–302. braun, v., & clarke, v. (2013). successful qualitative research: a practical guide for beginners. sage. brick, j. m., & williams, d. (2013). explaining rising nonresponse rates in crosssectional surveys. the annals of the american academy of political and social science, 645(1), 36–59. chicago state university [csu]. (2018). chicago state university office of institutional effectiveness fact book. chicago state university [csu]. (2022). strategic plan (2020–2025). https://www.csu.edu/strategicplanningresources/ clarke, v., & braun, v. (2013). teaching thematic analysis: overcoming challenges and developing strategies for effective learning. the psychologist, 26(2). cokley, k. (2002). the impact of college racial composition on african american students' academic self-concept: a replication and extension. journal of negro education, 71(4), 288–296. colen, c. g., pinchak, n. p., & barnett, k. s. (2021). racial disparities in health among college-educated african americans: can attendance at historically black colleges or universities reduce the risk of metabolic syndrome in midlife? american journal of epidemiology, 190(4), 553–561. https://doi.org/10.1093/aje/kwaa245 conrad, c., & gasman, m. (2015). educating a diverse nation. harvard university press. cossa, j., & barker, l. (2021). motivation to pursue a phd in computing: black students in historically black colleges and universities. journal of interdisciplinary studies in education, 10(1), 139–159. curtin, r., presser, s., & singer, e. (2000). the effects of response rate changes on the index of consumer sentiment. public opinion quarterly, 64(4), 413–428. de brey, c., musu, l., mcfarland, j., wilkinson-flicker, s., diliberti, m., zhang, a., branstetter, c., & wang, x. (2019). status and trends in the education of alumni perceptions of the educational benefits 113 racial and ethnic groups 2018. nces 2019–038. national center for education statistics. https://nces.ed.gov/pubs2019/2019038.pdf defreitas, s. c., & bravo, jr., a. (2012). the influence of involvement with faculty and mentoring on the self-efficacy and academic achievement of african american and latino college students. journal of the scholarship of teaching & learning, 12(4), 1–11. espinosa, l. l., & mitchell, t. (2020). the state of race and ethnicity in higher education. change: the magazine of higher learning, 52(2), 27–31. https://doi.org/10.1080/00091383.2020.1732769 espinosa, l. l., turk, j. m., & taylor, m. (2017). pulling back the curtain: enrollment and outcomes at minority serving institutions. american council on education. espinosa, l. l., turk. j. m., & taylor, m. (2018). minority serving institutions as engines of upward mobility. american council on education. fan, w., & yan, z. (2010). factors affecting response rates of the web survey: a systematic review. computers in human behavior, 26(2), 132–139. flores, s. m., & park, t. j. (2013). race, ethnicity, and college success: examining the continued significance of the minority-serving institution. educational researcher, 42(3), 115–128. fosnacht, k., sarraf, s., howe, e., & peck, l. k. (2017). how important are high response rates for college surveys? the review of higher education, 40(2), 245–265. fountaine, t. p. (2012). the impact of faculty-student interaction on black doctoral students attending historically black institutions. the journal of negro education, 81(2), 136–147. galotti, k. m., clare, l. r., mcmanus, c., & nixon, a. (2016). the academic experiences survey (aes): measuring perceptions of academic climate in liberal arts institutions. journal of the scholarship of teaching and learning, 16(5), 32–48. gasman, m., samayoa, a. c., & nettles, m. (2017). investing in student success: examining the return on investment for minority-serving institutions. ets research report series, 2017(1), 1–66. goodboy, a., martin, m., & johnson, z. (2015). the relationships between workplace bullying by graduate faculty with graduate students’ burnout and organizational citizenship behaviors. communication research reports, 32(3), 272–280. https://doi.org/10.1080/08824096.2015.1052904 alumni perceptions of the educational benefits 114 groves, r. m. (2006). nonresponse rates and nonresponse bias in household surveys. public opinion quarterly, 70(5), 646–675. hall, b., & closson, r. b. (2005). when the majority is the minority: white graduate students' social adjustment at a historically black university. journal of college student development, 46(1), 28–42. hardy, p. m., kaganda, e. j., & aruguete, m. s. (2019). below the surface: hbcu performance, social mobility, and college ranking. journal of black studies, 50(5), 468–483. https://doi.org/10.1177/0021934719847910 hardre, p. l., & hackett, s. m. (2015). understanding the graduate college experience: perceptual differences by degree type, point-in-program and disciplinary subgroups. learning environments research, 18(3), 453–468. harmon, n. (2012). the role of minority-serving institutions in national college completion goals. institute for higher education policy. hegji, a. (2017). programs for minority-serving institutions under the higher education act. crs report r43237, version 15. updated. congressional research service. hubbard, s. m., & stage, f. k. (2009). attitudes, perceptions, and preferences of faculty at hispanic serving and predominantly black institutions. the journal of higher education, 80(3), 270–289. hutchings, p. (2000). approaching the scholarship of teaching and learning. in hutchings, p. (ed.), opening lines: approaches to the scholarship of teaching and learning (pp. 1–10). the carnegie foundation for the advancement of teaching. jett, c. c. (2011). “i once was lost, but now am found”: the mathematics journey of an african american male mathematics doctoral student. journal of black studies, 42(7), 1125–1147. johnson, r. b., onwuegbuzie, a. j., & turner, l. a. (2007). toward a definition of mixed methods research. journal of mixed methods research, 1(2), 112– 133. johnson-bailey, j., valentine, t., cervero, r. m., & bowles, t. a. (2009). rooted in the soil: the social experiences of black graduate students at a southern research university. the journal of higher education, 80(2), 178–203. jones, b. (2019). predominantly black institutions: pathways to black student educational attainment. msis unplugged: the penn center for minority serving institutions. https://cmsi.gse.rutgers.edu/sites/default/files/pbis.pdf alumni perceptions of the educational benefits 115 keeter, s., miller, c., kohut, a., groves, r. m., & presser, s. (2000). consequences of reducing nonresponse in a national telephone survey. public opinion quarterly, 64(2), 125–148. king, j. e., council, t. m., fournillier, j. b., richardson, v., akua, c., mcclendon, n., neely, a. n., mason chisholm, g., russell, t. s., vieira da silva santos, f., & streeter, m. (2019). pedagogy for partisanship: research training for black graduate students in the black intellectual tradition. international journal of qualitative studies in education, 32(2), 188–209. https://doi.org/10.1080/09518398.2018.1548040 lambert, a. d., & miller, a. l. (2014). lower response rates on alumni surveys might not mean lower response representativeness. educational research quarterly, 37(3), 40-53. ma, j., pender, m., & welch, m. (2016). education pays 2016: the benefits of higher education for individuals and society. trends in higher education series. college board. martin, m. m., goodboy, a. k., & johnson, z. d. (2015). when professors bully graduate students: effects on student interest, instructional dissent, and intentions to leave graduate education. communication education, 64(4), 438–454. https://doi.org/10.1080/03634523.2015.1041995 massey, d. s., & tourangeau, r. (2013). where do we go from here? nonresponse and social measurement. the annals of the american academy of political and social science, 645(1), 222–236. miller, a. n., & orsillo, s. m. (2020). values, acceptance, and belongingness in graduate school: perspectives from underrepresented minority students. journal of contextual behavioral science, 15, 197–206. mushonga, d. r., & henneberger, a. k. (2020). protective factors associated with positive mental health in traditional and nontraditional black students. american journal of orthopsychiatry, 90(1), 147–160. https://doi.org/10.1037/ort0000409 national research council. (2013). monitoring progress toward successful k–12 stem education: a nation advancing? national academies press. national science foundation. (2015). women, minorities and persons with disabilities in science and engineering: 2015. national center for science and engineering statistics. nelson laird, t. f., bridges, b. k., morelon-quainoo, c. l., williams, j. m., & holmes, m. s. (2007). african american and hispanic student engagement at minority serving and predominantly white institutions. journal of college student development, 48(1), 39–56. doi:10.1353/csd.2007.0005 alumni perceptions of the educational benefits 116 orphan, c. m., & broom, s. (2021). life at the “people’s universities”: organizational identification and commitment among regional comprehensive university faculty members in the usa. higher education, 82(1), 181–201. https://doi.org/10.1007/s10734-020-00629-9 palmer, r. t., hilton, a. a., & fountaine, t. p. (eds.). (2012). black graduate education at historically black colleges and universities. routledge. palmer, r. t., walker, l. j., goings, r. b., troy, c., gipson, c. t., & commodore, f. (eds.). (2016). graduate education at historically black colleges and universities (hbcus): a student perspective. routledge. parker, k., horowitz, j. m., brown, a., fry, r., cohn, d. v., & igielnik, r. (2019, december 31). demographic and economic trends in urban, suburban and rural communities. https://www.pewsocialtrends.org/2018/05/22/demographic-and-economictrends-in-urban-suburban-and-rural-communities/ perna, l. w. (2005). the benefits of higher education: sex, racial/ethnic, and socioeconomic differences. the review of higher education, 29(1), 23–52. peytchev, a. (2013). consequences of survey nonresponse. the annals of the american academy of political and social science, 645(1), 88–111. pyne, j., & grodsky, e. (2020). inequality and opportunity in a perfect storm of graduate student debt. sociology of education, 93(1), 20–39. https://doi.org/10.1177/0038040719876245 scott, l., & sharp, l. a. (2019). black males who hold advanced degrees: critical factors that preclude and promote success. journal of negro education, 88(1), 44–61. https://doi.org/10.7709/jnegroeducation.88.1.0044 seifert, t. a., drummond, j., & pascarella, e. t. (2006). african-american students' experiences of good practices: a comparison of institutional type. journal of college student development, 47(2), 185–205. doi:10.1353/csd.2006.0023 shappie, a. t., & debb, s. m. (2019). african american student achievement and the historically black university: the role of student engagement. current psychology, 38(6), 1649–1661. siming, l., gao, j., xu, d., & shaf, k. (2015). factors leading to students’ satisfaction in the higher learning institutions. journal of education and practice, 6(31), 114–118. strayhorn, t. l. (2016). a return on investment analysis for black graduates of historically black colleges and universities: insights from three studies. center for minority serving institutions, university of pennsylvania. alumni perceptions of the educational benefits 117 terry, g., hayfield, n., clarke, v., & braun, v. (2017). thematic analysis. the sage handbook of qualitative research in psychology, 2, 17–37. trostel, p. (2015). it’s not just the money: the benefits of college education to individuals and to society. government & civic life, 4. https://digitalcommons.library.umaine.edu/mcspc_gov_civic/4 u.s. department of education [usde]. (2014). predominantly black institutions program—formula grants. https://www2.ed.gov/programs/pbihea/eligibility.html watkins, d., & gioia, d. (2015). mixed methods research. oxford university press. williams, k. l., russell, a., & summerville, k. (2021). centering blackness: an examination of culturally-affirming pedagogy and practices enacted by hbcu administrators and faculty members. innovative higher education, 46, 733– 757. https://doi.org/10.1007/s10755-021-09562-w alumni perceptions of the educational benefits 118 appendix table 1 participants’ demographics gender percentage (%) males 30 females prefer not to answer 67 3 race white 17 african american 64 hispanic asian 4.6 9 others 5.4 first generation graduate student no 48 yes 52 degrees obtained* master’s 79 doctorate graduate certificate 17 4 returning students no 69 yes 31 note. percentage does not add up to 100% because some answers are “select multiple.” alumni perceptions of the educational benefits 119 table 2 personal motivation and benefits factors in choosing field of graduate studies* percentage (%) required for career path 48 potential to increase earnings ability to contribute to the community via the chosen field change careers to best fit personal aspirations own aptitude in the field ability to balance work & family increasing representation of gender/racial/ethnic group in a designated profession 41 33 30 25 16 16 main reasons for enrollment* affordability 72 status as a minority institution 17 access to professional employment opportunities highly ranked program 17 17 benefits after obtaining a graduate degree* personal fulfillment 64 advanced knowledge in the discipline getting a new job improved leadership skills salary increase more credibility changed career path enhanced prestige increased job security work promotion 54 54 42.5 38.7 36.8 30.3 29 24.5 13.4 note. percentage does not add up to 100% because some answers are “select multiple.” alumni perceptions of the educational benefits 120 table 3 finances how were graduate studies financed* percentage (%) loans (federal or private) 75.9 personal/family earnings or savings scholarship, grant, assistantship, or fellowship employer reimbursement/assistance 36 29 13.4 amount borrowed $100,000 or more 19 $70,000-$99,999 6.7 $50,000-$69,999 $30,000-$49,999 $10,000-$29,999 less than $10,000 10.3 14 18 3.8 amount still owed all 43 some none 25 8.4 note. percentage does not add up to 100% because some answers are “select multiple.” alumni perceptions of the educational benefits 121 table 4 inclusive environment inclusive and respectful environment of socioeconomic background percentage (%) strongly agree or agree 88 neither agree nor disagree strongly disagree or disagree 9 3 inclusive and respectful environment of race/ethnicity strongly agree or agree 87 neither agree nor disagree strongly disagree or disagree 11 2 inclusive and respectful environment of gender identity or sexual orientation strongly agree or agree 84 neither agree nor disagree strongly disagree or disagree 14 2 inclusive and respectful environment of religious/spiritual beliefs strongly agree or agree 73 neither agree nor disagree strongly disagree or disagree 24 3 inclusive and respectful environment of political orientation strongly agree or agree 68 neither agree nor disagree strongly disagree or disagree 29 3 alumni perceptions of the educational benefits 122 table 5 satisfaction choice of major percentage (%) extremely satisfied or satisfied 88 neither satisfied nor dissatisfied extremely dissatisfied or dissatisfied 9 3 quality of the curriculum extremely satisfied or satisfied 88 neither satisfied nor dissatisfied extremely dissatisfied or dissatisfied 7 5 quality of graduate education extremely satisfied or satisfied 88.5 neither satisfied nor dissatisfied extremely dissatisfied or dissatisfied 5.75 5.75 cost worthy no 20 yes 80 microsoft word rovito final version.docx journal of effective teaching in higher education, vol. 2, no. 2 understanding the effectiveness of a peer coach model in research-intensive undergraduate courses michael j. rovito, university of central florida, michael.rovito@ucf.edu amanda koontz, university of central florida mary tripp, university of central florida kim schneider, university of central florida linda walters, university of central florida martha garcia, university of central florida abstract: the impact of peer coaches embedded in classrooms are outlined in the literature. there is, however, a lack of evidence on when their services should be introduced into the curriculum to have the most impact possible. the goals of this exploratory research project, therefore, were to aim to provide baseline evidence on the most effective and efficient application of ‘research coaches’. a total of 129 undergraduate students took part in a cross-sectional design by completing a series of original surveys on the research process and qualities about their research identity. our research suggests that peer coaches can most benefit those with the least previous research experiences as related to perceived increases in understanding of fields, research skills, previous knowledge of the subject of study and confidence in research abilities. such quantitative findings are reinforced through our qualitative findings, suggesting that overall, peer coaches and faculty can be perceived as complimentary support systems for effective research-focused course implementation. future implications of the use of ‘research coaches’ in the classroom are discussed. keywords: undergraduate research, peer coaches, curriculum, mentorship while the benefits of undergraduate research are clearly supported in the literature, the difficulty in providing research opportunities is often related to a lack of support for faculty (auchincloss et al., 2014; gregerman et al., 1998; morales et al., 2017). one way this has been addressed is through the use of peer coaches, sometimes called peer mentors (budge, 2006; linn et al., 2015). schneider et al. (2015) suggest that undergraduate peer coaching, when integrated into learning communities, increases both gpas and retention from the first into the second years in comparison to control groups without these experiences. kim and hannafin (2010) indicate another successful method to support faculty in fostering teaching excellence includes pedagogical scaffolding of undergraduate research throughout a specific curriculum [a specific illustration is the model developed within the concept of higher degree by research (hdr)]. other research on hdr has supported the supervision for one-on-one and the peer-to-peer learning practice lead undergraduates towards a more efficient preparation for postgraduate studies and job markets (e.g., batty & sinclar, 2014). additional dialogue about peer-to-peer mentoring benefits focuses on the importance of the early practice of students’ collaboration through co-authorship, emphasizing how students need to take on particular tasks of the “novice effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 43 researcher” to start adopting the associated identity (garbati & samuels, 2013: 367). this leads us to also reconsider the need for peer-to-peer collaboration from the earliest stages at the undergraduate levels. for example, in fields dedicated to the teaching and learning of second or multiple languages, peer-to-peer activities have constituted the grassroots of undergraduate language acquisition due to the need for constant interaction and communication among peers to reach the expected standards of proficiency and fluency (kissau & king, 2015). pedagogical studies in teaching english as a second language have established, as well, the strong pillar of peer mentoring strategies (nguyen, thi mai, & peter hudson, 2012). quality of instruction through a more structured paradigm of mentoring has received special attention because of the necessary professionalization, especially critical thinking and communication skills, that the current job market demands and the need for comprehensive preparation for collegiate students to succeed in their chosen career paths (reid, 2008). in combinations with diversifying student populations, the addition of undergraduate collaborative research experiences in higher education as early as possible may be particularly fruitful (nguyen, 2017). in considering the effectiveness of peer coaches, however, questions remain as to not just if an impact will occur as a result of the implementation of a peer coach model, but when should their services be introduced into the curriculum to have the most impact possible. as sandoval (2017) indicates that the exposure to research experiences earlier in a given curriculum may augment students’ epistemological beliefs, there is cause for concern that the effectiveness of a peer coach upon the highly-experienced students may be limited. therefore, a key question arises with the implementation of a peer coach model within undergraduate courses: are peer coaches more or less effective for students who have had previous exposure to the research process compared to those who have limited experience? in other words, would it be more effective to embed peer coaches into courses with undergraduate students who have less experience with the research process? the goals of this exploratory research project aim to provide baseline evidence on the most effective and efficient application of research coaches. review of the literature the following section reviews literature on prior findings on the implementation of various versions of peer mentoring/coaching models, offering context for the peer coaching model implemented for the purposes of this research study. albeit not exhaustive, the summarized literature supports the integration of inquiry-based learning, authentic research, and high-impact opportunities throughout undergraduate college programs (cahill et al., 2008; guy & malenczyk, 2008; ryan et al., 2008). extant literature indicates that the integration of research into the curriculum increases enhanced engagement with course content, higher quality outcomes, and skill development applicable for professional and/or graduate academic careers (elmes, 2002). the literature also suggests that embedded peer mentoring for traditional undergraduate research may encourage higher retention rates, better effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 44 professionalization, and enhanced engagement with course content (see elmes, 2002; reid, 2008; schneider et al., 2015). these models, which have proven successful across curriculums in higher education institutions, can efficiently assist in contributing to the overall preparation of students during their college years and beyond graduation. other research indicates that embedding authentic research experiences, specifically in early stages of learning and throughout the entire student’s tenure at the university, instead of just one or two semesters during their junior or senior years, have produced higher rates in the students’ development of confidence and overall success (schneider et al., 2015). however, in order to continually assess the efficiency of models and ascertain areas that may require some improvement, metrics need to be integrated into the development of the curriculum within and beyond the home institutions (watson, siska, & wolfel, 2013). consequently, identifying and building a methodology of evaluation may contribute to a higher level of adoption of the coaching and mentoring norms that can help direct the focus of coaching. norms facilitate teachers and learners grasp of the practicality and dexterity of applications and approaches (kiefer & neufeld, 2002); as such, more research is needed that contextualizes undergraduate students’ research experiences, in part to determine the norms of their experiences and thus to help direct the focus of peer coaching to improve effectiveness. one gap in our understanding of peer coach success is understanding the background of students who would most benefit from taking a course embedded with the peer coach model, which we entitle an embedded peer coach course (epcc). we, therefore, compared number of prior research experiences with perceived gains during the semester in courses involving embedded course research facilitated by a peer coach model. this information contributes to enabling decision-makers to focus the efforts of peer coaches within classes where impact can be maximized. implementation and methods design of the peer coaches model the grounding for a systematic implementation of different styles of classroom strategies has generated new approaches in pedagogical applications (song, 2016). in the process, specific rubrics for successful mentoring practices have been instrumental in the advance of enacting coaching and peer-to-peer collaboration, not only at the professional level in academia, but through its application in research-focused college experiences (nguyen & mai, 2017). therefore, an adapted version of a model developed at the university of north carolina-chapel hill (uncch) that focused on embedding graduate student research consultants into undergraduate courses (pukkila et al., 2013) served as the framework for this study. four faculty members from a variety of disciplines at a large university in the southeastern u.s. participated in the application of the epccs. research-focused effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 45 courses were operationally defined as undergraduate courses that include a research component where students construct and complete their own semesterlong research project. the research component was required to include, but was not limited to, the following student activities: 1) creating a research question based on literature analysis, 2) developing research hypotheses, 3) developing a priori methodological plans for hypothesis testing, 4) utilizing appropriate data collection and analysis methods for their field (or subject), and/or 5) communicating their project to their peers at annual campus undergraduate research showcase. the epccs were designed for peer coaches to assist the enrolled students with embedded undergraduate research projects. each faculty member was responsible for identifying, recruiting, training, and embedding their peer coaches into his or her epcc. criteria for peer coaches were flexibly defined as an advanced undergraduate or graduate student who possessed the skills necessary, as purported by the course instructor, to serve in the capacity of a coach. each instructor was responsible for the preparation of the peer coach according to the demands of the course. such assistance consisted of providing consult on core concepts of discipline-specific research, assistance in research project facilitation, guide in adjacent areas of knowledge required for genuine research experiences (i.e. writing and/or statistics counsel), or some combination of these tasks in the epcc. to study the effectiveness of the peer coach model in these epccs across all levels of student research, a pretest-posttest study design was employed to understand undergraduate students’ perceptions of their: 1) understanding of the field, 2) research skills, 3) ability to conduct research. our research aimed, therefore, to understand how peer coaches could most effectively support the implementation of research-focused undergraduate courses. this research study was approved by the university’s institutional review board. sampling methodology all students enrolled in the epccs were asked to participate in the study. following irb guidelines, they were enrolled in the study if they were age 18 or older and provided appropriate consent. a total of 12 coaches were assigned to the courses: 3 coaches to modern languages, 2 coaches to composition, 4 coaches to health sciences, and 3 coaches to marine biology. the model had 1 coach per 15-20 students in the course to ensure that students had access to the resource and support regardless of the class size. peer coaches were considered volunteers in following institutional policies but were provided a financial award to support the following classroom research related activities: reimbursement of travel to an academic conference related to their research area, funding for research supplies, and/or reimbursement of travel costs to conduct research or creative activities. in the program at unc-ch, graduate research consultants received a $1000 stipend to support research-exposure course effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 46 for a total of 30 hours over the course of the semester after attending a short training. the university where this current project was completed developed a version of this program that allowed both undergraduate and graduate students to serve as peer coaches in research-intensive courses for up to 30 hours of support during a single semester. peer coaches received 6 hours of training by the team of faculty who developed the epcc model prior to the start of the semester and a $500 stipend. course enrollees were given participation credit for taking the survey, in accordance with the instructor guidelines. students choosing to opt out of the study had the option to choose an open-ended questionnaire for participation credit during the same time students were completing the research surveys to preserve ethical principles. a research pi or co-pi, that was not the instructor of record for the respective course, distributed surveys and questionnaires. they then also gave the instructor of record a list of all students present when the surveys and questionnaires were distributed for participation credit but were not informed as to their participation choice. this was done out of respect for issues of coercion and confidentiality. instrumentation original preand post-surveys were created via adapting an existing science identity tool (anthony, walters, & mcgrady, 2017) to measure similar concepts across a wider range of disciplines as it pertains to developing a research identity for all disciplines. the survey gathered information on students’ research backgrounds (i.e., previous student experience), including if they had and, if so, the number of times they had: conducted a research project, participated universitywide research-based programming, attended a research conference, participated in service-learning, and/or had been a formal tutor/peer mentor. the questionnaire assessed attitudes and perceptions among participants pertaining to: 1) understanding of research in the field, 2) perceived research skills, 3) and perceived confidence in ability to conduct research. these three categories were assessed via a series of statements that the participant was asked to agree or disagree with on a scale of 0-10 where 0 = n/a, 1 meant strongly disagree, and 10 meant strongly agree. a series of 15 items made up the perceived research skills and perceived confidence in ability to conduct research composite variables, while 9 items made up the understanding of research in the field composite measure. results the final sample size consisted of 129 students, as 129 of the total 149 eligible and consenting students completed both the preand post-survey (86.57% response rate; see table 1 for aggregate participant demographic information). the four courses had the following number of participants complete both preand post-test assessments: health sciences, 45 out of 55, modern languages, 9 out of 11, composition, 14 out of 22, and marine biology, 56 out of 60. effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 47 table 1 demographic information of students who participated in the preand postquestionnaire. variable category percentage (%) age 18-20 26.8 21-23 58.5 24-26 9.8 27+ 4.9 race/ethnicity white/caucasian 65.9 black/african-american 4.8 asian/pacific islander 5.7 native american/ alaskan native 1.6 multiracial/mixed 3.2 did not indicate 18.8 hispanic yes 19.4 sex male 32.5 female 67.5 to explore the effect of embedded peer coaches in an epcc, participants were categorized into groups according to their self-reported previous experiences with research to determine if this impacted results. five items operationalized previous student experience, including if, at the time of entering the course, they had any research background previously in their undergraduate career, as mentioned previously. participant responses indicating ‘yes’ to any of the 5 items were summed up to represent a raw experience score. participants were further categorized into: “no previous”, “one previous”, “two previous”, and “three to five previous” experience (see table 2). table 2 percentage of previous experience categories variable category percentage (%) previous experience no previous 33.3 one previous 34.1 two previous 16.3 three to five previous 16.3 understanding of research in the field a total of 9 items that assessed participant familiarity and knowledge of research topics and ideas operationalized the understanding of research in the field construct. participant responses on both the preand post-survey were summed and then divided by the total number of items to calculate an average score. a wilcoxon paired sampled means test indicated that only the ‘one previous effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 48 experience’ group exhibited a statistically significant increase in scores from preto post-test (table 3). table 3 understanding of research in the field pre-post analysis variable previous experience category baseline 𝒙" postest 𝒙" wilcoxon p-value understanding research in the field none 7.29 7.46 .536 one 7.29 8.01 .006* two 7.84 7.96 .372 three to five 8.38 7.77 .436 * denotes statistical significance @ p≤ 0.05 an independent samples kruskal-wallis test (table 4) indicated baseline and posttest significant differences in mean scores present across ‘previous experience’ groups. dunn’s test post-hoc analyses indicated that, at the start of the semester, those students who had no or one previous experience had statistically significant mean score differences as compared to those students who had three to five previous experiences. post-test analyses indicted that although a significant difference existed across all groups, at the end of the semester only the ‘no previous experience’ group had a significant difference compared to those with three to five previous experiences. this reveals that the ‘one previous experience’ and ‘two previous experience’ groups shifted enough to not significantly differ from the ‘three-to-five experience’ group. table 4 understanding of research in the field between-group analysis variable previous experience category baseline 𝒙" postest 𝒙" understanding research in the field between experience groups all groups 0.13* .043* between individual experience groups (dunn’s post-hoc) no to one experience .708 .169 no to two experiences .292 .529 no to three to five experiences .006* .005* one to two experiences .174 .645 one to three to five experiences .002* .090 two to three to five experiences .143 .067 * denotes statistical significance @ p≤ 0.05 effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 49 perceived research skills a total of 15 items assessing perceived skills needed to conduct research were used to operationalize perceived research skills. participant responses on both the pre and post-survey were summed and then divided by the total number of items to calculate an average score. a wilcoxon paired sampled means test indicated that all groups, except the three to five previous experiences group, exhibited statistically significant increases in their scores from preto post-test (table 5). table 5 perceived research skills pre-post analysis variable previous experience category baseline 𝒙" postest 𝒙" wilcoxon p-value previous research skills no 7.29 7.46 <.001* one 7.29 8.01 <.001* two 7.84 7.96 .030* three to five 8.38 7.77 .895 * denotes statistical significance @ p≤ 0.05 an independent samples kruskal-wallis test (table 6) results indicated only baseline significant differences in mean scores present across ‘previous experience’ groups. dunn’s test post-hoc analyses indicated that, at baseline, those students who had no or only one previous experience had statistically significant mean score differences as compared to those students who had two previous experiences and three to five previous experiences. there were no statistically significant different scores at post-test across the previous experience groups. table 6 perceived research skills between-group analysis variable previous experience category baseline 𝒙" postest 𝒙" previous research skills between experience groups all groups <.001* .157 between individual experience groups (dunn’s post-hoc) no to one experience .189 no to two experiences .001* no to three to five experiences <.001* one to two experiences .027* one to three to five experiences .004* two to three to five experiences .571 * denotes statistical significance @ p≤ 0.05 effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 50 perceived confidence in ability to conduct research a total of 15 items that assessed participant perceived ability in the field operationalized the perceived research ability construct. this construct helps to differentiate between their perceived research skills and their confidence in actually applying those skills. participant responses on both the preand post-survey were summed and then divided by the total number of items to calculate an average score. a wilcoxon paired sampled means test indicated that the no previous and one previous experience groups exhibited statistically significant increases in their scores from preto post-test (table 7). with two research experiences, the results were marginally significant (p = 0.77). table 7 perceived research ability pre-post analysis variable previous experience category baseline 𝒙" postest 𝒙" wilcoxon p-value previous research ability no 4.87 6.89 .001* one 5.43 7.30 <.001* two 6.86 7.34 .077 three to five 7.06 7.41 .801 * denotes statistical significance @ p≤ 0.05 an independent samples kruskal-wallis test (table 8) results indicated only baseline significant differences in mean scores present across ‘previous experience’ groups. dunn’s test post-hoc analyses indicated that, at baseline, those students who had no previous experience had statistically significant mean score differences as compared to those students who had two to five previous experiences. there were no statistically significant different scores at post-test across the previous experience groups. table 8 perceived research ability between-group analysis variable previous experience category baseline 𝒙" postest 𝒙" previous research ability between experience groups all groups <.006* .055 between individual experience groups (dunn’s post-hoc) no to one experience .235 no to two experiences .005* no to three to five experiences .004* one to two experiences .067 one to three to five experiences .053 effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 51 variable previous experience category baseline 𝒙" postest 𝒙" two to three to five experiences .933 * denotes statistical significance @ p≤ 0.05 semesters attended students were also categorized by how many semesters they attended university: 0-3 semesters, >3 to <6 semesters, >6 to <8 semesters, and >8 semesters. at baseline, there were statistically significant differences across groups regarding perceived research skills and ability. however, no post-test significant differences were reported. wilcoxon paired sampled means test indicated that there were statistically significant differences preto post-test in the 0-3 and the >3 to <6 semesters attended group for perceived research skills and perceived research ability. the >6 to <8 group demonstrated significant preand post-test scores in understanding of research in the field and perceived research skills. finally, the >8 semesters attended group had significant preto post-test scores for all three categories (i.e. understanding of research in the field, perceived research skills, and perceived research ability). psychometric analysis of instrument a principal components confirmatory factor analysis was run on the three a prioridefined sections: understanding of research in the field, perceived research skills, and perceived research ability. three factors were produced that mirrored the original survey composite variables. low communality on extracted factors present and cronbach’s scores of α = .976, α = .944, and α = .822, respectively, offered construct validity and internal consistency to the data. content and face validity was confirmed through peer-conferencing techniques on survey creation. discussion and implications in examining how to effectively implement peer coaches, the findings of this research suggest that although multiple groups benefit, those with the fewest research experiences revealed the most significant changes in areas related to research identities. the findings suggest that students in epccs with the fewest previous research experiences benefit the most from peer coaches in undergraduate courses, as defined by shifts in perceived understanding of research in the field, research skills, and confidence in ability to conduct research. these findings support prior research that suggests that students and peer mentors benefit from integrating inquiry-based learning, authentic research, and high impact opportunities throughout the undergraduate college programs (cahill et al., 2008; guy & malenczyk, 2008; reid, 2008; ryan et al., 2008). in addition, our research offers viable recommendations for how peer coaches can be effectively implemented across disciplines, highlighting the commonalities for how to implement such a peer coach model. effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 52 while we know that genuine research experiences produce higher rates in the students’ development of confidence and overall success (schneider et al., 2015), this research expands our understanding to suggest that delimiting resources for coaches may be targeted for larger, lower level undergraduate courses because of the increased likelihood to have a significant impact. it is also logical that the use of peer coaches help faculty integrate these authentic research experiences in college classrooms. our research brings these two pedagogical practices together, reinforcing anthony, walters, and mcgrady’s (2017) findings, suggesting benefits from an increase in their perceptions of themselves as more confident, motivated, engaged, knowledgeable, and skillful at research as the result of interacting with peer coaches. our study demonstrates that increasing the support for a course design that involves peer coaches, especially in the early years of an undergraduate degree, can potentially lead to better learning outcomes for these students, which can eventually lead to increased “career-readiness” among the student body. to support the findings from this study, further research into the effects of peer coaches in early undergraduate courses will still be needed, recommended, and beneficial. peer coaches may be particularly effective because of: 1) their perceived help with the research process, 2) their accessibility to allow those with less experiences more availability to ask core questions on the research process that they may otherwise be embarrassed to ask the professor, and 3) their allowing for a leveling on topics that those with greater experiences already have had exposure to in their prior research. these points additionally give the professor more time to answer questions on areas of disciplinary expertise, which may be more helpful to those with more prior experiences. some limitations of our study include the lack of a control group to truly highlight the possible causal pathway between the implementation of this model and benefits to the student body. therefore, future studies could more specifically focus on potential applications and causal mechanisms driving the positives of this peer coach model. for example, we know women and minority students are particularly receptive to mentoring; to adopt an intersectional approach to researching the interconnections across demographics (crisp & cruz, 2007; gibbons & woodside 2014). further examination of how demographic and cultural background characteristics potentially intersect (e.g., gender, race/ethnicity, ses, firstgeneration, etc.) and influence the effectiveness of epccs would be especially beneficial. epccs may be an additional bridge for transfer students adjusting to a larger scale of educational systems; as based in the particularly large university at which this study took place, there is a distinct possibility that the peer coaching model is a viable option for helping diverse student populations navigate transitions into and through classes where they may otherwise not feel integrated into the classroom. some other limitations of the study include the regression artifact threat. as regression to the mean stems from in large part a nonprobability sample, there is little that could be done to correct for this possible source of systematic error. students cannot be randomly selected from the university population and then effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 53 allocated to course sections, let alone across different discipline’s courses. future applications can attempt to remedy this issue but may prove methodologically difficult. also, the amount of time and the exact use of the coaches will differ across each course that embeds them into the course. this does not provide a uniformity of use and then may detract from the general effectiveness of the coaches per course. future studies should look at the effectiveness of coach application to determine best practices. the combination of peer coaches and research experiences seems to offer an effective response to the economic constraints challenging the diversification of the academic community who wish to maintain a sustained high level of integrity, interaction, and participation among undergraduate students in early years of college experience and their counterpart faculty members. in particular, it appears that peer coaches especially help students in developing research skills, so that there are possibilities for effective usage across student populations. such distribution can offer insight into, for example, whether it can be more effective to have peer coaches in early programming versus capstone, since students may have increased experience at that given point. however, as our research still shows significant shifts across the three assessment categories for students with more than 8 semesters, it may be that coaches serve a different purpose in helping students who have attended school longer develop research skills specifically, since they already have started to develop disciplinary expertise. additional research can also explore whether the leveling across research experiences is due to an increased practical understanding of their knowledge in skills; this, in turn, may substantially increase their retention in their fields because they are able to match their abilities with practical application and needs for further skill development. to summarize, our research suggests that peer coaches can most benefit those with the least previous research experiences as related to perceived increases in understanding of fields, research skills, previous knowledge of the subject of study and confidence in research abilities. such quantitative findings are reinforced through our qualitative findings, suggesting that overall, peer coaches and faculty can be perceived as complimentary support systems for effective research-focused course implementation. in effectively implementing this model, coaches can be understood as accessible and approachable peer mentors that offer the first line of support for research projects, while faculty members can have the additional time to go in-depth with students on expertise, specialized knowledge and content. acknowledgments this research was funded by our university's offices of undergraduate research and quality enhancement plan during the academic year of 2014-2018. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 54 references auchincloss, l. c.., laursen, s. l., branchaw, j.l., eagan, k., graham, m., hanauer, d.i., . . . dolan, e.l. (2014). assessment of course-based undergraduate research experiences: a meeting report. cbe-life sciences education 13(1): 29-40. anthony, a. k., walters, l. j., & mcgrady, p. (2017). creating connections between authentic research and the development of science identities in undergraduate marine biology experiences. florida scientist 80(2-3): 61-76. ariza, a., & viáfara gonzález, j.j. (2009). interweaving autonomous learning and peer-tutoring in coaching efl student-teachers/la conexión entre aprendizaje autónomo y la tutoría entre pares como apoyo a estudiantes de inglés como lengua extranjera. profile 11(2); 85-104. batty, c. & sinclair, j. (2014). peer-to-peer learning in the higher degree by research context: a creative writing case study. new writing 11(3); 335-46. budge, s. (2006). peer mentoring in postsecondary education: implications for research and practice. journal of college reading and learning 37(1): 71-85. cahill, l., miller-cochran s., pantoja, v., & rodrigo, r.l. (2008). graduate student writing groups as peer mentoring communities. in m. f. eble & l. l. gaillet, (eds.) stories of mentoring: theory and praxis (pp.153-58) anderson, sc: parlor press crisp, g., & cruz, i. (2009). mentoring college students: a critical review of the literature between 1990 and 2007. research in higher education 50(6): 525545. elmes, d.g. (2002). lab courses for undergraduates: the benefits are clear. aps bulletin 15(7): 13-40. garbati, j. & samuels, b. (2013). publishing in educational research journals: are graduate students participating? journal of scholarly publishing 44(4): 35572. gregerman, s. r., lerner, j. s., hippel, w., jonides, j. & nagda, b. a. (1998). undergraduate student-faculty research partnerships affect student retention. the review of higher education 22(1): 55-72. gibbons, m. m., & woodside, m. (2014). addressing the needs of first-generation college students: lessons learned from adults from low-education families. journal of college counseling 17(1): 21-36. guy, a. e., jr. & malenczyk, r. (2008). a new paradigm for wpa mentoring? the case of new york university's expository writing program. in m. f. eble & l. l. gaillet, (eds.) stories of mentoring: theory and praxis (pp. 235-47) anderson, sc: parlor press. hamilton, d. (1977). writing coach. college composition and communication 28(2): 154-58. effectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 55 healy m. & jenkins a. (2009). developing undergraduate research and inquiry. new york, ny: the higher education academy. kiefer, k. & neufeld, j. (2002). making the most of response: reconciling coaching and evaluating roles for teachers across the curriculum. academic writing: interdisciplinary perspectives on communication across the curriculum. https://wac.colostate.edu/aw/articles/kiefer_neufeld_2002.pdf. kim, m. & hannfin m. (2011). scaffolding problem solving in technology-enhanced learning environments (teles): bridging research and theory with practice. computers and education 56: 403-417. kissau, s. p. & king, e. t. (2015). peer mentoring second language teachers: a mutually beneficial experience? foreign language annals 48(1): 143-60. linn, m. c., palmer, e., baranger, a., gerard, e. & stone, e. (2015). undergraduate research experiences: impacts and opportunities. science 347(6222): doi:10.1126/science.1261757. morales, d. x., grineski, s. e., & collins, t. w. (2017). faculty motivation to mentor students through undergraduate research programs: a study of enabling and constraining factors. research in higher education 58(5): 520544. nguyen, h. t. m. & hudson, p. (2012). peer group mentoring: preservice efl teachers' collaborations for enhancing practices. in a. honigsfeld, & m. g. dove (eds.), coteaching and other collaborative practices in the efl/esl classroom: rationale, research, reflections, and recommendations (pp. 231-40). charlotte, nc: information age publishing. nguyen, h. t. m. (2017). models of mentoring in language teacher education. new york, ny: springer. perlman b. & mccann l. (2005). undergraduate research experiences in psychology: a national study of courses and curricula. teaching of psychology 32(1): 5-14. pukkila p., arnold m., li a., & bickford d. (2013). the graduate research consultant program: embedding undergraduate research across the curriculum. cur 33(4): 28-33. ryan, h, reamer, d. enos, t. (2008). narrating our revision: a mentoring program's evolution. in m. f. eble & l. l. gaillet, (eds.) stories of mentoring: theory and praxis (pp. 289-300) anderson, sc: parlor press. reid, e. s. (2008). mentoring peer mentors: mentor education and support in the composition program. composition studies 36(2): 51-79. sandoval, c. (2017). the impact of undergraduate research experience: a focus on special consideration students. poster. panoma, ca: southern california conference for undergraduate research. schneider, k. r., bickel, a., & alison morrison-shetlar, a. (2015). planning and implementing a comprehensive student-centered research program for firsteffectiveness of a peer coach model journal of effective teaching in higher education, vol. 2, no. 2 56 year stem undergraduates. journal of college science teaching 44(3): 3743. song, k. h. (2016). systematic professional development training and its impact on teachers' attitudes toward ells: siop and guided coaching. tesol 7(4): 76799. microsoft word fulford.docx journal of effective teaching in higher education, vol. 2, no. 1 students as co-producers of queer pedagogy kathryn wymer, north carolina central university, kwymer@nccu.edu collie fulford, north carolina central university abstract. responding to concerns about a textbook reading that students perceived as heteronormative, cisnormative, and antifeminist, we formed a partnership between students and faculty to reflect on the situation and to workshop ways to move forward. our discussions were informed by our situation: a public hbcu in north carolina, a state that had been in the headlines for anti-lgbt legislation. many students reported that prior to our work they had not felt they had power to challenge the authoritative nature of texts in a classroom, even when they found those texts to be incorrect or inappropriate. this project empowered students to work with faculty and the publisher to change the textbook itself as well as the way certain rhetorical content was taught in our institution. keywords: queering; agency; student-faculty partnership; partnership pedagogy; course texts we want our students to feel empowered in our classrooms. we want them to create new knowledge, to challenge accepted ideas, and to develop as leaders and change makers. sometimes, however, the structures of authority embedded in the classroom can create blind spots for instructors. in the case described here, we discovered mid-semester that a handbook our department had adopted contained a writing sample that many students found offensive, claiming it perpetuated gender and sexuality stereotypes. that handbook, which was in use by our first-year writing program, became the topic of discussion in an upper-level english course on lgbtq literature. students in that course frequently reflected on concepts of heteronormativity, cisnormativity, and gender stereotypes as part of the course content. during one class period, students raised their first-year experience of reading a piece of student writing in the little seagull handbook 2e (bullock, brody, & weinberg, 2014). our instructional focus on the formatting of the piece, which was included to demonstrate apa style, had blinded us to the content students found offensive. for the purposes of class discussion, the handbook excerpt became a useful example of the pervasiveness of stereotyping, but it also created a moment in which the instructors could invite further reflection and student agency. throughout, students expressed multiple ideas for how to change not only the handbook but also the teaching related to it. the intensity of their investment in the process culminated in discussions that included the handbook’s authors and editor, and all involved at the publisher were highly responsive to our concerns. what unfolded became a complex case of student-faculty partnership in rhetorical intervention that had tangible results in and beyond this classroom. students as co-producers of queer pedagogy 46 journal of effective teaching in higher education, vol. 2, no. 1 in the following essay, we review literature on student-faculty partnerships for institutional change and examine the local social and political contexts that gave exigence to the actions we and our students initiated. we discuss our thought processes behind the initial class discussions and subsequent activities. our perspectives reflect that we approached this situation from different roles in the university. we further consider how some of what we did might be adapted productively in other contexts. our hope is that our work might help others consider how students working as partners with faculty members can effect change on multiple levels: in the classroom, in a writing program, at the university as a whole, and at a national publisher of college texts. student-faculty partnerships in reviewing international scholarship about student engagement and studentfaculty partnerships, healey, flint, and harrington (2014) discern four major domains in which such partnership approaches are used: • learning, teaching, and assessment; • subject-based research and inquiry; • scholarship of teaching and learning; • curriculum design and pedagogic consultancy. (p.36) the case involving kathryn’s literature course demonstrates further value from collaborations between students and faculty. classroom discussion turned into a coconstructed activist project with tangible external results. while this occasion evidenced several of the above domains in that students assessed a commonly assigned text, discussed the issues with it in subject-based terms, and provided pedagogical recommendations, this partnership also had an activist quality that resulted in actions with significance beyond one classroom and institution. a strong strand in the scholarship of teaching and learning positions students as potential change agents rather than mere consumers in higher education. kay, dunne, and hutchinson (2010) argue that student-researchers can use their agency and knowledge to contribute meaningfully as institutional problem solvers. stocks (2012) and ambos (2012) see undergraduate research conducted with faculty partners or sponsors as a key source for student-initiated institutional change. important in these approaches are the degrees to which students initiate projects and maintain agency throughout. fielding’s (2001) criticism of the ubiquitous and potentially manipulative use of student voices as merely data for institutional accountability highlights important ethical considerations for those interested in involvement with students. he discusses a spectrum of student roles in school improvement projects, from serving as informants and recipients to being the initiators, collaborators, and change-makers who share power equitably with faculty partners in “radical collegiality” (p. 129). mcculloch (2009) similarly proposes a way of framing students as “co-producers” of higher education. mcculloch forwards this term from public administration to counter consumerist metaphors for students. students as co-producers of queer pedagogy 47 journal of effective teaching in higher education, vol. 2, no. 1 other scholars of teaching and learning who recognize students as producers in student-faculty collaborations document students’ contributions to published knowledge-making (taylor & wilding, 2009; werder & otis, 2010; wymer, fulford, baskerville, & washington, 2012), their roles as producers of course materials (williams et al., 2011), and their influence on curricular design (bovill, cooksather, & felten, 2011; mihans, long, & felten 2008). the egalitarian intent behind such collaborations between students and faculty can also be understood through feminist scholarship of listening rhetoric, especially as posited by royster (1996) and ratcliffe (2005). royster raises listening as a crucial means for discourse across racial and status boundaries: how can we teach, engage in research, write about, and talk across boundaries with others instead of for, about, and around them? …. [w]hen do we listen? how do we listen? how do we demonstrate that we honor and respect the person talking and what that person is saying, or what the person might say if we valued someone other than ourselves having a turn to speak? (p.38) ratcliffe, building on royster’s work, defines rhetorical listening as a “code of crosscultural conduct” (p. 19) with moves that can help “approximate equal positioning” (p.128) in situations of power difference. royster and ratcliffe challenge us to acknowledge the different subject positions we occupy. in any given class, our hbcu has a racially diverse group of students, but the majority identify as african american; we are white faculty members. many of our students come from disadvantaged backgrounds, but many are from wealthy families. we have had in our classes young people who have served time in prison, and students from more privileged backgrounds, including the child of a senator and the child of a macarthur “genius grant” recipient. in the lgbtq literature course, many of the students identified openly, as collie does, as part of the queer community; whereas, others identified, as kathryn does, as allies to the queer community. we recognize what is powerful and diverse within our hbcu setting. listening closely to our students inspires us to perform fielding’s “radical collegiality” with them toward activist ends within this context of intersecting identities. methods the authors’ methods include naturalistic participant-observations of the case, participant member-checks, and collaborative rhetorical analysis of textual artifacts through a queer theory lens. because our initial goals for questioning and reflecting together (including with our students) were to better understand this case’s implications for our own teaching, this might be characterized initially as a stumbled-upon action research project. however, it seemed evident that the case could be instructive beyond our own classrooms. students as co-producers of queer pedagogy 48 journal of effective teaching in higher education, vol. 2, no. 1 to gather data and insights, we engaged in periodic reflective conversation and notetaking with each other during the teaching situation and in the months following. we involved a student from the class to co-analyze and present with us in order to member check the findings and distribute the power to make meaning from the situation. this reflects our principles about student agency and students as partners in institutional inquiry. these actions were triangulated by analyzing relevant textual artifacts including state news, educational policy, and the handbook in question. our textual analysis began with students’ insights informed through the course’s queer theory lens. social and political contexts for student action in march 2016, a few months before the course began, north carolina passed house bill 2 (hb2), labelled by legal observers and journalists as one of the most anti-lgbtq laws in the united states (yang, 2016). though its legal name was the “public facilities privacy & security act,” it became known nationwide, especially in right-wing media, as north carolina’s “bathroom bill” (williams, 2016). the bill’s provisions did more than simply target the rights of transgender people to use public restrooms. because it restricted the rights of lgbtq persons in a variety of ways, the ramifications were felt on north carolina’s campuses on multiple levels: concern among the lgbtq community about how the law would be implemented, immediate lawsuits against the unc system challenging the law’s provisions, and withdrawal of collegiate sporting tournaments by the ncaa and ciaa (tracy & blinder, 2016; peralta & portillo, 2016). unc system president margaret spellings received notice from the u.s. department of justice that hb2 put the university system in violation of title ix, the u.s. federal civil rights law barring educational programs from discriminating on the basis of sex, to which spellings (2016) replied that the university of north carolina was “in a difficult position” (p. 1) because it was subject to the laws of the state. as one of the 17 unc system campuses, our university was impacted in real and meaningful ways by hb2. this event rocked any complacency progressive north carolina educators might have had about the status of trans and queer students in our state. the suddenly and very public struggle prompted by the law’s passage made many of us want to ensure that our classrooms were hospitable spaces for all students. interacting with our students underscored for us how important an instructor’s role is in creating such space for productive conversation. engaging in honest conversation requires actively listening to students, and that act of listening reminded us that students who are sensitive to these issues can be powerful collaborators in our efforts to make education a site of not simply fairness, but also queer agency. our historical and institutional context is important. not only did this scenario unfold at a public institution in north carolina months after a highly controversial bill aimed at the lgbtq community had been passed, but we also teach at one of the nation’s historically black colleges and universities (hbcus). for our students who identify as queer and trans people of color (qtpoc), the stakes can be high. as garvey et al. (2018) point out, “qtpoc students are often faced with a tenuous choice regarding how they express their minoritized sexual, gender, and racial students as co-producers of queer pedagogy 49 journal of effective teaching in higher education, vol. 2, no. 1 identities within society and the extent to which they disclose their identities within collegiate environments” (p. 2). mobley and johnson (2015) have also called attention to the fact that hbcus have lagged behind their higher education counterparts in providing welcoming spaces for lgbtq individuals, noting in 2015, that only 21 out of the nation’s 102 hbcus had lgbt/ally organizations, though that number has increased somewhat in recent years. in terms of campus climate, mobley and johnson (2015) further point out that “rather than encouraging students to walk in their own truth and embrace their authentic selves, many hbcus compel students who identify as gay or lesbian to suppress these identities while on campus” (p. 79). though we believe, for many reasons, that our institution is pioneering new levels of support for lgbtq students (williams, 2018), we acknowledge that the culture of our campus as an hbcu may affect students’ perceptions of safety and support. even the class in which we began this project is something that students clearly had been wanting for a long time. when we offered the course “eng 3040: special topics: lgbt literature” in fall 2016, it was the first ever undergraduate course at our hbcu that focused exclusively on lgbtq literature, and it was long overdue. the deep interest students showed and the vibrant discussions that followed were clear indicators of their desire for more opportunities to talk about lgbtq issues in an academic setting. like all historically black colleges and universities, our classrooms are diverse spaces. the makeup of the class under discussion included students of different races, ages, and a variety of other factors that impacted their experiences. as a result, they brought to our discussion their own approaches to, and understanding of, discrimination informed by the intersections of race, class, gender identity, and sexual orientation. this diversity of experience yielded a rich discussion in the classroom, and it positively impacted our ability to collectively respond to the situation. student concerns about the handbook our primary textbook for the eng 3040 course, finding out: an introduction to lgbt studies, took as its approach “a recognition of how many queer cultures bring substantive, potentially transformative insights to bear on mainstream and dominant modes of being,” and the authors expressed the value of “how our queer visions help us see the world in intellectually, politically, and personally capacious ways” (gibson, alexander, & meem, 2014, p. xv). that approach challenges patterns of communication that tend to exclude from recognition people who do not identify as heterosexual or cisgender. in the early part of the semester, therefore, participants in the course began discussing heteronormative and cisnormative discourse and its queer erasure. it quickly came up in discussion that students wanted to examine the writing handbook our department had adopted as an example of such a text. as the students were all familiar with the little seagull handbook, having used it in their composition courses, they were eager to discuss it. handbooks are considered authoritative sources on pronoun use; students wanted to evaluate how the second edition of little seagull handled such language. students also chose to engage with students as co-producers of queer pedagogy 50 journal of effective teaching in higher education, vol. 2, no. 1 a sample paper provided in the handbook as a model of student writing. the title of the piece is “it’s in our genes: the biological basis of human mating behavior.” here is the opening abstract of the item in question: while cultural values and messages certainly play a part in the process of mate selection, the genetic and psychological predispositions developed by our ancestors play the biggest role in determining to whom we are attracted. women are attracted to strong, capable men with access to resources to help rear children. men find women attractive based on visual signs of youth, health, and, by implication, fertility. while perceptions of attractiveness are influenced by cultural norms and reinforced by advertisements and popular media, the persistence of mating behaviors that have no relationship to societal realities suggests that they are part of our biological heritage. (stonehill, 2014, p. 167) what follows this abstract is an introduction in which we are asked to imagine a young woman named jenny who is attracted to a “tall, muscular, and stylishly dressed” young man who owns a bmw (p. 168). this depiction is provided as a representative example of preferences in sexual attraction. while discussing the text, students pushed for an aggressive critique of the handbook and its editorial decisions. there was erasure of experience here. if the essay was claiming that biology was the basis for heterosexual attraction, it didn’t allow for the validity of any kind of same-sex attraction. they also felt it was important that the authors and editors of the textbook chose to share as an exemplar an essay that was heteronormative in its exclusion of the possibility of same-sex attraction. additionally, the gender stereotypes present in the essay’s assessment of attraction run counter to feminism’s advocacy on behalf of individual respect and equality, suggesting, by implication, that women would not be attracted to men without money and that men are only attracted to able-bodied youthful women. by reinforcing gender stereotypes, the essay also promoted the concept of binary gender roles, a cisnormative concept. as the conversation continued, our students also wanted to look at the guidance the handbook offered on pronoun usage. though there were admonitions in the text to avoid sexist language, guidance on pronouns did not embrace they/them for singular usage. alternate pronouns (ze, for example) were not even mentioned. students were aware of much debate in popular media about that kind of pronoun guidance, so they were quick to point out that a composition handbook really needed to address the issue in a helpful way. after we started this project of critically engaging the handbook, there was one interaction kathryn particularly remembers. she was talking with a transgender student who came to visit her office hours. he was not enrolled in the literature class, but he had been a student in her composition class two years before. kathryn briefly mentioned that there was an offensive essay in the composition handbook that some of her students had wanted to address in class discussion. without further elaboration from kathryn, the student immediately said, “i know exactly which essay you’re talking about.” he remembered with clarity the essay that had students as co-producers of queer pedagogy 51 journal of effective teaching in higher education, vol. 2, no. 1 taken kathryn several semesters to notice as a potential problem. lgbtq students weren’t seeing form over content the way kathryn had been. they saw an essay in a university-sanctioned textbook that erased and invalidated their existence, and there hadn’t been an opportunity for them to make their concerns heard. workshopping the problem during the second week of the semester, kathryn mentioned to collie that she and her students in eng 3040 had voiced criticism for the handbook used in our firstyear writing courses, noting the glaringly heteronormative sample text and narrow guidance about pronoun usage as their main objections. collie directs nccu’s first year writing program. a committee she led had selected that handbook, and she felt a responsibility to speak with the students. collie therefore volunteered to lead a workshop during one of the eng 3040 sessions to fully air students’ concerns and start discussing what actions we might take. to prepare, she read the sections in dispute and researched how other authoritative texts were working with pronouns. she discovered that the writing center journal referenced the guidelines for gender-fair use of language crafted by the national council of teachers of english (2002) and had an editorial policy of singular “they.” she brought an excerpt from the writing center journal's current submission guide and style sheet (n.d.) as a reference point for our discussion. as students worked through their responses to the text and how its guidance compared with current standard practice, collie explained her role in the original decision to adopt the handbook. she explained that she took responsibility, and that now kathryn and their class had raised the issue, she was equally concerned. collie wanted to hear students’ perspectives and to work with them to weigh our options for taking action. during our workshop, students expressed strong feelings, sharp insights, and a desire to move forward with some kind of formal response to the publisher. kathryn followed up by gathering students’ brief written perspectives. collie identified the proper channels at the publisher through which we could express our concerns. we decided to draft a letter [appendix a] including our collective analysis of the problems and our recommendations for specific changes to this handbook that would redress the problems we found. interactions with the textbook publisher representatives at norton were immediately responsive to our concerns, facilitating a face-to-face meeting with collie and also creating an opportunity for conversation with one of the handbook’s authors. fortunately, our discussion coincided with the publication of an updated version of the handbook, which meant that they were immediately able to implement our request to change the student writing sample. one author, michal brody, was especially responsive and wrote an email we were very pleased to be able to share with students: first, i want to personally thank you for getting students to truly and literally interact with the text. as an author, i’ve never received such detailed, students as co-producers of queer pedagogy 52 journal of effective teaching in higher education, vol. 2, no. 1 thoughtful, and heartfelt feedback, and i really appreciate the engagement. as part of the norton team, i also want to assure you that we are taking your recommendations very seriously, and we are making changes for the 3rd edition. in fact, a good bit of what your letter mentions has already been addressed in our revision process, including pronouns and singular they, so we’re particularly eager for you to see the results when the book comes out. as for the sample apa paper, let me mention that it was chosen quite a long time ago by one of the co-authors; since its purpose is to show formatting, its content hadn’t been reviewed for a while, and we didn’t realize how out of date and inappropriate it is. we appreciate very much your bringing the problem to our attention, and we are replacing the sample essay for the new edition. (m. brody, personal communication, sept. 23, 2016) students expressed how empowering it felt for them to be able to raise issues with people they felt to be authorities: instructors, program directors, authors, and textbook publishers. that their concerns were heard and resulted in change was something that several of them had not imagined was possible. sharing the results of our work clearly our work had impact in the specific course we taught, and the way we interrogated our textbook initiated an important conversation for our university’s writing program. we shared regular updates on our students’ efforts to make changes to the composition handbook at departmental meetings. our colleagues were appreciative of that work. however, we felt that the spirit of our project required an additional level of intervention among instructors. the students on this project communicated an urgent desire that our work could positively impact the way other instructors teach their courses. students expressed concern that just advocating for altering the textbook might not be enough to help the wider community of composition instructors teach in ways that made positive space for queer identity. of special concern was the concept of potential erasure of identity or exclusion in the classroom. after all, that had been at the heart of students’ concerns with the textbook itself. the version with the problem essay and the lack of inclusive pronoun guidance at the minimum gave tacit approval to heternormativity and cisnormativity. at worst it had the potential to enable teachers or other students to label certain practice, such as nonbinary pronoun usage, as “incorrect” and liable to be graded as such. collie took on the first iteration of this work in her presentation at the carolinas writing program administrators fall 2016 conference (fulford, 2016). at that point, we were in the midst of the case, and that conversation helped guide some of our subsequent discussion with the textbook publishers. it seemed important to follow up again after the spring semester, once the course had ended, the new textbook edition had appeared, and we had time to process some of the lessons learned. we were fortunate that one of our students, zachary brewer-kirby, was willing to continue to work with us in order to create a presentation at a 2017 summer institute held on our campus (wymer, fulford, & brewer-kirby, 2017). strategies for resistance, resilience, and hope: supporting qtpoc on college campuses was students as co-producers of queer pedagogy 53 journal of effective teaching in higher education, vol. 2, no. 1 presented by the consortium of higher education lgbt resource professionals and the national black justice coalition. it was exciting for our campus to host this symposium, and we were enthusiastic about sharing our work with a wider community of faculty, students, and staff who could consider how they might be able to employ similar interventions on their home campuses. takeaways this project has involved several components: • acknowledging the harm expressed by students who feel textually erased and offended. • recognizing that students are not always prepared, willing, or able to critique their learning materials unless space is made for them to offer a response. • identifying students’ criticism of a text as an exigence for rhetorical action. • gathering and channeling students’ insights into a rhetorical response to the situation. • collaborating with students, other faculty, and authors and staff at the publishing company in an attempt to queer the handbook. • imagining additional ways to queer our teaching practices so that students of various sexualities and gender identities can experience inclusion rather than erasure from our pedagogies. • reflecting on what kinds of pedagogical and curricular situations appear to “co-sign” viewpoints that do damage to our students, especially when those situations occur outside the scope of the material’s directly intended purpose (e.g., teaching apa style) • reflecting on what kinds of pedagogical and curricular contexts we can intentionally create in order to foster student agency. as instructors we were inspired by our students’ willingness to engage and push for changes in the way we taught our courses. “[p]artnership pedagogy is about being (radically) open to and creating possibilities for discovering and learning something that cannot be known in advance” (healey, flint, & harrington, 2014, p. 58). we suspect that our prior commitment to the principles of student-faculty partnership for research primed us to be open to impromptu partnership pedagogies when critical discussion arose during kathryn’s lgbtq literature class. given the intensity of students’ concerns as well as her own, kathryn reconfigured her planned course schedule to make space for further discussion and action. she also invited collie, a stakeholder to the issue, to join the group. this openness allowed students to expand upon their frustrations with a text and – with faculty sponsors – to consider ways of responding to it that might have significance beyond the classroom. kathryn did not know in advance how this decision would unfold or what the students would learn from engaging with the problem collaboratively, but she determined that it was important to try. taking a significant pause in a literature class to hold space for students to work things out is a case of partnership pedagogy fused with rhetorical listening. ratcliffe students as co-producers of queer pedagogy 54 journal of effective teaching in higher education, vol. 2, no. 1 (2005) points out that teachers may avoid tense issues such as those that highlight differences of race, gender, sexuality, and other subjectivities. when such a discussion’s ends are unknown, teachers may fear losing control of the class (p. 140). being open to listening pedagogically, however, offers a way to “ask the class why things are going awry. ask what can be done better. ask if [failure] is an isolated incident or if it represents a larger cultural pattern” (ratcliffe, 2005, p. 141). although the lesson in rhetorical agency was unplanned and unscripted, analyzing the larger cultural patterns of queer oppression aligned with the course’s theoretical underpinnings. beyond analysis, however, royster (1996) asks us to consider, “how do we translate listening into language and action, into the creation of an appropriate response?” (p. 38). it strikes us that the risks we took were necessary for producing a collective rhetorical response to one cultural pattern that needed to be resisted. mcculloch’s (2009) concept of students as co-producers of higher education and fielding’s (2001) exhortations to mind the power dynamics both apply to this case as well because students and faculty worked together as agents to change instructional materials and teaching practices that affect many people. our power differences within the institution were not flattened, but instead acknowledged and utilized strategically. the authority of qtpoc students (and their allied peers) to describe their lived experiences with the text worked in conjunction with our institutional authority to navigate the structures of publishers and programs. the students in eng 3040 thereby effected change with us, co-producing new practices and texts. we hope that sharing our case encourages other instructors to make that space to collaborate with students and other stakeholders to improve the work we do. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references ambos, e. (2012). undergraduate and faculty researchers change themselves, their educational institutions ... and cur. council on undergraduate research quarterly, 33(1), 4–5. bovill, c., cook-sather, a., & felten, p. (2011). students as co-creators of teaching approaches, course design and curricula: implications for academic developers. international journal for academic development, 16(2), 133– 145. https://doi.org/10.1080/1360144x.2011.568690 bullock, r., brody, m. & weinberg, f. (2014). the little seagull handbook (2nd ed.). new york: w.w. norton & company. fielding, m. (2001). students as radical agents of change. journal of educational change, 2, 123–41. https://doi.org/10.1023/a:1017949213447 students as co-producers of queer pedagogy 55 journal of effective teaching in higher education, vol. 2, no. 1 fulford, c. (2016). queering the handbook: taking rhetorical action with students. work in progress presented at carolinas writing program administrators conference, little switzerland, nc. garvey, j. c., mobley, jr. s. d., summerville, k. s., & moore, g. t. (2018). queer and trans* students of color: navigating identity disclosure and college contexts. the journal of higher education, 90(1), 150–178. https://doi.org/10.1080/00221546.2018.1449081 gibson, m. a., alexander, j., & meem, d. t. (2014). finding out: an introduction to lgbt studies (2nd ed.). los angeles, ca: sage. healey, m., flint, a., & k. harrington, k. (2014). engagement through partnership: students as partners in learning and teaching in higher education. york, united kingdom: the higher education academy. kay, j., dunne, e. & hutchinson, j. (2010). rethinking the values of higher education—students as change agents? [commissioned paper]. gloucester, united kingdom: quality assurance agency of higher education. retrieved from https://dera.ioe.ac.uk//1193/ mcculloch, a. (2009). the student as co-producer: learning from public administration about the student-university relationship. studies in higher education, 34(2), 171–83. https://doi.org/10.1080/03075070802562857 mihans, r., long, d., & felten, p. (2008). power and expertise: student-faculty collaboration in course design and the scholarship of teaching and learning. international journal for the scholarship of teaching and learning, 2(2). https://doi.org/10.20429/ijsotl.2008.020216 mobley, jr., s. d. & johnson, j. (2015). the role of hbcus in addressing the unique needs of lgbt students. new directions for higher education, 2015(170), 79–89. https://doi.org/10.1002/he.20133 national council of teachers of english. (2002, july 31). guidelines for gender-fair use of language [archived position statement]. retrieved from https://web.archive.org/web/20171127080051/http://www2.ncte.org/statem ent/genderfairuseoflang/ peralta, k. & portillo, e. (2016, september 30). ciaa will keep basketball tournaments in charlotte, but move other sports events over hb2. charlotte observer. retrieved from http://www.charlotteobserver.com/news/business/article105149121.html ratcliffe, k. (2005). rhetorical listening: identification, gender, whiteness. carbondale, il: southern illinois university press. royster, j. (1996). when the first voice you hear is not your own. college composition and communication, 47(1), 29–40. https://doi.org/10.2307/358272 spellings, m. (2016, may 9). letter to vanita gupta, principal deputy assistant attorney general. retrieved from students as co-producers of queer pedagogy 56 journal of effective teaching in higher education, vol. 2, no. 1 http://www.newsobserver.com/news/local/education/article76620312.ece/bi nary/unc's%20letter%20to%20the%20department%20of%20justice stocks, j. e. (2012). undergraduate researchers change their institutions. council on undergraduate research quarterly, 33(1), 6. stonehill, c. (2014). it’s in our genes: the biological basis of human mating behavior. in bullock, r., brody, m. & weinberg, f. the little seagull handbook (2nd ed.). new york: w.w. norton & company. taylor, p. & wilding, d. (2009). rethinking the values of higher education—the student as collaborator and producer? undergraduate research as a case study [commissioned paper]. gloucester, united kingdom: quality assurance agency of higher education. retrieved from https://dera.ioe.ac.uk//433/ tracy, m. & blinder, a. (2016, september 12). n.c.a.a. moves championship events from north carolina, citing anti-gay-rights law. new york times. retrieved from https://www.nytimes.com/2016/09/13/sports/ncaa-moveschampionship-events-from-north-carolina.html werder, c., & otis, m.m. (eds.). (2010). engaging student voices in the study of teaching and learning. sterling, va: stylus. williams, d. (2016, may 9). the truth about north carolina’s bathroom bill. the federalist. retrieved from http://thefederalist.com/2016/05/09/the-truthabout-north-carolinas-bathroom-bill/ williams, j., alder, d., cook, j., whinney, m., o’connell, r., duffin, w. & king, p. (2011). students and staff as educational partners in the development of quality assured online resources for medical education. in s. little (ed.), staff-student partnerships in higher education. (pp. 76–91). london, england: continuum. williams, s. (2018, june 21). it’s time for hbcus to address homophobia and transphobia on their campuses. the nation. retrieved from https://www.thenation.com/article/time-hbcus-address-homophobiatransphobia-campuses/ writing center journal. (n.d.). submission guide and style sheet. writing center journal. retrieved from http://www.writingcenterjournal.org/submissions wymer, k., fulford, c. j., baskerville, n, & washington, m. (2012). necessity and the unexpected: sotl student-faculty collaboration in writing program research. international journal for the scholarship of teaching and learning: 6(1). https://doi.org/10.20429/ijsotl.2012.060120 wymer, k., fulford, c., & brewer-kirby, z. (2017). emphasizing qtpoc voices in the classroom. presentation at the summer institute of the consortium of higher education lgbt resource professionals, durham, nc. yang, j. (2016, march 24). how north carolina signed a bill dubbed the most antilgbt law in the u.s. [transcript]. pbs news hour. retrieved from https://www.pbs.org/newshour/show/how-north-carolina-signed-a-billdubbed-most-anti-lgbt-law-in-the-u-s students as co-producers of queer pedagogy 57 journal of effective teaching in higher education, vol. 2, no. 1 appendix a: our letter to the publisher september 14, 2016 marilyn moller, composition editor & mary ann parrott, college sales representative w. w. norton & company, inc. 500 fifth avenue new york, new york 10110 dear marilyn moller and mary ann parrott: north carolina central university’s writing program has long had a good relationship with norton because of your history of responsive service and respectful, knowledgeable marketing of books at relatively affordable price points. we have therefore adopted the little seagull handbook with exercises, 2nd edition (isbn 978-0-393-93581-3) for several years now. students are required to purchase seagull for our two required introductory composition classes. we enrolled over 1100 new students this fall, almost all of whom take these courses. we understand that the new 2016 mla guidelines necessitate changes to seagull. how timely. although there are many useful features of the book, a number of students have called our attention to troubling content that should also be changed. we hope that the following analysis and recommendations thus arrive in good time for the authors to consider as they work on 3rd edition. students in nccu’s eng 3040: special topics in lgbt literature course have recently taken up a study of the book’s content, and they offered several critiques of the book. they especially find the apa sample paper and the guidance on pronouns to be wholly unacceptable for a required textbook. with respect to the apa sample student essay that appears as section apa-e, on pages 187-191 and in fuller format in the online companion, one student wrote, “it is an extremely biased and potentially highly offensive essay.” the title of the essay alone highlights the students’ concerns: “it’s in our genes: the biological basis of human mating behavior.” the essay goes on to explain heterosexual attraction between humans while erasing the possibility for homosexual attraction. by stating that there is a biological and evolutionary basis for this attraction, the only possible conclusion the essay can offer for homosexuality is that it is somehow outside the norm. this essay has very much upset students who, during class discussion, questioned whether it was the motive of the editors at norton to exclude lgbtqidentifying people. in response to this idea, one student advised the following: “in students as co-producers of queer pedagogy 58 journal of effective teaching in higher education, vol. 2, no. 1 future, the editors should be mindful that the essay examples should be carefully chosen because by their inclusion it leads to the assumption that the editors agree/endorse with the essay’s content. further, we are questioning your (the editor’s) motives!” if we were to retain our adoption of the text, nccu’s writing program would in essence also be continuing to endorse this reading. that is not something we feel we could continue to do. in addition to the problems with the essay’s heteronormative, cisnormative approach to human sexuality, this essay also contains multiple aspects that any feminist would find troubling. for instance, the essay implies that women are attracted to men with wealth and that men are attracted to women who look fertile. this essay must be replaced if nccu is to continue endorsing the little seagull handbook. pronoun guidance in the handbook is also inadequate. many students today reject the gender binary, and singular “they” is widely accepted as a way to address this situation. one student particularly advocated the model provided by the writing center journal as a strong inclusive example of how to consider pronoun usage: “wcj adopts the singular “they” as a gender-neutral term. we do not find a pronoun-antecedent agreement error in a sentence such as “each tutor identified their own strategies”; rather, we recognize the phrase “his or her own strategies” reflects a problematic, exclusive binary.” pronoun usage must be more inclusive and must indicate a positive acceptance of singular “they” if nccu is to continue to adopt the little seagull handbook. introduction of other, alternate pronouns would be welcome, also. we understand that this section’s author, michal brody, is at work addressing these issues and we are hopeful that our suggestions are in alignment with the planned revisions. overall our students advocate that throughout the text there be “inclusion of neutral definitions, summaries, and examples that do not single out a specific community.” we look forward to your reply and a resolution to these problems. the writing program will be meeting soon to revisit handbook selection, so communication about changes being made to the third edition would be important to our decisionmaking process. students as co-producers of queer pedagogy 59 journal of effective teaching in higher education, vol. 2, no. 1 sincerely, the students of eng 3040: special topics in lgbt literature, fall 2016, north carolina central university dr. kathryn wymer, associate professor of english composition and literature, department of language and literature, north carolina central university dr. collie fulford, associate professor of english rhetoric and composition, department of language and literature, north carolina central university microsoft word andzik.docx journal of effective teaching in higher education, vol. 2, no. 1 the effects of gaming on university student quiz performance natalie r. andzik, northern illinois university, nandzik@niu.edu corinne m. gist, elle e. smith, menglin xu, nancy a. neef, the ohio state university abstract. the use of competitive games to increase classroom engagement has become common practice among many teachers and university professors. however, it is unclear if using games, as an assessment tool, is a viable way to increase student performance. this study examined the effects of administering point-earning quizzes through a game-based system, kahoot!, visible to all in the classroom versus privately on an electronic device. the quiz scores of 56 undergraduate students, enrolled in one of two special education courses, were evaluated. a linear regression was used to compare student scores across the two conditions as well as performance over the course of a 15-week semester. no significant difference in quiz scores was found between the two conditions, and quiz scores in both conditions improved similarly over time. sixty-eight percent of the students reported preferring to take the quiz privately on an electric device as opposed to on kahoot! limitations and recommendations for practitioners are discussed. keywords: higher education, college instruction, game-based assessment, studentresponse system instructors in higher education seek to engage students in the learning process while simultaneously assessing their skills and knowledge. formative and summative assessments are commonly used in the college classrooms for this purpose (brookhart, 2004). summative, or formal, assessments include systematic ways of assessing student learning and providing instructors with information that is useful in making final decisions, such as grade assignments (brookhart, 2004). these assessments often include incentives to perform well (e.g., points) associated with them and may include quizzes, midterms, and final exams. informal, or formative, assessments refer to a wide variety of methods that teachers use for guiding student learning, instructional change, and other improvements (brookhart, 2004). examples of formative assessments include active responding, think-pairshare, and choral responding. a popular version of active responding, often used in the college classroom, includes classroom response systems (crs), also called student response systems (srs) (wang, zhu, & sætre, 2016). srs may employ individual remotes that students use to respond to questions that are projected on the screen and a handheld transmitter that collects and displays the student’s responses. other classroom response technologies are web-based and allow students to use cell phones or other personal mobile devices to answer questions (wang, 2015). srs allow instructors to instantaneously collect data from student responses and display the answers on a classroom projection screen where both students and the effects of gaming on university student quiz performance 110 journal of effective teaching in higher education, vol. 2, no. 1 instructor can see and discuss them (wang, 2015). researchers have reported many benefits to using srs in the classroom including improved attendance, increased focus among students, improved student engagement, increased learning performance, improved teaching, and improved interaction between teacher and students (see kay & lesage, 2009, for a review). a newer srs system that has become increasing popular is kahoot! kahoot! is an example of game-based student response systems (gsrs) that combine the benefits of srs with those of game-based learning. as a game-based system, kahoot! is intended to transform the classroom into a game show, where the students are the competitors and the instructor is the host (wang, zhu, & sætre, 2016). kahoot! gamifies the student response process by using a graphic user-interface, special audio, and competition. these effects are designed to attract and engage the students. game-based learning has been received positively when used in the university classrooms. for example, in a recent study, students reported that classroomlearning games increased their enjoyment of the learning process (crocco, offenholley, & hernadez, 2016; robinson, 2014). in addition, two experimental studies found that higher quiz and test grades were achieved when game-based learning was used as a study strategy (neef, perrin, haberlin & rodriques, 2011; robinson, 2014). despite the benefits, prior to considering game-based learning in classrooms and deciding on a medium for its delivery, practitioners should consider the competitive aspect of this learning strategy as well as the makeup of the students in the classroom. the element of competition can aid in student learning as evidenced by a study that used competitive game-based learning as a way of demonstrating learning and self-reported motivation among undergraduate students (cagiltay, ozcelik, & ozcelik, 2015). however, neef and colleagues (2011) used a different, cooperative approach with gaming, and although there was not a noticeable difference between the gaming and control conditions, the groups expressed they “felt bad” for the other team and expressed they wished them to do well on the quizzes. one obvious benefit when incorporating gsrs in the classroom is the natural increase in participation among students. university instructors and professors are often up against distracting technology (e.g., cell phones, internet) and likely decreased participation when teaching larger groups of students (rocca, 2010). public posting through a “score board” can be beneficial when capturing larger audiences of students and inadvertently pressures students to prepare in advance to avoid any possible embarrassing effects of placing last in comparison to their peers in a game. anecdotal reports suggest other benefits of gsrs when used as informal assessments or study tools (plump & larosa, 2017; wang, 2015). although few experimental studies have been conducted, it is reasonable to assume that gsrs could be used as a formal assessment tool. currently, only one study has looked at the effects of kahoot! in this way. wang, zhu, and sætre (2016) compared the effects of using kahoot! srs (i.e. clicker) and paper tests as a formal assessment. pre-and post-tests were used to assess the learning outcomes of the lectures and a questionnaire was given to obtain feedback on the students’ engagement, enjoyment, concentration, and motivation. the effects of gaming on university student quiz performance 111 journal of effective teaching in higher education, vol. 2, no. 1 students who took the kahoot! quiz reported higher levels of motivation, enjoyment, engagement, and concentration during the assessment than those who were assessed using paper tests or srs. to assess the learning outcomes, quiz scores were analyzed using the mann-whitney test. no significant difference was found between the quizzes taken on kahoot! and the quizzes taken on paper. the srs condition was not included in the learning outcome analysis. one limitation of the wang, zhu, and sætre (2016) study included only exposing the participants to one condition. therefore, the students who took the quiz using kahoot! could not compare the gamified approach to the srs or paper assessments. in addition, data were collected on only one quiz. finally, the researchers did not use any incentive (e.g., points, rewards) for the completion of the quiz. incentives have been shown to increase, even if temporarily, the motivation of students (bartel & hagel, 2014) the current study aimed to extend the work of wang, zhu, and sætre (2016) by increasing the number of assessments, having participants take the quizzes in both conditions, and including incentive (i.e., points in the class). this allowed the participants to compare their experiences in each condition and created a larger sample of quiz scores. the aims of this study were to (1) determine if there was a difference in scores when quizzes were taken on kahoot! versus privately on an electronic device, (2) determine if student performance changed over time, and (3) determine if students preferred one quiz method over the other. methods participants and setting fifty-six undergraduate students from a large midwestern university were included in this study. after obtaining university institutional board approval, students were recruited from either introduction to the special education profession (n=21) or applied behavior analysis (aba) for teachers (n=35), both instructed by the second author. data were collected during weekly class sessions (2 h 45 min) over a 15week semester. all sessions occurred on campus and were conducted in university classrooms. aba for teachers was designed to provide an overview of applied behavior analysis, and the aims of the course were to introduce students to the basic principles by which humans learn social, academic, physical, and other skills. these basic principles include, but are not limited to, positive and negative reinforcement, extinction, punishment, discrimination, and stimulus control. pre-service teachers took this course in their first semester after being accepted into the licensure program (typically their third year in college). introduction to the special education profession was a course that provided an overview of the education of exceptional learners. students learned about the various categories of special education, including the criteria for each category, prevalence, demographics, and typical interventions. in addition, students studied special education laws as well as special education as a professional discipline, including its history, current issues and the effects of gaming on university student quiz performance 112 journal of effective teaching in higher education, vol. 2, no. 1 challenges, and contemporary research-based instructional practices. this course was designed for pre-service teachers to take before their acceptance into the licensure program (often taken during their second year in college). both courses were introductory in nature and chapter quizzes were designed to (a) ensure students read the material prior to lecture, (b) assess the level of comprehension of the text of all students, (c) prompt students to critically think about the content prior to lecture and come prepared with questions. the introductory level of these courses highlights the primary level of learning— remembering and recalling facts, initial acquisition of new concepts, and understanding of the materials and how they affect learners with disabilities. each quiz was worth 10 points and accounted for 30% (introduction to special education) and 35% (aba for teachers) of the total grade in the course. procedure and materials an alternating treatment design was used in which the participants shifted between the two quiz conditions (johnston & pennypacker, 2009). students were randomly assigned to one of two groups (group a and group b). to determine if there were any outliers in the group, a pre-test was conducted to establish a baseline measure of the participants’ basic knowledge of the subject matter. no outliers were found. the groups were exposed to one of two conditions each week, private and gamebased. at the end of each class session, students were notified which quiz condition their group would be in the following week. at the beginning of each class, the instructor administered the quizzes, which included 10 multiple-choice questions based on the week’s assigned reading. each quiz had a time limit of 20 minutes and students could not change their answers once the selection was made. during the private condition, students took the quiz on a computer or personal electronic device (e.g., ipad, smart phone, laptop computer) in a nearby conference room supervised by either a graduate student or the classroom instructor. only the students taking the quiz could see their screen and answers. the group of students in the game-based condition took the same quiz as the group in the private condition, but these students used kahoot! as a group in the classroom. kahoot! is a free online platform used by teachers to assess the knowledge of their students. the quiz questions were displayed one by one, each for 2 minutes, or until all students had answered. if all students answered before the 2-minute time cap, the question automatically left the screen. the questions were projected on a large screen along with four possible responses. the students answered by choosing the corresponding symbol on the board that matched that on their personal electronic devices. after every question, the correct answer, a bar graph with the distribution of how the students answered, and the top three players were displayed on the main screen. students received individual feedback on their personal devices including the number of points received, their ranking compared to the rest of the group, how far the student was behind the student ranked above them, and the correct answer. at the end of a game-based session, the top threepoint earners’ names were displayed on the classroom screen. points were earned the effects of gaming on university student quiz performance 113 journal of effective teaching in higher education, vol. 2, no. 1 for answering quickly and correctly. the faster a student answered correctly, the more points they received. students in both conditions could retrieve answers to the quiz questions on the classroom website after the class period had ended. there was no data collected regarding which, if any, of the participants checked the answers online following the quizzes. each group participated in the private and public condition an average 6.5 times (range 6–8). although each quiz was created to be equally challenging, it was not possible to compare difficulty of the material across quizzes; therefore, analysis focused on within-unit comparison of quiz scores for private and public groups. although proctoring for each testing condition varied between an instructor or graduate student, the students’ experience in each testing condition were identical. interobserver agreement interobserver agreement (ioa) was assessed on all quizzes across all sessions. all quizzes were written and scored by the first, second, and third authors. the private group quiz scores were graded by a university online platform (i.e., canvas), and kahoot! scores were automatically graded by the website. each week, all scores were double-checked for accuracy by the first, second, and third authors and 100% agreement was found. social validity social validity was assessed through an anonymous questionnaire given at the conclusion of the study. one participant opted not to fill out the survey. participants were asked 11 questions, including 7 multiple-choice and 4 open response questions, and were provided space for additional comments. these questions were designed to assess the participants’ satisfaction with the procedures of the study (wolf, 1978). statistical analysis first, the quiz scores were re-scaled to fall into the range of 0 and 1, so that the scores collected from 12 quizzes were comparable. second, data were transformed from the original wide format to long format in order to fit the regression model. specifically, the variable of type (0, 1) was created to represent the private condition and the kahoot! condition respectively, the variable of time was created to indicate the week that the quiz was taken, and the variable of quiz scores (numeric score) was added to serve as dependent variable. the long format data contains 672 quiz scores (i.e., 56 students at 12 time points). third, descriptive statistics were run via spss 24.0 to produce mean and standard deviation (sd) for the private condition and kahoot! condition. finally, a linear regression model was fit using spss with the quiz outcome serving as the dependent variable and time and type serving as the independent variables. the effects of gaming on university student quiz performance 114 journal of effective teaching in higher education, vol. 2, no. 1 results thirty-nine of the 672 quiz scores were missing (rate = 5.8%); therefore, listwise deletion was adopted in the subsequent analyses due to the low missing rate. the mean scaled quiz outcome was .725 (sd = 0.183) for the private condition (n=337) and .707 (sd = 0.193) for the kahoot! condition (n=296), revealing no pronounced difference across quiz types. table 1 displays the results of the regression model. type had no significant effect on quiz outcome, β = -.018, se = 0.015, p = .216, which indicated that there is no prominent difference between the private condition and kahoot! condition after controlling for time. however, a significant time effect on quiz performance was detected, β = .006, se = 0.002, p = .003, which indicated that students experienced significant growth over the semester, regardless of quiz type. students quiz scores improved as the semester went on. type and time explained 1.6% of variance in quiz outcome. figure 1 displays the growth pattern of quiz performance by type. students progressed at similar speeds for both conditions. figure 1. quiz growth outcome by type. note. blue line: private; green line: kahoot! the effects of gaming on university student quiz performance 115 journal of effective teaching in higher education, vol. 2, no. 1 table 1 results of regression model β se t p intercept 0.684 0.017 39.833 <.001 type -0.018 0.015 -1.239 .216 time 0.006 0.002 2.976 .003 r2 = .016 note. type: 0: private condition; 1: kahoot! condition; time: the week that the quiz was taken. social validity table 2 displays the results of the questionnaire completed by 55 participants. when asked which assessment type the participants scored higher on, most indicated they did well on “both” (n=24) or that they did better on “private” (n=23). in response to which assessment type the participants studied for more, 95% reported “both” (n=52). when asked which quiz type increased their content knowledge, most participants reported “private” or “both” (n=20 for both responses). the majority (65%) of the participants reported they enjoyed the private condition over the kahoot! condition (n=38). most participants (73%) would prefer to take an assessment on a private device versus on kahoot! in the future. when asked if there was anything the participants disliked about the assessment conditions, the majority (89%) of the participant reported “yes” (n=49) for a dislike about kahoot! and 47% (n=26) reported “yes” for a dislike about private. the students provide additional information on what they liked and didn’t like about each condition as described below. the researchers coded the open-ended questions based on common themes across the answers. given that the participants could provide multiple answers for each question, the n score reflects the number of responses, not the total number of participants. the participants were asked to evaluate the components they liked for both assessment types. when asked about kahoot!, the majority (n=27) of the responses commented positively about the immediate feedback following each question. some of the participants reported that the points and scoring features of kahoot! made the assessment fun and motivating (n=10). fourteen participants reported that there was nothing they disliked about kahoot! the participants’ evaluations of the private condition highlighted that many participants (n=29) liked that the assessment was self-paced and there was not a per question time limit. additional positive responses included the non-distracting format, the similarity to traditional assessments (n=14), private scores (n=8), and the lack of pressure for the participants to respond quickly (n=7). the effects of gaming on university student quiz performance 116 journal of effective teaching in higher education, vol. 2, no. 1 table 2 social validity questionnaire responses question answers kahoot! private both on which quiz did you receive a higher score? 16% 42% 44% which quiz type did you study for more (duration, intensity, etc.)? 2% 4% 95% i feel my content knowledge about the chapter was stronger after taking the ______ quiz. 27% 36% 36% which quiz did you enjoy more? 25% 65% 4% in future classes, i would prefer to take quizzes on a _________format. 22% 73% n/a answers yes no did you dislike anything about the kahoot quizzes? n=49 (89%) n=6 (11%) did you dislike anything about the private quizzes? n=26 (47%) n=25 (45%) participants were asked to make suggestions for improving the assessments. the majority of comments about kahoot! recommended turning off or reducing the volume of the music during the assessment. many participants found the music distracting (n=24). additional recommendations including eliminating the time requirement for answers (n=8), being able to change answer selections (n=8), and removing the display of the number of people who answered incorrectly (n=6). several participants recommended using kahoot! for review instead of for a grade. the most commonly provided recommendation for the private condition was related to the format of the assessment. specifically, participants wanted to change answers on previously completed questions (n=21) and to skip and/or see all the questions at once (n=7). discussion similar to the findings of wang, zhu, and sætre (2016), there was no significant difference between private quiz performance and kahoot! quiz performance. students in both groups showed growth in quiz scores over the course of the semester, but these results were not related to the quiz condition. finally, students reported preparing for both testing conditions in the same way and reported a preference for the private condition over kahoot! this study extended the work of wang, zhu, and sætre (2016) by exposing students to both conditions multiple times. the effects of gaming on university student quiz performance 117 journal of effective teaching in higher education, vol. 2, no. 1 limitations when considering the results of this study, a few limitations should be considered. first, future researchers should consider using a larger sample with random sampling when comparing multiple methods for improving student performance on assessments. in addition, kahoot! carries some limitations as an assessment tool and should be used with caution. some students with disabilities could not participate in this study, as the nature of the group assessment does not allow for extended time or access to a distraction-free environment. the individuals with disabilities enrolled in these courses were offered pencil/paper tests to accommodate their needs and thus their scores were not included in the analysis. second, kahoot! audio was reportedly distracting to a lot of students as many requested the music be turned down, and several students muted the audio feedback on their personal devices. third, some students who are english language learners could not participate as they required more time to process the language presented on the screen and needed pauses for frequent clarification. implications for practitioners practitioners who adopt grss in their classroom should take the findings from this study into consideration. participants overwhelmingly preferred the assessments conducted in the private setting. the elements of kahoot! that are designed to be stimulating and enticing (e.g., music, feedback) were reported to be distracting to students and were not supportive of a typical quiz-taking environment. however, one important benefit to kahoot! was the immediate feedback students received during the assessment. students appreciated this immediacy of the answers provided when using kahoot! students assessed using the private condition were unable to access the answers to the quiz questions until after class. practitioners should consider providing immediate feedback during assessment whenever feasible. immediate feedback on a web-based platform could include availability of answers following the completion of a quiz, availability of answers following each question, or availability of answers after all students have completed a quiz. an additional option that does not include a web-based platform might have the instructor verbally reviewing the answers after an in-person assessment. in addition to immediate feedback, many participants indicated a desire to be able to go back to previous questions on the quiz. some students commented that they remembered the answer for a previous question after they submitted their answer or that they would have liked to skip and then return to a question. of course, if immediate feedback were given directly after the submission of an answer, backtracking would not be possible. practitioners might poll their students at the onset of the course to see where their preferences lie. an additional benefit of using kahoot! is the built-in data collection system. after each quiz is administered, a report of the results is generated and available for the instructor to download. the report includes specific information regarding how each student answered individual questions as well as percentage correct information for the effects of gaming on university student quiz performance 118 journal of effective teaching in higher education, vol. 2, no. 1 each question. these reports can provide the instructor with immediate information regarding the students’ comprehension of the content and highlight what areas may need to be re-taught or reviewed. this feature is a benefit whether kahoot! is being used as a formal or informal assessment. conclusion gsrs provide a fun and innovative way of getting students to be excited about learning in the classroom. the addition of games in the classroom can be refreshing to students in university and k-12 schools. however, when using a public, gamebased platform for formal assessment, students reported the format to be less preferable when compared to a private, more traditional platform. the findings from this study may indicate that kahoot! is best used to informally assess student learning and not as a formal method of assessment. practitioners should use kahoot! as a strategy for formally assessing their students with caution; however, game-based learning has its benefits and should be considered when reviewing content, preparing students for an upcoming exam, or other non-weight bearing activities to supplement instruction. kahoot! may also be most beneficial when used as a baseline assessment, a presentation tool to keep students engaged and check for understanding, or as a review or study tool for formal assessments. when developing formal assessments, instructors should consider learners with special needs and should also consider immediacy of feedback and allow for students to move freely between quiz questions during an assessment. references bartel, a., & hagel, g. (2014). engaging students with a mobile game-based learning system in university education. engineering education towards openness and sustainability: proceedings of the ieee global engineering education conference (educon) (pp. 957–960). istanbul: ieee. doi:10.1109/educon.2014.6826215 brookhart, s.m. (2004). assessment theory for college classrooms. new directions for teaching and learning, 100, 5–14. doi: 10.1002/tl.165 cagiltay, n. e., ozcelik, e., & ozcelik, n. s. (2015). the effect of competition on learning in games. computers & education, 87, 35–41. crocco, f., offenholley, k., and hernadez, c. (2016). a proof-of-concept study of game-based learning in higher education. simulation & gaming, 4, 403–422. doi: 10.1177/1046878116632484 johnston, j.m., & pennypacker, h.s. (2009) strategies and tactics of behavioral research (3rd ed.). new york, ny: routledge. kay, r. h., & lesage, a. (2009). examining the benefits and challenges of using audience response systems: a review of the literature. computers & education, 53, 819–827. doi: 10.1016/j.compedu.2009.05.001 neef, n.a., perrin, j.a., haberlin, a.t., & rodriques, l.c. (2011). studying as fun and games: effects on college students’ quiz performance. journal of applied behavior analysis, 44, 897–901. doi: 10.1901/jaba.2011.44-897 plump, c. m., & larosa, j. (2017). using kahoot! in the classroom to create engagement and active learning: a game-based technology solution for the effects of gaming on university student quiz performance 119 journal of effective teaching in higher education, vol. 2, no. 1 elearning novices. management teaching review, 2, 151–158. doi:10.1177/2379298116689783 rocca, k. a. (2010). student participation in the college classroom: an extended multidisciplinary literature review. communication education, 59, 185–213. robinson, f. f (2014). it’s a game! evaluation of a classroom game to enhance learning in an introductory counseling course. innovative teaching, 3, 10. doi:10.2466/o7.08.it.3.10 wang, a.i. (2015) the wear out effect of a game-based student response system. computers and education, 82, 217–227. doi: 10.1016/j.compedu.2014.11.004 wang, a. i., zhu, m., & sætre, r. (2016). the effect of digitizing and gamifying quizzing in classrooms. in t. connolly & l. boyle (eds.), proceedings of the 10th european conference on games based learning (pp. 729–737). paisley, scotland: academic conferences and publishing internatioal. wolf, m. m. (1978). social validity: the case for subjective measurement or how applied behavior analysis is finding its heart. journal of applied behavior analysis, 11, 203–214. doi: 10.1901/jaba.1978.11-203 microsoft word thibaut_final.docx journal of effective teaching in higher education, vol. 6, no. 1 https://doi.org/10.36021/jethe.v6i1.301 traditional, mixed-mode, and online instruction for microbial metabolism before and during covid-19 dylan thibaut, university of central florida, lake erie college of osteopathic medicine-bradenton kersten schroeder, university of central florida, kersten.schroeder@ucf.edu abstract. as covid-19 restrictions are removed, instructors are faced with two questions: should the course be taught online, via mixed-mode, or through traditional methods, and what effect did online instruction have on students in the course in regards regarding enrolled across eight semesters of metabolism courses using traditional, mixed-mode, and online methods. post-hoc analysis of a repeated-measures anova determined that while mixed-mode outperformed traditional methods on metabolism exams involving introductory concepts on the first exam (4.47 ±4.24, p= 0.012) and on the cumulative final exam (8.15 ±4.24, p= <0.001), traditional methods were superior to both mixed-mode (9.21 ±4.24, p= <0.001) and online methods (6.37 ±5.76, p= 0.006) at teaching alternative pathways of metabolism and lipid, nucleotide, and amino acid synthesis pathways. all methods showed poor student scores on glycolysis, tca cycle, oxidative phosphorylation topics. online methods performed nearly equally to mixed-mode, though there was not a significant difference on the first exam’s content as was found with mixed-mode. it is evident that, based on this study, a different approach to teaching basic central metabolic cycles such as glycolysis, the tca cycle, and the electron transport chain is necessary to improve student understanding of these topics. keywords: mixed-mode; covid-19; online; metabolism; microbial with regulations on instruction during covid-19 relaxing, instructors of upper-level undergraduate science courses like microbial metabolism are faced with two major questions: how did covid-19 impact the education of students and how should teaching transition from online instruction now that covid-19 no longer requires instruction to be fully online? it is no surprise that covid-19 has had dramatic effects on instructor perception and teaching approaches: teachers are often not seeing online instruction as beneficial, there are increased feelings of burnout, and there are difficulties in transitioning to online teaching (abdelmola et al., 2021; daumiller et al., 2021; dietrich et al., 2020). this difficulty was similarly felt by students, with questions of accessibility for students with disabilities, psychological health, and the influence of diverse backgrounds on learning all making teaching stem a challenge (gin et al., 2021; yu, 2021; hasan & bao, 2020; lynn et al., 2020). undergraduates have been shown to have decreased participation and engagement in college science courses before the pandemic; this has only gotten worse during the pandemic (wester et al., 2021; perets et al., 2020). even with attempts to adapt biochemistry and metabolism courses to novel online instruction, there are reports that some traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 82 methods have not been effective or have been inconclusive to improving student outcome (ossai et al., 2020). despite these negatives, however, there have been multiple successes with instruction for biochemistry and metabolism courses switching to online formats (kapil et al., 2021; contreras-rodríguez et al., 2020). with effective preparation and successful activities to apply biochemistry course content, there is research showing that students can perform just as well online as face-to-face instruction (keppetipola & patchen, 2021). even the laboratory content of biochemistry was shown to effectively be performed in online formats without having students experience gaps in knowledge (costabile, 2020; zewail-foote, 2020). successful strategies have included instructors and teaching assistants being available for students to assist in biochemistry, encouraging participation and collaboration amongst the students, and being prepared with strategies adapted to online teaching (procko et al., 2020). despite the negatives introduced by the covid-19 pandemic, there is evidence supporting that metabolism and biochemistry courses may be viable in an online format. conflicting ideas presented here call for an investigation as to how student scores were affected in metabolism courses when taught fully online versus different instructional methods of the past. this question is important for instructors as they transition away from the online-only courses during covid-19. instructors are faced with either continuing to teach online-only, going back to traditional methods of instruction, or trying less commonly used mixed-mode approaches. this scenario offers a prime opportunity for instructors to explore different teaching methods. unfortunately, the conflicting research surrounding each method of instruction (traditional, mixed-mode, and online instruction methods) as well as the lack of studies concerning online instruction for such high-level undergraduate sciences makes this question a difficult one for an instructor to make. while instructors may be familiar with online methods of instruction due to the recent pandemic or traditional teaching methods using lectures before the pandemic, the term mixed-mode may be less known. mixed-mode is a method of instruction which provides approximately half of class time online and half of class time in person, acting as a sort of mix between traditional in-person instruction and online methods of teaching. though there are several approaches to this method of instruction, arguably the most common structure is that of a flipped classroom: students learn lecture material delivered online, read their textbooks or readings, and listen to videos outside of class while in class, students apply the information using active learning collaborative activities (sajid et al., 2016). flipped classroom mixed-mode designs have shown gains in student problem-solving skills in upperlevel sciences, higher scores, and more preparation of students (klegeris, 2020; styers et al., 2018; gross et al., 2015). regarding metabolism and biochemistry courses specifically, students have higher grades, difficult concepts have been better understood by students, and students have improved upon their cooperative learning abilities (ren et al., 2020; francis et al., 2020; jafarkhani & jamebozorg, 2020; ojennus, 2016). with research supporting mixed-mode approaches like this, traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 83 a switch to mixed-mode instruction methods for teachers of metabolism courses stands as a valid choice. traditional lecturing also is a method many instructors are used to, and most certainly will be a comfortable pick for instructors returning to teaching in classrooms without covid-19 restrictions. one may argue that this method has been the most used form of teaching biochemistry and metabolism courses, and thus, it makes little sense to change what has been used in the past. for instructors wishing to branch off from lectures alone, beneficial results have been seen when instructors using traditional lecture teaching give short activities between lectures to check for student understanding (ghorbani & ghazvini, 2016; miller et al., 2013). instructors of metabolism courses can use online technology to enhance their traditional lecture content, allowing students to further increase understanding of lectures without requiring a change to another teaching method (henly & reid, 2001). traditional lectures combining technology may best for metabolism education post-covid-19, but this is ultimately something future research will examine. ultimately, instructors are left confused as to which instruction method to use as more questions arise from research than answers. one problem concerning research in different methods of instruction is that of consistency, as an overall increase in student gains from mixed-mode methods does not in itself mean that all metabolism topics are best taught in that format. another problem arises choosing between traditional or online instruction: how does online instruction differ from the traditional lecture-based approach of the past, and is it viable to continue teaching metabolism courses online? this study wishes to answer these questions. first, this study wishes to evaluate multiple semesters of data of students enrolled in undergraduate metabolism courses to understand which instructional method produces better scores for students. second, the effect of covid-19 online-only instruction will be evaluated to see if scores differ markedly from mixed-mode and traditional instructional methods; this will help instructors understand the effect covid-19 had on student understanding of metabolism concepts. finally, this study explores whether each teaching method is consistently better or worse for teaching metabolism topics based on performance on exams. methodology student participants from eight semesters of microbial metabolism were chosen for analysis. all eight semesters were taught by the same instructor at the same university. by choosing multiple semesters, data from both spring and fall was included, therefore limiting influence of the time of the course as a factor in student scores. two of the eight semesters were taught using a traditional teaching method (n= 367), three semesters were taught with a mixed-mode method (n= 429), and three semesters were taught using an online method of instruction during the covid-19 pandemic (n= 152). in total, 948 students’ scores on each of the four exams were used for analysis. traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 84 the semesters used for analysis included fall semester and spring semester metabolism courses to help eliminate time of semester instruction as a factor in student score. additionally, students were only included if there was an exam score available for all four exams, meaning that students who withdrew from the course were not included for analysis. semesters taught during covid-19 had markedly lower student enrollment and smaller class sizes in an online environment. more semesters would have been used for analysis to help balance the number of students being compared to be more equal across the three instruction method groups if possible, however, online-only education during covid-19 was restricted to those semesters alone. all microbial metabolism semesters had four exams covering the same learning objectives. the same question bank was consistently used across all semesters to preserve comparability. each exam had the same number of questions in every semester to make the scores more comparable as well. in person exams were proctored by the instructor and teaching assistants and online exams were proctored by proctoring software that locks down the exam takers browser and video monitors them during the exam. exam 1 content covered basic microorganism introductory material and review content from prerequisite science courses. exam 2 content covered central metabolism including enzymes, reactions, and substrates involved in glycolysis, the tca cycle, and oxidative phosphorylation. exam 3 covers alternative pathways of metabolism as well as pathways for lipid, nucleotide, and amino acid synthesis. exam 4 is the final, cumulative exam of the course with content from all previous exams as well as additional metabolic pathway content and quorum sensing not covered in exam 3. all student grade data was collected with approval by an irb, with grade data being de-identified before analysis. all data analysis and interpretation was performed by a separate researcher than the instructor to limit bias in interpretation. the course instructor always included an extra-credit question on each exam, allowing the total score on each exam to be slightly above 100%; this was consistent across all semesters. beyond this, the lowest exam score (except the final exam, exam 4) was always dropped from a student’s final grade calculation in each semester. this lowest score dropping caused students to ultimately get a zero on some exams on purpose as they were aware of the dropped score. these zeroes were viable grades for statistics and as such were still included in the analysis. to consistently teach each instructional method, the curriculum for each method was developed and used the same way for each semester. at the university where this study was conducted, the metabolism course used one of two schedules: classes on monday, wednesday, and friday for a 50-minute periods, or classes on tuesday and thursday for a 75-minute periods. the course was taught using one of these two options depending on the semester that the course was taught. the traditional method used every class time for lectures taught by the instructor of the course. for the mixed-mode method semesters, a flipped classroom approach was used modeling previous research (johanson, 2017). there would be one day in which class was online where students would watch pre-recorded lectures and take traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 85 a quiz on the content. the following day of the class was taught in person in which students applied the learned material from online to scenarios and activities. instruction during covid-19 was entirely online. students had online lectures on half of the days and online collaborative activities applying the material on the other half. as covid-19 progressed, literature on effective methods of instruction for biochemistry education online were considered (lapitan jr. et al., 2021). this approach also included the collaborative aspect to improve student biochemistry outcomes, increase positive perception of the course, and make the online environment more akin to what it would be like if the course were in person (wilson, 2021; fernández-santander, 2008). lecture content for the online course was mainly pre-recorded videos, allowing them to start or pause the videos according to their needs. this also gave students a resource to refer to when studying the material. a repeated measures anova was performed comparing grades on each of the four exams across each of the students enrolled in traditional, mixed-mode, and online semesters. results compared between the groups provides evidence supporting whether a difference exists between the teaching methods as well as whether that difference is consistent across all exams in the course. the repeated measures anova was checked for assumptions with mauchly’s test, and a type iii sum of squares design was used when performing the repeated measures anova. regarding post-hoc analysis, the holm method was used. in addition, a separate average for each exam was found to see how students perform on metabolism exams. this analysis was designed to highlight the topics most missed by students overall, making it easier for instructors to identify necessary changes. results upon analysis of the results via repeated measures anova, mauchly’s test of sphericity indicated a violation, χ²(5)= 405.60, p= <0.001. to compensate for this, a greenhouse-geisser correction was applied. the repeated measures analysis of variance with a within-subjects factor of semester instruction method (traditional, mixed-mode, online) and a between-subjects factor as exam (exam 1, exam 2, exam 3, exam 4) summarized in table 1 and table 2 showed semester instruction method had an effect on exam score f(2, 4.89)= 29.13, p= <0.001, η²p= 0.06. descriptive statistics are summarized in table 3 and represented in boxplots in figure 1. traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 86 table 1 between subjects effects cases sum of squares df mean square f p η² η² p semester 1079.69 2 539.84 0.83 0.44 8.38e -4 0.002 residuals 614478.22 945 650.24 note: type iii sum of squares. table 2 within subjects effects cases sphericity correction sum of squares df mean square f p η²p exam greenhousegeisser 67551.68 2.44 27657.84 112.05 < .001 0.11 exam ✻ semester greenhousegeisser 35119.17 4.89 7189.46 29.13 < .001 0.06 residuals greenhousegeisser 569697.58 2308.07 246.83 note: type iii sum of squares. ᵃ mauchly's test of sphericity indicates that the assumption of sphericity is violated (p < .05). table 3 descriptive statistics by instruction method exam semester mean sd n 1 mixed-mode 81.59 12.92 429 online 80.88 14.25 152 traditional 77.13 15.95 367 2 mixed-mode 67.66 19.29 429 online 68.16 19.20 152 traditional 67.81 23.98 367 3 mixed-mode 68.63 20.57 429 online 71.47 14.92 152 traditional 77.84 19.14 367 4 mixed-mode 71.78 14.26 429 online 71.82 14.33 152 traditional 63.63 16.26 367 traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 87 figure 1 boxplots note: data is divided based on exam number and compares the different results of each method of instruction. post hoc analysis results are summarized in table 4. a post hoc analysis with holm correction showed that exam scores on exam 1 were higher for participants taught using mixed-mode instruction versus traditional instruction (4.47 ±4.24, p= 0.012), but exam scores did not differ significantly between traditional and online instruction nor between mixed-mode and online teaching. no significant differences were found with exam 2 scores between semesters. exam scores on exam 3 were lower for participants taught using mixed-mode instruction versus traditional instruction (9.21 ±4.24, p= <0.001) and were lower for participants taught using online instruction versus traditional instruction (6.37 ±5.76, p= 0.006). exam 3 scores did not differ significantly between mixed-mode and online instruction (p= 1.000). exam scores on exam 4 were higher for participants taught using mixedmode instruction versus traditional instruction (8.15 ±4.24, p= <0.001) and were lower for participants taught using online instruction versus traditional instruction (8.19 ±5.76, p= 0.006). exam 4 scores did not differ significantly between mixedmode and online instruction. traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 88 table 4 post hoc analysis 95% ci: δ mean group 1 group 2 δ mean (group 21) lower upper se t p holm exam 1 mixedmode online 0.71 -4.92 6.34 1.67 0.43 1.00 traditional 4.47 0.22 8.71 1.26 3.55 0.01* online traditional 3.76 -2.00 9.51 1.71 2.20 0.50 exam 2 mixedmode online -0.49 -6.13 5.14 1.67 -0.30 1.00 traditional -0.14 -4.39 4.10 1.26 -0.11 1.00 online traditional 0.35 -5.40 6.11 1.71 0.21 1.00 exam 3 mixedmode online -2.84 -8.47 2.80 1.67 -1.70 1.00 traditional -9.21 -13.45 -4.97 1.26 -7.32 <0.001* online traditional -6.37 -12.13 -0.62 1.71 -3.73 0.006* exam 4 mixedmode online -0.04 -5.67 5.59 1.67 -0.02 1.00 traditional 8.15 3.90 12.39 1.26 6.47 <0.001* online traditional 8.19 2.43 13.94 1.71 4.80 <0.001* note: *p= <0.05 to determine how students overall performed on each exam all 948 students were combined into a single group, and an average exam score and standard deviation for each exam was found. findings are summarized in table 5. overall, students performed worst on exam 2 (m= 67.80, sd= 21.20). exam 4 scores were overall low as well (m= 68.63, 15.58). this lower score may be due to the cumulative nature of the exam, as it covered all content of the course. students who already failed a previous test or who did not understand a section previously most likely struggled with those same subjects when they appeared on the final, exam 4. exam 1 had the highest overall average (m= 79.75, sd= 14.51) which is unsurprising considering the prerequisite information included on this exam. exam 3’s average score was 72.65% (sd= 19.65). overall class grades were not used for statistical analysis; while the exams were graded the same way during all eight semesters, there were additional pre and post quizzes, extra assignments, and other differences in grading the course during mixed-mode and online semesters which may have artificially inflated the final grades of the students. by comparing exams rather than final grade in the class, this variable is eliminated, and only the exams that were consistent across all semesters were considered. traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 89 table 5: descriptive statistics with all students combined exam 1 exam 2 exam 3 exam 4 n 948 948 948 948 mean 79.75 67.80 72.65 68.63 std. deviation 14.51 21.20 19.65 15.58 discussion to answer the central questions of this study, an examination of two ideas was necessary. firstly, which instructional method or methods achieved higher scores compared to others on each exam of the microbial metabolism course? secondly, did students consistently get higher scores in that instructional method across the different exams of the course compared to other methods, or was the instructional method inconsistent in generating better scores across the four exams of the microbial metabolism course? results indicate that, rather than being consistently higher in score across all exams, instruction using mixed-mode methods produced higher scores on two of the four exams of the microbial metabolism course when compared to traditional instruction. mixed-mode instruction outperformed traditional instruction on exam 1, the test covering basic microorganism introductory material and review content from previous science courses, and exam 4, the cumulative final exam with additional metabolism concepts not covered in exam 3. interestingly, online instruction was closer to the average score found with traditional teaching, though it was not significantly different from either traditional instruction (p= 0.50) or mixed-mode instruction (p= 1.00). this establishes evidence towards mixed-mode teaching being effective at introducing basic microbial metabolism concepts in exam 1 as well as being a positive influence on reviewing past prerequisite material covered on the first exam of the course in comparison to online or traditional teaching. exam 4, which reviewed all content of the course with a final exam plus additional metabolic processes, similarly showed that mixed-mode teaching produced better results on exams compared to traditional teaching. exam 1 and exam 4 were both associated with reviewing past content (either from before the course as is the case with exam 1 or the entire semester’s content with exam 4), so results may support mixed-mode teaching serves a beneficial purpose in reviewing content. although mixed-mode teaching overall outperformed traditional instruction on half of the exams of this study, the other exams show a problematic result. regardless of method, students performed the same on exam 2 (central metabolism including enzymes, reactions, and substrates involved in glycolysis, the kreb’s cycle, and oxidative phosphorylation). this was the only exam with an average score of between 60-69% for all three methods. results suggest one of two conclusions as traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 90 to why teaching method did not change student score. firstly, the mixed-mode approach to teaching basic metabolic cycles may not be effective in the form it is being taught and if it were modified, it could possibly show higher scores. another possible conclusion is that central metabolism cycles are consistently difficult to teach regardless of teaching method. exam 3 showed the inverse of what occurred with exam 1 and exam 4, with traditional teaching methods leading to higher scores compared to the scores found when teaching with mixed-mode methods (p= <0.001). exam 3 covered topics including alternative pathways of metabolism as well as pathways for lipid, nucleotide, and amino acid synthesis; this raises a question as to what made the traditional teaching more effective compared to mixed-mode teaching when teaching these topics. as these metabolic processes are complicated with many steps, it may be that more guidance from instructors is necessary. the self-directed active learning of the mixed-mode course may either not be effectively applying these concepts, or the approach of mixed-mode teaching may not have enough teacher guidance to be effective with students. online teaching during the pandemic produced approximately the same scores as mixed-mode, performing better than traditional methods on exam 4 and performing worse than traditional methods during exam 3. strangely, exam 1 did not show a large enough difference in mean score to be substantially different from traditional methods (p= 0.502) despite the mean average online score on exam 1 being higher. regardless of this finding, online instruction managed to have students with higher scores than student scores during traditional methods on exam 4 by 8.19% (p= <0.001). despite the forced transition to online instruction, students were able to perform on-par or better than students who were taught by traditional and mixed-mode methods (except in exam 3, which students had higher scores when taught with traditional methods). students taught with online methods performed most like those taught with mixed-mode methods. though it may be due to the similarity with mixed-mode and online instruction which led to their close results, an alternative explanation may be the increased withdrawals from the semesters taught online during covid-19. with a higher-than-normal withdrawal rate from the microbial metabolism course, some of the lowest scoring students never completed the course and therefore did not have exam results available to compare in this study; this may have influenced the results in the online semesters. synthesizing the data from this study, one sees that despite mixed-mode instruction outperforming other methods on half of the exams of the microbial metabolism course, it is inconsistent in its effectiveness for certain exam topics. an alternative explanation may be that the approach used by the instructor was a poor way of adapting that teaching method. for example, the mixed-mode and online teaching may have both been taught in a similar, ineffective way for the material covered on exam 3. changes to the activities performed in the mixed-mode courses or modifications of online instruction may be necessary to improve the results. the opposite issue may have made the traditional method of teaching less effective on exam 1 and exam 4: traditional instructional methods may just have been poorly performed by the instructor rather than being the traditional method itself being traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 91 problematic. as this study used 948 students across eight semesters, this problem was hopefully mitigated. exam 2, which covered basic central metabolism cycles and topics, produced the same results across all teaching methods. combining all student scores across all teaching methods together and finding the average score of students, exam 2 had the lowest average score as well (67.80%). it is evident that a different approach to teaching basic central metabolic cycles such as glycolysis, the tca cycle, and the electron transport chain is necessary to improve student understanding of these topics. conclusion mixed-mode instruction for upper-level sciences for undergraduates is commonly seen as the more effective method of teaching compared to traditional teaching regarding overall student achievement, but just having an overall better outcome does not mean that mixed-mode is consistently better throughout an entire semester of teaching. the results of this study suggest that while mixed-mode instruction produced the strongest scores of students in the first and last exams of the microbial metabolism course, it was not effective at teaching alternative pathways of metabolism and pathways for lipid, nucleotide, and amino acid synthesis compared to traditional teaching methods during the two exams in the middle of the course. rather than suggesting that mixed-mode instruction is the optimal teaching method for microbial metabolism, results of this study suggest either changing the method of instruction from mixed-mode to traditional depending on the subject material of an exam or improving on the teaching approach when covering topics of metabolic cycles such as glycolysis, the kreb’s cycle, and alternative metabolic pathways to change the average or poor results seen in scores on exam 2 and exam 3. there is merit to using traditional instruction for certain metabolism concepts that should not be disregarded. an approach to teaching which embraces traditional teaching at some points, such as when alternative pathways are taught, and mixed-mode teaching at other points, such as at the start and end of the course, should be explored in future research. with central metabolism cycles being so challenging for students, research prevents several novel approaches to help students learn the material more effectively. some research-supported approaches have included interactive arts-and-crafts type activities to learn steps of metabolism, metabolic cycle board games, or interactive physical games (frança & campos, 2021; fishovitz et al., 2020; rose, 2011). alternatively, a case-based approach may work, as it connects the individual steps of metabolic cycles to real-world medical conditions and has been shown to be effective for learning biochemistry and metabolism (thibaut & schroeder., 2022; garcía-ponce et al., 2021; joshi et al., 2014; kulak & newton, 2014; nair et al., 2013). beyond this, additional media and representation methods can help cement student understanding of metabolic cycle content (long et al., 2021; wikandari et al., 2021). while it is unclear why students performed the worst on the exam covering these metabolic cycles, there may be individual student factors which can explored to improve understand of processes like the electron transport chain traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 92 (darabi et al., 2015). perhaps game-based learning, different active learning methods, or other ideas may be explored for this purpose. the proposed ideas here offer different approaches for instructors to pursue in their transition back to normal teaching after the covid-19 pandemic. while the online semesters during covid-19 managed to still do better than traditional instruction methods on the final exam, the same effect that mixed-mode instruction had compared to traditional was not as apparent on exam 1. the cause of this remains a question, though this study provides some evidence supporting that online teaching of microbial metabolism did not substantially lower student scores. it is possible that students performed worse in the prerequisite courses for microbial metabolism during covid-19, with their foundational knowledge covered on exam 1 less than that of students in other semesters. while not effecting the first semester of online instruction, it influenced the other semesters of online instruction. regardless of these results, it is the instructor’s decision as to how different teaching methods are used in the classroom. whether traditional methods or mixed-mode methods are adopted, an approach bringing in active learning, technology, or fun methods of approaching metabolism should be explored. giving up on traditional methods or fully switching to mixed-mode is not what this study suggests; instead, there is a need to be more open to adaptability. twenty-first century skills are expected from students, and it is up to instructors to bridge the gap in their science courses and use mutability in instruction to supply that need. conflict of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references abdelmola, a. o., makeen, a., hanafi, h. m., & ageeli, e. (2021). e-learning during covid-19 pandemic, faculty perceptions, challenges, and recommendations. mededpublish, 10, 112. https://doi.org/10.15694/mep.2021.000112.1 contreras-rodríguez, l. e., sandoval-hernández, a. g., maya-hoyos, m., & soto, c. y. (2020). remote learning of biochemistry during the covid-19 pandemic: case of undergraduate students in bogota, colombia. https://doi.org/10.21203/rs.3.rs-72902/v1 costabile, m. (2020). using online simulations to teach biochemistry laboratory content during covid-19. biochemistry and molecular biology education, 48(5), 509–510. https://doi.org/10.1002/bmb.21427 darabi, a., arrastia-lloyd, m. c., nelson, d. w., liang, x., & farrell, j. (2015). learning how the electron transport chain works: independent and interactive effects of instructional strategies and learners’ characteristics. traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 93 advances in health sciences education, 20(5), 1135–1148. https://doi.org/10.1007/s10459-015-9592-2 daumiller, m., rinas, r., hein, j., janke, s., dickhäuser, o., & dresel, m. (2021). shifting from face-to-face to online teaching during covid-19: the role of university faculty achievement goals for attitudes towards this sudden change, and their relevance for burnout/engagement and student evaluations of teaching quality. computers in human behavior, 118, 106677. https://doi.org/10.1016/j.chb.2020.106677 dietrich, n., kentheswaran, k., ahmadi, a., teychené, j., bessière, y., alfenore, s., & hébrard, g. (2020). attempts, successes, and failures of distance learning in the time of covid-19. journal of chemical education, 97(9), 2448–2457. https://doi.org/10.1021/acs.jchemed.0c00717 fernández-santander, a. (2008). cooperative learning combined with short periods of lecturing: a good alternative in teaching biochemistry. biochemistry and molecular biology education, 36(1), 34–38. https://doi.org/10.1002/bmb.20141 fishovitz, j., crawford, g. l., & kloepper, k. d. (2020). guided heads-up: a collaborative game that promotes metacognition and synthesis of material while emphasizing higher-order thinking. journal of chemical education, 97(3), 681–688. https://dx.doi.org/10.1021/acs.jchemed.9b00904 frança, v., & campos, w. f. (2021). interactive metabolism, a simple and robust active learning tool that improves the biochemistry knowledge of undergraduate students. advances in physiology education, 45(2), 353–364. https://doi.org/10.1152/advan.00042.2020 francis, n., morgan, a., holm, s., davey, r., bodger, o., & dudley, e. (2020). adopting a flipped classroom approach for teaching molar calculations to biochemistry and genetics students. biochemistry and molecular biology education, 48(3), 220–226. https://doi.org/10.1002/bmb.21328 garcía-ponce, á. l., martínez-poveda, b., blanco-lópez, á., quesada, a. r., suárez, f., alonso-carrión, f. j., & medina, m. á. (2021). a problem-/case-based learning approach as an useful tool for studying glycogen metabolism and its regulation. biochemistry and molecular biology education, 49(2), 236–241. https://doi.org/10.1002/bmb.21449 ghorbani, a., & ghazvini, k. (2016). using paper presentation breaks during didactic lectures improves learning of physiology in undergraduate students. advances in physiology education, 40(1), 93–97. https://doi.org/10.1152/advan.00137.2015 gin, l. e., guerrero, f. a., brownell, s. e., & cooper, k. m. (2021). covid-19 and undergraduates with disabilities: challenges resulting from the rapid transition to online course delivery for students with disabilities in traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 94 undergraduate stem at large-enrollment institutions. cbe—life sciences education, 20(3), ar36. https://doi.org/10.1187/cbe.21-02-0028 gross, d., pietri, e. s., anderson, g., moyano-camihort, k., & graham, m. j. (2015). increased preclass preparation underlies student outcome improvement in the flipped classroom. cbe—life sciences education, 14(4), ar36. https://doi.org/10.1187/cbe.15-02-0040 hasan, n., & bao, y. (2020). impact of “e-learning crack-up” perception on psychological distress among college students during covid-19 pandemic: a mediating role of “fear of academic year loss”. children and youth services review, 118, 105355. https://doi.org/10.1016/j.childyouth.2020.105355 henly, d. c., & reid, a. e. (2008). use of the web to provide learning support for a large metabolism and nutrition class. biochemistry and molecular biology education, 29(6), 229–233. https://doi.org/10.1111/j.15393429.2001.tb00129.x jafarkhani, f., & jamebozorg, z. (2020). comparing cooperative flipped learning with individual flipped learning in a biochemistry course. journal of medicine and life, 13(3), 399. https://doi.org/10.25122/jml-2019-0149 johanson, k. e. (2017). a topic-based approach for teaching metabolism in a flipped classroom. the faseb journal, 31, 751.13. https://doi.org/10.1096/fasebj.31.1_supplement.751.13 joshi, k. b., nilawar, a. n., & thorat, a. p. (2014). effect of case based learning in understanding clinical biochemistry. ijbar, 5(10), 516–518. https://doi.org/10.7439/ijbar.v5i10.920 kapil, a., gonzalez, l., & sikora, a. (2021). analysis of student learning gains in a biochemistry cure course during the mandatory covid-19 shift to online learning. the faseb journal, 35. https://doi.org/10.1096/fasebj.2021.35.s1.03494 keppetipola, n., & patchen, t. (2021). titrating teaching: an interdisciplinary case study of online and face-to-face undergraduate biochemistry instruction. journal of microbiology & biology education, 22(1), ev22i1–2603. https://doi.org/10.1128/jmbe.v22i1.2603 klegeris, a. (2020). mixed-mode instruction using active learning in small teams improves generic problem-solving skills of university students. journal of further and higher education, 1–15. https://doi.org/10.1080/0309877x.2020.1826036 kulak, v., & newton, g. (2014). a guide to using case-based learning in biochemistry education. biochemistry and molecular biology education, 42(6), 457–473. https://doi.org/10.1002/bmb.20823 traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 95 lapitan jr, l. d., tiangco, c. e., sumalinog, d. a. g., sabarillo, n. s., & diaz, j. m. (2021). an effective blended online teaching and learning strategy during the covid-19 pandemic. education for chemical engineers, 35, 116–131. https://doi.org/10.1016/j.ece.2021.01.012 long, s., andreopoulos, s., patterson, s., jenkinson, j., & ng, d. p. (2021). metabolism in motion: engaging biochemistry students with animation. journal of chemical education, 98(5), 1795–1800. https://doi.org/10.1021/acs.jchemed.0c01498 lynn, m. a., templeton, d. c., ross, a. d., gehret, a. u., bida, m., sanger, t. j., & pagano, t. (2020). successes and challenges in teaching chemistry to deaf and hard-of-hearing students in the time of covid-19. journal of chemical education, 97(9), 3322–3326. https://doi.org/10.1021/acs.jchemed.0c00602 miller, c. j., mcnear, j., & metz, m. j. (2013). a comparison of traditional and engaging lecture methods in a large, professional-level course. advances in physiology education, 37(4), 347–355. https://doi.org/10.1152/advan.00050.2013 nair, s. p., shah, t., seth, s., pandit, n., & shah, g. v. (2013). case based learning: a method for better understanding of biochemistry in medical students. journal of clinical and diagnostic research: jcdr, 7(8), 1576. https://doi.org/10.7860/jcdr/2013/5795.3212 ojennus, d. d. (2016). assessment of learning gains in a flipped biochemistry classroom. biochemistry and molecular biology education, 44(1), 20–27. https://doi.org/10.1002/bmb.20926 ossai, p. a. u., ogwu, c., ichipi-ifukor, p. c., & achuba, f. i. (2020). disposition and challenges of alternative to physical classroom teaching and learning in a pandemic; an appraisal of biochemistry students covid-19 learning with whatsapp. galician medical journal, 27(4), e2020411-e2020411. https://doi.org/10.21802/gmj.2020.4.11 perets, e. a., chabeda, d., gong, a. z., huang, x., fung, t. s., ng, k. y., & yan, e. c. (2020). impact of the emergency transition to remote teaching on student engagement in a non-stem undergraduate chemistry course in the time of covid-19. journal of chemical education, 97(9), 2439–2447. https://doi.org/10.1021/acs.jchemed.0c00879 procko, k., bell, j. k., benore, m. a., booth, r. e., moore, v. d. g., dries, d. r., & provost, j. j. (2020). moving biochemistry and molecular biology courses online in times of disruption: recommended practices and resources-a collaboration with the faculty community and asbmb. biochemistry and molecular biology education. 48(5), 421–427. https://doi.org/10.1002/bmb.21354 traditional, mixed-mode, and online instruction for microbial metabolism journal of effective teaching in higher education, vol. 6, no. 1 96 ren, x., lin sui, l., fei guan, y., kong, y., liu, b., fang ding, y., ... & kong, l. (2020). flipped classroom is an approach of basic medical education for undergraduate students. https://doi.org/10.1002/bmb.21427 rose, t. m. (2011). a board game to assist pharmacy students in learning metabolic pathways. american journal of pharmaceutical education, 75(9), 183. https://doi.org/10.5688/ajpe759183 sajid, m. r., laheji, a. f., abothenain, f., salam, y., aljayar, d., & obeidat, a. (2016). can blended learning and the flipped classroom improve student learning and satisfaction in saudi arabia? international journal of medical education, 7, 281. https://doi.org/10.5116/ijme.57a7.83d4 styers, m. l., van zandt, p. a., & hayden, k. l. (2018). active learning in flipped life science courses promotes development of critical thinking skills. cbe— life sciences education, 17(3), ar39. https://doi.org/10.1187/cbe.16-110332 thibaut, d., & schroeder, k. t. (2022). design of a semester-long case-based active learning curriculum for medical biochemistry courses during covid19. journal of chemical education, 99(7), 2541–2547. https://doi.org/10.1021/acs.jchemed.1c01126 wester, e. r., walsh, l. l., arango-caro, s., & callis-duehl, k. l. (2021). student engagement declines in stem undergraduates during covid-19–driven remote learning. journal of microbiology & biology education, 22(1), ev22i1– 2385. https://doi.org/10.1128/jmbe.v22i1.2385 wikandari, r., putro, a. w., suroto, d. a., purwandari, f. a., & setyaningsih, w. (2021). combining a flipped learning approach and an animated video to improve first-year undergraduate students’ understanding of electron transport chains in a biochemistry course. journal of chemical education, 98(7), 2236–2242. https://doi.org/10.1021/acs.jchemed.0c01477 wilson, k. j. (2021). online discussions in biochemistry to increase peer interactions and student interest. biochemistry and molecular biology education, 49(2), 298–300. https://doi.org/10.1002/bmb.21487 yu, z. (2021). the effects of gender, educational level, and personality on online learning outcomes during the covid-19 pandemic. international journal of educational technology in higher education, 18(1), 1–17. https://doi.org/10.1186/s41239-021-00252-3 zewail-foote, m. (2020). pivoting an upper-level, project-based biochemistry laboratory class to online learning during covid-19: enhancing research skills and using community outreach to engage undergraduate students. journal of chemical education, 97(9), 2727–2732. https://doi.org/10.1021/acs.jchemed.0c00543 microsoft word craig_final.docx journal of effective teaching in higher education, vol. 5, no. 2 https://doi.org/10.36021/jethe.v5i2.287 navigating the new normal: teaching in the time of covid clay m. craig, texas state university, cmc365@txstate.edu andrea m. bergstrom, coastal carolina university mary e. brooks, west texas a&m university shannon bichard, texas tech university abstract. the importance of staying relevant when teaching communication and advertising courses is hardly a new concept nor is the use of modern and evolving technologies to achieve this goal. however, empirically assessing college students’ perceptions of online learning when it is a forced choice is an underexplored area. to address this gap and consider the perspectives of diverse student populations, this paper analyzes data collected from a large communication class at a hispanic serving institution. the purpose of this research is to better understand students’ perceptions of teaching technologies, class sizes, benefits and hindrances of online learning, preferred course delivery methods, social adaptations, and perceptions of how their university has handled the transition. data revealed face-to-face courses were the most desired modality with no race or first-generation (fg) differences; however, gender differences were present. class size impacted modality preference as well as preference for the inclusion of specific online tools. additionally, a longitudinal comparison of student data between 2018 and 2020, before and after the onset of the global pandemic, revealed significant differences in student preparedness, faculty interaction, and institutional communication. keywords: pedagogy, covid-19, survey, diversity, online learning the covid-19 pandemic originated in december 2019 in china but quickly spread to countries in all corners of the world and became a global pandemic that continues to impact all areas of life, including higher education. as the pandemic escalated, prompting a national and international lockdown, students experienced a cataclysmic shift in the fall of 2020 across higher education institutions which offered primarily online delivery methods. previously, scholars have viewed online learning as a more modern form of distance learning with minimal live interaction between faculty and students (benson, 2002; conrad, 2002). more recently scholars believe there is a relationship between distance education and online learning, but that they are not necessarily one in the same (moore et al., 2011). the improvements to digital media and content delivery over the last decade have drastically changed how online learning is conducted and discussed. these online environments can include a variety of educational practices, but are often characterized by student-centered, active learning techniques including simulations, games, and new media on mobile platforms (keengwe & kidd, 2010). research examining online learning has reported improved learning based on test scores, student engagement, enhanced understanding, and a stronger sense of community (nguyen, 2015). additional benefits of online learning navigating the new normal 2 journal of effective teaching in higher education, vol. 5, no. 2 include convenience (fedynich, 2013), participation (morrison et al., 2019), and cost-effectiveness for universities (steen, 2008). while online learning provides numerous benefits, it is not without consequences and limitations. one disadvantage is that many students view online courses as individualized learning contributing to a sense of isolation from professors, classmates, and course material (boling, et al., 2012). others have noted how online learning hinders conversations with others and creates an impersonal atmosphere (kear, 2010; vonderwell, 2003). while the examination of online learning is extensive, an area that needs examination is how the pandemic midsemester forced transition from face-to-face to online course delivery impacted students. to address this gap and consider the perspectives of diverse student populations, this paper analyzes data collected from a large hispanic-serving southwestern university during a global pandemic to better understand students’ preferred course delivery methods, perceptions of teaching technology, class sizes, benefits and hindrances of online learning, social adaptations, and opinions of how their university handled the transition. modalities for over two decades a widely accepted practice across several universities has included online content delivery (manathunga, 2002; wernet et al., 2000) and the use of internet-based course content to replace classroom instruction (ahern & elhindi, 2002; brower, 2003; ponzurick et al., 2000) to meet the shifting demands of their student population. in fact, 77% of accredited institutions of higher education have turned to distance learning programs to deliver course work to students separated by time and physical space (parker et al., 2011). the modality of course delivery can be divided into five broad categories (finkelstein, 2006): 1. face-to-face (f2f) which is a traditional format where students and faculty are both physically present in the classroom at the same time for content delivery. 2. hybrid where part of the class is delivered in person and part is delivered online. 3. synchronous online which allows for live interaction between the instructor and the students (e.g., audio conferencing, videoconferencing, web chats etc.). 4. asynchronous online which involves significant delays in time between instruction and its receipt (e.g., email, prerecorded videos, discussion forums etc.). navigating the new normal 3 journal of effective teaching in higher education, vol. 5, no. 2 5. hyflex which integrates online and face-to-face instruction to create learning environments where students can attend in-person or from a distance simultaneously (angelone et al., 2020). as institutions were adapting to the evolving longevity of the covid-19 pandemic, they were forced to determine which modalities made the most sense given their resources and what was best for students. while it was clear that f2f was not a viable option, they had to assess which of the other options students would prefer. previous research has compared online and f2f course delivery; however, arias et al. (2018) notes the lack of random selection and modality preferences as limitations of accurate comparisons in delivery modality. if students are able to self-select modes of pedagogical delivery, the validity of comparative studies are limited although student self-selection of modality may not always be to their own advantage academically, socially, or financially. coates and humphries (2001) found younger students who actively select online courses may be at a disadvantage compared to more mature students. alpert et al.’s study (2015), an exception to the lack of random sampling comparing f2f, hybrid, and online, found that online students underperform relative to hybrid and f2f, and more specifically, disadvantaged students in the hybrid and online sections did worse than those in the f2f section. beyond performance, students have expressed concerns with online course delivery due to lack of motivation, challenges understanding the material, and decreased communication with the professor (alawamleh et al., 2020). in contrast, students appreciate the ease of participation facilitated by mixed learning environments from blogs to learning management systems to online polls (morrison et al., 2019). teaching technologies while the importance of knowledge acquisition and learning has not changed, the tools being utilized have evolved. a 2002 pew internet and american life project report found that college students were early adopters of the internet, which enhanced their education (jones, 2002). educational practices of discussion, training, storytelling, and using video for assignments and research have been around for decades (altun, 2017), but how these practices are delivered and shared has evolved with online technological advances. while several tools can be used across modalities, academicians are divided on how and if online course material should be delivered differently than f2f. some advocate creating online courses that emulate traditional f2f classroom experiences while others argue that equivalency is impossible because the medium shifts the dynamics of teaching and learning within virtual classrooms (donovan et al., 2011). henderson et al. (2018) explored the effectiveness of 14 teaching tools in f2f versus online courses for mba students. the authors found that f2f classes rated guest speakers and team presentations as more effective while online classes viewed tutorials, textbooks, and online activities as more effective for academic success. the two most important factors that demonstrated a significant impact on navigating the new normal 4 journal of effective teaching in higher education, vol. 5, no. 2 the delivery of online content included the technology used and the design of the course (kampov-polevoi, 2010). assessing the forced shift to online classes during the covid-19 pandemic, major (2020) revealed that zoom has a positive impact on online learning experiences due to the direct communication among students and lecturers, which resembles f2f teaching. in an online course, discussion is an important component that can impact effectiveness (maddix, 2012) and can take many forms from in-class chats and polls to asynchronous discussion boards through learning management systems. as the proliferation of online teaching tools continues, students’ perceptions of the benefits of current tools should be assessed. diverse student populations and varied educational experiences it is important to note that the student experience is not universal and cannot be treated as such. in fact, the population of students enrolled in colleges and universities nationwide has shifted notably within the past decade. the rise in hispanic student enrollment in higher education reflects overall demographic shifts within the u.s. population, and a record number of hispanic students (3.8 million) enrolled in colleges in 2019, increasing 287% since 2000 (mora, 2022). highlighting the importance of researching perceptions at a hispanic serving institution because the number of such universities is likely to increase in the future. additionally, by the 2015–2016 academic year, first-generation (fg) students, defined as those whose parents had not obtained an education beyond high school (tate et al., 2015), accounted for more than half (56%) of all undergraduate students nationally (rti international, 2019). educators should recognize that fg students report feeling less prepared for college-level courses and are in greater need of remedial coursework (reid & moore, 2008). furthermore, beyond being the first among their families to attend college, fg students are also more likely to be part of racial and ethnic minority groups (bui, 2002) and tend to have been raised in lower income households than their peer counterparts (gibbons & woodside, 2014). these two additional factors should be considered within the context of higher education and facilitating teaching and learning experiences for fg students. previous research examining the impact of gender on performance and satisfaction in online learning environments is sparse and conflicting (dousay & trujillo, 2019). bolliger and supanakorn (2011) found that females prefer multimodal learning (consistent with online learning) more than males while wehrwein et al. (2007) found exactly the opposite. luik (2011) notes how it was previously thought that females and males think, feel, and behave differently with technology because of different technology use preferences, habits, and computer literacy characteristics based on gender. technology use for males often involves leisure activities while females use technology to complete specific tasks (dousay & trujillo, 2019). this may indicate that female learners are more at ease using technology specifically for learning purposes (luik, 2011; nistor, 2013) as opposed to technology use for a hobby or entertainment activity, a frequent occurrence for males. navigating the new normal 5 journal of effective teaching in higher education, vol. 5, no. 2 online class size prior to the pandemic, educators wrestled with how to best deliver quality educational experiences to their students while adapting to changing pressures both internal and external to the university. one such pressure comes from university administrators who are concerned with fiscal responsibility (as well as student learning) and view increased class sizes and alternative modalities as a way to save schools and programs money (mitchell & leachman, 2015; tomei, 2006). this poses a challenge for faculty, as research suggests large class sizes lead to higher dropout rates, lower retention and attendance, increased cheating, reduced breadth and depth of subject matter, less instructor–student interaction, less instructor feedback, increased reliance on the lecture, and less student involvement (russell & curtis, 2013; saiz, 2014). class size and its impact on online course delivery has been studied with mixed results (russell & curtis, 2013). bettinger et al. (2017) found negligible impact of class size on grades and retention while cavanaugh (2005) found the addition of even a single student increased instruction time dramatically. based on existing research, there is room for additional inquiry related to student perceptions of online class sizes and effective teaching tools. institutional response the covid-19 crisis influenced students and higher education institutions across the u.s. in early 2020 with an abrupt nationwide transition to online learning. the disruption of society due to covid-19 continued as universities struggled to satisfy the conflicting wants and needs of their varied communities (kennedy, 2020), attempting to balance concerns about reopening so that businesses can begin to recover or to keep classrooms empty to mitigate bringing large groups of students together in confined spaces. given the unprecedented health and safety concern, universities scrambled to create safe and healthy learning environments through masks, gallons of soap and disinfectants, minimized classroom capacities, and social distancing educational campaigns. the constant evolution of information about the virus forced best practices to adapt quickly and university plans to change frequently. academic institutions had to make difficult decisions not only regarding educational delivery but also in how they responded to the larger social, cultural, and financial implications of the pandemic. based on the previous research, the following research questions are proposed: rq1: what are students’ general perceptions of course modality and online teaching tools in pandemic contexts? rq2: are there any (a) gender, (b) racial, or (c) fg differences in modality preference in pandemic contexts? rq3: does the preference of (a) modality or (b) online tools differ based on class size in pandemic contexts? navigating the new normal 6 journal of effective teaching in higher education, vol. 5, no. 2 rq4: how do students perceive (a) university allocation of resources, (b) level of preparedness, (c) level of interactions, and (d) the university’s response to covid-19? rq5: how have students’ perceptions of (a) university allocation of resources, (b) level of preparedness, and (c) level of interactions changed over time? methods to address the above-mentioned research questions, an electronic survey was administered during the fall 2020 semester to students at a hispanic serving institution. after receiving irb approval, the survey was administered, resulting in 332 participants recruited from a large introductory advertising course. the questionnaire was designed and administered through qualtrics, and respondents received in-class participation credit for completing the survey. the sample consisted of predominantly females (66%) and most students were juniors (40.7%) and seniors (32.2%). the majority of participants were caucasian (51.2%) followed by latino (27.1%) and african american (13%). the age of the sample ranged from 18 to 37, with a mean age of 21.55 (sd = 2.79). additionally, 34.3% identified as fg college students. beyond demographic items, the following measures were used. modality. the categories of modality of course delivery were adapted from finkelstein’s (2006) prior research (f2f, hybrid, synchronous online, asynchronous online, and hyflex). participants were asked a series of questions regarding the five modalities, including if they have ever taken a college course using any of the modalities (yes or no), were they currently taking college courses using any of the five modalities (yes or no), and the format they preferred for course delivery (fivepoint likert scale from “like a great deal” to “dislike a great deal”). additionally, participants were asked which modality they preferred based on class size where they selected only their top preference: large survey class (100+), average (20– 40), small seminar (<12), and skills/technology-based class. online teaching tools. henderson at al.’s (2018) nine online teaching tools were assessed by students using a five-point likert scale ranging from “extremely effective” to “not effective at all.” the tools included pre-recorded lectures, breakout rooms (in-class activities, virtual class sessions, virtual office hours, and online group projects, current events/examples, discussion posts on learning management systems, online polls/chat tool, and online class discussions. additionally, participants were asked if they viewed each online teaching tool as effective based on class size (large survey class (100+), average (20–40), small seminar (<12), and skills/technology-based class) where they selected all that applied. student engagement. student engagement was assessed based on the national survey of student engagement (nsse) survey using a five-point likert scale from navigating the new normal 7 journal of effective teaching in higher education, vol. 5, no. 2 “never” to “always.” each subcategory used the following question: during the current school year, about how often have you done the following? 1. student preparedness: (a.) asked questions or contributed to course discussions in other ways, (b.) asked another student to help you understand course material, (c.) explained course material to one or more students, (d.) prepared for exams by discussing or working through course material with other students, (e.) worked with other students on course projects or assignments, and f. given a course presentation. 2. critical thinking: (a.) combined ideas from different courses when completing assignments, (b.) connected your learning to societal problems or issues, (c.) included diverse perspectives (political, religious, racial/ethnic, gender, etc.) in course discussions or assignments, (d.) examined the strengths and weaknesses of your own views on a topic or issue, (e.) tried to better understand someone else's views by imagining how an issue looks from their perspective, (f.) learned something that changed the way you understand an issue or concept, and (g.) connected ideas from your courses to your prior experiences and knowledge. 3. faculty engagement: (a.) talked about career plans with a faculty member, (b.) worked with a faculty member on activities other than coursework, (c.) discussed course topics, ideas, or concepts with a faculty member outside of class, (d.) discussed your academic performance with a faculty member. quality of interactions. the quality of interactions was examined by having students respond to the following statement: “indicate the quality of your interactions with the following people at your institution since you started college” for the following groups: (a.) students in your major, (b.) other students on campus, (c.) academic advisors, (d.) faculty in your major, (e.) other faculty on campus, (f.) student services staff, and g. other administrative staff and offices (using a five-point likert scale ranging from “horrible” to “excellent »). institutional focus. participants provided their perceptions of how well the institution focused on different initiatives on a five-point likert scale ranging from “not well at all” to “extremely well.” the specific items assessed were as follows: (a.) spending significant amounts of time studying and on academic work, (b.) providing support to help students succeed academically, (c.) using learning support services, (d.) encouraging contact among students from different backgrounds, (e.) providing opportunities to be involved socially, (f.) providing support for your overall well-being (recreation, health care, counseling, etc.), (g.) helping manage your non-academic responsibilities (work, family, etc.), (h.) attending campus activities and events, and (i.) attending events that address important social, economic, or political issues. additionally, participants were asked specifically how the institution was handling the pandemic based on a five-point likert scale ranging from “not well at all” to navigating the new normal 8 journal of effective teaching in higher education, vol. 5, no. 2 “extremely well” for the following items: (a.) providing access to technology to those in need, (b.) providing technical support to help students succeed, (c.) appropriately accommodating students during the pandemic, (d.) providing timely and relevant communication with the campus community overall, (e.) providing timely and relevant communication with the campus community about the university’s response during the pandemic, and (f.) providing financial support for tuition and housing. discussion of findings the purpose of this study was to evaluate students’ preferences and perceptions related to the forced shift to online learning during the global covid-19 pandemic. the first research question examined participants’ general perceptions of the modality of classes. analysis revealed face-to-face (f2f) was the most preferred (m = 3.89, sd = 1.24) followed by synchronous online (m = 3.71, sd = 1.20), asynchronous online (m = 3.71, sd = 1.20), hybrid (m = 3.30, sd = 1.30), with hyflex (m = 3.29, sd = 1.22) being the least preferred modality. in fact, 52.6% liked f2f a great deal while only 17.2% had the same sentiment about hyflex. based on the data, f2f courses were the most preferred class modality across the student sample, indicating that students will likely want to return to traditional classroom learning environments once it is safe to do so. when f2f courses are not an option, students prefer to meet with their classes in synchronous online course sessions, probably due to the real time communication, instantaneous responses, and connections with their academic community. if the trend to provide more online and hybrid courses continue, concerns about lack of motivation, challenges understanding the material, and decreased communication with the professor (alawamleh et al., 2020) will need to be addressed. beyond modality, faculty must determine the number and type of online tools to use in their classes. the second part of research question one evaluated students’ perceptions of the effectiveness of online teaching tools during the pandemic. data showed that students viewed online polls/chat tools to be the most effective (m = 3.87, sd = 1.03) followed by virtual class sessions (m = 3.70, sd = 1.03), online class discussion (m = 3.68, sd = 1.08), virtual office hours (m = 3.66, sd = 1.10), discussion posts on lms (m = 3.42, sd = 1.21), current events/examples (m = 3.38, sd = 1.13), breakout rooms (m = 2.86, sd = 1.25), and online semester long projects (m = 2.41, sd = 1.28). the tools viewed most effective supports maddix’s (2012) argument that in online classes the most important determinant of success is discussion and interaction not only between faculty and students, but between students as well. while there is not a “one size fits all” approach to online tools, at least one tool should help facilitate discussions in real time and allow for interpersonal interactions between students and faculty to take into account students’ perceptions of these pedagogical tools. the second research question examined if there were any differences in modality preferences between gender, race, and fg students. results showed differences in gender but not race or fg students. such findings indicate that understanding the demographic breakdowns at universities and within individual majors and programs navigating the new normal 9 journal of effective teaching in higher education, vol. 5, no. 2 may prove to be a useful tool for educators looking to best match student enrollment preferences with demographic characteristics. using independent sample t-tests, analysis revealed significant differences between males and females for f2f and synchronous online, with no significant differences for asynchronous online (t(325) = 1.87, p = .06), hybrid (t(324) = .08, p = .94), or hyflex (t(324) = .46, p = .64). males (m = 4.10, sd = 1.08) preferred f2f significantly more than females (m = 3.77, sd = 1.30) t(248.62) = 2.46, p = .02; however, females preferred synchronous online (m = 3.81, sd = 1.2) significantly more than males (m = 3.51, sd = 1.19), t(325) = 2.16, p = .03. in contrast, oneway between subjects anovas were conducted for each modality revealing no differences in modality preference based on race or fg status. results for race and modality preferences include the following: f2f [f(6, 324) = 1.32, p = 0.25], synchronous [f(6, 325) = 1.59, p = 0.15], asynchronous [f(6, 325) = 1.85, p = 0.48], hybrid [f(6, 324) = .22, p = 0.97], and hyflex [f(6, 324) = .16, p = 0.99]. similarly, the results for fg students are as follows: f2f [f(1, 329) = 1.39, p = 0.24], synchronous [f(1, 330) = .09, p = 0.77], asynchronous [f(1, 330) = .05, p = 0.82], hybrid [f(1, 329) = .52, p = 0.47], and hyflex [f(1, 329) = .22, p = 0.64]. the next research question (rq3) examined the impact of class size on modality and online tool preferences. students overwhelmingly preferred f2f for traditional/average size (41%), seminar (47.9%), and skills/technology based (57.5%) classes while for large survey classes there wasn’t a clear preference with only 28% selecting synchronous online and 25% choosing f2f. it should be noted that this survey was conducted during the first fully remote semester during the pandemic. as such, the desire to go back to normal may have influenced the preference for f2f courses. class size impacted students’ perceptions of online tool effectiveness. for large classes, most students found pre-recorded lectures (70.2%) and online polls/chats (64.8%) effective. in contrast, for traditional/average sized classes, discussion posts on lms (59.9%) and online class discussions (60.2) were the most effective. for seminar and skills-based classes, virtual office hours were selected the most with 48.2% and 35.5%, respectively. table 1 provides a complete breakdown of online tool preferences based on class size. these findings shift the focus from the outcomes regarding retention and attendance to what tools might help improve these outcomes. as faculty and administrators are grappling with pressure to increase class sizes, shifts in the online teaching tools used in larger classes with emphasis on prerecorded lectures and polls/chats are needed. the fourth research question assessed students’ perceptions of the university’s focus and allocation of resources, students’ level of preparedness and critical thinking, and students’ level of interaction (see table 2 for detailed results). overall, students felt that the university was strongest in providing learning support services and providing support to help students succeed academically. this is not surprising as most institutions had these services and opportunities prior to the pandemic. the areas where students felt there was a need for improvement were navigating the new normal 10 journal of effective teaching in higher education, vol. 5, no. 2 with helping to manage non-academic responsibilities and encouraging the attendance at campus activities and events. this reflects an opportunity for universities to provide a more holistic level of student support that goes beyond focusing solely on academics. table 1 online tool preferences based on class size tool large traditional seminar skills pre-recorded lectures 233 (70.2%) 151 (45.5%) 100 (30.1%) 111 (33.4%) breakout rooms 79 (23.8%) 164 (49.4%) 154 (46.4%) 71 (21.4%) virtual class sessions 179 (53.9%) 192 (57.8%) 126 (38%) 84 (25.3%) virtual office hours 181 (54.5%) 184 (55.4%) 160 (48.2%) 118 (35.5%) online group projects 72 (21.7%) 138 (41.6%) 144 (43.4%) 75 (22.6%) current events/examples 152 (45.8%) 181 (54.5%) 140 (42.2%) 88 (26.5%) discussion posts on lms 149 (44.9%) 199 (59.9%) 132 (39.8%) 91 (27.4%) online polls/chat 215 (64.8%) 189 (56.9%) 127 (38.3%) 91 (27.4%) online discussion 149 (44.9%) 200 (60.2%) 154 (46.4%) 96 (28.9%) note. values represent the number of participants that viewed the tool as effective for each class size. participants could select multiple tools for each class size, so percentages do not equal 100. regarding student preparedness and critical thinking, the most common activity students engaged in was asking questions or contributing to course discussions while the least common was attending an art exhibit, play, or other arts performances. students overwhelmingly thought they were excelling in criticalthinking areas with the lowest ratings related to including diverse perspectives in course discussions or assignments (m = 3.38) and the highest being connecting ideas from your courses to your prior experiences and knowledge (m = 3.65). navigating the new normal 11 journal of effective teaching in higher education, vol. 5, no. 2 the third part of research question four examined the frequency and quality of interactions students had with other members of the university community. data revealed limited interaction with faculty about academic performance, career plans, course topics outside of class or working with faculty on activities other than coursework. while interaction with faculty may not be frequent, the quality of interaction with faculty in their major was rated highest followed interactions with academic advisors, students in their major, other students on campus, other faculty and college staff, and administrative staff and offices. the pandemic limited the amount and type of interaction students had with university faculty and staff, but the quality of the interaction was still viewed as favorable. this finding provides evidence that even if faculty and staff are not interacting with students in person, the quality of the interaction and exchange has a positive influence on student perceptions. next, students’ perceptions of how well the university focused on different aspects of student life during the pandemic was evaluated. data shows several areas for improvement with students’ perceptions all falling below average. the highest praise was for providing timely and relevant communication with the campus community (m = 3.15, sd = 1.11) and providing technical support to help students succeed (m = 3.14, sd = 1.12) followed by providing timely and relevant communication with the campus community about the university’s response (m = 3.11, sd = 1.16), providing access to technology to those in need (m = 3.07, sd = 1.13), appropriately accommodating students during the pandemic (m = 2.98, sd = 1.21), and, finally, providing financial support for tuition and housing (m = 2.80, sd = 1.27). results demonstrate the need for universities to have a clear communication plan about each of the different initiatives they are using so that students are informed and feel as though the institution is taking a proactive approach to ensuring student wellbeing. research question five examined whether students’ perceptions changed over two years regarding the university allocation of resources, student level of preparedness and critical thinking, and overall quality of interactions. data collected in fall 2018 from the same course during the same week of the semester was compared with the fall 2020 data. prior to comparative analysis, a series of independent samples t-tests revealed no significant differences in race (t(623) = .39, p = .70), gender (t(622) = 1.32, p = .31), gpa (t(623) = 1.02, p = .31), age (t(617) = .29, p = .78), or class level (t(623) = .47, p = .64). this allowed for a direct comparison between the two samples. interestingly, students’ positive perceptions of the university’s allocation of resources were significantly higher in 2020 than 2018 (see table 2). in contrast, students’ self-reported levels of preparedness and critical thinking was significantly lower in 2020 compared to 2018. the researchers believe this finding is especially worthy of consideration as this may be an important indicator of the ongoing toll that the prolonged conditions of the global pandemic are having on student learning within the realm of higher education. navigating the new normal 12 journal of effective teaching in higher education, vol. 5, no. 2 while the level of faculty interaction was reported to be significantly lower in 2020 than in 2018, students rated the quality of interactions with their faculty as significantly higher and responded with overall praise for prompt and thorough communication during the pandemic. because these are unprecedented times within higher education, such findings indicate the commitment of university educators and higher education as a whole to continue to serve students to the best of their ability, and findings indicate that students recognize and appreciate these efforts. table 2 results of t-tests & descriptive by year outcome year year *.05 2018 2020 **.01 m sd n m sd n t df *** .001 prepared 2+ drafts 2.88 1.25 292 2.48 1.31 331 3.89 618.54 *** attended art 2.52 1.30 293 1.64 1.11 330 9.04 577.20 *** asked student for help 3.07 1.06 293 2.74 1.24 331 3.59 621.39 *** explained course material 3.22 1.03 293 2.71 1.17 332 5.73 623 *** prepared for exams with other students 3.25 1.11 293 2.66 1.33 331 5.95 619.4 *** worked with others on course projects 3.24 1.05 293 2.8 1.26 331 4.70 619.95 *** gave a presentation 2.58 1.35 293 2.3 1.31 332 2.62 623 ** combined ideas 3.74 0.93 293 3.51 1.12 331 2.89 619.2 ** connected learning to societal problems 3.66 0.98 293 3.47 1.10 331 2.33 622 * navigating the new normal 13 journal of effective teaching in higher education, vol. 5, no. 2 included diverse perspectives in course discussion or assignments 3.66 1.01 293 3.38 1.17 331 3.11 621.71 ** examined own views on a topic or issue 3.69 0.92 293 3.49 1.07 331 2.59 621.69 ** tried to understand others' views 3.88 0.89 293 3.55 1.07 331 4.25 619.22 *** learned something that changed the way you understand an issue 3.76 0.86 293 3.41 1.04 331 4.60 618.92 *** connected ideas from courses to prior knowledge 3.86 0.88 293 3.65 1.04 331 2.78 621.12 ** faculty interaction outcome year year 2018 2020 m sd n m sd n t df career plans 2.68 1.17 293 2.24 1.17 331 4.65 622 *** activities outside of course 2.37 1.27 293 2 1.17 331 3.73 622 *** course topics outside of class 2.69 1.22 293 2.23 1.22 331 4.77 622 *** academic performance 2.86 1.16 293 2.39 1.23 331 4.92 619.87 *** quality of interactions navigating the new normal 14 journal of effective teaching in higher education, vol. 5, no. 2 outcome year year m sd n m sd n t df academic advisors 3.72 1.10 289 3.9 1.03 322 2.175 609 * other faculty on campus 3.5 1.10 283 3.71 1.00 300 -2.35 581 * student services 3.49 1.08 283 3.71 1.00 298 -2.58 579 ** other administration 3.39 1.15 285 3.69 0.99 311 -3.42 561.60 *** institution emphasis outcome year year 2018 2020 m sd n m sd n t df using learning support services 2.99 0.67 293 3.42 1.04 331 -6.21 572.07 *** encouraging contact among students from different backgrounds 2.82 0.79 293 3.31 1.082 331 -6.46 600.45 *** providing opportunities to be involved socially 2.98 0.67 293 3.26 1.11 331 -3.83 553.99 *** providing support for overall wellbeing 2.85 0.73 293 3.25 1.09 331 -5.55 582.18 *** navigating the new normal 15 journal of effective teaching in higher education, vol. 5, no. 2 helping manage nonacademic responsibilities 2.67 0.87 293 2.93 1.14 331 -3.16 608.26 ** attending campus activities and events 2.93 0.72 293 3.09 1.09 331 -2.13 579.36 * attending events that address important social, economic, or political issues 2.86 0.74 293 3.11 1.13 331 -3.42 575.37 *** implications and future research the current research revealed a number of interesting findings regarding students’ perceptions of online classes, preferences for teaching tools, and their perceptions of interactions with university faculty and staff members during the pandemic. while this research offers an important snapshot of student perceptions at the start of the covid-19 pandemic, it can also help universities and faculty evolve their approach to course material delivery. modality preference revealed f2f was the most preferred overall. when examined based on class type, students overwhelmingly preferred seminar-style courses and skills/technology focused classes in the f2f learning environment while synchronously online was preferred for large survey courses. this is likely due to the less personalized nature of interactions within these classes and a lower likelihood of engaging as individuals. students need to engage with their peers, and when it is necessary to move online, educators must find appropriate tools to offer them connection. this body of work contributes to the sparse and conflicting previous research examining the impact of demographic characteristics on performance and satisfaction in online learning environments (dousay & trujillo, 2019). female student’s preference for synchronous online learning more is consisted with bolliger and supanakorn’s (2011) finding that females prefer multimodal learning more than males. this could be due to the fact that female learners are more at ease using technology specifically for learning purposes (luik, 2011; nistor, 2013). while there were no racial or fg differences, it is important that future research continues to explore these characteristics as college student populations continue to diversify. navigating the new normal 16 journal of effective teaching in higher education, vol. 5, no. 2 in terms of online teaching tools, students found online polls/chat tools to be the most effective tool within their online classroom spaces. in contrast, breakout rooms and semester-long projects were perceived by students to be the least beneficial. this study supports previous findings that students prefer a more interactive learning environment that allows them to connect with their peers and professors (henderson et al., 2018; major, 2020). future research should continue to explore students’ rationale for citing these preferences for learning modalities and tools and examine whether students’ preferences are in fact correlated with academic performance and success. within these varied class structures, students’ perceived effectiveness of the specific online tools also varied based on class size, making it important for instructors to consider both the course modality as well as the size of the class before choosing specific online engagement tools. matching online tools successfully with specific courses of varied size is an important consideration as it appears to impact student likelihood for engagement in online environments. further study should address the circumstances and tools that facilitate the most effective online learning and interaction. students’ self-reported critical thinking measures revealed that they believed they were meeting or exceeding expectations; however, it is important to note that there was a decline in their self-perceptions from 2018 to 2020. one possible reason for this decrease is that students’ level of preparedness also showed a steep decline, specifically in relation to working with other students. a possible side effect of the isolating nature of forced online classes is the inability of students to form social bonds digitally in the same way they used to in person. future research is needed to examine the impact that online course work has on forming peer relationships among students. as data for this analysis is based on students’ selfperceptions, future research should consider incorporating additional measures beyond self-report items for critical thinking and preparedness in order to externally assess students in these areas both for f2f and online courses. when exploring students’ perceptions of the university during the pandemic, students indicated that resources were effectively allocated for learning support and academics; conversely, they noted fewer resources for non-academic support (such as campus activities and events). this is understandable due to the health and safety limitations during the pandemic. it is important to understand moving forward that students appreciate a more holistic commitment to their well-being, beyond just in academics. additionally, findings revealed more limited interactions with university faculty, advisors, and other staff members during the forced shift to online classes although the quality of interaction with individuals from these groups was perceived positively by students. this is an important finding in regard to the additional efforts that many areas of higher education undertook in order to aid in facilitating the continued student success within unprecedented teaching and learning circumstances. future research exploring the specific nature of these interactions may provide additional insights as to best practices to carry over beyond the pandemic, if and when it finally concludes. navigating the new normal 17 journal of effective teaching in higher education, vol. 5, no. 2 conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references ahern, t. c., & el-hindi, a. e. (2002). improving the instructional congruency of a computer-mediated small-group discussion: a case study in design and delivery. journal of research on computing in education, 32(3), 385–400. alawamleh, m. al-twait, l. m., & al-saht, g. r. (2020). the effect of online learning on communication between instructors and students during covid-19 pandemic. asian education and development studies, 11(2), 380–400. https://doi.org/10.1108/aeds-06-2020-0131 alpert, w. t., couch, k. a., & harmon, o. r. (2015). a randomized assessment of online learning [unpublished manuscript]. department of economics, university of connecticut. altun, m. (2017). the effects of teacher commitment on student achievement: a case study in iraq. international journal of academic research in business and social sciences, 7(11), 417–426. angelone, l., warner, z., & zydney, j. m. (2020). optimizing the technological design of a blended synchronous learning environment. online learning, 24(3), 222–240. https://doi.org/10.24059/olj.v24i3.2180 arias, j. j., swinton, j., & anderson, k. (2018). online vs. face-to-face: a comparison of student outcomes with random assignment. e-journal of business education and scholarship of teaching, 12(2), 1–23. benson, a. (2002). using online learning to meet workforce demand: a case study of stakeholder influence. quarterly review of distance education, 3(4), 443−452. bettinger, e., doss, c., loeb, s., rogers, a., & taylor, e. (2017). the effects of class size in online college courses: experimental evidence. economics of education review, 58, 68–85. https://doi.org/10.1016/j.econedurev.2017.03.006 boling, e. c., hough, m., krinsky, h., saleem, h., & stevens, m. (2012). cutting the distance in distance education: perspectives on what promotes positive, online learning experiences. the internet and higher education, 15(2), 118– 126. https://doi.org/10.1016/j.iheduc.2011.11.006 bolliger, d. u., & supanakorn, s. (2011). learning styles and student perceptions of the use of interactive online tutorials. british journal of educational navigating the new normal 18 journal of effective teaching in higher education, vol. 5, no. 2 technology, 42, 470–481. https://doi.org/10.1111/j.14678535.2009.01037.x brower, h. h., (2003). on emulating classroom discussion in a distance-delivered obhr curse: creating an on-line learning community. academy of management learning and education, 2(1), 22–36. https://www.jstor.org/stable/40214163 bui, k. v. t. (2002). first-generation college students at a four-year university: background characteristics, reasons for pursuing higher education, and firstyear experiences. college student journal, 36, 3–11. cavanaugh, j. (2005). teaching online—a time comparison. online journal of distance learning administration, 8(1). https://www.semanticscholar.org/paper/teaching-online-a-timecomparison-cavanaugh/2acaf51103ac360b5731ccfa5ba0854b436b1e41 coates, d., & humphreys, b. r. (2001). evaluation of computer-assisted instruction in principles of economics, educational technology & society, 4(2), 133–144. https://www.jstor.org/stable/jeductechsoci.4.2.133 conrad, d. (2002). deep in the hearts of learners: insights into the nature of online community. journal of distance education, 17(1), 1−19. donovan, j., mader, c. e., & shinsky, j. (2011). constructive student feedback: online vs. traditional course evaluations. distance education report, 15(4), 283−296. dousay, t. a., & trujillo, n. p. (2019). an examination of gender and situational interest in multimedia learning environments. british journal of educational technology, 50(2), 876–887. https://doi.org/10.1111/bjet.12610 fedynich, l. v. (2013). teaching beyond the classroom walls: the pros and cons of cyber learning. journal of instructional pedagogies, 13(1). http://www.aabri.com/manuscripts/131701.pdf finkelstein, j. e. (2006). learning in real time: synchronous teaching and learning online (vol. 5). john wiley & sons. gibbons, m. m., & woodside, m. (2014). addressing the needs of first-generation college students: lessons learned from adults from low-education families. journal of college counseling, 17(1), 21–36. https://doi.org/10.1002/j.21611882.2014.00045.x henderson, k., lyons, b., & grace, b. (2018). picking pedagogical practices students prefer: an analysis of the effectiveness of teaching tools in face-toface versus online delivery. business education innovation journal, 10(1), 47–50. navigating the new normal 19 journal of effective teaching in higher education, vol. 5, no. 2 jones, s. (2002). the internet goes to college: how students are living in the future with today’s technology. pew internet and american life project. http://www.pewinternet.org/reports/toc.asp?report=71 kampov-polevoi, j. (2010). considerations for supporting faculty in transitioning a course to online format. online journal of distance learning administration, 13(2). https://www.semanticscholar.org/paper/considerations-forsupporting-faculty-in-a-course-kampovpolevoi/93f50a6b30ee68c4047e34f73db91ac8c46766e7 kear, k. (2010). social presence in online learning communities. proceedings of the seventh international conference on networked learning 2010. aalborg, denmark. https://www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2010/abstracts/ pdfs/kear.pdf keengwe, j., & kidd, t. t. (2010). towards best practices in online learning and teaching in higher education. merlot journal of online learning and teaching, 6(2), 533–541. kennedy, m. (2020). the chaos of coming back. american school & university, 93(1), 22–25. luik, p. (2011). would boys and girls benefit from gender-specific educational software? british journal of educational technology, 42, 128–144. https://doi.org/10.1111/j.1467-8535.2009.01005.x maddix, m. a. (2012). generating and facilitating effective online learning through discussion. christian education journal, 9(2), 372–385. https://doi.org/10.1177/073989131200900209 major, c. (2020). innovations in teaching and learning during a time of crisis. innovative higher education, 45(4), 265–266. https://doi.org/10.1007/s10755-020-09514-w manathunga, c. (2002). designing online learning modules: an australian example of teacher education. international journal of instructional media, 29(2), 185–195. mitchell, m., & leachman, m. (2015, may 13). years of cuts threaten to put college out of reach for more students. center on budget and policy priorities. http://www.cbpp.org/research/state-budget-and-tax/years-of-cuts-threatento-put-college-out-of-reach-for-more-students moore, j. l., dickson-deane, c., & galyen, k. (2011). e-learning, online learning, and distance learning environments: are they the same? the internet and navigating the new normal 20 journal of effective teaching in higher education, vol. 5, no. 2 higher education, 14(2), 129–135. https://doi.org/10.1016/j.iheduc.2010.10.001 mora, l. (2022). hispanic enrollment reaches new high at four-year colleges in the u.s., but affordability remains an obstacle. pew research center. https://www.pewresearch.org/fact-tank/2022/10/07/hispanic-enrollmentreaches-new-high-at-four-year-colleges-in-the-u-s-but-affordability-remainsan-obstacle/ morrison, g. r., ross, s. j., morrison, j. r., & kalman, h. k. (2019). designing effective instruction. john wiley & sons. nguyen, t. (2015). the effectiveness of online learning: beyond no significant difference and future horizons. merlot journal of online learning and teaching, 11(2). https://jolt.merlot.org/vol11no2/nguyen_0615.pdf nistor, n. (2013). stability of attitudes and participation in online university courses: gender and location effects. computers and education, 68, 284– 292. https://doi.org/10.1016/j.compedu.2013.05.016 parker, k., lenhart, a., & moore, k. (2011). the digital revolution and higher education. pew research center internet & technology. http://www.pewinternet.org/2011/08/28/the-digital-revolution-and-highereducation. ponzurick, t. g., france, k. r., & logar, c. m. (2000). delivering graduate marketing education: an analysis of face-to-face versus distance education. journal of marketing education, 22(3), 180–187. reid, m. j., & moore, j. l., iii. (2008). college readiness and academic preparation for postsecondary education: oral histories of first-generation urban college students. urban education, 43(2), 240–261. https://doi.org/10.1177/0042085907312346 rti international. (2019). first-generation college students: demographic characteristics and postsecondary enrollment. washington, dc: naspa. https://firstgen.naspa.org/files/dmfile/factsheet-01.pdf russell, v., & curtis, w. (2013). comparing a largeand small-scale online language course: an examination of teacher and learner perceptions. internet and higher education, 16, 1–13. https://doi.org/10.1016/j.iheduc.2012.07.002 saiz, m. (2014). economies of scale and large classes. thought & action, 149–159. steen, h. l. (2008). effective elearning design. merlot journal of online learning and teaching, 4(4), 526–532. https://jolt.merlot.org/vol4no4/steen_1208.pdf navigating the new normal 21 journal of effective teaching in higher education, vol. 5, no. 2 tate, k. a., caperton, w., kaiser, d., pruitt, n. t., white, h., & hall, e. (2015). an exploration of first-generation college students’ career development beliefs and experiences. journal of career development, 42, 1–17. https://doi.org/10.1177/0894845314565025 tomei, l. (2006). the impact of online teaching on faculty load: computing the ideal class size for online courses. journal of technology and teacher education, 14(3), 531–541. https://eric.ed.gov/?id=ej729637 vonderwell, s. (2003). an examination of asynchronous communication experiences and perspectives of students in an online course: a case study, the internet and higher education, 6(1), 77–90. https://doi.org/10.1016/s1096-7516(02)00164-1 wehrwein, e. a., lujan, h. l., & dicarlo, s. e. (2007). gender differences in learning style preferences among undergraduate physiology students. advances in physiology education, 48, 153–157. https://doi.org/10.1152/advan.00060.2006 wernet, s. p., olliges, r. h., & delicayh, t. a. (2000). postcourse evaluations of webct (web course tools) classes by social work students. research on social work practice, 10(4), 487–505. https://doi.org/10.1177/104973150001000408 microsoft word leibold_final.docx journal of effective teaching in higher education, vol. 5, no. 2 https://doi.org/10.36021/jethe.v5i2.302 student perceptions of teacher online feedback: an integrative review nancyruth leibold, southwest minnesota state university, nancyruth.leibold@smsu.edu laura m. schwarz, minnesota state university, mankato abstract. teacher feedback is a core feature of the student educational experience and impacts student development. educators can learn about teaching practice from student perceptions of teacher online feedback (tof). this integrative research review explores student perceptions of tof in higher education using modified prisma guidelines. the researchers searched six academic databases and included 12 articles for deep analysis and evaluation after three rounds of article reviews. students reported a preference for timely tof to help improve their performance. participants noted a variety of preferred feedback mediums, such as video, text, and audio. support for the community of inquiry (coi) theoretical framework was present in the research findings. the studies reviewed ranged from high to lower levels of evidence. scholarship of teaching and learning (sotl) implications stress that online teacher feedback is a complex and contextual skill to develop. the teacher’s online feedback method and medium should match the student population’s preference. research findings about student perceptions of tof in higher education are critical for application by faculty to advance the sotl. keywords: community of inquiry (coi); teacher online social presence; teacher online feedback; feedback medium in recent years, interest in the evidence base of best practices for teachers giving online feedback to students continues to expand. online higher education continues to grow (national center for education statistics, 2022), necessitating improved faculty knowledge and skills for online teaching practices. the teacher’s skill of giving quality online feedback is crucial to enhancing learner development (leibold & schwarz, 2015). the present study focuses on student viewpoints of helpful and effective teacher online feedback (tof). background of online feedback feedback is information or data from another person specific to an assignment or skill performance that may include validation or explain gaps in performance (douglas et al., 2016; kamiya, 2018). feedback should be timely, clear, easy to understand, and purposeful in supporting improvement and development (leibold & schwarz, 2015). online feedback is information or data provided online with 24/7 access. formative assessments throughout a course provide frequent feedback opportunities to note flourishing performance areas and areas needing correction (steele & holbeck, 2018; wiggins, 2012). summative feedback is an evaluation at the end of an event or time (jug et al., 2019). both formative and summative feedback are opportunities for providing helpful dialogue and tips to promote student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 23 learner development. knowledge of student perceptions of tof provides data about feedback strategies that help and those that can be improved. the community of inquiry (coi) framework guides teaching, including feedback practices. the coi framework includes three vital concepts: teaching presence, social presence, and cognitive presence. these concepts should be applied together and are mutually essential in developing deep and meaningful online experiences (annand, 2011), inclusive of providing feedback. teaching presence means designing and implementing cognitive and social processes to promote individualized learning and meaning (anderson et al., 2001). social presence is projecting emotional expression, communicating openly, and creating a cohesive community of inquiry (garrison et al., 1999). garrison et al. (1999) define cognitive presence as the degree to which participants within a community of inquiry build meaning through continued communications. cognitive presence over sustained periods promotes critical thinking skills (garrison et al., 1999). various online feedback mediums exist, including handwritten and scanned notes, typed text notes, automated software, and audio and video recordings. espasa et al. (2022) reported that students preferred video feedback, but the medium type (text, audio, or video) did not impact student performance. sarcona et al. (2020) reported that most students preferred text feedback over audio feedback and noted an association with perceived learning style. the researchers of this study sought to identify the viewpoints of students who receive tof. purpose and research questions this integrative review aimed to evaluate student perceptions of tof and strategies for providing quality feedback to the population of students in higher education. effective tof is an increasingly important aspect of teaching praxis and is in the early research evidence development stages. instructional and learning practices are quickly changing with the modern-day use of technology. teacher practice and research in higher education must also progress to reflect changes. feedback from teachers influences student experience and performance (adams, 2019). the research questions (rq) were: • rq1: what are student perceptions of tof in higher education? • rq2: what are student perceptions of text, audio, and video feedback in higher education? • rq3: what are student perceptions about teacher online social presence in higher education? • rq4: what are the qualities of the study methods and designs? student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 24 methodology a mixed-methods, integrative review study design guided the investigation of student perceptions of tof. the integrative review study design is a broad approach that allows for the inclusion of experimental and non-experimental research in the exploration (whittemore & knafl, 2005) of tof research. an integrative review is well suited to educational research because it amalgamates experimental and non-experimental evidence findings from various online feedback strategies. the mixed-methods approach incorporates qualitative and quantitative data allowing for the analysis and synthesis of multi-perspective research evidence into themes and findings. the preferred reporting items for systematic reviews and meta-analyses (prisma) are guidelines for reviewing and reporting systematic reviews and meta-analyses (2021). however, researchers may also use prisma for integrative and other research reviews (prisma, 2021). the investigators used a modification of the prisma framework for reporting in this integrative review study. the key aspects of integrative reviews are identification, screening, eligibility, and included articles. moher et al. (2009) recommend including the type of review (for example, systemic review or meta-analysis) in the article‘s title, as done in this study. prisma recommends describing the search process for articles, inclusion/exclusion criteria (see figure 1), review phases to make the included article selections (see figure 2), participants, interventions, study appraisals, article synthesis, limitations, conclusions, and implications for practice (moher et al., 2009). the peer-reviewed article search process included six major databases from 2014 to 2021 to find research on teacher online feedback. search terms/phrases used were feedback, online feedback, online feedback by faculty in higher education, online teacher feedback and higher education, tof, online faculty feedback, and online educator feedback. the researchers used expanded searches and the boolean operators “or” and “and” to group search phrases and found 772 research journal articles from the database searches. next, researchers excluded 701 of the 772 publications by checking the title and abstract for search terms, inclusion/exclusion criteria, and duplicates. the remaining 71 full-text articles were retrieved, read twice, and evaluated using inclusion/exclusion criteria. see figures 1 and 2 for more information about the inclusion and exclusion criteria and prisma flowchart. there was an exclusion of 59 of the 71 articles by this phase’s end. researchers reread all articles and discussed them, leading to a final inclusion count of 12 research journal articles. extraction of data and data analysis the researchers created a summary table that included the complete reference for each of the 12 research articles and read each article several times. next, researchers recorded details in the table, including study method, design, research questions, hypothesis/hypotheses, sample size, sampling strategy, and the country of study origin. also included were statistical data or qualitative analysis, levels of evidence, study rigor, possible risk of bias notes, and significant results. the student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 25 researchers likewise noted insights that each study offered. each researcher independently completed the table, followed by meetings to compare results for a consensus and emerging themes. the table was helpful to the researchers in extracting, organizing, and analyzing data to discover common themes. figure 1 study inclusion and exclusion criteria evaluating study quality the researchers used a modified integrative review of the qualitative, quantitative, and mixed methods studies framework focused on methods and concepts described by hong and pluye (2019) to assess study quality. the methods dimension refers to the trust in the approach related to sampling and the risk of bias. the conceptual dimension refers to the study’s insightfulness. credibility, generalizability, transferability, dependability, and confirmability were key aspects under review in the studies. researchers reviewed studies for internal and external validity, reliability, and objectivity. the reporting quality of meaning accuracy, completeness, and transparency is specific to the research manuscript’s quality. for example, should participants drop out of the study during the data collection phase, do the researchers describe the process for handling the data or a reason for dropping out? researchers should also describe the data synthesis related to the purpose (hong & pluye, 2019). the researchers combined the seven levels of student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 26 figure 2 prisma literature search and inclusion flow chart evidence (see figure 3) characterized by melnyk and fineout-overholt (2019) with the framework by hong and pluye (2019) to assess study quality. the quality assessment of each research article included methodology, conceptualization, reporting quality, purpose, and data synthesis. research on online teacher feedback is in the early stages, and this study sought to evaluate recent research. student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 27 figure 3 levels of evidence results rq1). what are student perceptions of tof to students in higher education? timely feedback was perceived as desirable by students. parkes and fletcher (2017) studied video versus written feedback. students preferred a quick turnaround time of audio feedback with a simple recording instead of sophisticated or edited recordings that took additional time to receive. gredler (2018) investigated student feedback preferences, with timely feedback emerging as a minor theme. along with being timely, students commented that the feedback should include why points were deducted and include examples. most (86%) participants in jones and blankenship’s (2014) study findings partially or totally agreed that feedback was timely enough to use in subsequent course assignments. on the contrary, douglas et al. (2016) found that 50% of students perceived less timely feedback, while 20% perceived that feedback was timely. further, the timeliness of the feedback was a factor in their perception of a negative or positive feedback experience. see table 1 for study summaries. student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 28 most studies highlighted the developmental value of online feedback to students for improving their work. most of jones and blankenship’s participants perceived feedback helpful in reaching course goals (96%) and enhancing course performance (95%). likewise, marshall et al. (2020) determined that student writing growth also resulted from video feedback. student comments indicated that the feedback helped them learn from their mistakes and grow as writers. online feedback was also found valuable in that it was motivating. pan and shao (2020) determined a significant positive effect of online feedback on learning motivation and engagement (n = 312) although learning engagement partially mediated the relationship of learning motivation. douglas et al. (2016) studied what learners (n = 321) perceive as feedback and the educational value of the feedback, specifically formative versus summative feedback. most learners found summative feedback vital. however, only 24% of students were able to recognize formative feedback as necessary to improve, and 34% wanted only summative feedback. students did not always connect that they should use formative feedback to improve their skills and knowledge. this finding contrasts with the findings of other researchers who found that feedback was helpful in development, hence of value. a variety of other feedback preferences emerged in the studies. gredler’s (2018) participants preferred clear, detailed, constructive, and supportive written feedback placed near the assignment content. jones and blankenship (2014) studied preference for a medium of feedback and feedback helpfulness. students most favored the award of a numerical grade but found using a rubric with a summary of comments and corrected spelling and grammar helpful. table 1 research summaries of student perceptions of tof authors year purpose & feedback type level of evidence, methodology, design & sampling student perceptions 1 ali (2016) determine the effect of video feedback on enhancing students’ writing skills and structure and determine students’ perceptions of video feedback on their writing. level ii randomized mixed methods, preposttest survey, independent t-tests. convenience sample (n = 63) undergraduate students (33 experimental groupvideo feedback, 30 control group-written feedback) in egypt. most (94%) students perceived video feedback as personal and individualized, felt valued, and paid more attention to videos, perceived videos as motivating. student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 29 authors year purpose & feedback type level of evidence, methodology, design & sampling student perceptions 2 borup et al. (2014) determine students’ perceptions of instructor social presence with video-only feedback versus written-only feedback. level iv non-randomized mixed methods, independent t-tests. n = 229 undergraduate elementary and secondary education students in in blended educational technology courses (99 control group [n = 75] written only, 130 experimental group [n = 105] video only in usa. no statistical difference between groups. qualitative findings indicated video feedback perceived as more helpful in establishing instructor social presence than written-only feedback. 3 dias & trumpy (2014) determine student satisfaction with individual written/group written feedback verses individual written/group audio feedback. level iv non-randomized, quasi-experimental, quantitative, onetailed t-test. n = 99 undergraduate business communication students (control group n = 49 individual and group written feedback, experimental group n = 50 individual written feedback and group audio feedback) in usa. perceived written individual/group audio feedback as enhancing teacher social presence and showing genuine concern of instructor regarding whether students learned. 4 douglas et al. (2016) determine learners’ perceptions of feedback, including what constitutes feedback, educational value, level vi mixed methods openended survey, thematic analysis. n = 321 first and second-year undergraduate education, biomedical science, exercise 24% perceived formative comments important in linking feedback to improvement, but 34% wanted only summative feedback. did not always perceive formative student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 30 authors year purpose & feedback type level of evidence, methodology, design & sampling student perceptions how and why they actively seek it, and responses to feedback. science, environmental health, health science, medical radiation science, and nursing majors in australia. feedback helpful in improving learning. tended to perceive summative feedback as important. 5 gredler (2018) determine students’ preferences for written teacher feedback. level iv sequential explanatory research design with a mixed methods survey; descriptive frequencies and qualitative data analysis for themes; n = 93 online graduate (95.6%) and undergraduate social science, health sciences, and “other” majors in usa. perceived detailed and supportive feedback placed close to content is helpful in improving writing skills and understanding deductions. preferred clear, detailed, constructive, and supportive feedback. 6 jones & blankenship (2014) examine student perceptions about online assignment feedback, feedback type preferred, helpfulness of and satisfaction with the feedback. level iv quasi-experimental research design, convenience sample with one treatment group, quantitative survey. n = 70 legal and applied business students in a southern university serving the hispanic population in usa. perceived rank order of usefulness (highest to lowest): assignment grade, rubric, summary of comments, corrected spelling and grammar. 96% found feedback useful in reaching course performance goals and 92% satisfied with the amount of feedback. 7 london (2019) study the effects of audio feedback versus text feedback in online level iii quantitative, quasiexperimental posttest design using the coiq. n = 202 rnbsn undergraduate nursing students (one preferred text-only feedback over audio feedback. text feedback was positively correlated with student satisfaction. student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 31 authors year purpose & feedback type level of evidence, methodology, design & sampling student perceptions discussion boards. group n = 100 received audio-only feedback, the other group n = 102 received text-only feedback for online discussions) in the usa. audio feedback was negatively correlated with student satisfaction. 8 marshall et al. (2020) explore students’ perceptions of both written and video feedback and the impact on writing growth. level vi qualitative, descriptive, semistructured interviews about feedback received. used atlas, a coding and qualitative analysis program. n = 26 graduate students in hybrid research methods course in usa. three themes of video feedback perceptions were 1) positive; 2) helped improve instructor social presence; and 3) student growth was a result. perceived video feedback as more specific, personable, and less threatening than written. 9 pan & shao (2020) determine student perceptions of tof and the relationship with motivation to learn. level iv qualitative cross-sectional design with interventional group only, reliability coefficient. n = 312 undergraduates in hybrid english courses (ell) in china. significant positive correlation/effect of online feedback perceived as influencing both learning motivation and learning engagement. 10 parkes & fletcher (2017) examine student perceptions of online audio feedback. level iv quantitative longitudinal, posttest survey with likert scale. n = 225 postgraduate computer education students in australia. audio feedback was preferred and with higher rated perceptions of quality and quantity versus past written feedback. preferred prompt feedback via simple recording over sophisticated or student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 32 authors year purpose & feedback type level of evidence, methodology, design & sampling student perceptions edited recording they had to wait for. 11 seckman (2018) evaluate impact of video versus text-based feedback on teaching, social, and cognitive presence in online learning communities. level iii quasi-experimental cross-sectional interventional study using coiq. convenience sample n = 100 students (37 undergraduate and 63 graduate students) in usa. the video feedback group perceived better awareness of teaching, social, and cognitive presence than the group who received written feedback 12 west & turner (2016) determine student perceptions of feedback including preferences for video versus written feedback, understanding feedback, time spent reviewing feedback, and instructor rapport. level iv mixed methods with post-survey. chisquare and one-way analysis of variance for quantitative data. key themes and issues analyzed for qualitative data. convenience sample, n = 142 undergraduate first-year bs education students in australia. most (61%) favored video verses written (21%) feedback. comments on video included “clearer and less ambiguous than other forms of feedback…improved both the quality and quantity of the feedback…greater rapport with their tutor and…greater insight into the assessment process” (p. 400). rq2). what are student perceptions of text, audio, and video feedback in higher education? students perceived video feedback as a positive and/or superior medium of online feedback in several studies. ali (2016) studied student perceptions (n = 63) of online video feedback’s effects on their writing. most (94%) perceived the video feedback as personal and individualized, made them feel valued, and paid more attention to the videos than other online feedback mediums. students indicated the videos were “succinct, unambiguous, multimodal, personal, feed forward, and motivating” (ali, 2016, p. 119). however, some students (9%) preferred written feedback because video feedback was more time-consuming to view. marshall et al. student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 33 (2020) explored student perceptions of video and written feedback on their research proposals. findings were similar to ali’s (2016) in that most students preferred video feedback over written feedback. students perceived video feedback as more specific and personable, and it helped them grow as writers. likewise, west and turner’s (2016) study (n = 142) found that 61% of students preferred video feedback versus 21% who preferred written feedback. students commented that the video was less ambiguous, provided better quality and quantity of feedback, fostered a greater rapport with their tutors, and provided better insight into evaluative processes. borup et al.’s (2014) qualitative findings indicated that students (n = 22) perceived video feedback as more effective for forming instructor social presence than written feedback. students described videos as more conducive to conveying emotions, conversation, interaction, and connection. three of the articles examined compared audio to written feedback. parkes and fletcher (2017) explored student perceptions (n = 225) of audio feedback mediums. participants preferred audio feedback over the written feedback they had previously received, with higher quality and quantity ratings. in contrast, london (2019) found that students (n = 202) preferred text-only feedback over audio feedback for discussion boards. a third study by dias and trumpy (2014) comparing audio versus written feedback had different findings. this study examined students’ satisfaction with individual and group written feedback (n = 49) versus individual written and group audio feedback (n = 50). the student group who received individual written and group audio feedback perceived that the instructor genuinely wanted the students to learn. rq3). what are student perceptions about teacher online social presence in higher education? coi is a framework that “represents a process of creating a deep and meaningful learning experience through the development of three interdependent elements— social, cognitive, and teaching presence“ (community of inquiry, n.d.). coi arose as a theme in some articles related to perceptions of tof. borup et al.’s (2014) qualitative findings were that students found video-only feedback more helpful in forming instructor social presence than text-only feedback. with video feedback, “instructors could better speak with emotions, talk in a conversational manner, and create a sense of closeness with students“ (p. 232). however, quantitative findings indicated no statistically significant difference in perceptions of social presence between the experimental group (n = 105), who received video feedback, and the control group (n = 75), who received written-only feedback. this finding was possibly due to the blended course delivery negating instructor presence through feedback. marshall et al. (2020) also found that students perceived video feedback as promoting instructor presence. students’ comments indicated that video feedback helped build the student-faculty relationship. in a study of online audio versus written feedback, a combination of group audio feedback and individual written feedback was found to enhance teacher social presence (dias & trumpy, 2014). seckman (2018) used the coi survey questionnaire (coiq) to study the effect of student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 34 video versus text feedback and found that the video feedback group perceived better teaching, social, and cognitive presence than the group who received text feedback. london used the coiq to measure differences between audio and textonly feedback regarding perceptions of teaching, social, and cognitive presence. in contrast to the aforementioned studies, london found that teaching presence increased with text-only feedback but decreased with audio-only feedback. perceptions of social and cognitive presence were not affected by either text or audio feedback mediums. rq4). what are the qualities of the study methods and designs? the fourth research question, “what are the qualities of the study methods and designs?,“ addresses levels of evidence and rigor of study method and design. see the previous figure 3 describing the seven levels of evidence by melnyk and fineout-overholt (2019). the levels of evidence of the 12 studies ranged from level ii to level vi. three of the 12 studies had high evidence levels (see table 1). nine studies had moderate or lower evidence levels (see table 2). table 2 study rigor and limitations authors year study limitations presented by authors 1 ali (2016) small sample negates generalizability, short study duration 2 borup et al. (2014) findings not generalizable, in-person interactions possibly affecting social presence of video feedback; instructor feedback limited to student projects 3 dias & trumpy (2014) authors noted internal validity limitations with survey; survey questions may not indicate actual satisfaction with feedback, response homogeneity creating inaccuracies, potential bias in answering questions, and differing response rates between sections; external validity limitations: generalizability of study’s populations, potential differences in perceptions over time (studied over two terms) 4 douglas et al. (2016) student perceptions affected by other experiences 5 gredler (2018) small sample and participant self-selection; native-english speaking graduate students experienced in online learning [limits generalizability] 6 jones & blankenship (2014) small convenience sample limits generalizability student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 35 authors year study limitations presented by authors 7 london (2019) online rn-bsn sample from one public university limits generalizability 8 marshall et al. (2020) authors noted study design with interviews conducted after the semester concluded, in-person interactions [possibly taints results of perceptions of online feedback] 9 pan & shao (2020) self-reported data may impact results accuracy; crosssectional design possibly biasing results; small sample size/narrow focus of english language learning course [limits generalizability] 10 parkes & fletcher (2017) authors recommended trialing audio feedback first; study audio feedback with diverse groups and undergraduates as this study was conducted with post-graduates 11 seckman (2018) sample limited to online informatics course in singlelocation limits generalizability; self-reported data impacting accuracy of results; multiple sections of same course with varying faculty may impact feedback quality/consistency 12 west & turner (2016) first semester undergraduate participants may have strongly preferred video feedback based on novelty; longer videos equaled greater time spent reviewing/perceptions of greater value of video feedback over written feedback discussion this review sought to critically analyze and summarize the literature from 20142020 regarding students’ perceptions of tof. the online educational modality is increasing (national center for education statistics, 2022), and so is the evidence of the effectiveness of tof. student perceptions of tof timely, clear, detailed, and supportive feedback were emerging themes. desiring timely feedback was a common finding (marshall et al., 2020; gredler, 2018; pan & shao, 2020; parkes & fletcher, 2017). timely feedback to support improvement and development is consistent with previous work by leibold and schwarz (2015) and kamiya (2018). clear, detailed quality and supportive feedback was a common positive student perception (gredler, 2018; pan & shao, 2020; parkes & fletcher, 2017). ninety-five percent of respondents reported that their work improved due to tof (jones & blankenship, 2014). previous work regarding the importance of formative and summative course feedback is consistent with this study (jug et al., student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 36 2019; steele & holbeck, 2018; wiggins, 2012). both formative and summative feedback are opportunities for providing clear, detailed, and supportive feedback. an insightful finding in this review was related to bridging feedback with improving performance. douglas et al. (2016) found that some students needed to connect using feedback to improve their skills and knowledge base. however, ali (2016) reported improvement in performance by students who received balanced feedback on what was done well and how to improve. on the contrary, jones and blankenship (2014) found that 56% of participants did not care if they had positive feedback but wanted to know what to improve. providing students with specifics on how to improve is desired by students (jones & blankenship, 2014). student perceptions of feedback medium researchers gained insight into the preferences of varied student groups for specific feedback modalities. rn-bsn students preferred text-only feedback to audio feedback (london, 2019), while first-year students liked video feedback as it was specific and personable (ali, 2016). west and turner (2016) reported that the video medium was less ambiguous for undergraduates. seckman (2018) found that nursing students favored video feedback regarding teaching, social, and cognitive presence. the variance of findings of medium preference is consistent with espasa et al. (2022), who reported a preference for video feedback, and sarcona et al. (2020), who found that students preferred text feedback. population and in-person contact may impact feedback medium inclinations. one possible explanation is that different populations or course purposes (nursing versus english majors or undergraduate versus graduate levels) may result in unique preferences. another possible reason, as borup et al. (2014) acknowledge, is that the teachers may have in-person contact with students, although the feedback was online. based on the evidence, student populations have varied feedback preferences. feedback and community of inquiry support for the coi theoretical framework of teacher online social presence was present in the findings. online feedback contributes to teaching, social, and cognitive presence, though results varied by study and feedback medium. the results of this study are consistent with anderson et al. (2001), who stress the importance of individualizing student learning to promote teaching, social, and cognitive presence. interactions within the coi framework (annand, 2011) and communications (garrison et al., 1999) are in the study findings. perceptions of video feedback were that it was more helpful in establishing teacher social presence than written-only feedback (borup et al., 2014). seckman (2018) found that the video feedback group had a greater perception of teaching, social, and cognitive presence than the written feedback group. video feedback was perceived to promote teacher presence (marshall et al., 2020). dias and trumpy (2014) found enhanced teacher social presence by combining group audio feedback and individual written feedback. london (2019) found an interesting result that audio feedback decreased student perception of teaching presence and text feedback increased teaching presence. varied study findings may relate to mediators such as student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 37 teaching presence, research design, population, education level, and measures. overall, the findings in this study found connections to the coi framework and feedback research. quality appraisals of study methods and designs only one study had control and experimental groups with randomization that allows for causality (ali, 2016). most studies had small sample sizes and were of moderate rigor. although higher levels of evidence are best, educational research rigor includes evaluating the research and being thorough, responsible, reasonable, and accurate (williams & kimmons, 2022). research reviewed in this study displayed responsibility by authors self-reporting possible biases and limited generalizability, such as in-person contact with students (borup et al., 2014). in-person contact to clarify feedback and social presence are not negative teacher behaviors but could impact research data. having in-person live communication with a student may skew research data when focusing on online feedback methods because of the differences between online feedback and in-person discussions. in some research articles, whether in-person contact may or may not have occurred was not clearly stated. the differences in communication may impact research results as a confounding variable affecting the independent and dependent variables. however, researchers could clarify in research reports if there was contact. additionally, inperson contact with teachers on campus may impact student perceptions even when a course is online. for example, the student may have more live, in-person contact with the teacher and the ability to clarify feedback instead of total online communication with a teacher. therefore, researchers should indicate if they had in-person contact with student study participants in research reports. implications for future research are to use larger sample sizes, provide thorough research processes, and randomize participants into control and experimental groups. strengths and limitations a literature gap exists in randomized experimental designs to study student perceptions of tof. the study by ali (2016) was the only one with randomized experimental design rigor used for cause and effect. the best practice for determining cause and effect from an intervention is to use the random assignment of study participants to the control and experimental groups (melnyk & fineoutoverholt, 2019). control and experimental groups with randomization and larger sample sizes would improve the rigor of future studies. coi, as a part of the study, is both a strength and a limitation. the coi framework used by some researchers was a strength. further studies that examine the use of coi could provide valuable data. future research recommendations include the additional analysis of all three constructs, teaching, social, and cognitive presence, and the relationships between the three. a possible limitation of this study is that coi and social presence were intentionally omitted as literature search terms to discover how much the coi framework was in the literature. the researchers found that coi was present in the research literature about feedback. possibly, if the researchers had included coi or teacher online social presence as search terms, it student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 38 would have substantially impacted the study results. the coi presence may be a strength as it allowed the researchers to find the use of coi without inputting coi literature search terms. the researchers intentionally did not search for the coi framework or teacher online social presence to evaluate if it would surface. some studies did not address extraneous variables that may impact results. one example of an extraneous variable is the procedure for not allowing the control and experimental groups to contaminate the results by conversing. another extraneous variable example is how email, phone, web conference, or in-person follow-up clarification of feedback might have occurred. additional communication is appropriate for teacher behavior but may have impacted some results. more research is necessary to compare in-person contact to online-only contact groups to examine the effect of the contact variable on student perceptions of tof. for example, does in-person contact result in more teaching or social presence than online contact? describing feedback types in studies and research procedures would also help identify implications. for example, studies could include the procedure for video feedback. specifically, researchers should indicate if the assignment is shown on the screen with video feedback, a summary of feedback comments, the face of the teacher, or a combination of the face of the teacher and paper or comments. to explain the feedback processes, researchers should improve the research procedures' reporting quality (completeness and description). teaching practice recommendations teachers should provide coaching tips for using feedback to improve future student performance. one way to shift energy to betterment is by saying, “please use this feedback to help improve the next revision of this paper.” teachers may also include specific details, such as “please use this feedback to help improve ____ for your next revision of the assignment.” online educators can use the evidence findings to practice timely, clear, detailed, and supportive feedback to help students improve their performance. teachers should know their student populations to know which feedback medium is preferred. for example, when a student population prefers video feedback, the teacher should consider incorporating this modality into the course. educators could survey the students about their preferred feedback format in the online setting. online teachers who have in-person contact with students can incorporate inperson feedback meetings. implications for teaching practice include encouraging students to email or phone for clarification and assistance and communicating availability and a timeframe for answering emails and messages, such as 24–48 hours. other teaching implications include being present in online discussions and crafting announcements with a personalized approach, such as asking students about something they wrote and addressing students by their preferred names and pronouns. student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 39 conclusions effective tof is a vital skill to improve student performance. timely tof as a student preference is a key finding of the study. teachers should strive to provide timely feedback to students. the researchers did not search for the coi framework, yet found several studies that support the use of the coi framework. parkes and fletcher (2017) reported that students prefer timely feedback over advanced and complex audio feedback. researchers found diverse student preferences for text, audio, and video feedback in the integrative review. the researchers assert that the context of the student population may explain the differences in feedback medium preferences. another conclusion was that most studies had moderate research method levels and design rigor. the researchers recommend that future researchers use the highest level of research methods and design to improve the quality of evidence. level two research that includes the random assignment of participants to a control group or experimental group for an intervention is the best research method and design to examine cause and effect (melnyk & fineoutoverholt, 2019). teachers should craft feedback to help students develop their skills and coach students to use the feedback to improve their performance. by considering the student population and their preferences, teachers can select the most effective methods and media for providing online feedback. future scholarship in tof is warranted to develop the body of evidence. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references adams, g. (2019). a narrative study of the experience of feedback on a professional doctorate: “a kind of flowing conversation.” studies in continuing education, 41(2), 191–206. https://doi.org/10.1080/0158037x.2018.1526782 ali, a. d. (2016). effectiveness of using screencast feedback on efl students’ writing and perception. english language teaching, 9(8), 106–121. https://eric.ed.gov/contentdelivery/servlet/ericservlet?accno=ej1106616 anderson, t., rourke, l., garrison, d. r., & archer, w. (2001). assessing teaching presence in a computer conferencing context. journal of asynchronous learning networks, 5(2), 538–551. https://doi.org/10.24059/olj.v5i2.1875 annand, d. (2011). social presence within the community of inquiry framework. the international review of research in open and distance learning, 12(5), 40–56. https://files.eric.ed.gov/fulltext/ej963922.pdf borup, j., west, r. e., thomas, r. a., & graham, c. r. (2014). examining the impact of video feedback on instructor social presence in blended courses. student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 40 international review of research in open & distance learning, 15(3), 232– 256. https://doi.org/10.19173/irrodl.v15i3.1821 community of inquiry. (n.d.). coi framework. coi. https://coi.athabascau.ca/coimodel/ dias, l. p., & trumpy, r. (2014). online instructor’s use of audio feedback to increase social presence and student satisfaction. journal of educators online, 11(2), 1–19. https://doi.org/10.9743/jeo.2014.2.5 douglas, t., salter, s., iglesias, m., dowlman, m., & eri, r. (2016). the feedback process: perspectives of first and second year undergraduate students in the disciplines of education, health science and nursing. journal of university teaching & learning practice, 13(1), 1–19. https://doi.org/10.53761/1.13.1.3 espasa, a., mayordomo, r. m., guasch, t., & martinez-melo, m. (2022). does the type of feedback channel used in online learning environments matter? students’ perceptions and impact on learning. active learning in higher education, 23(1), 49–63. https://doi.org/10.1177/1469787419891307 garrison, d. r., anderson, t., & archer, w. (1999). critical inquiry in a text-based environment: computer conferencing in higher education. the internet and higher education, 2(2), 87–105. https://doi.org/10.1016/s10967516(00)00016-6 gredler, j. j. (2018). postsecondary online students’ preferences for text–based instructor feedback. international journal of teaching & learning in higher education, 30(2), 195–206. https://www.isetl.org/ijtlhe/pdf/ijtlhe3002.pdf hong, q. n., & pluye, p. (2019). a conceptual framework for critical appraisal in systematic mixed studies reviews. journal of mixed methods research, 13(4), 446–460. https://doi.org/10.1177/1558689818770058 jones, i., & blankenship, d. (2014). what do you mean you never got any feedback? research in higher education journal, 24, 1–9. https://files.eric.ed.gov/fulltext/ej1064104.pdf jug, r., jiang, x., & bean, s. m. (2019). giving and receiving effective feedback: a review article and how-to guide. archives of pathology & laboratory medicine, 143(2), 244–250. https://doi.org/10.5858/arpa.2018-0058-ra kamiya, n. (2018). an analysis of the meaning of “natural” concerning oral corrective feedback. tesl-ej, 22(1), 1–12. https://teslej.org/wordpress/issues/volume22/ej85/ej85a4/ student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 41 leibold, n., & schwarz, l. m. (2015). the art of giving online feedback. the journal of effective teaching, 15(1), 19–31. https://uncw.edu/jet/articles/vol15_1/leibold.pdf london, j. (2019). use of asynchronous audio feedback in discussion boards with online rn-bsn students. nurse educator, 44(6), 308–311. https://doi.org/10.1097/nne.0000000000000633 marshall, d. t., love, s. m., & scott, l. (2020). “it’s not like he was being a robot”: student perceptions of video-based writing feedback in online graduate coursework. international journal for the scholarship of teaching and learning, 14(1). https://eric.ed.gov/contentdelivery/servlet/ericservlet?accno=ej1256273 melnyk, b. m., & fineout-overholt, e. (2019). evidence-based practice in nursing and healthcare: a guide to best practice (4th ed.). wolters kluwer. moher, d., liberati, a., tetzlaff, j., altman d. g., for the prisma group. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. bmj, 29(7716), e78–336. https://doi.org/10.1136/bmj.b2535 national center for education statistics. (2022). undergraduate enrollment. condition of education. u.s. department of education, institute of education sciences. https://nces.ed.gov/programs/coe/indicator/cha pan, x., & shao, h. (2020). teacher online feedback and learning motivation: learning engagement as a mediator. social behavior & personality: an international journal, 48(6), 1–10. https://doi.org/10.2224/sbp.9118 parkes, m., & fletcher, p. (2017). a longitudinal, quantitative study of student attitudes towards audio feedback for assessment. assessment & evaluation in higher education, 42(7), 1046–1053. https://doi.org/10.1080/02602938.2016.1224810 preferred reporting items for systematic reviews and meta-analyses (prisma). (2021, march 2). welcome to prisma! http://prisma-statement.org/ sarcona, a., dirhan, d., & davidson, p. (2020) an overview of audio and written feedback from students’ and instructors’ perspective. educational media international, 57(1), 47–60. https://doi.org/10.1080/09523987.2020.1744853 seckman, c. (2018). impact of interactive video communication versus text-based feedback on teaching, social, and cognitive presence in online learning communities. nurse educator, 43, 18–22. https://doi.org/10.1097/nne.0000000000000448 student perceptions of teaching online feedback journal of effective teaching in higher education, vol. 5, no. 2 42 steele, j., & holbeck, r. (2018). five elements that impact quality feedback in the online asynchronous classroom. journal of educators online, 15(3), 176–180. https://doi.org/10.9743/jeo.2018.15.3.10 west, j., & turner, w. (2016). enhancing the assessment experience: improving student perceptions, engagement and understanding using online video feedback. innovations in education and teaching international, 53(4), 400– 410. http://doi.org/10.1080/14703297.2014.1003954 williams, d. d., & kimmons, r. (2022). qualitative rigor. in r. kimmons (ed.), education research across multiple paradigms (1st ed., 2.3). byu open textbook network. https://open.byu.edu/education_research wiggins, g. (2012). seven keys to effective feedback. educational leadership, 70(1), 10–16. http://www.ascd.org/publications/educationalleadership/sept12/vol70/num01/seven-keys-to-effective-feedback.aspx whittemore, r., & knafl, k. (2005). the integrative review: updated methodology. journal of advanced nursing, 52(5), 546–553. https://doi.org/10.1111/j.1365-2648.2005.03621.x microsoft word hollander_etal_final.docx journal of effective teaching in higher education, vol. 5, no. 1 https://doi.org/10.36021/jethe.v5i1.111 the importance of college reading: professors’ and students’ views pam hollander, worcester state university, phollander@worcester.edu duke dawson, worcester state university charlotte haller, worcester state university erika briesacher, worcester state university caroline collins, worcester state university kristina rearick, worcester state university elise lemieux, worcester state university abstract. professors and students have contradictory views of course reading. professors believe that reading outside of the classroom is essential in optimizing learning. however, students often find the readings to be time-consuming, not necessary to pass the class, and an option rather than a requirement. we surveyed 449 undergraduate university students and interviewed 17 university faculty to determine the perceived value of reading assignments in college classes. this preliminary study yielded that faculty could benefit from professional development workshops that will help them determine how to incorporate strategies to increase the students’ completion of assigned readings. by investigating students’ thoughts and concerns, professors can better understand how to make assigned readings more attractive to students. keywords: college reading; student perceptions; interdisciplinary while writing-across-the-curriculum has been accepted in university academic departments since the early 1990s, reading-across-the-curriculum has received fewer productive resources or attention. as with writing, college professors find that students need to work on reading to gain general college-level skills as well as discipline-specific reading skills. neither college-level writing nor reading are necessarily taught before college. despite this, reading woes at the college level have not been able to gain the status of writing woes, which led to the writingacross-the-curriculum (wac) movement. the wac movement has increased resources for college writing instruction, including first-year and writing-in-thedisciplines classes. with these improvements, shortcomings in the development of reading skills have become more apparent. professors complain that students are not completing the assigned reading, and students admit they skip the assigned reading, leaving us to wonder about the correlation between the two. this research article investigates the difficulty college professors and students face regarding assigned reading to determine the potential causes of these reading issues. the results of this study reflect the localized experiences of our students, who are largely first-generation workingand middle-class students. however, most of their experiences and the observations of our professors align with what other researchers found from a cross-section of colleges. we—two literacy education, one science education, and three history education faculty—approached this subject because we have genuine concerns about how a possible mismatch between professors’ and students’ beliefs about college reading journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 91 may be impacting both instruction and learning outcomes in our classes. we all heard professors worrying about the reading not getting done and, at the same time, had heard students talking about not needing to and not doing all the reading. we set out to document our anecdotes with empirical research by interviewing professors from different subject areas followed by surveying students online. there are several factors that seem to contribute to students’ difficulties with college level reading. students receive little direct instruction in how to approach reading after elementary school, professors are at a loss about how to motivate students to read, and professors either do not realize that they need to provide direct guidance in the art of reading or struggle to find ways to convey disciplinespecific reading strategies (horning, 2013b; odom, 2013). reading at the college level is not necessarily a natural process but is an art that can be taught. students also have little experience that prepares them for discipline-specific college-level reading, and while professors in great numbers worry about reading, they are perplexed about what to do about it. we began our work by gathering data that would help us communicate our concerns. our overarching questions were as follows: 1. “what common issues and problems do we as professors face with student reading?” 2. “what content-specific issues/problems do we as professors face with student reading?” these questions led us to question more specifically our students’ experiences with reading. this resulted in the following additional questions being posed: 3. “what gets in students’ way when it comes to reading?” 4. “what can we do to make our students’ experiences with reading better?” after developing these research questions, we proceeded in three ways: 1) by educating ourselves more about the role of reading on our campus and other campuses through workshops with faculty sponsored by our center for teaching and learning; 2) by interviewing professors about their experiences with student reading; and 3) by gathering quantitative and qualitative data about students’ perceptions about the reading done for classes on campus. we chose to research both science and history with hopes that this same research can be done in other disciplines in the future. college-level reading although many professors assign reading and expect students to comply without any immediate extrinsic reward (aside from doing well in class because of knowledge from the reading), others intentionally give quizzes or questions directly linked to the readings in order to motivate students to read. the latter professors journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 92 find that these strategies help with reading completion (brown et al., 2016; phillips & compton, 2016). previous research has found that such assignments need to count as 20% or more to have any effect on students’ reading cooperation (nilson, 2010). questions or writing assignments in correlation with assigned readings have not been shown to have universal impacts on students’ understanding of reading. odom (2013) reported that writing across the curriculum faculty fellows using writing assignments, which acted as quizzes, did not produce favorable effects. students’ perceptions of these writing assignments as quizzes seemed to hurt their effectiveness. odom concluded that students were used to seeing these kinds of quiz-like questions and answered them the way they always had in the past—in the most superficial way possible (p. 10). in the sciences, it is important that students have some training in reading and critically analyzing published information in scientific journals. whereas undergraduates are accustomed to reading textbooks and taking notes in lectures, they often find it difficult to understand research articles in the basic biomedical sciences. while there is a general agreement among scientists that comprehension of scientific papers and communication of scientific concepts are two of the most important skills that we can teach undergraduates, few undergraduate biology courses make such comprehension explicit course goals or attempt to teach these skills (brownell et al., 2013). literature in the area of college-level reading points to purposeful reading, direct instruction, authentic assignments, personal and “real world” connections, and intertextuality as tools for helping college students with college-level reading (gogan, 2013; odom, 2013). odom (2013) also found that writing across the curriculum faculty fellows who attempted to use writing to improve students’ reading were more successful when they consciously changed the assignments that go along with readings to be more than a reading check. she alluded to the idea of direct instruction in how to read a particular kind of piece but spends more time on the areas of personal and real-world connections and authentic assignments: when the faculty made changes not just in how they assessed student reading compliance, but rather in how they asked students to approach their reading, they found real improvement in students’ comprehension of material and their ability to use what they read to their advantage throughout the course. (p. 10) odom (2013) quotes an english professor who used to give reading quizzes but now guides new open reading reflections by giving this direction: “i did specify that students were not to summarize the reading but to work through their reactions to the text” (p. 10). this professor makes the reading assignment more personally engaging and directed and does provide a bit of guidance. however, the professor did not provide direct instruction or mentoring in academic reading. journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 93 another professor in odom’s (2013) study had students make connections between their reading and current events. odom believes that establishing a personal or real-world experience is important for reading. certainly, this idea of the importance of a personal connection with one’s reading has been established in educational theory and research on motivation and learning (willingham, 2010). gogan’s (2013) study of a “required, writing-intensive course” supports the idea of direct instruction as helpful in increasing a student’s ability to read in college-level subject area classes. gogan reported that an assignment where students chose a scholarly article in their major and then dissected it using genre study was found to be perceived by 60.4% of students as having “helped me prepare for academic reading in my discipline” and was reported to have great impact a year later by students who were interviewed about the assignment. college science class reading to help students approach the challenge of reading research articles in the basic biomedical sciences, rangachari and mierson (1995) developed a checklist to guide students in the analysis of different components of a research article. in their study, students were assigned an article, usually a short communication, where techniques were familiar to them. the students were asked to use a checklist to help them critically analyze the article, write a paper assessing the article, and respond to a questionnaire evaluating the experience and their ability to understand the article. students had positive responses to the questionnaire and rated the experience as helpful. a writing intensive course format was used in a study by brownell et al. (2013) to boost undergraduate biology students’ confidence in their abilities to read scientific literature and communicate what they learned to scientific audiences. in the course, students discussed primary articles in class, summarized articles from the new york times, typed articles about the topic, and, as a final assignment, wrote a review article on a topic of the students’ choosing in neuroimmunology. preand postcourse questionnaires showed that the students’ perceptions of and confidence in their abilities to read scientific literature increased following the course. college history class reading sam wineburg (1999) responds to the misconceptions students have about history being simply the memorization of events and dates when he makes this observation: “the funny thing is that when you ask historians what they do, a different picture emerges. they see themselves as detectives searching for evidence among primary sources to a mystery that can never be completely solved” (p. 2). historical thinking, which has at its center the critical reading of texts, is “unnatural,” as wineburg terms it, for many college students. history professors thus work to make the work of historians explicit and accessible to their students. they identify the implicit questions (when, why, who, where, what’s missing, and what’s the broader context) that historians ask of the texts they encounter and ask journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 94 students to follow their lead. history professors also work to expose students to a range of historical texts, although individuals may prioritize some over others: textbooks, a variety of primary sources, and secondary sources. the goal is to get students to understand how to read for argument rather than simply for facts. ultimately, students should be able to understand historiography, or the history of how history has been written, and be able to situate their own work in the context of what has already been written. nonetheless, history professors frequently bemoan the difficulties in getting their students to read, and they struggle to balance the rewards of uncoverage with the needs of covering core material. greene (1993) pointed to a disconnect between the way that history professors and their students think about reading, the value of sources to their writing, and the ways that other texts are employed in students’ writing. the professors want students to stay true to the meaning of the original text while using it in the service of constructing interpretive arguments. traditional history college classes followed the “lecture-textbook-test” model, but history professors are increasingly experimenting with other methods to foster student learning. many of these approaches center on attempting to train students to “read like historians,” employing more active stances toward reading such as annotating (simpson & nist, 1990); critical evaluation of source, context, and evidence (hynd et al., 2004); and prioritizing information (pace, 2004). a number of professors explicitly teach their students reading strategies, such as “predatory reading,” which gives students permission to skim and the skills to identify argumentation (rael, 2004). these methods, however, require substantial and effective reading by students, and many students do not appear up to the task. students appreciate the focused detective work and exploration allowed when classes explore a particular event, question, or experience. when there are in-class readings, which are necessarily short, these exercises can work well. however, if students are unable or unwilling to read longer documents in preparation for class, these exercises become especially time-consuming, and professors fear that students are losing out on the full narrative of the field. how to persuade students that this work is worth the extra effort is something that has not generally been addressed by the history literature. additionally, research investigating the transition to higher education has shown that students are often unmoved to try new methods of studying and reading in college (armstrong & newman, 2011). history professors generally agree on the value of incorporating primary sources into the classroom, often in conjunction with a traditional textbook, although some faculty have ceased to assign the textbook altogether. there is less of a clear path to incorporating peerreviewed books and articles by historians, especially in the survey courses, as these readings require yet another skill set to read dense, detailed academic prose. journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 95 methods faculty interviews after gaining institutional review board approval through our university, we combined qualitative and quantitative methods for this research. we decided to use phenomenological interviewing because we wanted to find out how the professors constructed the experience of assigning reading and interacting with students around texts for the class. phenomenological interviewing allows researchers to get at the meaning participants make of a particular experience (seidman, 1991), and we coded the data to find themes. we interviewed 10 science professors at westview university (a pseudonym). each professor was interviewed once for a period of 30–45 minutes; the professors were from the following fields: chemistry (2), biology (4), physics (2), and earth science (2). in the history phase of our study, we repeated the same procedure with seven history professors. these professors teach a variety of history courses, but we did not categorize them the way we did the science professors, whose expertise focused on very different fields of science. our interview questions can be found in the appendix. the questions covered the general areas of student reading compliance, reading assignments, and assessment of quality of reading. student surveys in order to triangulate our data from professors with data from students, we needed to discover why students made decisions about reading. since we wanted to gain an understanding of how many students were reading and how much they were reading, we decided to use a survey format which would result in both qualitative and quantitative data about students’ experiences with course reading. an anonymous online survey with 15 questions, including short answer, multiple choice, and demographic information questions, was distributed to the entire undergraduate student body (6,000 students) via email. we received 327 completed surveys from the science class phase and 121 completed surveys for our history phase. in order to complete the survey, students had to have taken a science class or a history class within the past two semesters. see table 1 for information about students who completed surveys. the actual survey can be found in the appendix. table 1 student profiles science students history students • 327 completed surveys • 30 majors represented • 122 completed surveys • 22 majors represented journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 96 • similar distribution of years (1st, 2nd, 3rd, 4th) • 75% (245 students) were age 18– 22 • 79% (258 students) were female, 21% (69 students) were male • similar distribution of years (1st, 2nd, 3rd, 4th) • 68% (82 students) were age 18– 22 • 66% (80 students) were female, 34% (41 students) were male results overlapping findings from professor interviews results reflected that the majority of both the science and history professors had concerns about the effectiveness of student reading, both in terms of their completion and comprehension of the reading. it was also found that the majority of professors were concerned about student compliance. professors reported that students wait until after lectures to do the reading. they also reported that student reading compliance was more of a concern in the introductory and general education (lasc) classes. professors reported that students were more likely to do reading if they earned a concrete reward for doing so, usually a grade. in general, students seemed to make calculations about the importance of reading for classes. in other words, they make judgements about whether it is worth their time to do a particular reading. according to one history professor, “learning is a result of trust and delight. we need to gain the trust of the students that the reading is worthwhile.” both groups of professors found that students had trouble meeting faculty expectations that they would read like practitioners in those fields. one science professor said, “once you’re beyond it, it’s hard to see it as the student. what we want them to do is learn to read like experts in the field. we know how to do that, but it’s difficult to remember how it was before we learned to do that.” in general, professors found that students do not have foundational concepts and lack experience with reading the genres favored in each field. additionally, both groups of professors felt that students prefer videos, podcasts, and visuals in general over reading. many students have more experience with digital and interactive visual online formats over textbook reading. science-specific findings the science professors believed students need more preparation before being able to read science journal articles. they expressed concern that students need training in how to read the genre of journal articles, focusing on the different sections of a research study. the professors also mentioned the importance of specialized vocabulary necessary for students to learn (including math concepts), so they can read successfully in science classes. some of this highly specialized vocabulary may not be seen in other parts of students’ lives, so they have a harder time connecting it to pre-existing knowledge. journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 97 history-specific findings history professors expressed concern for students’ lack of ability to read for the argument a writer is presenting as opposed to only reading for information. professors would like to see students make more of a connection between history classes and other classes, as they read and do classwork. history students are asked to look at many sources on the same material; therefore, students need to be able to understand different sources in relation to each other. they also need to understand complex context surrounding historical texts. the history professors expressed concern about how students often read unskeptically, accepting all history information as the truth. this belief, according to one professor, “is an obstacle to critical interrogation of historical sources and interpretation.” as with science texts, history also has its own unique vocabulary, which professors also saw as a challenge for students. history professors also pointed to a lack of confidence students have in class discussions. they believed students had a fear of being wrong in their interpretations or looking unintelligent during class discussions. student surveys we found that across science and history students, there were no significant differences regarding the number of pages assigned per week (see table 2). likewise, the percentage of students who answered that they “did not complete the assigned reading on most days” was spectacularly high for both history (70%) and science (> 60%) (see table 3). in probing likely reasons for skipping or skimming reading, both science and history majors pointed to a perceived difficulty of the reading. twenty-five percent of history students and 40% of science students reported that the readings they skipped were either difficult or very difficult (see table 4). but both student groups reported that time was a major factor in skipping readings (44% of science students and 50% of history students). notably, only science students explained their skipped readings, choosing “reading was not necessary for me to get good grades on quizzes or exams” (39% of science students) and “reading was not interesting to me” (38% of science students) (see table 5). returning to the question of difficulty, we noted few significant differences when asking students why they found their reading difficult—roughly the same percentage of each group agreed that “text organization/writing style was too hard/dense to work through” (21% science; 18% history) and “text had too many new/difficult/unfamiliar vocabulary words” (18% science; 18% history), but science students were more likely to complain of a lack of background knowledge as a contributor to a feeling of having difficulty with reading (19% science; 11% history) (see table 6). journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 98 table 2 quantity of reading assigned number of pages assigned per week science students history students 20–40 pages 12%(n = 39) 17% (n = 21) 10–20 pages 31% (n = 99) 29% (n = 35) 20–30 pages 25% (n = 80) 26% (n = 32) 1–10 pages 21% (n = 87) 12% (n = 14) table 3 completion of assigned reading science students history students % of students that did not complete the assigned reading on most days > 60% (n = 206) 70% (n = 76) read a portion of the reading on most days 25% (n = 82) skim quickly over the reading most days 27% (n = 33) table 4 perceived level of difficulty science students history students very easy 4% (n = 14) easy 10% (n = 33) neutral 50% (n = 60) not too difficult 45% (n = 142) difficult/ very difficult 40% (n = 126) 25% (n = 26) table 5 reasons for not reading science students history students level of difficulty 18% (n = 48) 10% (n = 10) reading took too long (time constraints) 44% (n = 110) 50% (n = 52) journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 99 reading was not necessary for me to get good grades on quizzes or exams. 39% (n = 105) reading was not interesting to me. 38% (n = 94) table 6 reasons why reading was difficult science students history students text organization/writing style was too hard/dense to work through 21% (n = 60) 18% (n = 19) text had too many new/difficult/unfamiliar vocabulary words 18% (n = 52) 18% (n = 19) missing background knowledge 19% (n = 46) 11% (n = 12) comparison of professors’ and students’ beliefs as figure 1 (pictured below) shows, professors and students in our study had differing perceptions about the importance of reading. students’ decisions were driven by much more pragmatic concerns about immediate consequences of doing or not doing reading and time constraints. professors, on the other hand, held to their beliefs that reading is a long-term investment that is important because of its impacts on a wide variety of practical and intellectual activities, skills, and ways of being. professors did share students’ focus on grades, but students had in their mind a grade they would be okay with, even if it was not the highest grade they could get. in contrast, professors felt that students could get higher grades if they did read and could not support students avoiding reading because they did not care about getting that higher grade. professors of both science and history are concerned about the underlying reasons that students might struggle with college-level reading, and they want to help students with those skills. students do not seem to be aware of or concerned about their lack of these skills. as our survey results show, most students do not consider the assigned texts to be too hard. based on the professors’ concerns and the literature on college-level reading, which characterizes college reading as a very complex activity (horning, 2007), students may not be aware of what they are missing during their interactions with a text. journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 100 figure 1 comparison of professors’ and students’ beliefs discussion the student surveys affirmed what we and the science professors we interviewed believe about student reading: a lot of it is not getting done. the reasons for this go beyond issues within our control as professors. students have competing demands on their time that include work schedules and other classes. there is also pressure from social media, including acquiring the habit of getting information in small, often visual and auditory bites, which may essentially be training students not to dedicate longer blocks of time to reading (horning, 2013a). complexity of text is a contradictory issue between professors and students. in their interviews, professors reported that the dense, advanced language is a challenge for students who are new to college-level texts and can hinder the completion of reading assignments. the students, however, attribute lack of time and the reflection of reading completion in their grade. if we accept that there is some truth to both these views, professors are in a position to help students by explicitly teaching reading skills, restricting the amount of reading assigned for each class, and assigning work that allows students to apply their knowledge of the text. a genre that may need to be reexamined in reading skills is textbooks. research shows that 68% of u.s. faculty required textbooks for classes (collins-dogrul & reading for class will improve grade both time constraints are an issue personal engagement helps with reading application and accountability raise the priority of reading for students students don’t feel by and large that texts are too hard reading’s significance has to do with connection to grades reading is not necessary in all classes to get the desired grade journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 101 saldana, 2019), and while the science professors interviewed had begun helping students to learn how to read journal articles, the skills required to read textbooks received no special attention. there may be an assumption that students already know how to read textbooks, but information collected via interviews with professors indicated that some students may need support with college level reading, including textbooks. direct teaching about the parts of and genre of textbooks holds promise for student reading. the professors we interviewed were very aware of the often-misaligned perceptions of professors and students when it comes to reading. for example, several of the science professors noted that students completed readings only after the lecture because they perceived that the professor would tell them what was important to focus on in the lecture, while the professor generally considered the lecture as a reinforcement of the reading. the survey revealed that students believed the reading was not necessary to achieve the desired grade. repetition of information in powerpoint slides and inclass discussions of main ideas from the reading may be viewed as redundancy, leading students to neglect the reading because it will be covered in class. it is interesting to think about how students perceive the actions we take. our study points to the importance of considering how we as professors might more accurately present ourselves and our values when it comes to reading. the professors we interviewed, in line with other researchers’ reports, are in agreement that more relatable material correlates with better compliance. holding students accountable also seems to be a highly effective motivator. when students are required to apply material in written or oral assignments, there is a higher rate of engagement with reading. additionally, students seem to do better when they have a guide to reading (e.g., guiding questions, a podcast of lecture). finally, professors and study skills experts find that students benefit from mentorship about reading different genres. conclusions this study has implications for college teaching on a number of levels. students in large numbers are not engaged with their assigned reading. the professors’ and students’ perceptions collected in these interviews and surveys show a misalignment between the two groups. although professors have a sense that students are not fully engaged with their reading for class, students’ reasons for not complying include time constraints and perceptions that reading is not necessary or valued by the professor. if we as professors really do value reading, we have some public relations to attend to. college administrators and professors can benefit from our findings about students’ struggles to read college texts. such struggles include not only text difficulty but also fewer tangible issues of reading expectation for each discipline, such as time constraints. students would benefit from the teaching of college-level reading journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 102 strategies and genre study in discipline-area classes, mirroring the wac concerns over writing for particular disciplines. faculty could benefit from professional development workshops to help them incorporate these strategies into their classes. workload is a major factor for students. those who work with college students could benefit from this finding about students’ time conflicts. an increasing number of students work while attending college (lewis, 2020). at a state university like ours, the number of students working and the number of hours they work are very high. regarding students’ and professors’ perceptions about the importance of reading, those tasks that are considered most important will be done first within a very crowded schedule. if reading is very important to students, we need to make sure we can communicate that to them effectively while also being mindful of the quantity of reading assigned and how students can prioritize it. professors in our study and other researchers urge a consideration of balance between the quantity and quality to address both time availability and the effectiveness of reading that does get done. aagaard et al. (2014) found that students were more likely to do reading if it was what they considered short and would be used in class. there is a recognition that quality of reading and engagement is very important and should be considered in relation to quantity of reading. the lack of explicit classes with “reading” in the title is suggestive of the stigma that seems to exist in talking about college reading. at one time, college writing also received far less attention than it does today. the wac movement gained traction with the recognition that writing scientific lab reports was very different from writing political science essays. writing is something concrete that is produced and handed in for a grade. reading is often a personal experience that happens in the interior of the mind. but reading is just as important as writing and is a part of the same skill set that students need (flexibility and the ability to interact with language in different situations) to make them effective communicators in life. in order for students to achieve these skills, college reading must pivot to “across-thecurriculum” as well. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references aagaard, l., conner, t. w., jr., & skidmore, r. l. (2014). college textbook reading assignments and class time activity. journal of the scholarship of teaching and learning, 14(3), 132–145. retrieved from https://eric.ed.gov/?id=ej1035084 journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 103 armstrong, s. l., & newman, m. (2011). teaching textual conversations: intertextuality in the college reading classroom. journal of college reading and learning, 41(2), 6–21. https://doi.org/10.1080/10790195.2011.10850339 brown, c. a., danvers, k., & doran, d. t. (2016). student perceptions on using guided reading questions to motivate student reading in the flipped classroom. accounting education, 25(3), 256–271. https://doi.org/10.1080/09639284.2016.1165124 brownell, s. e., price, j. v., & steinman, l. (2013). a writing-intensive course improves biology undergraduates’ perception and confidence of their abilities to read scientific literature and communicate science. advances in physiology education, 37(1), 70–79. https://doi.org/10.1152/advan.00138.2012 collins-dogrul, j., & saldaña, k. (2019). text selection and course design: faculty perspectives on critical reading and critical thinking. journal of the scholarship of teaching and learning, 19(2), 88–101. retrieved from https://eric.ed.gov/?id=ej1214008 gogan, b. (2013). reading at the threshold. across the disciplines, 10(4). https://doi.org/10.37514/atd-j.2013.10.4.13 greene, s. (1993). the role of task in the development of academic thinking through reading and writing in a college history course. research in the teaching of english, 27(1), 46–75. http://www.jstor.org/stable/40171212 horning, a. s. (2007). reading across the curriculum as the key to student success. across the disciplines, 4. https://doi.org/10.37514/atd-j.2007.4.1.08 horning, a. s. (2013a). reading, writing, and digitizing: understanding literacy in the electronic age. cambridge scholars publishing. horning, a. s. (2013b). elephants, pornography and safe sex: understanding and addressing students’ reading problems across the curriculum. across the disciplines, 10(4). https://doi.org/10.37514/atd-j.2013.10.4.11 hynd, c., holschuh, j. p., & hubbard, b. p. (2004). thinking like a historian: college students’ reading of multiple historical documents. journal of literacy research, 36(2), 141–176. https://doi.org/10.1207%2fs15548430jlr3602_2 lewis, j. s. (2020). reconsidering the effects of work on college student leadership development: an empirical perspective. journal of college student development, 61(5), 539–557. https://doi.org/10.1353/csd.2020.0054 journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 104 pace, d. (2004). decoding the reading of history: an example of the process. new directions of teaching & learning, 2004(98), 13–21. https://doi.org/10.1002/tl.143 phillips, a., & compton, m. (2016). improving reading compliance with whole class qualitative quiz questions. compass: journal of learning and teaching, 9(13). https://doi.org/10.21100/compass.v9i13.335 nilson, n. b. (2010). teaching at its best: a research-based resource for college instructors. jossey-bass. odom, m. l. (2013). not just for writing anymore: what wac can teach us about reading to learn. across the disciplines, 10(4). https://doi.org/10.37514/atd-j.2013.10.4.16 rangachari, p. k., & mierson, s. (1995). a checklist to help students analyze published articles in basic medical sciences. the american journal of physiology, 268(6), s21–25. https://doi.org/10.1152/advances.1995.268.6.s21 rael, p. (2004). “predatory” reading. in reading, writing, and researching for history: a guide for college students, bowdoin college. retrieved from https://courses.bowdoin.edu/writing-guides/reading/predatory-reading/ seidman, i. e. (1991). interviewing as qualitative research: a guide for researchers in education and the social sciences. teachers college press. simpson, m. l., & nist, s. l. (1990). textbook annotation: an effective and efficient study strategy for college students. journal of reading, 34(2), 122–129. http://www.jstor.org/stable/40032053 willingham, d. t. (2010). why don't students like school?: a cognitive scientist answers questions about how the mind works and what it means for the classroom. john wiley & sons. wineburg, s. (1999). historical thinking and other unnatural acts. phi delta kaplan, 80(7). 488–499. journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 105 appendix interview questions for science and history professors how much reading do you assign for each class? how is student compliance for reading completion in your classes? are you happy with the way students are completing reading in your classes? how is student compliance for reading completion in your classes? in what ways do you evaluate student reading compliance? are students held responsible for the readings? if so, in what way(s)? what kinds of reading assignments do you assign, including digital and multimedia texts? (genres) what kinds of assignments do you give students to go along with reading? what challenges do you think students might face with the reading in your classes? what has worked well in the area of reading in your classes? what has not worked well in the area of reading in your classes? what kind of guidance and support do you give your students in the area of reading? when do you give this guidance and support? before reading? during reading? after reading? what special challenges do students face because they are reading in the area of science? what is valued in science in the area of reading and writing—in other words what is science literacy? student survey for science and history dear student, please take a few moments to complete this survey regarding your experience(s) with reading in science classes here at westview university (wsu).1 your input will be very valuable for a research study being conducted here at wsu. please note that all your responses are confidential and completely anonymous. if you would like to be entered into a random drawing for a $50 gift card from the wsu bookstore, please enter your name and email at the end of this survey. your anonymity will still remain if you decide to enter the drawing. there is no way for the researchers to link your name to your responses. thank you faculty learning community team journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 106 question answers are you 18 or older? ____ yes ____ no did you complete a science course in fall 2014? ____ yes ____ no (if answered ‘no’ student will be asked about spring 2014) did you complete a science course in spring 2014? ____ yes ____ no (if answered ‘no’ student will be directed to the end of the survey) *when answering questions about the reading in your science class, please think of the class that met earliest during the school week if you took more than one. what area of science and level was the course? (drop down menu with area of science and level 100-400 listed)* how much reading was assigned for this class per week? ____ 1-10 pages ____ 10-20 pages ____ 20-30 pages ____ 30-40 pages ____ 50+ pages how much of the assigned reading did you do for this class? ____ all reading every day ____ all reading most days journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 107 ____ a portion of reading most days ____ skim quickly over the reading most days ____ none of the reading most days what type of texts did you read for this class (choose all that apply)? ____ textbook ____ journal articles ____ newspaper articles ____ other (please specify): _____________________________ how difficult is the reading for this class based on a scale of 15? very difficult (1) difficult (2) neutral (3) easy (4) very easy (5) if you did not do all the reading for the course, please indicate any of these factors which contributed to your decision (choose all that apply): ____ reading seemed too difficult ____ reading was not required for me to participate in class discussions ____ reading was not necessary for me to get good grades on quizzed or exams ____ reading was not interesting to me ____ reading took too long ____ reading required a journal log ____ reading did not relate to in-class activities ____ na (i did all the assigned readings) ____ other (please specify): if you did not do all the reading for the course, which of these features of the readings ____ text had too many new/difficult/unfamiliar vocabulary words ____ text assumed background knowledge i did not have journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 108 themselves had the biggest impact on your decision (choose all that apply): ____ text organization/writing style was too hard/dense to work through ____ text discussed concepts that were too advance/difficult for me to grasp ____ na (decision was based only on factors unrelated to the difficulty of the text itself – e.g., time constraints or lack of interest) ____ na (i did all the assigned readings) ____ other (please specify): what is your gender? ____ male ____ female age range: ____ 18-22 ____ 23-30 ____ 31-40 ____ 41+ college year: ____ first-year ____ sophomore ____ junior ____ senior what is your major (1st major only) (drop down menu with all undergraduate majors listed) ___________________________ journal of effective teaching in higher education, vol. 5, no. 1 the importance of college reading: professors’ and students’ views 109 if you would like to be entered into the random drawing for a $50 gift card to the wsu bookstore, please enter your name and email address. please remember that the researchers cannot link your name to your responses to this survey. name: __________________________________________ email address: _____________________________________________ if you would be willing to be interviewed about reading in science classes in more depth please enter your name and email address. if not, click on &#39;next&#39; please remember that the researchers cannot link your name to your responses to this survey. name: __________________________________________ email address: _____________________________________________ thank you for completing this survey. your input is very valuable. microsoft word warren.docx journal of effective teaching in higher education, vol. 3, no. 2 culturally responsive perceptions and practices of instructors at a minorityserving institution: a mixed methods study jeffrey m. warren, north carolina central university camille l. goins, university of north carolina at pembroke, camille.goins@uncp.edu leslie a. locklear, university of north carolina at pembroke dana l. unger, university of north carolina at pembroke tiffany m. locklear, university of north carolina at pembroke gerald neal, university of north carolina at pembroke claudia nickolson, university of north carolina at pembroke gretchen g. robinson, university of north carolina at pembroke abstract. the purpose of this mixed-methods study was to examine the culturally responsive perceptions and practices of instructors at a public, minority-serving institution located in the southeast quadrant of the united states. survey data were collected from 34 undergraduate and graduate faculty participants. findings from a hierarchical regression analysis indicated that race or ethnicity and deficit ideology were predictive of instructor-student relationships and effectively communicating expectations. additionally, a thematic analysis of participant responses suggested instructors believe students do not value higher education, and academic advisors should take on a more expansive role. participants minimized the role they play in promoting student success. based on these findings, the authors suggest that universities develop strategic plans to address inequitable policies and practices. furthermore, instructors must challenge beliefs that are detrimental to culturally responsive instruction. a discussion of the findings and implications for culturally responsive instruction in higher education, particularly at minority serving institutions, are included. keywords: culturally responsive instruction, deficit ideology, instructors, higher education according to the u.s. department of education (2016), in the past five years the racial and ethnic minority population has grown by 3%, more than 12% of americans speak spanish as a primary language, more than 50% of college students are women, and 47% of college students are considered nontraditional. as college students in the twenty-first century become more diverse, universities, microcosms of society, remain places where structures are antiquated and align with a shrinking societal mainstream (osman et al., 2018). as such, “the right to learn in ways that develop both individual competence and a democratic community has been a myth rather than a reality for many americans” (darling-hammond, 2006, p. 13). according to osman et al. (2018): the university as a whole: its curriculum, the classroom and its dynamics, the intellectual spaces of knowledge production and sharing, the methodologies—in other words, the pedagogical theories and practices of culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 77 the university—including the governance structures of higher education in general, all need to be questioned (p. 394). osman et al. (2018) stated, “we are particularly keen to return to the idea that higher education and social justice are inherently intertwined and can be achieved through adopting appropriate pedagogical strategies and stances” (p. 394). when social and educational justice is advanced through culturally relevant pedagogical practices, students are positioned to overcome barriers and succeed. however, faculty and instructors often deliver instruction through a privileged lens and in turn fail to adequately meet the needs of diverse students. as such, this article explores instructors’ perceptions of student challenges and recommendations for student support and delivery of culturally responsive instructional practices. culturally responsive instruction according to hammond (2015), culturally responsive instruction is “the process of using familiar cultural information and processes to scaffold learning” (p.156). this instructional approach emphasizes relationships and social awareness and combines affective and cognitive domain functioning to make culture relevant during learning. culturally responsive instruction connects education and social justice and provides a platform to affect social change. it is the instructor’s role to inspire and instill values in all students that lead to equality (bassey, 2016; fairclough, 2007; gay, 2010; ladson-billings, 2002). culturally responsive instructors use cultural characteristics, experiences, and the perspectives of diverse students in a manner that is meaningful to cultural, experiential, and emotional development. according to ladson-billings (2002), instructors are tasked with helping students “develop a broader sociopolitical consciousness that allows them [students] to critique the cultural norms, values, mores, and institutions that produce and maintain social inequalities” (p. 162). culturally responsive instructors acknowledge a relationship between social justice and active learning by engaging students emotionally, cognitively, and behaviorally as socially responsible citizens. morality, ethics, and dispositions become intimately linked to the learning process, helping students to recognize and understand differences between cultures and identify strands of similarity that bind them with others. deficit ideology according to gorski (2016), deficit ideology (di) is the belief that disproportionate outcomes of individuals from underrepresented groups are the result of cultural, intellectual, and moral deficiencies. this ideology identifies the individual as the problem rather than larger systemic issues. di is a means to pathologize marginalized students and is used to explain their lack of academic achievement (sleeter & grant, 2009). according to gorski (2016), di explains and justifies inequitable academic outcomes by pointing to potential deficits in a student’s home environment, community, or culture. as such, the sociopolitical context of the education system and the impact of racism, bias, and economic injustice that succinctly explain academic outcome inequalities are overlooked. gay (2018) culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 78 stressed the importance of understanding and honoring student’s abilities within the classroom in order to create an environment for student growth. the instructor’s ability to understand students’ differing cultures can play a role in student achievement. however, when instructors adopt a di, they typically believe there is very little they can do to help students achieve (sleeter & grant, 2009). university instructors who hold a deficit worldview may acknowledge macro-level factors (e.g., k-12 schooling, economic struggles, family responsibilities) impacting students but believe these factors are outside of their influence. culturally responsive instruction is important, but systemic change is required in order for marginalized students to succeed (gay, 2018). addressing di among instructors can be a first step in advancing culturally responsive instruction and affect social change. purpose and rationale research exploring instructor beliefs of student challenges and culturally responsive instruction is sparse. given the continued increases in diversity at colleges and universities, including minority-serving institutions, this area of inquiry is critical to ensure instructors are delivering relevant and effective instruction that meets the needs of all students. the purpose of this study is to better understand instructors’ culturally responsive teaching perceptions and practices. this research was guided by the following research questions: is there a significant difference in the frequency and desired use of culturally responsive instructional practices? does race/ethnicity and di predict preparedness, frequency of use, or desired use of culturally responsive instructional practices? based on our relevant experiences and a review of literature, the following hypotheses were derived: hypothesis #1: a significant difference exists between the frequency and desire to use culturally responsive practices. hypothesis #2: race/ethnicity and di are significantly related and predictive of culturally responsive instructional practices. additionally, we aimed to better understand instructors’ perceptions of student challenges and their recommendations for supporting students. method research team the researchers were members of a special interest group (sig) through their university’s teaching a learning arm of academic affairs. three members were associate professors, while four were assistant professors; one was a coordinator of a federally funded teacher education program. all researchers held terminal degrees in either education or counseling. in terms of race/ethnicity, three identified as american indian, four identified as white, and one was african american. a content analysis and other quantitative analyses were completed by two of the researchers. a thematic analysis of the qualitative data was conducted by four different members of the research team. culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 79 design a convergent mixed-method design was employed to explore culturally responsive practices and perceptions of instructors at a minority-serving institution. convergent designs provide researchers an opportunity to compare quantitative and qualitative research findings. we concurrently collected all quantitative and qualitative data and independently analyzed them, a routine practice when using a convergence model (creswell & plano clark, 2017). data analysis for integration occurred after all data were collected. once all data were analyzed, the results of the quantitative and qualitative analyses were merged in order to broadly interpret the findings and best understand the phenomena. by utilizing this design, we were able to gain a more thorough or comprehensive understanding of faculty perceptions and culturally responsive instruction. participants university instructors (n = 34) employed at a minority serving public institution in the southern quadrant of the united states participated in this study. participants identified their rank as lecturer (n = 11, 32.4%), assistant professor (n = 9, 26.5%), associate professor (n = 10, 29.4%), and full professor (n = 2, 5.9%). two (5.9%) participants indicated other for rank. most participants (n = 23, 67.6%) held doctoral degrees while 32.3% (n = 11) held master’s degrees. regarding affiliation, 52.9% (n = 18) of participants were instructors in an arts and sciences academic unit (e.g., college of arts and sciences). other instructors were affiliated with education (n = 6, 17.6%) and health sciences (n = 5, 14.7%); two (5.9%) participants indicated other. in terms of teaching responsibility, most participants indicated they taught at the undergraduate level (n = 23, 67.6%). four (11.8%) participants reported teaching at the graduate level. participants’ years of experience as a university instructor ranged from 1–33 years (m = 14.4, sd = 9.4). twenty-two (64.7%) participants identified as white; the remaining instructors identified as african american (n = 2, 5.9%), american indian (n = 2, 5.9%), asian (n = 2, 5.9%), multi-racial (n = 1, 2.9%), and non-residential (n = 1, 2.9%). procedures existent literature was reviewed to better understand teaching practices in higher education. then a research protocol was developed and submitted to the university’s institutional review board (irb) for review. once the study was approved, we disseminated an email that contained a qualtrics survey link that directed potential participants to a research packet. the research packet included informed consent, demographic questionnaire and prompts, and two surveys. the email was distributed to all faculty via the official university faculty listserv. additionally, the email was distributed to all department chairs with a request to disseminate the study to faculty members in their respective departments. the study remained open and available for three weeks. out of approximately 400 instructors, 34 participants completed the survey, a response rate of 8.5%. culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 80 data sources demographic questionnaire participants completed a demographic questionnaire developed by the researchers. the questionnaire included items related to race/ethnicity, earned degree, years of instructional experience, academic rank, and unit affiliation. faculty perception prompts the faculty perception prompts (fpp), developed by the researchers, are two openended questions about perceived student challenges and support strategies. prompt 1 read, “in a sentence or two, please describe why you think some students on our campus struggle academically or are unsuccessful.” prompt 2 stated, “describe what you believe is the #1 thing that can be done to support the students who struggle academically on our campus.” culturally responsive teacher preparedness scale (crtps, hsiao, 2015) the crtps is an 18-item, self-report measure designed to assess the culturally responsive preparedness of pre-service teachers. three subscales comprise the crtps: curriculum and instruction, relationships and expectation establishment, and group belonging formation. the curriculum and instruction subscale includes eight items designed to measure the quality of multicultural curricula, instructional resources, and the instructional methods used to meet student need. six items comprise the relationships and expectation establishment subscale which assesses instructor communication and efforts to engage students in academic success. finally, the group belonging formation subscale uses four items to measure instructional climate. an example of an item on the crtps is, “i am able to communicate expectations of success to culturally diverse students.” participants use a 6-point likert-type scale ranging from 1 (unprepared) to 6 (fully prepared) to report their level of preparedness for cultural responsiveness. scores on the crpts range from 18 (minimum) to 108 (maximum). hsiao (2015) found the crtps and its subscales reliable measures of preparedness for culturally responsive instruction. internal consistency coefficients for the crtps and the curriculum and instruction, relationships and expectation establishment, and group belonging formation subscales are .95, .91, .91, and .88 respectively. for this study, internal consistency for the crtps was .93, while coefficient alphas for the subscales were .89 (curriculum and instruction), .88 (relationships and expectation establishment), and .83 (group belonging formation). culturally relevant teaching survey (crts, rhodes, 2013) the crts is a 34-item, self-report measure of cultural responsiveness in adult learning environments. participants use a 5-point likert-type scale ranging from 1 (never) to 5 (always) to report their frequency of use and desired use of 17 instructional practices. "i use surveys to find out about my students classroom culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 81 preferences," is an example of an item on the crts. the frequency use of subscale assesses the degree to which participants utilize culturally responsive teaching practices. the desired use of subscale measures participants' preferences for incorporating culturally responsive practices in the classroom setting. full scale scores range from 34 to 170, with higher scores representing culturally responsive instruction. the crts has demonstrated convergent validity with the multicultural teaching competency scale (spanierman et al., 2011). in two studies reported by rhodes (2016), both subscales of the crts were found reliable; alpha coefficients ranged from .78 to .88. the crts was found to have acceptable internal consistency in the current study. alpha coefficients for the frequency use of and desired use of subscales were .88 and .95 respectively. the full-scale internal coefficient was .95. data analysis the goal of the quantitative phase was to explore the predictive relationships of race/ethnicity and di and culturally responsive instructional practices. in this phase, a content analysis, as outlined by krippendorff (2013), was conducted as well as other quantitative analyses. qualitative data were transformed; dummy variables derived from the content analysis of the data collected from prompt 1 of the fpp were used to determine pearson product-moment correlation coefficients and conduct a hierarchical regression analysis. these additional analyses were conducted using spss. a separate, qualitative phase offered additional insights into instructor perceptions of student challenges and recommended supports. the purpose of the qualitative phase of this study was to identify themes in the participants’ responses on the fpp. content analysis of prompt responses participants’ responses to prompt 1 of the fpp were analyzed using content analysis. content analysis is an objective, scientific technique used by researchers to make valid and replicable inferences about texts or other written material (krippendorff, 2013). as such, the goal of the content analysis conducted in this study was to explore participant perceptions and beliefs within the context of di. coding frame development. a coding frame was developed based on the concept of di as described by gorski (2016). the definition of di was operationalized within the context of academic struggles and student success in higher education. based on the recommendations of krippendorff (2013), coding instructions were designed to systematically determine whether participant responses to the prompt were rooted in di. examples of possible responses and relevant coding were provided to assist in accurately identifying di. responses that contained any aspect of deficit thinking was dummy coded with a 1. comments that did not include deficit thinking were dummy coded with a 0. coding procedures. the coding team tested the established coding frame using three cases (8.8%) from the sample. this practice is consistent with content culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 82 analyses conducted by barrio minton et al. (2014) and hays et al. (2016). coding occurring during the pilot test indicated consistency in the application of the coding frame. as a result, the remainder of the cases were coded independently by the team members. once all cases were coded, krippendorff’s alpha was used to determine intercoder reliability. the analysis indicated an alpha level of .80, suggesting a minimally acceptable level of reliability (krippendorff, 2013). all discrepancies in the coding were discussed by the team until consensus was reached. once the team agreed on the coding of all responses, the results of the content analysis were used for further investigation. thematic analysis a thematic analysis, as described by corbin and strauss (2015), was utilized during analysis of the qualitative data. data collected from the fpp were placed in microsoft word to assist with comparing typed data responses. the coding team reviewed each case independently and then met collectively to discuss emergent themes and coding discrepancies. each case was read and re-read to identify and code textual and structural descriptors that pertain to instructors’ perceptions of student challenges and recommended support strategies. to enhance trustworthiness, the members of the coding team engaged in bracketing and crossreferenced personal notes using a shared word document. this process allowed the team to reach consensus and agree on a final set of themes and related codes. findings initial analysis preliminary analyses were conducted in order to become familiar with the sample including demographic information and the constructs under investigation. due to small sample sizes of the race/ethnicity subgroups, participant responses for this demographic variable were collapsed into two distinct racial/ethnic categories: white instructors and instructors of color. as such, participants who identified as white remained in the white subgroup. participants who identified as african american, american indian, asian, multi-racial, and non-residential were placed in the instructors of color subgroup. the aggregation of data by majority and minority categories demonstrated in this study is comparable to warren and hale (2020). in order to determine the likelihood of finding significant effects if they exist, an analysis of power was conducted. with alpha level set at .05 and an effect size of .35, the analysis yielded a power of .84 for a sample of 34 participants. additionally, tests of collinearity and error were conducted to ensure basic assumptions were met. descriptive statistics including mean and standard deviation as well as correlation coefficients are provided in table 1. culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 83 table 1 alpha coefficients and descriptive statistics for measures of cultural responsiveness __________________________________________________________________ 2 3 4 5 6 7 m sd 1. c/i .67** .56** .93** .69** .45** .59** 37.44 8.06 2. ree .84** .88** .40* .25 .34* 25.88 4.24 3. gbf .79** .32 .25 .30 22.26 2.19 4. crtps .61** .40* .53** 85.59 12.99 5. fou .73** .91** 55.35 12.48 6. du .95** 62.06 15.93 7. crts 117.41 26.47 __________________________________________________________________ note. *p < .05 **p = .01, c/i = curriculum and instruction, ree = relationships and expectation establishment, gbf = group belonging formation, crtps = culturally responsive teacher preparedness scale, fou = frequency of use, du = desired use, crts = culturally relevant teaching survey. main analysis a paired sample t-test was used to compare the frequency of use of culturally responsive instructional practices to the desired use of culturally responsive instructional practices. there was a significant difference between the frequency of (m = 55.35, sd = 12.48) and desire to (m = 62.06, sd = 15.93) deliver culturally responsive instructional practices; t(33) = -3.59, p = .001. pearson product-moment correlation coefficients were calculated to evaluate the relationships between race/ethnicity (dummy coded; race/ethnicity majority as comparison) and di (dummy coded; di as comparison) and several criterion variables: curriculum and instruction, relationships and expectation establishment, group belonging formation, and the frequency use of and desired use of culturally responsive instructional practices. relationships and expectation establishment was negatively and significantly related to both race (r = -.35, p < .05) and di (r = .43, p < .05). the other criterion variables were not significantly related to either race or di and therefore were not included in additional analyses. a hierarchical regression analysis was conducted to explore the degree to which race and di predict relationships and expectation establishment (see table 2). in the first step of the analysis, the demographic variable race/ethnicity was entered, as suggested by cohen et al. (2003). in the second step, di was entered while the predictor variable in step 1 was held constant. culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 84 table 2 summary of hierarchical regression analysis for variables predicting relationship and expectation establishment __________________________________________________________________ predictor variable b se b β r² ∆ r² step 1 .12 .12 race/ethnicity -3.02 1.45 -.35* constant 27.83 1.17 step 2 .23 .12 race/ethnicity -2.07 1.44 -.24 deficit ideology -3.06 1.42 -.36* constant 29.49 1.38 _________________________________________________________________ note. *p < .05. step 1 of the hierarchical regression analysis indicated a significant relationship between race/ethnicity and relationship and expectation establishment, f(1, 32) = 4.32, p < .05., r2 = .12. race/ethnicity accounts for 12% of the variance explained in relationship and expectation establishment. a combination of race/ethnicity and di in step 2 was significant related with relationship and expectation establishment, f(1, 31) = 4.72, p = .05, r2 = .23 (adjusted r2 = .17). di increased the variance explained in relationship and expectation establishment by 12%. thematic analysis two central themes emerged during this investigation, which sought to understand instructor perceptions of student challenges and recommended support strategies. the themes emphasize responsibility and were identified as (a) instructor responsibility and (b) delegated responsibility. instructor responsibility participants discussed their perceptions related to their role and responsibility as instructors, including their work as academic advisors. two subthemes further articulated (a) academic advising and (b) university support. academic advising. the ways in which instructors thought academic advising would best serve students varied. instructors called for “one-on-one meetings” that would include “meeting with advisees, checking on their progress and midterm grades, and actively engaging with students to address their struggles.” other participants recommended instructors should “build trust and rapport” during the advising process. participants also suggested that advising should focus on the whole student, not merely academics. for example, some participants suggested advising could include social-emotional development to help students address “the root of their particular problem.” still some participants advocated for “trained professional advisors” to provide student support. culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 85 university support. participants described ways the university could foster instructor responsibility. participants suggested the university could train advisors to actively engage with students and provide professional development on “creative” teaching approaches. other participants recommended the development of resources to help instructors work with the whole student and encouraged the university to emphasize the value of instructor-student relationships. delegated responsibility university instructors also discussed delegated responsibilities described as tasks or behaviors deemed the responsibility of others. two subthemes included (a) the lack of academic preparedness and (b) the perception of student motivation. academic preparation. most instructors suggested the lack of academic preparedness led students to struggle in college. one participant shared, “they (students) have not received adequate education prior to entering university studies.” other participants suggested students are “too needy” and are not sufficiently skilled in writing, studying, communication, and time management. additionally, participants suggested that students are the “victims of accountability testing” when explaining their lack of preparation for college level academic work. one participant suggested ineffective instructors/instruction contributes to academic struggles. student motivation. while academic preparedness was a concern for participants, they also noted other factors that impeded student success. many participants believed students lack motivation, grit, emotional intelligence, and interest. for example, one participant commented, “apathy, i don’t think they have a clear grasp of what their education can do for them and are thus apathetic about attending class and meeting deadlines.” other participants suggested students “do not place a high value on education.” participants believed that “poor family values” and “too many personal problems” were roots of the problem. a few participants acknowledged that many students are faced with “the juggling of multiple responsibilities (parenting, jobs, etc.).” however, one participant stated, “they [students] need to put academics first.” discussion intentional analyses were conducted to answer the research questions and evaluate hypotheses related to culturally responsive instructional practices, di, and instructor perceptions of student challenges and recommended support strategies. the first hypothesis that suggested a significant difference exists between the frequency and desire to use culturally responsive practices was supported. the mean score was significantly higher for desired use as compared to frequency of use of culturally responsive instructional practices. instructors may genuinely have a vested interest in refining the manner in which instruction is delivered to meet the needs of culturally diverse students. however, they may not possess the requisite knowledge and skills to strengthen their instructional practices. culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 86 the second hypothesis, race/ethnicity and di are significantly related and predictive of culturally responsive practices, was partially supported. race/ethnicity and di did not predict desired use or frequency of use of culturally responsive practices. however, race/ethnicity and di were related to and predictive of participants’ responses on the relationship and expectation establishment subscale of the crtps. white instructors and those with a di were more likely to have lower scores of relationship and expectation establishment than instructors of color. gorski (2016) noted that di is a view held by many in the educational system. according to civitillo et al. (2018), instructor beliefs can marginalize students while they attempt to deliver culturally responsive instruction. di may inhibit the ability of instructors to form relationships and effectively communicate expectations to students. di may stem from biases held by white faculty that prohibit them from understanding and connecting with minoritized students. various responses to the faulty perception prompts indicate a lack of understanding of minoritized students’ sense of community and perceptions of time and responsibilities. participants considered students were “too needy” and displayed “apathy.” they were also concerned about a lack of responsibility and worried that expectations of behavior founded in white, western culture were at odds with a diverse student body. past research indicates that teachers with limited experiences and understanding of inequality will find it difficult to understand students with different experiences (darling-hammond et al., 2002). additionally, some instructors may not possess the requisite knowledge, attitudes, and skills to effectively connect with marginalized students. tinto (2017) suggested that instructors can foster self-confidence and a sense of belonging among students. however, this seems most achievable when white instructors are able to navigate and neutralize their privileged position. in terms of participants’ perceptions, a divergent interplay between accepting and declining instructional responsibility for challenges students experience was prevalent. participants were eager to support students yet not willing to fully do so within the context of instruction. according to hammond (2015), “one of the goals of education is not simply to fill students with facts and information but to help them learn how to learn” (p. 12). similarly, many participants advocated for their role to extend beyond instruction and span academic support and guidance. while the role of academic advising in student success was highlighted, the relationships instructors form with students in the classroom often are more meaningful (tinto, 2017). when reviewing the quantitative and qualitative aspects of this work, it appears as if university instructors are in a quagmire. participants indicated in the quantitative responses they would like to deliver more culturally responsive instruction, yet countered that desire in qualitative responses that delegated the responsibility elsewhere and had bold threads of di. additionally, qualitative results suggested participants have difficulty relating to and communicating expectations with students, perhaps due to their privileged positions and di that were found in the quantitative responses. these conclusions were supported by findings that suggested participants believe students are incompetent and don’t value education. instructors appear to lack flexibility and the willingness to meet student needs and culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 87 therefore place responsibility elsewhere. these findings suggest grave instructional barriers exist that translate into educational and social injustices, keeping those in privileged positions in power. limitations this study presents valuable findings; however, there are a few limitations. first, the sample may not represent all msis; therefore, the findings may be difficult to generalize. only 34 participants took part in this study, yet this sample size is sufficient to detect a medium effect. additionally, the crtps was designed for use with pre-service teachers, not university instructors, and, therefore, may impede the validity of the findings. finally, participants self-reported, which may have led participants to provide socially desirable responses. implications the findings of this study offer insight into the perceptions and culturally responsive practices of university instructors. several implications for supporting the diverse needs of students emerge for universities and instructors when considering these findings. foundationally, these findings demonstrate a critical need for institutions of higher education to embrace culturally responsive instruction. colleges and universities are encouraged to conduct an institutional analysis of policy and practices of all academic units and identify misalignments with culturally responsive practice. furthermore, an evaluation of current instructional practices is required to develop a targeted and intentional, strategic plan focused on realizing culturally responsive pedagogy in all college classrooms. ongoing professional development is central to a strategic plan emphasizing culturally responsive instruction. professional development should aim to effect changes in instructors’ knowledge, attitudes, and skills related to culturally responsive instruction. for example, a shift to deliberate culturally responsive instruction requires a change in instructor mindset that can only occur through intentional, individual reflection. it is imperative instructors vanquish perspectives akin to di and begin to view all students as possessing assets that can be molded and developed within classroom instruction. instructors are encouraged to develop professional relationships with students and seek to understand their strengths and needs. assessing faculty instructional needs and dispositions to address di is needed prior to providing pedagogical strategies. professional development opportunities should allow for extensive course and syllabi revisioning as instructors work through the complexities of reshaping their classroom environments to meet the needs of all students. hammond (2015) stated, “the culturally-responsive teacher tries to create an environment that communicates care, support, and belonging in ways that students recognize” (p. 20). instructors also are encouraged to actively involve students in course development and revisions. changes in knowledge, attitudes, and skills that culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 88 advance culturally responsive learning environments will impact curricula, faculty engagement, and advising and have endless influence on student development and societal change. furthermore, professional development centered on utilizing the transparency equity framework (winkelmes, 2015) as a teaching practice could potentially enhance the success of a diverse student population. the transparency project focuses on a set of teaching practices that assist faculty with framing intentional dialogue with students about their learning processes including the purpose and design of assignments and class agendas, the unfolding of class discussions, the testing and development of students’ understanding, and the involvement of students in applying an established grading criterion (winkelmes, 2015). academic advisors are valuable contributors to the success of students. the findings of this study suggest that expanded roles of academic advisors may better meet the needs of students. as such, it is important for universities to consider the duties, responsibilities, caseloads, and training of advisors within the context of other student services (ricks & warren, 2020). universities are encouraged to develop and implement structured advising programs that encourage frequent oneon-one meetings; cultivate student-advisor relationships; foster supportive, student-centered dialogue; and emphasize the cultural assets and identity of students. finally, advisors should have prerequisite knowledge to assess for socialemotional concerns and make appropriate referrals to student services offices as necessary. future research while this study offers insight into instructors’ perceptions and pedagogical practices, as noted research on culturally responsive instruction in higher education is sparse. a replication of this study using a larger sample size will allow for additional analysis and insight in instructional practices and the role race, relationships, expectations, and di play in college classrooms. additionally, researchers are encouraged to explore the role di plays in hindering the development of culturally responsive learning environments at msis. non-msis should consider the findings of this study and their implications for faculty development and student success on their campuses. a longitudinal study that examines the impact of culturally responsive teaching on student success, retention, and graduation rates is recommended. finally, future research should include student perceptions of culturally responsive instruction in college and their beliefs of academic support needs. conclusion promoting student success is a shared responsibility and involves students, instructors, advisors, and myriad others on a university campus. this study explored the perceptions and practices of instructors at an msi and highlighted the nuanced approaches to instruction that often present as barriers to student achievement. it is important that universities establish supportive measures to culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 89 address instructor knowledge, attitudes, and skills that are detrimental to culturally responsive instruction and hinder student growth and development. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references barrio minton, c. a., wachter morris, c. a., & yaites, l. d. (2014). pedagogy in counselor education: a 10-year content analysis of journals. counselor education and supervision, 53, 162–177. https://doi.org/10.1002/j.15566978.2014.00055.x bassey, m. (2016). culturally responsive teaching: implications for educational justice. education sciences, 6(4), 35. https://doi.org/10.3390/educsci6040035 civitillo, s., juang, l. p., badra, m., & schachner, m. k. (2018). the interplay between culturally responsive teaching, cultural diversity beliefs, and selfreflection: a multiple case study. teaching and teacher education, 77, 341– 351. https://doi.org/10.1016/j.tate.2018.11.002 cohen, j., cohen, p., west, s. g., & aiken, l. s. (2003). applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). lawrence erlbaum associates. corbin, j., & strauss, a. (2015). basics of qualitative research: techniques and procedures for developing grounded theory (4th ed.). sage. creswell, j. w., & plano clark, v. l. (2017). designing and conducting mixed methods research (3rd ed.). sage. darling-hammond, l. (2006) securing the right to learn: policy and practice for powerful teaching and learning. educational researcher, 35(7), 13–24. https://doi.org/10.3102/0013189x035007013 darling-hammond, l., french, j., and garcia-lopez, s., eds. (2002). learning to teach for social justice. new york: teachers college press. fairclough, a. (2007). a class of their own: black teachers in the segregated south. belknap press of harvard university press. gay, g. (2010). culturally responsive teaching: theory, research, and practice. teachers college press. culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 90 gay, g. (2018). culturally responsive teaching: theory, research, and practice (3rd ed.). teachers college press. gorski, p. (2016). poverty and the ideological imperative: a call to unhook from deficit and grit ideology and to strive for structural ideology in teacher education. journal of education for teaching, 42(4), 378–386. https://doi.org/10.1080/02607476.2016.1215546 hammond, z. (2015). culturally responsive teaching and the brain: promoting authentic engagement and rigor among culturally and linguistically diverse students. corwin. hays, d. g., wood, c., dahl, h., & kirk-jenkins, a. (2016). methodological rigor in journal of counseling & development qualitative research articles: a 15-year review. journal of counseling & development, 94, 172–183. https://doi.org/10.1002/jcad.12074 hsiao, y. (2015). the culturally responsive teacher preparedness scale: an exploratory study. contemporary issues in education research, 8(4), 241– 250. krippendorff, k. (2013). content analysis: an introduction to its methodology (3rd ed.). sage. ladson-billings, g. (2002). but that's just good teaching! the case for culturally relevant pedagogy. in s. j. denbo & l. m. beaulieu (eds.), improving schools for african american students: a reader for educational leaders (pp. 95– 102). charles c thomas publisher. osman, r., ojo, e. & hornsby, d. (2018). transforming higher education towards a socially just pedagogy. journal of human behavior in the social environment, 28(4), 393–396. https://doi.org/10.1080/10911359.2018.1476641 rhodes, christy m. (2016, june 2–5). validation of the culturally responsive teaching survey [paper presentation]. adult education research conference, charlotte, nc, united states. https://newprairiepress.org/aerc/2016/papers/34 ricks, j., & warren, j. m. (2020). transitioning to college: experiences of successful first-generation college students. [manuscript accepted for publication]. journal of educational research and practice. sleeter, c. e., & grant, c. a. (2009). making choices for multicultural education: five approaches to race, class, and gender (6th ed.). wiley. spanierman, l. b., oh, e., heppner, p. p., neville, h. a., mobley, m., wright, c. v., dillon, f. r., & navarro, r. (2011). the multicultural teaching competency culturally responsive perceptions and practices of instructors journal of effective teaching in higher education, vol. 3, no. 2 91 scale: development and initial validation. urban education, 46(3), 440– 464. https://doi.org/10.1177/0042085910377442 tinto, v. (2017). through the eyes of students. journal of college student retention: research, theory & practice, 19(3), 254–269. https://doi.org/10.1177/1521025115621917 warren, j. m., & hale, r. w. (2020). predicting grit and resilience: exploring college students’ academic rational beliefs. journal of college counseling, 23, 154– 167. https://doi.org/10.1002/jocc.12156 winkelmes, m. a. (2015). equity of access and equity of experience in higher education. the national teaching & learning forum, 24(3), 1–4. u.s. department of education. (2016). advancing diversity and inclusion in higher education: key data highlights focusing on race and ethnicity and promising practices. u.s. department of education. https://www2.ed.gov/rschstat/research/pubs/advancing-diversityinclusion.pdf culturally responsive teaching online 18 facilitating culturally responsive teaching through online courses and coaching trish lopez, university of arkansas, trishl@uark.edu janet penner-williams, university of arkansas, rebecca carpenter de cortina, university of arkansas abstract. teacher professional development and education programs are enhancing job-embedded experiences to address the disparity between theory and implementation. simultaneously, higher education is now offering online courses to attract geographically distant educators, especially in high-needs fields such as teaching english learners and culturally and linguistically diverse students. there is a need to investigate what online teacher professional development and education programs can do to promote teachers’ application of what they learn. this pilot study utilized the inventory of situationally and culturally responsive teaching (iscrt) to investigate 23 in-service teachers’ culturally responsive teaching (crt) practices before and after receiving online coursework and coaching. when compared to the control group, treatment teachers’ scores on four of the five iscrt standards—joint productive activity, language and literacy development, challenging activities, and instructional conversations—as well as the composite were statistically significant. findings suggest online crt coursework with complementary instructional coaching supports teachers’ implementation of new knowledge and pedagogy. keywords: culturally and linguistically diverse; english learners; instructional coaching; online teacher learning; culturally responsive teaching teacher educators in higher education are charged with addressing a two-pronged trend from research, which is reflected in the increasing demand for online academic courses (allen & seaman, 2013; soffer & nachmias, 2018) that do not forego the application of job-embedded experiential learning (daniel & pray, 2017; webster-wright, 2009). this is especially true in the areas of culturally responsive teaching (crt) for culturally and linguistically diverse (cld) students, including english learners (els). culturally responsive teaching refers to “using the cultural characteristics, experiences, and perspectives of ethnically diverse students as conduits for teaching them more effectively” (gay, 2002, p. 106) while simultaneously supporting cultural identity, native language, and cultural connections (siwatu, 2007). in response, purposefully designed online coursework has been implemented as a way to provide needed professional development (pd) to teachers of els (sutcher et al., 2019), especially in rural areas where teachers may have difficulty accessing it (hansen-thomas et al., 2016). specific attention within the pd should move “beyond just good teaching” to incorporate knowledge of language and culture, skills in collaboration and leadership, and training on specific instructional strategies for els (he et al., 2011, p. 15; master et al., 2016). teacher education programs in higher education must ensure online el coursework is not only interactive, but also effective at changing crt practices. teacher quality is perceived to be a major predictor of student success (organization for economic cooperation and development [oecd], 2005; rivkin et al., 2005), and teachers of cld students should incorporate crt to provide quality instruction for all students. this pilot study investigates the effects of an online job-embedded pd program that focuses on the implementation of crt practices through applied, interactive coursework, and instructional coaching. the program is designed to provide in-service teachers the competencies and training needed to pass the english to speakers of other languages praxis exam and earn an english as a second language (esl) endorsement. the study was designed to examine this question: does online coursework with coaching support teachers’ implementation of culturally responsive teaching? as such, it adds to previous research on crt, online coursework, and instructional coaching while touching on collaborative online learning communities (cullen et al., 2013). the article provides insights that may be of interest to teacher educators looking to implement a hybrid online program, those who wish to incorporate a coaching component in online courses or those who hope to increase implementation of crt. culturally responsive teaching an assets-based perspective toward cld students’ cultural and linguistic differences promotes instructional equity and improves academic and personal results for students (garcía & kleifgen, 2010; gay, 2010; villegas & lucas, 2002). gay (2013) advocated crt as a method for improving achievement through accepting differences as not only normative but also valuable to societal and personal development. crt situates learning in students’ lived experiences and frames of reference to contextualize learning, set high expectations, and create equal status among all students (gay, 2002; herrera, 2016; ladson-billings, 2014). however, scholars have highlighted the need for concrete examples of what crt looks like in practice (howard & rodriguez-minkoff, 2017; sleeter, 2012). one method to identify crt in practice is the inventory of situationally and culturally responsive teaching (iscrt), a rubric with twenty-two indicators of crt placed on a continuum from not observed to integrating (herrera & kavimandan, 2019). the iscrt is based on the standards for effective pedagogy and learning from the center for research on education, diversity, & excellence (crede). the five standards focus on teachers and students producing together, cross-curricular language and literacy development, connecting content to students’ lives, teaching complex thinking, and teaching through conversations (herrera & kavimandan, 2019). considering these five standards enables teachers to view instruction through a culturally responsive lens and requires them to address content learning alongside language development in order to further challenge els (bunch, 2013; o’hara & pritchard, 2008). the online coursework developed for this study incorporated components of the iscrt with the state’s english language proficiency (elp) standards. to achieve equitable education, teachers need to improve upon elp standards by incorporating appropriate instructional scaffolds for each student (coady et al., 2016) and applying crt. crt applied to online coursework and coaching coursework provided concrete illustrations of what crt looked like in a variety of grade levels, content areas, and student language proficiency levels through instructor-provided examples and participants’ applications that were shared with others through discussion boards, shared videos, virtual classrooms, and recorded audio comments. the courses were based on the state’s competencies for esl teachers, grades k-12, and the content requirements of the english to speakers of other languages praxis ii exam. ample research provided a framework for crt methods and content knowledge (fenner & snyder, 2017; freeman & freeman, 2014; herrera 2016; wright, 2015). for example, in the course second language methodologies, teachers focused on the standard of developing language and literacy across the curriculum. after reading material on the topic, teachers commented on the importance of both content and language objectives in content area classrooms through asynchronously recorded audio-visual comments. later in the course, the topic was revisited when the teachers were asked to submit a lesson plan that incorporated an instructional strategy with content and language objectives. after receiving feedback from the course instructors, the teachers recorded themselves teaching the lesson and then reflected on the lesson’s effectiveness for els. providing teachers with the opportunity to learn through experience and reflect on that learning aligns with research on professional learning (webster-wright, 2009) and helps teachers apply, enact, and adjust crt in their everyday pedagogy. in addition to providing concrete examples of crt, the instructors also created the coursework following extant research on effective practices for online instruction and professional development. research since 2008 suggests that high-quality online coursework should be well designed, provide engaging interaction, and have well-prepared, resource-supported instructors (sun & chen, 2016). the online coursework incorporated these as well as the seven features of high-quality pd proposed by darling-hammond and colleagues (2017). the pd was content focused and active learning was involved as teachers applied course content to their current classroom. collaboration was achieved through use of asynchronously recorded comments and discussion boards. models of effective practice were provided through videos and exemplar lesson plans. instructional coaching was conducted as a support for the coursework. reflection was required for the majority of the pd assignments and feedback was provided by the instructors. the pd was sustained, as participants completed four eight-week long asynchronous online courses over the course of one academic year. coaching provided an additional level of practice and guidance to reach high levels of implementation with appropriate scaffolds for content standards and students’ levels of english proficiency. coaching has gained traction in the international education arena as a potentially beneficial aid for teachers and administrators to implement learned knowledge and skills in instruction (rebora, 2019; van nieuwerburgh, 2012). while there is dispute over the definition, coaching usually refers to helping a person improve performance or achieve goals through structured one-on-one conversations (van nieuwerburgh, 2012). this study employed instructional coaching (knight, 2007). instructional coaching is based on implementing a partnership approach to analyze current reality, set goals, implement new strategies, and provide support until the goals are met (knight, 2018). there is research support for instructional coaching as an effective tool for teachers’ practice and student achievement outcomes (knight et al., 2018) in terms of increasing teachers’ use of a new practice (cornett & knight, 2009; teemant et al., 2011), implementing visible learning (knight, 2019), and implementing principles of learning for diverse students (teemant, 2014). furthermore, a meta-analysis of 60 studies of teacher coaching programs within the u.s. and other countries found large positive effects of coaching on teachers’ instructional practice and a positive effect on student achievement (kraft et al., 2018). kraft and colleagues (2018) advocated for web-based, virtual coaching to decrease cost and time constraints while increasing teacher-coach contact, suggesting there is an opportunity for coaching to be applied within an online course structure. within the online courses in this study, in-service teacher participants met with an instructional coach in a dialogical approach to work towards a culturally responsive goal correlated to an iscrt indicator. participants met with coaches face-to-face or virtually for one to three coaching sessions. the online coursework incorporated video, observation, and self-reflection to support instructional coaching. additional communication between coaches and participants occurred in a semi-structured way through follow-up emails and phone calls. a brief, illustrative example of one teacher’s work in coaching sessions shows how instructional coaching allowed course instructors and participants to collaboratively mediate course material. based on an initial observation and a coaching conversation a pre-kindergarten teacher selected incorporating descriptive language into math, targeting the use of comparative adjectives. she created visual representations, found real objects that represented these terms, and used books in the classroom to reinforce these concepts. through continued conversations and additional coaching sessions, the teacher reflected that students were in progress to achieving proficiency of these comparative words. instructional coaching allowed this teacher to apply the language acquisition process into the specific context of a pre-kindergarten classroom. methods data collection and analysis twenty-three in-service teachers from seven different school districts in a southeastern state took part in this study. these pk-12 grade teachers self-selected to enroll in the crt coursework, coaching, and research. verbal and written consent to participate in the study was obtained from each teacher. most participants were female (22/23, 96%). of the 14 participants who identified race, 79% identified as white, 7% identified as hispanic/latina, 7% identified as two or more races, and 7% identified as native american, american indian, or alaskan native. regarding years of teaching experience, more participants fell in the 0–9 year range of experience (78%), and fewer participants (22%) fell into the 10–20 year range. these teachers work in a state with a 24.7% rate of poverty for children under 18 years old (american community survey, 2018). during the study, the majority of teachers (96%) worked in the northwest region where the majority of els living in the state attend school (office of education policy, 2019). this pilot study implemented a nonequivalent group, pretest-posttest design (gribbons & herman, 1997; patten, 2000). the participants described above, who voluntarily enrolled in coursework and coaching, were defined as the treatment group. program directors and research associates served as coaches. coaches were paired with treatment teachers based on the subject area in which a treatment teacher was being coached. all treatment teachers were coached. to establish the control group, each treatment teacher was asked to identify a control teacher who met the following criteria: (a) employed by the same district, (b) taught at the same grade level and/or subject, (c) did not have an esl endorsement, and (d) had a classroom with similar student demographics. treatment and control teachers were observed twice using the iscrt (herrera & kavimandan, 2019), allowing for a thorough comparison of teachers’ level of enactment of crt practices before and after the treatment teachers received coursework and coaching. the first observation occurred within the first three weeks of the start of coursework (fall 2018) to determine a baseline for comparison. similarly, the second observation occurred within the last three weeks of the treatment teachers completing the coursework (spring 2019). the iscrt is a valid, reliable measure of teachers’ culturally responsive teaching practices (herrera et al., 2011; penner-williams et al., 2019). additional observations for coaching conversations occurred between the two scored observations. the iscrt includes 22 observable indicators categorized under five standards. the trained observers rated teacher performance on each of the 22 indicators using a scale of zero to four. a score of zero suggests the indicator was not observed, while four indicates the teacher integrated the instructional practice throughout the lesson. the scores were then averaged together based on their associated standard. additionally, a quantitative composite score for each teacher was calculated by averaging all 22 indicators. trained observers conducted both preand post-treatment observations in two pairs. a cronbach's alpha analysis assessing the internal reliability of the instrument yielded a composite score alpha of 0.94. inter-rater reliability was tested using a single-rating, absolute-agreement, two-way random-effects intra-class correlation coefficient for each pair. for one pair, the intra-class correlation coefficient was 0.71, indicating moderate reliability. the other pair had good reliability with an intra-class correlation coefficient of 0.90. the same rater observed the same teachers for both preand post-observations. data collected from the iscrt were analyzed using independent samples t-tests in two ways to investigate if there was an effect on treatment teachers’ crt practices. first, treatment teachers’ scores on the five standards and composite were compared to control teachers’ scores. the purpose of this test was to examine if the treatment and control groups were statistically significantly different in their implementation of crt before treatment. second, pre-observation scores for the treatment and control teachers were subtracted from the post-observation scores to create a difference score. this difference score was analyzed using an independent samples t-test separated by the treatment grouping variable to determine if there were statistically significant differences between the treatment and control groups across the two trials. in addition to the iscrt observations, treatment teachers completed surveys at the end of each course. surveys were designed to elicit teachers’ perceptions of the effectiveness of the courses in increasing their knowledge and skills related to crt practices and course topics. the surveys were tailored to the material covered in each course and featured one to three likert-scale questions. the likert scale ranged from 1-“not effective” to 4-“very effective.” treatment teachers’ responses on the questionnaire for likert-scale questions were aggregated to obtain an overall percentage of effectiveness as reported by the participants. at the end of each survey, there was an open-ended, optional question for additional comments, and teachers responded to open-ended questions on coaching at the end of the year-long program. the anecdotal, open-response data was also collected and included to provide a more complete understanding of teachers’ ratings and their impressions of the coaching process. change in treatment teachers’ culturally and linguistically responsive teaching practices analyses of the descriptive characteristics of the difference values on the five standards and composite score on the iscrt divided by group (control, treatment) revealed normal distributions (shapiro-wilk statistic, min. p = 0.103) with acceptable skewness (min. = -0.48, max. = 0.71), kurtosis (min. = -1.09, max. = 1.76), and variability. additional visual analyses of the histograms were also conducted. the scores in all standards and on the composite score were relatively low, but there were noticeable differences between means which were explored to determine if there was statistical significance. as teacher participants were not randomly assigned to treatment and control groups, the first analysis sought to provide evidence that the two groups were essentially equivalent in levels of implemented crt before the coursework with coaching began. independent samples t-tests were conducted on the five standards and composite score to compare the mean scores of treatment (n = 23) and control (n = 23) groups on the pre-observation scores, and variability was examined using levene’s test of homogeneity of variance. the results of the t-tests (table 1) suggest the groups were not statistically significantly different in their observed levels of crt before coursework with coaching began. the treatment and control groups also had similar variability in all standards and the composite score. table 1 iscrt standard and composite scores, control and treatment, pre-observation standard control n = 23 m (sd) treatment n = 23 m (sd) p levene’s test for equality of variances (p-value) joint productive activity, pre 1.40 (0.47) 1.37 (0.45) 0.850 0.733 lang. & lit. development, pre 1.08 (0.58) 1.24 (0.63) 0.366 0.993 contextualization, pre 0.64 (0.40) 0.83 (0.50) 0.166 0.257 challenging activities, pre 1.48 (0.47) 1.39 (0.50) 0.545 0.239 instructional conversation, pre 1.30 (0.60) 1.27 (0.53) 0.877 0.787 iscrt composite, pre 1.17 (0.38) 1.20 (0.41) 0.773 0.329 after this initial analysis establishing the relatively equivalent implementation of crt for the treatment and control groups, an independent samples t-test was conducted on the difference score between the preand post-observation scores for each standard and composite scores to explore the impact of coursework supported by coaching on teachers’ crt over time. the results of the independent samples t-test (table 2) suggest the treatment and control groups were statistically significantly different in their observed levels of crt after coursework with coaching. table 2 iscrt scores, control and treatment differences, post observation standard control n = 23 m (sd) treatment n = 23 m (sd) sig. t-test p-value levene’s test for equality of variances (p-value) effect size (cohen’s da) joint productive activity -0.01 (0.62) 0.76 (0.58) <0.001 0.914 1.27 lang. & lit. development 0.00 (0.74) 0.54 (0.59) 0.009 0.268 0.81 contextualization 0.13 (0.63) 0.42 (0.84) 0.191 0.258 - challenging activities -0.26 (0.49) 0.67 (0.53) <0.001 0.697 1.82 instructional conversations 0.52 (0.69) 0.83 (0.60) <0.001 0.828 1.20 iscrt composite 0.18 (0.75) 1.08 (0.75) <0.001 0.875 0.83 note: acohen (1988) for all standards and the composite score, with the exception of the standard contextualization, there were significant results for the difference in scores in favor of the treatment group. combining the results of the significance test with an examination of the means suggests there were significant differences between the groups’ iscrt observation scores, with the treatment teachers’ scores increasing in observed levels of crt over time. effect sizes were also calculated using cohen’s d (1988). cohen’s d effect size ranged from 0.81 for the standard language and literacy development to 1.82 for the standard challenging activities. this suggests the treatment and control groups differed by 0.81 to 1.82 standard deviations on four standards: joint productive activity (d = 1.27); language and literacy development (d = 0.81), challenging activities (d = 1.82), and instructional conversations (d = 1.20), as well as the composite score (d = 0.83). additional visual analyses of the plots of the preand post-estimated marginal means separated by control and treatment supported these findings. these analyses suggest there is a positive difference for the treatment group across trials on the composite score and the standards joint productive activity, language and literacy development, challenging activities, and instructional conversation. treatment teachers’ perceptions of effectiveness of the online courses treatment teachers’ responses on likert-scale questions regarding the effectiveness of each course were aggregated to obtain an overall percentage of perceived effectiveness. the number of effectiveness questions per course ranged from one to three, so responses were averaged by participant per course before being aggregated for the overall course effectiveness rating. percentages of survey responses indicating “effective” or “very effective” are reported in figure 1, and table 3 displays the ratings, means, and standard deviations for these survey questions. figure 1 treatment teachers’ perceived effectiveness ratings note: effectiveness ratings by course. table 3 ratings, means, and standard deviations for effectiveness survey questions effectiveness rating survey question 1 2 3 4 n mean (sd) second language assessment creating and implementing assessments to enhance learning and success for english learners 0 0 11 10 21 3.48 (0.50) using information gained from various assessment types for english learner instruction 0 1 8 12 21 3.52 (0.59) second language acquisition identifying the stages of language development 0 3 7 9 19 3.32 (0.73) the relationship between first and second language development 0 0 9 10 19 3.53 (0.50) strategies to simultaneously promote content learning and second language acquisition 0 2 10 7 19 3.26 (0.64) second language methodologies creating lesson plans, learning goals and language objectives to further develop english skills across the curriculum 0 1 4 18 23 3.74 (0.53) implementing instructional strategies that allow cld students and english learners to demonstrate student learning. 0 0 4 19 23 3.83 (0.38) integrating instructional practices that apply knowledge of teaching academic vocabulary and target language. 0 0 6 17 23 3.74 (0.44) teaching people of other cultures dealing with students from different cultures. 0 2 7 11 20 3.45 (0.67) each course was perceived by the participants to be effective, with effectiveness rating for each course ranging between 90% and 98.55%. in the open-ended survey responses, treatment teachers expressed that they felt supported by the instructors and the courses were informative though challenging. treatment teachers’ perceptions of coaching participants received a median of two face-to-face instructional coaching sessions. of the 23 teachers enrolled, 18 met with an instructional coach twice, three met once, and two met for three times. participants were asked to respond to an open-ended question to reflect on the effectiveness of coaching. they expressed that the coaching sessions helped develop their goals and provided useful feedback. one teacher situated her response in the broader demands placed on teachers: in education, we’re often given a list of things that we need to accomplish daily, weekly, and annually. the coaching experience allows us to take the list of expectations and walk with someone else through the process of applying it appropriately to your age group of students and the diverse needs that you have in the classroom. teachers that commented on the coaching experience felt the coach helped them navigate implementing the newly learned information into their classrooms. discussion the purpose of this pilot study was to determine the effectiveness of a purposefully designed job-embedded online pd program for in-service teachers that incorporated interactive crt coursework, practical field experience, and instructional coaching. through a dialogical framework in the form of discussion boards, asynchronous audio-visual comments, and coaching conversations, treatment teachers increased their crt knowledge and skill set for el and cld students. teachers also reflected on their crt through coursework assignments and post-observation reflections. the results of the structured classroom observations are the first measure of this program’s effectiveness. while both treatment and control teachers had essentially equivalent levels of implementing crt practices in the beginning, treatment teachers had positive, statistically significant differences in their observed levels of crt after completing coursework and coaching. the largest effects of intervention were in relation to iscrt standards: (a) joint productive activity, in which the teacher facilitates learning through academic, linguistic, and cultural commonalities while students work in collaborative groups; (b) challenging activities, in which the teacher simultaneously motivates and challenges students; and (c) instructional conversation, in which the presence of unscripted dialogical communication between teachers and students bridges prior and new knowledge (herrera & kavimandan, 2019). the standard of language and literature development, in which the teacher takes an assets-based perspective to develop language by utilizing the students’ funds of knowledge (amanti et al., 2005), also showed a smaller positive effect. the overall iscrt composite score for all treatment teachers was positively statistically significant, but this difference could have been even greater if the values of contextualization would have also increased. both control and treatment teachers showed a decrease in this standard, which is often difficult for teachers who are new to learning about and implementing crt (herrera et al., 2011; wyatt, 2015). contextualization is difficult because of the dual challenges of limited time to get to know students and of finding a mutual point of reference to unify diverse perspectives (wyatt, 2015). no teacher self-selected a culturally responsive goal on contextualization for the coaching sessions. perhaps with this additional support, treatment teachers would have implemented higher levels of crt on this standard and their composite scores. treatment teachers perceived all courses as effective. the courses second language assessment and second language methodologies were rated “very effective,” with averages of 97.62% and 98.55% respectively. the courses teaching people of other cultures and second language acquisition were rated effective, with respective averages of 90.00% and 91.23%. treatment teachers shared they found second language methodologies gave them greater insights on creating lesson plans, learning goals, and language objectives while implementing instructional strategies specifically for el and cld students. the teaching people of other cultures course stressed the foundational nature of cultural awareness when working with cld students. second language assessment spoke to the creation and implementation of effective assessments in order to evaluate and enhance el learning while the scope of second language acquisition was to learn the stages of language development to promote content learning and acquisition. teachers’ perception of the program as providing strong preparation increases a sense of efficacy (podolsky et al., 2017). overall, treatment teachers believed the coursework was a relevant program component that increased their crt knowledge and learning of cld and el instructional strategies. the coursework, due to intentional planning of interactive platforms, such as discussion boards and asynchronous audio-visual comments, functioned as an online, facilitative learning community (cullen et al., 2013). accordingly, the courses provided treatment teachers with a shared online meeting place where they were able to (a) facilitate knowledge gains, (b) apply learning, (c) reflect on their own professional development, and (d) examine how their newfound knowledge and skill set affects their students’ learning. through this pilot study, it appears that instructional coaching played a significant and supportive role as participants navigated learning how to implement the various crt theories and strategies learned. the dialogical approach used by the instructors utilizes personalization in creating and meeting the goals the teacher sets through professional interaction and partnership (knight, 2018). throughout the program, participants and coaches worked together to discuss goals, strategies, and methodology. with this guidance teachers had the unique opportunity to formatively practice what they had learned within their school context, simultaneously building efficacy as they shared ideas in this dialogical approach. the present study is not without limitations. the selection of both the treatment and the control groups were not random. the treatment group teachers may have certain characteristics that make them more open to changing instructional practices as indicated by their willingness to enroll in the coursework. the treatment teacher purposely chose the corresponding control teacher. to try to minimize the effects of these non-random groups, the researchers created parameters for the selection of a control teacher. while it is impossible to control for the multitude of factors that influence teacher performance, the statistical equivalence between treatment and control groups on the pre-observation suggest that these groups were relatively equivalent at the beginning of the study in terms of crt. an additional limitation to the study is the small sample size. the study was intended to be a pilot of the program’s design and features and to contribute to future programmatic changes. the pretest-posttest analysis allowed for a thorough comparison to be made between the treatment and control groups. accordingly, the researchers are confident this study presents promising results that may be applied to a larger group in future research. implications with an increasing number of el and cld students comprising the student body in u.s. pk-12th schools, the achievement gap between els and non-els is gaining attention (crawford, 2013; gibson, 2016). research supports teacher quality as a determiner of student achievement, and crt is widely accepted as quality teaching for els (garcía & kleifgen, 2010; gay, 2013, 2010; herrera, 2016; ladson-billings, 2014; villegas & lucas, 2002). previous research, however, has not focused on comprehensive changes in teacher crt practices after online pd courses and coaching. in response, the current study examined and compared the change in treatment teacher use of crt practices after online courses and coaching. the evidence from this pilot study points to a positive change in teachers’ use of crt practices. a purposively selected control group of teachers who did not participate in the pd and did not increase use of crt techniques point to a correlation between treatment and increased crt practices. future research could build off of these findings by separating the coaching and coursework as variables or using a comparative interrupted time series design (kim & steiner, 2016) to observe teachers’ implementation of crt with and without coaching. in addition, the standard of contextualization should be addressed specifically in future coursework and coaching. with a growing population of k-12 culturally diverse students (u.s. department of education, national center for education statistics [nces], 2019) teachers in higher education, particularly those preparing future teachers and those working with in-service teachers, need to provide elements of crt within their coursework. teachers in higher education may also need to take the coach’s role in assisting teachers to adopt new crt practices, especially in the area of contextualization. preservice teachers need to apply crt practices in their field placements and in-service teachers need to be taught and supported in applying crt practices in their current classroom. while further research is needed, this pilot study suggests that quality pd may increase teachers’ use of crt and, in turn, result in higher quality teaching for el and cld students. furthermore, this study provides evidence that quality online pd for teachers of el and cld students should focus on crt and use dialogical coaching to support teachers. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references allen, i. e., & seaman, j. (2013). changing course: ten years of tracking online education in the united states [pdf file]. retrieved from https://files.eric.ed.gov/fulltext/ed541571.pdf amanti, c., gonzalez, n., & moll, l. c. (2005). funds of knowledge: theorizing practices in households, communities, and classrooms. routledge. american community survey (2018). poverty status in the past 12 months, table s1701. 2018 american community survey 1-year estimates. retrieved from https://www.census.gov/programs-surveys/acs bunch, g. g. (2013). pedagogical language knowledge: preparing mainstream teachers for english learners in the new standards era. review of research in education, 37(1), 298–341. https://doi.org/10.3102/0091732x12461772 coady, m. r., harper, c., & de jong, e. (2016). aiming for equity: preparing mainstream teachers for inclusion or inclusive classrooms? tesol quarterly, 50(2), 340–368. https://doi.org/10.1002/tesq.223 cohen, s. s. (1988). practical statistics. edward arnold. cornett, j., & knight, j. (2009, april). studying the impact of instructional coaching. paper presented at the annual meeting of the american educational research association, san diego, ca. https://www.instructionalcoaching.com/research/ crawford, e. r. (2013). bridging the english learner achievement gap: essential lessons for school leaders. journal of educational administration, 51(6), 871–874. https://doi.org/10.1108/jea-02-2013-0026 cullen, r., kullman, j., & wild, c. (2013). online collaborative learning on an esl teacher education programme. elt journal, 67(4), 425–434. https://doi.org/10.1093/elt/cct032 daniel, s. m., & pray, l. (2017). learning to teach english language learners: a study of elementary school teachers’ sense-making in an ell endorsement program. tesol quarterly, 51(4), 787–819. https://doi.org/10.1002/tesq.347 darling-hammond, l., hyler, m. e., & gardner, m. (2017). effective teacher professional development. learning policy institute. fenner, d. s., & snyder, s. (2017). unlocking english learners’ potential: strategies for making content accessible. corwin. freeman, d. e., & freeman, y. s. (2014). essential linguistics: what teachers need to know to teach esl, reading, spelling, grammar (2nd ed.). heinemann. garcía, o., & kleifgen, j. a. (2010). educating emergent bilinguals: policies, programs, and practices for english language learners. teachers college press. gay, g. (2002). preparing for culturally responsive teaching. journal of teacher education, 53(2), 106–116. gay, g. (2010). culturally responsive teaching: theory, research, and practice (2nd ed.). teachers college press. gay, g. (2013). teaching to and through cultural diversity. curriculum inquiry, 43(1), 48–70. https://doi.org/10.1111/curi.12002 gibson, c. (2016). bridging english language learner achievement gaps through effective vocabulary development strategies. english language teaching, 9(9), 134–138. gribbons, b., & herman, j. (1997). true and quasi-experimental designs. practical assessment, research, & evaluation, 5(14). https://doi.org/10.7275/fs4z-nb61 hansen-thomas, h., richins, l., kakkar, k., & okeyo, c. (2016). i do not feel i am properly trained to help them! rural teachers’ perceptions of challenges and needs with english-language learners. professional development in education, 42(2), 308–324. https://doi.org/10.1080/19415257.2014.973528 he, y., prater, k., & steed, t. (2011). moving beyond “just good teaching”: esl professional development for all teachers. professional development in education, 37(1). https://doi.org/10.1080/19415250903467199  herrera, s. (2016). biography-driven culturally responsive teaching (2nd ed.). teachers college press. herrera, s. g., & kavimandan, s. k. (2019, october). inventory of situationally and culturally responsive teaching. in center for intercultural and multilingual advocacy [cima] training session conducted at kansas state university. herrera, s. g., perez, d. r., kavimandan, s. k., holmes, m. a., & miller, s. s. (2011, april). beyond reductionism and quick fixes: quantitatively measuring effective pedagogy in the instruction of cld students. paper presented at the annual meeting of the american educational research association, new orleans, la. howard, t., & rodriguez-minkoff, a. (2017). culturally relevant pedagogy 20 years later: progress or pontificating? what have we learned, and where do we go? teachers college record, 119(1). kim, y., & steiner, p. (2016). quasi-experimental designs for causal inference. educational psychologist, 51(3–4), 395–405. https://doi.org/10.1080/00461520.2016.1207177 knight, d., hock, m., skrtic, t. m., bradley, b. a., & knight, j. (2018). evaluation of video-based instructional coaching for middle school teachers: evidence from a multiple baseline study. the educational forum, 82(4), 425–442. https://doi.org/10.1080/00131725.2018.1474985 knight, j. (2007). instructional coaching: a partnership approach to improving instruction. corwin. knight, j. (2018). the impact cycle: what instructional coaches should do to foster powerful improvements in teaching. corwin. knight, j. (2019). instructional coaching for implementing visible learning: a model for translating research into practice. education sciences, 9(2), 101. https://doi.org/10.3390/educsci9020101 kraft, m. a., blazar, d., & hogan, d. (2018). the effect of teacher coaching on instruction and achievement: a meta-analysis of the causal evidence. review of educational research, 88(4), 547–588. ladson-billings, g. (2014). culturally relevant pedagogy 2.0: a.k.a. the remix. harvard educational review, 84(1), 74–84. master, b., loeb, s., whitney, c., & wyckoff, j. (2016). different skills? identifying differentially effective teachers of english language learners. elementary school journal, 117(2), 261–284. https://doi.org/10.1086/688871 office of education policy. (2019). a brief history of english language learners in arkansas. policy briefs, 16(4). http://www.officeforeducationpolicy.org/a-brief-history-of-english-language-learners-in-arkansas/ o’hara, s., & pritchard, r. h. (2008). meeting the challenge of diversity: professional development for teacher educators. teacher education quarterly, 35(1), 43–61. organization for economic cooperation and development [oecd] (2005). teachers matter: attracting, developing and retaining effective teachers. oecd. patten, m. l. (2000). understanding research methods: an overview of the essentials (2nd ed.). pyrczak pub. penner-williams, j., diaz, e., & gonzales worthen, d. (2019). sustainability of teacher growth from professional development in culturally and linguistically responsive instructional practices. teaching and teacher education, 86, 1–13. podolsky, a., kini, t., bishop, j., & darling-hammond, l. (2017). sticky schools: how to find and keep teachers in the classroom. phi delta kappan, 98(8), 19–25. rebora, a. (2019). expanding on coaching’s potential. educational leadership, 77(3), 9. rivkin, s. g., hanushek, e. a., & kain, j. f. (2005). teachers, schools, and academic achievement. econometrica, 73(2), 417–458. https://doi.org/10.1111/j.1468-0262.2005.00584.x siwatu, k. o. (2007). preservice teachers’ culturally responsive teaching self-efficacy and outcome expectancy beliefs. teaching and teacher education, 23(7), 1086–1101. https://doi.org/10.1016/j.tate.2006.07.011 sleeter, c. (2012). confronting the marginalization of culturally responsive pedagogy. urban education, 47(3), 562–584. https://doi.org/10.1177/0042085911431472 soffer, t., & nachmias, r. (2018). effectiveness of learning in online academic courses compared with face‐to‐face courses in higher education. journal of computer assisted learning, 34(5), 534–543. https://doi.org/10.1111/jcal.12258 sun, a., & chen, x. (2016). online education and its effective practice: a research review. journal of information technology education: research, 15, 157–190. http://www.informingscience.org/publications/3502 sutcher, l., darling-hammond, l., & carver-thomas, d. (2019). understanding teacher shortages: an analysis of teacher supply and demand in the united states. education policy analysis archives, 27(35). http://dx.doi.org/10.14507/epaa.27.3696 teemant, a. (2014). a mixed-methods investigation of instructional coaching for teachers of diverse learners. urban education, 49(5), 574–604. https://doi.org/10.1177/0042085913481362 teemant, a., wink, j., & serena, t. (2011). effects of coaching on teacher use of sociocultural instructional practices. teaching and teacher education, 27(4), 683–693. u.s. department of education, national center for education statistics [nces], (2019). the condition of education 2019. english language learners. fast facts. https://nces.ed.gov/fastfacts/display.asp?id=96 van nieuwerburgh, c. (2012). coaching in education: getting better results for students, educators, and parents. karnac books. https://doi.org/10.4324/9780429473036 villegas, a. m., & lucas, t. (2002). preparing culturally responsive teachers: rethinking the curriculum. journal of teacher education, 53(1), 20–32. https://doi.org/10.1177/0022487102053001003 webster-wright, a. (2009). reframing professional development through understanding authentic professional learning. review of educational research, 79(2), 702–739. https://doi.org/10.3102/0034654308330970 wright, w. e. (2015). foundations for teaching english language learners: research, theory, policy, and practice (2nd ed.). caslon publishing. wyatt, t. (2015). understanding the process of contextualization. multicultural learning and teaching, 10(1), 111–132. https://doi.org/10.1515/mlt-2013-0026 percent effective or very effective teaching people of other cultures second language methodologies second language acquisition second language assessment 0.9 0.98550000000000004 0.9123 0.97619999999999996 journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.216 journal of effective teaching in higher education, vol. 4, no. 3 microsoft word kosal_jethe_fall2021_final.docx journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.126 role of mindfulness practices in the college classroom erica f. kosal, north carolina state university, efkosal@ncsu.edu abstract. the following study examined the effect of mindful practices on college student attitudes and learning comprehension using two sections of an introductory biology course taught by the same instructor. one section used lecture complemented with active learning formats while the other section additionally included mindful practices. comparisons were made between the students’ surveys and quiz/exam scores. students in the mindful section also kept journals. results showed no significant differences in quiz and exam scores between the two groups; however, students in the mindfulness section found value in the practices both in and outside the classroom. students reported a gain in their attitude towards the value of mindfulness practices as well as the use of mindfulness practices and reported a decrease in anxiety levels. students identified these practices as helping them with focused attention, peace, and relaxation. at the end of the semester, over 90% of students agreed that mindfulness has a valuable place in the college classroom. keywords: mindfulness; contemplative practice; student learning; innovative pedagogy learning and memory are necessary for a student’s success. various strategies have been employed over the years to help student comprehension of material and topics, including active learning, using case studies, and flipping the class. tools that can be harnessed to help students of all backgrounds, those that are academically underprepared as well as the academically gifted, white students and students of color, along with students of different socioeconomic groups, should be utilized in such a way that allows all students to see learning gains. one such emerging device to be considered in the pedagogical toolbox is the use of mindfulness practices, where we can see its application to the educational field just beginning. mindfulness is predicted to have a positive impact on teaching and student learning. mindfulness is focusing on the present with concentration and attentiveness to the present. instead of seeing all that may be around a person, a person centers in on a specific and particular part of the environment. when first introduced in the literature, mindfulness was described in this way as bringing attention to the present moment or experience (kabat-zinn, 2013); however, since then, mindfulness has also been described as a metacognitive skill that requires monitoring and control such that self-regulation and attention is given. it is a state of alertness, and where the inhibition of elaborative processing and orientation to experience occurs, so that a person can focus or concentrate on something specific (bishop et al., 2004). during the 2000s, much research on mindfulness confirmed its effectiveness with helping physical and mental health. for example, reduced anxiety and depression role of mindfulness 148 journal of effective teaching in higher education, vol. 4, no. 3 have been shown in numerous studies (e.g., roemer & orsillo, 2002). this may be related to how mindfulness can interact on the cellular level to reduce the negative effects of stress. in one study, mindfulness practices were shown to reverse the expression of genes that are involved in inflammatory reactions that are triggered by stress (buric et al., 2017). much of modern disease, such as cancer and diabetes, can be linked to inflammation (tian et al., 2014), and mindfulness practices promise to help reduce such inflammation. other physical benefits connected to mindfulness practices include better heart health, such as heart rate control (delizonna et al., 2009), and, in one study, immune response increases (as compared to control groups) following the influenza vaccination (davidson et al., 2003). research on the effects of mindfulness practice in the education field has been conducted primarily with elementary school children (e.g., zenner et al., 2014), but its potential to help with learning and memory is suggested for all age groups, including college students. for example, neuroimaging studies suggest that the gray matter in the brain, which is related to learning and memory (among other things), increases with mindfulness practice (e.g., vestergaard-poulsen et al., 2009) along with increases in volume of the hippocampus, another area of the brain associated with learning and memory (holzel et al., 2011). specifically, within the higher education field, mindfulness practices have been shown to help students both with their mental health (gray et al., 2018; lynch et al., 2018) and with their learning. for example, meditation has been shown to help students connect with their internal processes, grow their self-compassion, and, in turn, allow for connection to others in the classroom. this has helped student learning in college (martin, 2018). in another study, students in a semester-long course practiced mindfulness techniques in and out of the classroom and reported that these activities helped reduce stress and anxiety (vilvens et al., 2021). when taking a semester-long course specifically on mindfulness, undergraduate students showed a signification reduction in their perceived stress levels and procrastination as well as increases in self-compassion and sleep quality (gray, 2021). while these studies reveal the potential of mindfulness practices in the classroom, there is still not a lot of research on how learning and memory are influenced by mindfulness practices in the college population. there have been a few recent exceptions to the above, in which researchers have considered mindfulness and college student learning gains. one study considered if cognitive mindfulness intervention could improve student learning. it was determined in this study that interventions did not result in better same-day learning but that students performed better on the final exam items that were taught on those intervention days (cavanah et al., 2021). in another study, with only a two-week mindfulness training course, mind wandering was reduced and working memory capacity improved (mrazek et al., 2013). that is, students were able to remember list items in prescribed order better than prior to the training. additionally, these students were given gre reading-comprehension tests before and after the mindfulness training, and their scores significantly improved. in a third study, college students were taught basic mindfulness skills in class over a role of mindfulness 149 journal of effective teaching in higher education, vol. 4, no. 3 semester and self-reported that these practices enhanced their learning experience (mapel, 2012). a final notable study conducted with college students compared two groups of students to one another (hall, 1999). students were randomly assigned to one of two study groups that met regularly (one hour a week) to study throughout the first semester of college. the treatment group practiced meditation for 10 minutes at the start and conclusion of each one-hour study group session. they were also instructed to meditate at home and before exams. the control group did not do any mindfulness practices. the groups did not differ in grade point average (gpa) at the beginning of the study, but at the end of the second academic semester, the meditation group had significantly higher gpa scores compared to the control group. these studies show the potential of mindfulness practice to positively influence student learning gains over a semester and course. mindfulness may allow students to improve their ability to maintain preparedness and keep attention during a course and/or may improve the ability to process information quickly and accurately (jha et al., 2007). sustained attention is essential for learning and academic achievement (spira & fischel, 2005) and this could lead to a positive impact on academic achievement. the current study aims to contribute to the understanding of how mindfulness can be used in the college classroom in which biology was the subject matter, specifically by addressing two questions: (1) do mindfulness practices and activities help students in their attitudes towards the subject (biology) and/or their personal confidence? (2) do mindfulness practices help students comprehend material more fully because they would be more attentive in class? we predicted that students would find value with incorporating the practice into the classroom setting as well as earn higher scores on quizzes and exams compared to their peers who did not have the benefit of mindful practice. methods two sections of the course evolution (bio 222) were taught in the fall semester of 2018. this course is an introductory biology course intended for students majoring in biological sciences. one section, a, contained 25 students and was taught in the morning (9:30–11:10 am), and the other section, b, contained 20 students and was taught in the afternoon (1:00–2:30 pm). the author taught both sections, covering the same content in the same order and using the same quizzes, exams, homework assignments, case studies, and in-class problems. the only difference was the insertion of mindfulness practices into one of the sections, section a, which was chosen randomly. additionally, this section heard a short presentation on the benefits of mindfulness practices for physical health, mental health, and the potential for its use in learning comprehension and reduction in test anxiety. this section is referred to as the mindfulness or treatment group. section b, without role of mindfulness 150 journal of effective teaching in higher education, vol. 4, no. 3 such mindfulness practices, is referred to as the control group. the student composition in these sections were similar in many ways (table 1). table 1 participating student demographics of the two sections of an introductory biology course involved in the study section a: section b: mindfulness/trt control characteristics males 10 4 females 15 16 caucasian 14 11 african-american/black 8 5 latinx/asian/middle3 4 eastern mean age 19 20* % transfer 20 55 mean grade earned b b in pre-requisite course *note. there were 4 students in this section who indicated the category of 22+ as their final age. these may have been 30 years old or 25 years old and it was still recorded as 22. students in this study were members of the traditional campus program, which serves 1,110+ students at a suburban liberal arts college in eastern north carolina. of these students, 57% are males and 43% are females, and 62% of the student body is of minority standing. there is a strong athletic program, which draws 41% of the student body (59% of the students are non-athletes), and most of the students are from north carolina (65%). there is also a substantial international student population with students coming from 42 countries across the world. at the start of the semester, both sections were informed of the author’s interest in studying student attitudes, motivation, and learning preferences for a study and consent forms were provided for participation. both sections were given the same surveys at the beginning of the semester and the end of the semester (see figure 1). these surveys were developed internally with the help of colleagues in the psychology department, were irb-approved, and aimed to ask questions on knowledge of mindfulness practices and perceived value of such interventions. additionally, students were given the mindfulness attention awareness scale instrument (maas; developed by brown & ryan, 2003) at the beginning of the semester. this 15-question instrument aims to measure qualities of consciousness role of mindfulness 151 journal of effective teaching in higher education, vol. 4, no. 3 that are associated with a sense of well-being and self-awareness. the maas considers day-to-day experiences and how students self-report their feelings towards statements such as “i break or spill things because of carelessness, not paying attention, or thinking of something else” and “i get so focused on the goal i want to achieve that i lose touch with what i’m doing right now to get there.” all items are rated on a five-point scale to capture respondents’ perceptions at that moment (1 = “almost always” to 5 = “almost never”). higher scores reflect higher levels of perceived mindfulness. the maas instrument was chosen because it has been verified as reliable and has been validated (mackillop & anderson, 2007), it is commonly used in studies across the country, and the questions seemed straight forward and appropriate for the college population. finally, students in the mindfulness section were asked to reflect on their experiences four times throughout the semester in a journal. journal prompts included the following: • which mindfulness practices did you like and why? elaborate on how you felt while engaging in the activities. • explain how your attitude towards and/or your journey with mindfulness practices evolved and changed over the semester. if it did not, explain why that may have been the case. • did you personally find value in engaging in mindfulness practices during class? explain. • describe your opinion on the value of including mindfulness practices in the college classroom. figure 1 survey given to the students at the start of the semester. twenty-three surveys were completed in section a and 20 in section b. 1. are you familiar with, or have participated in “mindfulness practices?” (very familiar, somewhat familiar, not familiar; if no, a description is provided in the next question) 2. mindfulness can be defined in a few ways, but is commonly used to describe practices that allow you to be aware in the present. it is paying attention, on purpose, in the present moment and knowing what is on your mind. what are your initial feelings about how mindfulness practices will help you with (helpful, somewhat helpful, not helpful): a. anxiety or depression or negative emotions b. studying for biology c. learning biology content role of mindfulness 152 journal of effective teaching in higher education, vol. 4, no. 3 3. i believe mindfulness can help me directly with relieving anxiety and helping omit depression. (yes, no) 4. i believe mindfulness can help me concentrate better and focus more, therefore helping me with learning more effectively. (yes, no) 5. i believe mindfulness has a place in the classroom. in other words, i see the value of using mindfulness practices during class. (yes, no) 6. given the following list, do you think the activity listed is a mindfulness practice? (yes or no) a. yoga b. meditation c. slow walks in nature d. concentrating on music e. reflective writing f. concentrated body movements (like moving your fingers slowly in a particular pattern) 7. overall, i see the value of mindfulness on a scale of 1 to 10 as a _____ (1 is of no value and 10 is of much value). the only difference in pedagogy was the addition of mindful practices in section a, the treatment section. these practices and activities were varied but included breathing exercises, short meditation sessions (3-5 minutes), reflective writing exercises, columbian hypnosis, focus exercises on sounds of an audio recording, and focus exercises on nature sounds as well as classical music. course content covered was the same as the control section. occasionally, the control section was released a few minutes early from class to keep the sections at the same place content-wise. results beginning of semester surveys when both sections of students were asked about mindfulness practices, there were some differences in perceived potential. for example, when asked what the students’ initial feelings towards mindfulness practices were and if they would help a student with anxiety, studying for biology, and learning biology content, most students in the control group felt such practices would be “helpful,” and most students in the treatment group felt such practices would be “somewhat helpful.” both sections placed the value of mindfulness practices highest in terms of being helpful for learning biology content. results show that 65% of the treatment group and 60% of the control group thought these practices would be most helpful with learning biology content (figure 2). role of mindfulness 153 journal of effective teaching in higher education, vol. 4, no. 3 figure 2 student responses from the beginning of the semester survey on the potential value of mindfulness practices to help with anxiety, studying, and learning biology content note: notice that the treatment group had more “somewhat helpful” responses and that the control group had more “helpful” responses. both sections help that mindfulness practices would be most helpful with learning biology content. both sections expressed similar opinions regarding mindfulness potential. students could answer “yes” or “no” to several opinion questions with the following results: • 70% of the treatment group and 65% of the control group said “yes” to “i believe mindfulness can help me directly with relieving anxiety and helping omit depression.” • 96% of the treatment group and 100% of the control group replied “yes” to “i believe mindfulness can help me concentrate better and focus more, therefore helping me with learning more effectively.” • 74% of the treatment group and 100% of the control group say “yes” to “i believe mindfulness has a place in the classroom. in other words, i see the value of using mindfulness practices during class.” students were then asked to give a score to the statement “i see the value of mindfulness” from 1 (lowest) to 10 (highest). the treatment group’s mean was 7.5 and the control group’s mean was 7.9. when considering the range of scores 0 10 20 30 40 50 60 70 % treatment helpful % treatment somewhat helpful % control helpful % control somewhat helpful pe rc en ta ge will help with anxiety will help with studying will help with learning content role of mindfulness 154 journal of effective teaching in higher education, vol. 4, no. 3 selected by the students, the control group had more frequency scored higher ranked numbers than the treatment group (figure 3). figure 3 students in each section gave a number rank to the statement “i see the value of mindfulness” from 1 (lowest) to 10 (highest). when considering data collected from the maas instrument, both sections were again comparable. the treatment group’s mean score was 57, with a range of 24– 85. the control group’s mean score was also 57, with a range of 27–81. higher scores are considered to show higher levels of dispositional mindfulness, in other words, that students are aware of and pay attention to what is taking place in the present. the maximum score a student could report would be 90—this is ranking each of the 15 items with a 6 being equated with a “almost never” ranking. because 57 is not far from the middle of this instrument of 45, students at the beginning of the semester were not particularly mindful. mid-semester and end-of-semester surveys when considering the mindfulness treatment section, student attitudes after the first exam reflected positive attitudes. results indicated that 64% percent of the students found the mindfulness meditation session immediately before the exam to be helpful, and 88% liked the mindfulness practices during class time in general. when identifying the biggest benefit students found from these practices, 35% of them wrote in that they found the benefit of focus, 30% reported calmness, 30% 0 5 10 15 20 25 30 35 40 rank 4 rank 5 rank 6 rank 7 rank 8 rank 9 rank 10 pe rc en ta ge treatment group control group role of mindfulness 155 journal of effective teaching in higher education, vol. 4, no. 3 stated a clear and relaxed mindset, and 5% found the time to be restful. these were words the students chose themselves, which can be categorized into two main areas: focused attention at 35–40% and peaceful/relaxed nature at 60–65%. an interesting result was found to the question “do you feel like you did better on this exam in terms of anxiety levels than in the past (on other exams in other courses) based on the mindfulness practices we have been doing?” the answers were split here with 48% reporting “yes” and 48% reporting “no” (the remaining 4% replied “i don’t know”). as the semester progressed, it became evident from their journaling that during this early part of the semester, students were still learning how to use mindfulness meditation before a quiz or an exam to their benefit. at the start of the semester, they were novices and found their minds wandering before an exam, but as the semester progressed, they were able to successfully use the 5 minutes before an exam to relax and clear their minds. students ranked the value of mindfulness practices at the start of semester on a scale of 1 (lowest) to 10 (highest) and gave a mean of 7.3 (with a median of 7) after the first exam. at the end of the semester this ranking rose to a mean of 8.4 (with a median of 9). additionally, at the end of the semester, 87.5% of the students who took the survey (n = 24) found value in the mindfulness practices when it came to reducing anxiety or negative emotions. the results showed that 54% of the students found value in the mindfulness practices for helping with studying for biology, and 62.5% found value in the practices for helping to learn biology content. when asked if mindfulness has a place in the classroom, 92% agree that there was value for its use. the majority of students in the mindfulness section, 71% said they have an improved opinion about the potential value of mindfulness overall. quiz and exam scores there were no statistically significant differences found between the two groups of students when comparing mean scores using t-tests. the average overall quiz scores on the three quizzes given throughout the semester were 74% for both the mindfulness section and the control section. when considering individual quiz data, there were some variations detected but none of statistical significance. on the first quiz, students in the mindfulness section scored between 53% and 100%, with the mean score of 75%; the control section students ranged from 47% to 100% on the quiz with a mean of 77% (t-test ns, t = 0.349). on the second quiz, the range of scores for the students in the mindfulness section was 28% to 96%, and in the control section, students ranged from 39% to 86%. the mean for the mindfulness section was 67% and 68% for the control section (t-test ns, t = 0.2037). the scores for the third quiz were also not statistically significant (t-test ns, t = 0.937) with students in the mindfulness section earning between 33% and 100%, with a mean of 81%, and students in the control section ranging from 42% to 100% with a mean of 76%. this last quiz did seem to show more of a potential difference leaning toward the mindfulness section faring better, and it would be interesting to have an additional quiz if time was available to test this trend. role of mindfulness 156 journal of effective teaching in higher education, vol. 4, no. 3 there were also no statistically significant differences in exam scores between the two groups. overall, the means for exams for the mindfulness section was 78% and for the control section, 78.5%. for the first exam, the mindfulness group of students earned a mean of 76% and a range of 58% to 92%, and the median was 75%. the control group of students earned a mean of 77%, ranging from 58% to 96%, and a median of 78% (t-test ns, t=0.265). for the second exam, the mindfulness group scored a mean of 78%, with a range of 55% to 100%, and a median of 73%. the control section had a mean of 78%, a range of 47% to 100%, but a higher median of 78%; however, this was not statistically significant (t-test ns, t = 0.034). the third exam showed the reverse pattern with the median being higher for the mindfulness section of students at 85% and for the control at 82%, but the means were comparable (mindfulness group with a mean of 84% and the control group at 83%). the ranges were 50% to 100% for the mindfulness section, and 64% to 100% for the control section. these were not statistically significant (ttest ns, t = 0.1129). likewise, final exam scores were also comparable between the sections and not statistically significant (t-test ns, t = 0.26215). the mindfulness section of students earned a mean of 74% with a range of 50% to 99%, and the control group of students earned a mean of 76% with a range of 40% to 100%. the median for the mindfulness section was 77% and for the control section, 76%. journal responses students reflected on the practices used during class time at two main intervals during the semester: after the first exam and at the end of the semester; they also answered prompts two other times throughout the semester. their positive reflections at 4.5 weeks into the semester was a testament to the fast nature of acceptance and the value students found. their attitudes continued to grow stronger and more positive as the semester progressed with students commenting that they were engaging in such practices outside of class time to help them in other areas as well. of the 23 students who responded at the first major check-in after the first exam, 91% (n = 21) found the practices enjoyable and helpful. some sample excerpts from the journals are included below. these students are all 19–20 years of age. most commented on being unsure or skeptical of the mindfulness practices but growing to enjoy them and even look forward to them in class. student 1 (female): “at the beginning, i didn’t think that the techniques and practices could help me focus more, but the more and more we do them, the easier i feel like it is to focus in class. it has also reduced my stress a little when it comes to assignments and the quiz.” student 2 (female): “at first, i thought that mindfulness wasn’t very beneficial. now, i see and value the importance of mindfulness. the videos that we watch in class allow me to unwind and clear my mind so i can better absorb material.” role of mindfulness 157 journal of effective teaching in higher education, vol. 4, no. 3 student 3 (male): “i think the role of mindfulness in a classroom is very important. it can help you concentrate more on one thing and help you to focus and understand more clearly. at first, i wasn’t too sure how well these mindfulness practices would work, but after trying these practices during class, [they] have helped me focus in class more.” at the end of the semester, their fourth journal entries reflected the same spirit, but with stronger enthusiasm. when asked to write about which practices they found most helpful and least helpful, student responses varied widely. students could comment on multiple practices; therefore, the percentages reported may be higher collectively than 100%. the short meditation sessions were valued the most at 30%, followed closely by the breathing exercises at 27%. the columbian hypnosis practice, which was quite opposite to these quiet and still practices, had the students moving and giggling and was favored at a value of 17%. reflective writing and listening to sounds both earned 10%, and listening to songs and moving an imaginary ball were scored at 3% each (fig 4). figure 4 students self-reported the exercises they preferred throughout the semester and could comment on multiple practices if they found value in more than one. note: numbers reported are in percentages. practices that were not favored by students also varied widely. the columbian hypnosis as well as the reflective writing exercises were tied at 30% each for least favored, followed closely by moving the imaginary ball at 26%. listening to nature sounds, breathing exercises, and short meditations were also included with a value of 4% each. 30 27 17 10 3 3 short meditation breathing exercises columbian hypnosis reflective writing listening to sounds moving imaginary ball role of mindfulness 158 journal of effective teaching in higher education, vol. 4, no. 3 students wrote about how they felt during class time when they engaged in mindfulness practices. students could list multiple ideas or just focus on one theme. percentages reported came from categorizing their paragraphs into subject areas. the most commonly reported was peacefulness/calmness/relaxation at 43%, followed by being present or focused at 21%. the following themes were reported at 7% each: comforted, tiredness, “good,” and distracted. the last two themes were reported at 4% each: bored and awareness of how their body felt. sample student reflections are included below. these students are all 19–20 years of age. student 1 (female): “when it comes to the mindfulness exercises, i find that there is so much value in them, especially for college students. to begin with, these exercises provide students with a better understanding of peace and meditation which i feel is so essential for providing a positive way of dealing with stress of test and being overwhelmed. in addition, the practices provide people to get a better understanding of themselves. meaning, that you can now understand what is causing you stress and why it’s not as big of a predicament as you may think it might be. lastly, the practice enhances focus which has really aided me in having the capability to achieving multiple complex tasks all at once.” student 2 (male): “my attitude has completely transformed towards mindfulness throughout the semester. i used to think that mindfulness was not effective, and that it was a waste of time. now, i see the importance of mindfulness. occasionally, we need a break from our daily lives, and our minds just need a moment to rest. ever since participating in these practices, i have seen a change in my mindset. i am more positive and motivated to do work, and i look forward to the mindfulness activity.” student 3 (female): “at the beginning of the semester, i did not feel like that the mindfulness practices would help me focus better or perform better on the tests. i thought that these mindfulness practices wouldn’t help at all and that they were pointless. however, i did not take these practices seriously enough in the beginning. i didn’t take them seriously on the first quiz, and i was anxious and forgot all the material that i studied. i performed poorly and took the practices more seriously after that quiz, and they seemed to help. after i started to take the mindfulness practices more seriously, i realized that they are helpful. over the course of the semester, class by class, i learned that these practices help to relieve my stress and help me to focus more. i have performed better on the quizzes and test progressively through the semester as we did more mindfulness practices. i feel like there are benefits to partaking in these mindfulness practices. my attitude towards these practices is that they can be beneficial to students in their classes and outside the classroom with various tasks they perform.” students wrote about engagement in mindfulness practices outside of the classroom. sixty-eight percent of students said they use mindfulness on their own as a result of their exposure to it from class, and 32% did not. many students commented on their appreciation of the practice in class and how it helped them to role of mindfulness 159 journal of effective teaching in higher education, vol. 4, no. 3 focus just on class itself (and ignore other courses and stressful assignments). two students did not see much value in the practice, commenting that it seemed a waste of time, took away time from covering material related to the course content, or made them feel more anxious when asked to participate because they felt silly. these two students are male students, and although the sample size is very low, it may be worthwhile to examine gender differences in the future. discussion when beginning the semester, both sections of the course were comparable in most aspects with a few exceptions. there were fewer men and more transfer students in the control section, and their views of the potential of mindfulness practices on student gains was higher when compared to the treatment group. this control group seemed more motivated overall in terms of coming to class prepared, asking questions during class, and attending outside supplemental instruction from a peer mentor. although not formally analyzed, the author sensed this motivation had to do with maturity level. the many transfer students in the control section were older, two came with bachelor’s degrees earned already (and were returning to school for a career change), one student was a mother of three children, and two students were taking the course for a second time for the goal of earning a higher grade. these factors likely played into the statistically non-significant test and quiz score results found between the control and treatment sections. the potential, however, to see a difference given a more homogenous set of students for a future study remains. it should also be noted that during the semester of study, the college experienced a significant interruption of flow to the semester as a result of two hurricane storms. several days of the semester were cancelled, and many students were without power and electricity (and in some cases stranded as they went home to be with family only to then be blocked from returning to campus because of flooding and highway/road closures). it is very likely that these weather events played into students being able to concentrate and study biology as effectively as they would have done in a “typical” semester. time and length of exposure to mindfulness practice has been shown to play a role in student learning gains (see cavanah et al., 2021). in this way, it would be helpful to explore specific content and mindfulness intervention correlations on final exam questions. in a multi-semester study, cavanah et al. (2021) found students did not perform better initially on learning content on days where mindfulness intervention was conducted; however, students did show improvement on final exam learning gains on content from the original mindfulness intervention days compared to the control group. it is possible that my students in the current study had a similar gain; however, this was not measured. in its current form, the prediction that students would perform better on quizzes and tests when engaging in mindfulness practices throughout the semester is not supported. from self-assessment, the majority of students (62.5%) in the mindfulness treatment section did report the practices were helping them to learn biology role of mindfulness 160 journal of effective teaching in higher education, vol. 4, no. 3 content. so perhaps with a non-disrupted semester, or with a focus on final exam questions compared to early exam questions, more direct gains in learning could be found. this should be studied more fully in the future. the second prediction that students would find value with incorporating the practice into the classroom was supported. a very clear result found in this study was the value of mindfulness on student attitudes. students gave higher ranking scores to the value of mindfulness practices as the semester progressed, and the vast majority of students (92%) said they believe mindfulness has a place in the classroom. additionally, 87.5% of the students found the practices helped with reducing anxiety or negative emotions. students wrote in their journals that they didn’t engage fully and/or seriously in the practices at the start of the semester but then began to see them of more value and found them helpful with focusing and relieving stress. this shows the relevance of using the exercises as often as possible. most of the exercises used in class took 3 to 5 minutes each and, therefore, a small investment has potentially huge payoffs. from the end of the semester journaling, students preferred different practices. this supports the idea that incorporating a variety of mindfulness practices in the classroom is helpful. some individuals will connect with one type of exercise and others with something else. this is especially important when first being introduced to mindfulness. students need to explore and find a style that suites them as they move forward with the goal of incorporating such practice into their daily lives. in general, students self-report that the mindfulness practices help them focus their attention during class and/or that by engaging in such practices they enter into a more relaxed or peaceful state. it would be interesting to consider in future research if specific mindfulness practices were more effective in helping with student learning gains than others. although mindfulness practices did not show a boost in student comprehension of the biology subject material in this current study, there is still potential it may exist, and the study should be repeated with more homogeneous groups of students and with a semester free from cancellations. the role that mindfulness practices did show in terms of student attitudes is of significant value. mindfulness helps students focus, feel good about themselves, and release stress and anxiety. many students report continuing using these practices outside of the classroom and that mindfulness practice is of value to them. a survey result of 92% approval rating for a practice should call us all in the educational field to consider how to incorporate mindfulness into our own classrooms. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. role of mindfulness 161 journal of effective teaching in higher education, vol. 4, no. 3 references bishop, s. r., lau, m., shapiro, s., carlson, l., anderson, n. d., carmody, j., segal, z. v., abbey, s., speca, m., veltin, d., & devins, g. (2004). mindfulness: a proposed operational definition. psychology: science and practice, 11(3), health module, 230–241. https://doi.org/10.1093/clipsy.bph077 buric, i., farias, m., jong, j., mee, c., & brazil, i. a. (2017). what is the molecular signature of mind-body interventions? a systematic review of gene expression changes induced by meditation and related practices. frontiers in immunology, 8(8). http://doi:10.3389/fimmu.2017.00670 brown, k. w., & ryan, r. m. (2003). the benefits of being present: mindfulness and its role in psychological well-being. journal of personality and social psychology, 84(4), 822–848. doi:10.1037/0022-3514.84.4.822 cavanah, s. r., lang, j. m., birk, j. l., fulwiler, c. e., & urry, h. l. (2021). a multicourse, multisemester investigation of the impact of cognitive reappraisal and mindfulness instruction on shortand long-term learning in the college classroom. scholarship of teaching and learning in psychology, 7(1), 14–38. http://dx.doi.org/10.1037/stl0000174 davidson, r. j., kabat-zinn, j., schumacher, j., rosenkranz, m., muller, d., santorelli, s. f., urbanowski, f., harrington, a., bonus, k., & sheridan, j. f. (2003). alternations in brain and immune function produced by mindfulness meditation. psychosomatic medicine, 65, 564–570. https://doi.org/10.1097/01.psy.0000077505.67574.e3 delizonna, l. l., williams, r. p., & langer, e. j. (2009). the effect of mindfulness on heart rate control. journal of adult development, 16, 61–65. https://doi.org/10.1007/s10804-009-9050-6 gray, l., font, s., unrau, y., & dawson, a. (2018). the effectiveness of a brief mindfulness-based intervention for college freshmen who have aged out of foster care. innovative higher education, 43, 339–352. https://doi.org/10.1007/s10755-018-9433-3 gray, l. a. (2021). mindfulness in the college classroom and wellness promotion: the impact of mindfulness curriculum on self-reported health and well-being in university students. the international journal of learning in higher education, 28(2), 99–109. https://doi.org/10.18848/23277955/cgp/v28i02/99-109 hall, p. d. (1999). the effect of meditation on the academic performance of african american college students. journal of black studies, 29(3), 408–415. https://doi.org/10.1177/002193479902900305 role of mindfulness 162 journal of effective teaching in higher education, vol. 4, no. 3 holzel, b. k., carmody, j., vangel, m., congleton, c., yerramsetti, s. m., gard, t., & lazar, s. w. (2011). mindfulness practice leads to increases in regional brain gray matter density. psychiatry research, 191(1), 36–43. https://doi.org/10.1016/j.pscychresns.2010.08.006 jha, a. p., krompinger, j., & baime, m. j. (2007). mindfulness training modifies subsystems of attention. cognitive, affective & behavioral neuroscience, 7(2), 109–119. https://doi.org/10.3758/cabn.7.2.109 kabat-zinn, j. (2013). full catastrophe living: using the wisdom of your mind to face stress, pain and illness. bantam. lynch, s., gander, m-l., nahar, a., kohls, n., & walach, h. (2018). mindfulnessbased coping with university life: a randomized wait-list controlled study. sage open, 8(1). https://doi.org/10.1177/2158244018758379 mackillop, j., & anderson, e. j. (2007). further psychometric validation of the mindful attention awareness scale (maas). journal of psychopathology and behavioral assessment, 29(4), 289–293. https://doi.org/10.1007/s10862007-9045-1 mapel, t. (2012). mindfulness and education: students’ experience of learning mindfulness in a tertiary classroom. new zealand journal of educational studies, 47(1), 19–32. martin, m. (2018). mindfulness and transformation in a college classroom. adult learning, 29(1), 5–10. https://doi.org/10.1177/1045159517744752 mrazek, m. d., franklin, m. s., tarchin phillips, d., baird, b. & schooler, j. w. (2013). mindfulness training improves working memory capacity and gre performance while reducing mind wandering. psychological science, 20(10), 1–6. http://www.jstor.org/stable/23409426 roemer, l., & orsillo, s. m. (2002). expanding our conceptualization of and treatment for generalized anxiety disorder: integrating mindfulness/acceptance-based approaches with existing cognitive-behavioral models. clinical psychology: science & practice, 9, 54–69. https://doi.org/10.1093/clipsy.9.1.54 spira, e. g., & fischel, j. e. (2005). the impact of preschool inattention, hyperactivity, and impulsivity on social and academic development: a review. canadian journal of psychology, 46, 755–773. https://doi.org/10.1111/j.1469-7610.2005.01466.x tian, r., li, d., hou, g., & yuan, t. f. (2014). a possible change process of inflammatory cytokines in the prolonged chronic stress and its ultimate implications for health. the scientific world journal, 2014, 1–8. role of mindfulness 163 journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.1155/2014/780616 vestergaard-poulsen, p., van beek, m., skewes, j., bjarkam, d. r., stubberup, m., & bertelsen, j. (2009). long-term meditation is associated with increased gray matter density in the brain stem. neuroreport, 20(2), 170–174. doi:10.1097/wnr.0b013e328320012a vilvens, h. l., frame, d. l., & owen, p. c. (2021). promoting the inclusion of mindfulness and contemplative practices in the college classroom. pedagogy in health promotion, 7(2), 148–158. https://doi.org/10.1177/2373379920925849 zenner, c., herrnleben-kurz, s., & walach, h. (2014). mindfulness-based interventions in schools-a systematic review and meta-analysis. frontiers in psychology, 5(603), 1–20. https://doi.org/10.3389/fpsyg.2014.00603 microsoft word davidson final version.doc journal of effective teaching in higher education, vol. 1, no. 2 “nothing is set in stone”: incorporating futurology into college courses skylar davidson, chattanooga state community college, skylar.davidson@chattanoogastate.edu abstract. “futurology” is the process of forecasting or designing the future, whether that be the near future or far future. college courses rarely explicitly include futurology, even though the content of many courses has implications for the future. to build knowledge on how future-oriented content can be incorporated into educational curriculum, i studied a college course in which students used futurology activities as part of an active learning setup. my study suggests that future-oriented content is appealing to students and helps students develop a sense of control over the future. in order to better introduce students to futurology, instructors must attend carefully to students' unfamiliarity with structured future-oriented thinking, as well as to fatalistic attitudes about the future. keywords: futurology, active learning, pedagogy despite the fact that all academic disciplines have a stake in the future, college teaching rarely explicitly incorporates future-oriented content. this is surprising given that future-oriented content can help students develop constructive hope about modern social and technological problems (e.g., climate change) rather than denying that such problems exist (ojala, 2015). education about the future is rare despite its importance and despite many attempts to make it more mainstream (slaughter, 2008). since education about the future is unfamiliar to many students, this study seeks to provide some general guidance to instructors who incorporate future-oriented content into their courses. this study evaluates the benefits and challenges college students experience when they engage in future-oriented activities and discusses how an instructor can effectively respond to students' experiences with future-oriented content. literature review and hypotheses this study bridges two literatures: the literature on active learning and the literature on teaching futurology. futurology is the process of forecasting or designing the future (serra del pino, 1998), and futurology encompasses the shortrange, mid-range, and long-range future. a future-oriented focus is essential for life in the modern world of rapid social and technological change (inayatullah, 2008; masini, 2011). future-oriented education is based on students using creative thinking to manipulate uncertain situations to people's benefit rather than on students simply absorbing and reciting existing knowledge (bishop and strong, 2010). futures education is an aspect of active learning, which is an educational technique based on multidimensional flows of information rather than a transfer of information from teacher to student (huggins & stamatel, 2015). active learning approaches encourage students to develop a sense of responsibility for their learning and allow students to not only practice technical skills but also practice nothing is set in stone 137 journal of effective teaching in higher education, vol. 1, no. 2 communication and decision-making skills (huggins & stamatel, 2015). futureoriented learning is thus inherently active learning, since it requires students to be participants in using evidence to construct the future. creative thinking and problem solving are essential parts of our modern world, which is experiencing exponential social and technological change (bishop & strong, 2010). active-learning educational methods fulfill this need because they utilize class time for concept engagement rather than content delivery (huggins & stamatel, 2015). active learning techniques increase students' creative and critical thinking by allowing them to practice problem-solving (huggins & stamatel, 2015; mccarthy, 2016). in courses based on active learning, there is still a transfer of core content; however, this transfer is accomplished not through lecture but instead either through delivery outside the classroom or through activities. this way, class sessions can be devoted to actively engaging with material instead of passively absorbing it. the literature on active learning points to improvements in student outcomes compared to lecture (e.g., eglitis, buntman, & alexander, 2016; huggins & stamatel, 2015; luna & winters, 2017). historically, education has been based on transmission of information, with instructors expecting students to be able to regurgitate existing knowledge rather than to develop new answers (bishop and strong, 2010). however, the solutions to modern social problems such as pollution and terrorism must be created rather than found among existing knowledge (bishop & strong, 2010; inayatullah, 2008); thus, instructors must provide more opportunities for students to build upon existing knowledge and use their creativity. futurology can help people build preferred futures despite the weight of the industrial era on society (inayatullah, 2008). education generally does not explicitly incorporate future-oriented content (slaughter, 2008). when future-oriented content is incorporated into education, it is frequently unspecific or hasty, or it ignores diverse possibilities in favor of attempting to predict only one potential outcome (hicks, 2004). it is important that educators incorporate more comprehensive and structured futures content into courses because future-oriented education helps students think logically about social problems so that they can develop plans of action for themselves or society instead of feeling fearful and overwhelmed (masini, 2011; slaughter, 2008). in addition, future-oriented active learning can help students understand the concerns and experiences of people from diverse backgrounds (masini, 2011). futureoriented education is thus a method for achieving critical thinking outcomes in students. “critical thinking” refers to skills that enable students to be informed consumers and users of information; both within and outside the student's field of study (huber & kuncel, 2015). students grow their critical thinking skills during their college careers; however, careful attention must be paid to ensuring that students have the opportunity to grow these skills throughout college, rather than emphasizing their development just at the beginning of students' college experience or toward the end of their degree (huber & kuncel, 2015; hall 2017). because future-oriented education emphasizes creative and original thought, it not only helps students grow as professionals in their own disciplines but also helps them grow their critical thinking skills in general (masini, 2011). as a result, futureoriented education can be implemented across fields and in both lower-division and upper-division courses. nothing is set in stone 138 journal of effective teaching in higher education, vol. 1, no. 2 many techniques for future-oriented analysis have been developed (e.g., inayatullah, 2008; saleh et al., 2008). for reader convenience, a description of some of these techniques as they can be adapted for classroom activities is available in the appendix. all of these futurology activities, like many other types of active learning techniques, can be adapted to fit instructors' unique needs and learning goals. for example, an instructor can modify the scope of an activity to fit into a particular time limit; can decide whether students should practice an activity alone or in groups; can decide whether an activity will be completed in class or out of class; or other factors. due to the diversity of ways in which instructors can use futurology activities, as well as the fact that the activities are suitable for use in diverse fields of study, the purpose of this study is not to suggest specific ways of using these activities. instead, this study identifies some general benefits and challenges students experience upon encountering future-oriented content for the first time, so as to inform emerging future-oriented pedagogy across fields. based on the literature, i hypothesize the following: (1) futurology will give students a sense of control over the future. (2) students will need structure and repetition in order to feel comfortable with futurology due to their unfamiliarity with it. research method this study was conducted at a large, rural, public university in the united states with approximately 23,000 undergraduates. in compliance with the university's guidelines regarding classroom research, all the study procedures were reviewed and approved by the institutional review board (irb) prior to the implementation of the study. data were collected in a sociology class, age and society, which had 21 enrolled students. the students ranged from first years to seniors, and there was a mixture of majors, though most students were either sociology or psychology majors. this class met for 50 minutes per session, three days a week, for 13 weeks. two types of data were collected for this study: class session notes and self-report survey data from students. this survey data is qualitative, from open-ended questions. following each class session, i wrote detailed notes about what occurred in class that day. these notes incorporated a description of the discussion, activity, lecture, or other content for the day and also an account of students' reactions to the content. the other source of data used for this study comes from a survey; 14 of the 21 enrolled students were present in class the day the survey was administered, and all gave consent for me to use their responses. the surveys were completely anonymous (no names or other identifying information were collected on the forms). the irb required that i leave the room while the surveys were administered to ensure the anonymity of the data collection process. this data collection strategy allowed me to not only test my hypotheses, but also identify and explain themes as they emerged organically in the course. nothing is set in stone 139 journal of effective teaching in higher education, vol. 1, no. 2 the survey questions follow: (1) what have been the most useful aspects of learning and practicing futurology so far this semester? (2) what challenges have you experienced while learning or practicing futurology in this course? (3) one of the goals of this course is for you to practice thinking about the future in a structured way. do you think this way of thinking will benefit you, either in your other courses or in your non-academic life? (this question was followed by three options a student could circle, “yes,” “no,” and “maybe/unsure,” as well as space to elaborate on one's opinion.) (4) is there anything else you would like to share about your experience with futurology, or with this course (age and society) in general? (5) instead of a consent form, there was a question at the end of the survey that allowed students to mark yes or no to the question of whether i could use their anonymous responses in a published paper or presentation. results and discussion various themes emerged from this study, both from the notes and from the survey. the results section begins by focusing on the survey. the three tables provide tallies of the number of students who made a specific type of response to a certain survey question. in these tables, the “comment” heading shows themes that emerged from the qualitative comments, and the “frequency” heading shows the number of students who made a response fitting into that theme. the interpretation of the tables incorporates some specific direct quotes from students and observations from my notes. following the interpretation of the survey responses is an interpretation of other key themes that emerged from the notes. the benefits of learning and practicing futurology table 1 responses to survey question about benefits of using futurology comment frequency thinking of the future in tangible ways 4 understanding the steps needed to achieve a desired outcome 4 reference to the usefulness of specific futurology activities 4 acknowledging that there are multiple potential future outcomes 3 organizational skills 2 thinking about the future more often 2 according to the first question, students had diverse viewpoints on the benefits of learning and practicing futurology. as shown in table 1, the greatest number of students identified how the future-oriented content changed their worldview, such nothing is set in stone 140 journal of effective teaching in higher education, vol. 1, no. 2 as allowing them to think of the future in more tangible ways and understanding the steps needed to achieve a desired outcome. one representative comment is thus: “thinking about the steps needed to be taken in order to achieve whatever goal (future outcome) we expect to happen [has] been a useful tool for myself in organizing my thoughts and taking into consideration what steps i need to take in order to achieve my desired outcome for myself when i graduate college.” in addition, many students also made references to specific futurology activities. students have often made the link between specific futurology activities and a concern for the future of society more broadly, which is what serra del pino (1998) advocated for; as one student commented, “studying futurology has allowed me to develop a more complete understanding of what the future may have in store for us. by learning new techniques of futurology i have been able to think of the future in more tangible and concrete ways.” in addition to the major themes outlined in table 1, some responses were idiosyncratic to only one student, such as finding videos from the course useful; finding visual depictions of futures useful; and practicing futurology both during class sessions and in out-of-class written work. the benefits of learning and practicing futurology table 2 responses to survey question about challenges of using futurology comment frequency discomfort with the inability to predict or plan for the future perfectly 6 student indicated no significant challenges 3 irrelevant comment (explanation of idiosyncratic academic challenge unrelated to futurology) 3 difficulty with abstractness 2 there was more homogeneity of viewpoints with regard to the challenges of learning and practicing futurology. as shown in table 2, the main source of concern for students was difficulty in making precise predictions. as one representative comment stated: “one of the challenges about learning futurology is that our thoughts are just thoughts. what i mean by this is that we can only predict or anticipate what will happen in the future. this makes futurology so difficult to analyze and to brainstorm because there are so many possible outcomes and we don't know for sure what is the outcome that will happen.” my own notes corroborated this fact: students expressed discomfort with uncertainty and ambiguity. students commented positively on more concrete activities that leave students with a tangible product in place when the class period is over, such as a futures wheel (a visual depiction of the direct and indirect consequences of a proposed action). students more often verbally critiqued more open-ended activities such as scenario planning (visioning different potential future outcomes) because of their lack of immediate impact. two students indicated difficulty with abstract content. for example, one student wrote: “after we analyze where do we go from there? not dealing with concretes is a challenge. if all this is hypothetical with no true equation or applied methods how nothing is set in stone 141 journal of effective teaching in higher education, vol. 1, no. 2 is this helpful?” since nothing of substantive social importance can be predicted with exact precision, futurology instead focuses itself on helping people identify diverse potential futures and identify how to reach preferred futures out of these (dator, 1995). educators covering futurology are faced with combating stereotypical ideas about the future as something that can be predicted with precise accuracy. transparency can help instructors encourage engagement and comfort with new instructional techniques. instructors using active learning methods in class often fear student resistance to novel educational content (seidel and tanner, 2013). in order to help students see the value in new teaching methods, it can be useful for instructors to explicitly share the reasoning for the pedagogical tools they use (seidel & tanner, 2013). this transparency not only helps students understand how the instructor is working to maximize their learning but also improves instructor rapport with students through treating students more like colleagues than underlings—the transparency respects students' intelligence and right to know why their participation is requested (seidel & tanner, 2013). furthermore, active learning allows students to understand and practice the process of becoming an expert in one's field (wallace et al., 2014). instructors thus may give students opportunities to critique and improve upon future-oriented analysis, with the goal of not only helping students feel more comfortable with it but also encouraging students to take an active role in constructing knowledge. among students who indicated no significant challenges in the survey question, the responses suggested that students had interpreted the question as if it were asking whether the course expectations were difficult to follow; for example: “[no challenges.] i have learned a lot in this class and i enjoy the material and setup.” this is likely a function of the fact that the setup of this survey was very similar to an ordinary midsemester evaluation in which students comment on the instructor's clarity and the course expectations more generally. it is also possible that these comments reflect the fact that some students are more comfortable with uncertainty and ambiguity than others. the benefits of learning futurology in other courses or in personal life table 3 responses to survey question about benefits of learning futurology in other courses or personal life comment frequency better able to prepare for the future 6 organizational and critical thinking skills 5 acknowledging diverse possibilities for the future 5 comment as to the usefulness of a particular futurology activity 3 more able to take on a long-term perspective 2 when asked whether they believe future-oriented thinking will benefit them in other courses or in their personal life, 11 responded yes and 3 responded maybe/unsure. nothing is set in stone 142 journal of effective teaching in higher education, vol. 1, no. 2 none responded no. this result shows that students overwhelmingly believe that future-oriented education is useful. the lack of “no” responses also suggests that instructors who choose to take on a future-oriented perspective are unlikely to receive resistance from students with regard to that ideological perspective. in this survey question, five students mentioned academic benefits in the comments section, 4 mentioned personal benefits, and 6 mentioned benefits without specifying the context. only one student did not elaborate in the comments section. as shown in table 3, there were diverse reasons students thought future-oriented thinking is or may be useful. futures thinking allows some students to feel more optimistic due to a greater sense of control over the future. as one student wrote: “this class is extremely important in and out of the university setting. it helps people our age realize that there still can be a happy, successful life waiting for them when they are over 65. looking to the future in a more positive, less stereotyped way, makes it more exciting.” some students feel as if futurology helps them understand the many options available to them; as one student wrote: “i can see [this course] benefiting me in life. it reminds me that nothing is set in stone. there are millions of possible futures.” among students who commented as to the usefulness of particular futurology activities, two referenced the futures wheel specifically, and both commented on its ability to help them organize and brainstorm ideas for other situations. the third focused on visual representations of futures more generally: “the importance of visual representations is often understated. futurology uses them heavily.” similarly, five students commented on how futures content helps them develop organizational and critical thinking skills, indicating the value of futures content in contributing to academic and personal development more generally. visual depictions and organizational skills can reinforce each other. my notes indicate that visual depictions of futures such as the futures wheel on the boards allowed for better comparison of differing opinions compared to presentations of work that did not have a substantial visual element. this is because it was easier to refer to points made earlier in the period when those points were visible for the whole period, as opposed to when multiple groups expressed points solely in the more fleeting verbal format. attending to unfamiliarity with futurology various themes emerged from the class session notes as well. one major theme was that futures thinking is new to students. this observation comes primarily from the observations of class sessions and is corroborated by comments in surveys. as one student wrote, “[this class] has been an experience unlike any class i have taken. before this, i would never have thought to take a futurology approach to anything.” my notes indicate that i have frequently needed to repeat directions and provide suggestions to students who are using futurology activities during in-class work, especially the first time an activity is introduced. students had the most difficulty getting started at the beginning of a class period with new activities, though most students displayed more confidence by the end of the class period. classroom dynamics more generally are relevant. many students feel intimidated about publicly admitting a lack of knowledge or understanding in class (roehling et nothing is set in stone 143 journal of effective teaching in higher education, vol. 1, no. 2 al., 2011). when describing a new futurology activity, i always asked whether students had questions before allowing them to start working, but usually no one spoke up. once they got to work and i began circulating around the room, there were questions, indicating that either students did not realize difficulties until actually practicing the activity or that many students did not want to ask questions in front of the whole class. when students did ask questions in front of the class, it was the same few people who felt comfortable speaking up publicly. since futurology activities are new to many students, instructors who use them should anticipate difficulty and provide opportunities for students to get help in a nonthreatening way. combatting fatalism it was apparent early on in this course that many students easily fell into a fatalistic attitude, in which they perceived the future as something that “happens to people” but failed to address how people shape technological and social development. for example, in a discussion from the first month of class about age and technology, students spontaneously brought up issues such as “the effects of internet on teenagers” or “the effects of robots on elder care,” but rarely did students spontaneously address how people or organizations influence technology. to address this fatalistic attitude, i provided activities that required students to identify their preferred futures and brainstorm ways of achieving them. these activities encouraged students to start thinking of the future as something that can be constructed. students latched on to the concept of alternative futures (inayatullah, 2008) as a way of addressing fatalism. “alternative futures” refers to the fact that there are diverse plausible possibilities for the future instead of a single trajectory. for example, during an activity about the future of retirement, students visioned some alternatives to the economic status quo, such as a 20-hour work week instead of a 40-hour work week or telecommuting being available to a substantial portion of the workforce. during the discussion this day, students acknowledged that such alternatives are plausible, though their implementation would be dependent on many social and technological factors. serra del pino (1998, p. 492) wrote that he tells students that “futurology involves a lot of common sense, plus some techniques and rules, with indispensable megadoses of creativity, imagination, novelty, and, why not admit it, some drops of absurdity.” though students are sometimes uncomfortable with the “hypothetical” nature of alternative futures, as explained earlier in this paper, instructors should help students use the concept of alternative futures so that students can better evaluate their preferred futures. the students dealt with alternative futures more easily as the semester went on. the primary way i noticed this shift is the fact that during class discussions that occurred toward the end of the semester, students more frequently offered potential solutions to problems spontaneously. in other words, they initiated solution-oriented discussion without my prompting, whereas earlier in the semester, i more frequently needed to prompt them to do so. for example, during the next-to-last week of the semester, the class discussed sudbury schools, which are schools with no standard curriculum. most of the students who spoke during this discussion were critical of this model, preferring a school setting that had at least some standard curriculum in addition to unstructured time. when speaking up nothing is set in stone 144 journal of effective teaching in higher education, vol. 1, no. 2 to critique the sudbury model, most students added to their critique a potential way to modify the school setting so that it had both structure and autonomy, such as having a half day of structured coursework and a half day of unstructured time, or incorporating arts education into a standard curriculum. this solution-oriented style was in sharp contrast to some earlier class discussions, such as the abovementioned discussion on age and technology from the first month. conclusion, implications, and recommendations this study has shown support for both of my hypotheses. futurology is appealing to students and helps students feel a sense of control over the future through encouraging them to identify preferred futures and ways to achieve them. because most students are unfamiliar with futurology, instructors must closely guide them through futurology activities in order to achieve this outcome. in particular, instructors should combat stereotypical and fatalistic visions of the future and provide an environment in which students feel comfortable trying new activities. as with any active learning setup, instructors who incorporate futurology into a course should monitor students' reaction to new teaching techniques and be able to articulate why the active learning methods are being used (cavanagh et al., 2016; seidel & tanner, 2013). finally, the results of this exploratory study should be extended to other academic fields. the president's council of advisors on science and technology (pcast) in the united states indicates that a major problem discouraging students from becoming science, technology, engineering, or math (stem) majors is uninspiring introductory courses (pcast, 2012). including more active learning and discoverybased research courses into stem education may help inspire more students to become stem majors (pcast, 2012). since active learning methods in stem classes improve student grades and reduce the risk of student failure (armbruster et al., 2009; freeman et al., 2014), future-oriented education would further this progress. further research on futurology can build upon this study to evaluate how particular futurology activities can be used effectively in different settings, such as different academic fields, different class sizes, and different levels of education (introductory versus advanced classes). conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. nothing is set in stone 145 journal of effective teaching in higher education, vol. 1, no. 2 references armbruster, peter, maya patel, erika johnson, and martha weiss. (2009). active learning and student-centered pedagogy improve student attitudes and performance in introductory biology. cbe—life sciences education 8, 203213. doi: 10.1187/cbe.09-03-0025 bishop, peter c., and kay e. strong. (2010). why teach the future? journal of futures studies 14(4), 99-106. http://jfsdigital.org/wpcontent/uploads/2014/01/144-s02.pdf cavanagh, andrew j, et al. (2016). student buy-in to active learning in a college science course. cbe—life sciences education 15. doi: 10.1187/cbe.16-070212 dator, jim. (1995). what futures studies is, and is not. hawaii research center for futures studies. retrieved from http://www.futures.hawaii.edu/publications/futures-studies/whatfsis.pdf eglitis, daina s., fran l. buntman, and dameon v. alexander. (2016). social issues and problem-based learning in sociology: opportunities and challenges in the undergraduate classroom. teaching sociology 44(3), 212-220. https://doi.org/10.1177/0092055x16643572 freeman, scott, et al. (2014). active learning increases student performance in science, engineering, and mathematics. pnas 111(23):8410-8415. doi: 10.1073/pnas.1319030111 hall, sarah k. (2017). practise makes perfect: developing critical thinking and effective writing skills in undergraduate science students. 3rd international conference on higher education advances, head’17, universitat polit`ecnica de val`encia, val`encia. http://dx.doi.org/10.4995/head17.2017.5512 hicks, david. (2004). teaching for tomorrow: how can futures studies contribute to peace education? journal of peace education 1(2):165-178. http://dx.doi.org/10.1080/1740020042000253721 huber, christopher r., and nathan r. kuncel. (2015). does college teach critical thinking? a meta-analysis. review of educational research 86(2):431-468. huggins, christopher m., and janet p. stamatel. (2015). an exploratory study comparing the effectiveness of lecturing versus team-based learning. teaching sociology 43(3), 227-235. https://doi.org/10.1177/0092055x15581929 inayatullah, sohail. (2008). six pillars: futures thinking for transforming. foresight 10(1), 4-21. https://doi.org/10.1108/14636680810855991 lloyd, david, and john wallace. (2004). imagining the future of science education: the case for making futures studies explicit in student learning. studies in science education 40, 139-178. http://dx.doi.org/10.1080/03057260408560205 luna, yvonne m., and stephanie a. winters. (2017). “why did you blend my learning?”: a comparion of student success in lecture and blended learning introduction to sociology courses.” teaching sociology, 45(2), 116-130. https://doi.org/10.1177/0092055x16685373 masini, eleonora barbieri. (2011). how to teach futures studies: some experiences. journal of futures studies 15(4), 111-120. retrieved from http://jfsdigital.org/articles-and-essays/2011-2/vol-15-no-4-june/ nothing is set in stone 146 journal of effective teaching in higher education, vol. 1, no. 2 mccarthy, josh. 2016. reflections on a flipped classroom in first year higher education. issues in educational research, 26(2), 332-350. http://www.iier.org.au/iier26/mccarthy-j.html ojala, maria. (2015). hope in the face of climate change: associations with environmental engagement and student perceptions of teachers' emotion communication style and future orientation. journal of environmental education, 46, 133-148. http://dx.doi.org/10.1080/00958964.2015.1021662 president's council of advisors on science and technology. 2012. engage to excel: producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. retrieved from https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/fac t_sheet_final.pdf roehling, patricia vincent, vander kooi, thomas lee, dykema, stephanie, quisenberry, brooke, and vandlen, chelsea. (2011). engaging the millennial generation in class discussions. college teaching, 59, 1-6. seidel, shannon b., and kimberly d. tanner. (2013). “what if students revolt?”— considering student resistance: origins, options, and opportunities for investigation. cbe—life sciences education 12, 586-595. doi: 10.1187/cbe13-09-0190 serra del pino, jordi. (1998). the challenge of teaching futures studies. american behavioral scientist 42, 484-492. https://doi.org/10.1177/0002764298042003020 saleh, mohamed, nedaa agami, ahmed omran, and hisham el-shishiny. (2008). a survey on futures studies methods.” cairo: faculty of computers & information-cairo university-infos, 38-46. slaughter, richard a. (2008). futures education: catalyst for our times. journal of futures studies 12(3), 15-30. retrieved from http://jfsdigital.org/articlesand-essays/2008-2/vol-12-no-3-february/ wallace, michael l., joshua d. walker, anne m. braseby, and michael s. sweet. (2014). “now, what happens during class?”: using team-based learning to optimize the role of expertise within the flipped classroom.” journal on excellence in college teaching 25(3&4), 253-273. retrieved from http://celt.miamioh.edu/ject/issue.php?v=25&n=3%20and%204 nothing is set in stone 147 journal of effective teaching in higher education, vol. 1, no. 2 appendix futurology techniques adapted for class activities technique description benefits cautions & limitations future-oriented discussion future-oriented discussion regarding an issue or topic that is not immediately implicated in building solutions or taking action. covers general information about a topic. helps students clarify their own perspective on a topic. subject to groupthink. less effective than more structured methods, such as backcasting, at helping students feel a sense of ownership over the future. future-oriented roleplay students act out the roles of people or organizations in the future and engage in debate or problemsolving. the duration is up to the instructor— anything from part of one class session to an entire semester. permits debate. builds public speaking skills. instructor must help students manage emotions constructively, to prevent students from bringing roleplay conflicts into real-life interactions with classmates. students may not have an opportunity to express their own viewpoints if roles are assigned. backcasting determine a desired occurrence at a particular point in the future. then work backward to determine what would need to happen between now and then in order for the desired occurrence to come about. follow by providing recommendations for action in the near future that the scope is up to the instructor—this could be a short exercise lasting less than one class session, intended to get across basic information, or an opportunity for long-term planning and potentially actual action. helps people feel a sense of ownership over the future. may not provide opportunities to evaluate radical social or technological change. nothing is set in stone 148 journal of effective teaching in higher education, vol. 1, no. 2 technique description benefits cautions & limitations can help lead up to the desired future occurrence. trend analysis look at information from the past and present to forecast expectations for the future. helps people familiarize themselves with historical and current events and see underlying patterns. if using statistical techniques, instructors must ensure availability of computers and software. may not take into account wildcards (low-probability events that would cause substantial effects if they did occur). delphi method conventionally, questionnaires would be distributed to a set of experts. then, anonymous responses would be compiled and sent to the same experts, who would have the opportunity to change their initial responses in response to the information in the compilation. this general format could be adapted to allow all students in one class to participate once they are familiar enough with the topic in question. anonymity permits quiet or shy people to have their point of view heard. giving all responses equal weight prevents loud people or people with strong personalities from dominating. allowing people to express their own thoughts before hearing others' thoughts prevents groupthink. since all anonymous responses are treated equally, poorly prepared students can take up time with irrelevant or illogical content. this technique would thus be more suitable for a class in which the instructor knows most students are well prepared. when doing this in person in larger classes, it may require more than one class session or portion thereof to allow time to compile responses. scenario planning construct several potential future possibilities regarding a specific topic. helps people recognize multiple potential futures, thus mitigating narrowto avoid idealistic wishful thinking in favor of plausible futures, instructors must create a nothing is set in stone 149 journal of effective teaching in higher education, vol. 1, no. 2 technique description benefits cautions & limitations mindedness. structure for students' responses (e.g., requiring them to relate their ideas to past successes or failures and requiring them to evaluate necessary steps to completion and a timeline to completion). futures wheel (see figure 1) visual depiction of potential futures. begin with an event or issue in the center, construct branches around it with direct potential consequences, and then construct branches around the direct consequences with consequences of those (that is, indirect consequences of the original event or issue). suitable for initial brainstorming about an issue. permits people to see multiple pros and cons. visual depiction is unsuitable for substantial detail. this method does not provide strategies for action. stakeholder analysis (see figure 2) identify people or organizations involved in an issue, then evaluate their characteristics and interests in a chart or graph. for example, construct a two-axis grid with one axis for level of power and one axis for level of interest and determine the appropriate place helps people understand different roles people and entities play in an issue. helps people realize where to direct attention or action. may need to be repeated in order to evaluate change over time. nothing is set in stone 150 journal of effective teaching in higher education, vol. 1, no. 2 technique description benefits cautions & limitations for each stakeholder. futurewatching choose a topic and skim a variety of resources (peerreviewed journals, news, blogs, businesses' websites, etc.) to gain some general information on trends, viewpoints, concerns, and hopes. can help students explore a topic they are less familiar with. provides an opportunity for instructors to teach students how to evaluate the quality of different sources of information. this technique can incorporate an expansive quantity of information. instructors must provide guidelines and limits to keep students on track. causal layered analysis in four layers (described in detail by inayatullah [2008]), move beyond the talking points on an issue to explore deeper beliefs and motives. encourages students to understand why opposing groups think and behave as they do. this method frequently incorporates an artistic or poetic element, allowing students to practice diverse communication skills. this method requires students to identify and critique deeply held assumptions about how the world works. instructors must push students to not only acknowledge their biases but also manage the emotions that come along with them. nothing is set in stone 151 journal of effective teaching in higher education, vol. 1, no. 2 figure 1. a futures wheel constructed by a group of students that shows potential consequences arising from the construction of an integrated daycare and senior center. nothing is set in stone 152 journal of effective teaching in higher education, vol. 1, no. 2 figure 2. a stakeholder analysis constructed by a group of students that shows the power and interest various people and groups have over guiding automation use in the workplace. microsoft word suart_final.docx journal of effective teaching in higher education, vol. 6, no. 1 https://doi.org/10.36021/jethe.v6i1.320 modalities of faculty engagement with the scholarship of teaching and learning celeste suart, mcmaster university, suartce@mcmaster.ca martha cassidy-neumiller, mcmaster university kelsey harvey, mcmaster university abstract. this article reports on a mixed-methods study examining the ways in which faculty and staff engage with the scholarship of teaching and learning (sotl) at a medium-sized research-intensive university in southern ontario, canada. survey data was collected from fifty-six faculty and staff respondents, along with eight faculty completing follow-up semi-structured interviews. we found respondents used multiple engagement modalities to stay informed on sotl literature, carry out sotl research, and disseminate their findings. barriers to sotl participation include lack of dedicated time, limited formal sotl training, and inexperience with different disciplinary norms found in sotl articles. participants emphasized the importance of collaborative sotl inquiry, highlighting in particular the benefits of partnering with students on scholarly projects. additionally, participants underscored the importance of implementing evidence-based teaching strategies. our findings mirror trends in the literature regarding sotl engagement activities, barriers to participation, and faculty perceptions of sotl. this study contributes novel insight into the ways faculty choose to engage with sotl and common obstacles, as well as suggestions for how teaching and learning centers can use engagement data to better support faculty and staff sotl scholars. keywords: scholarship of teaching and learning (sotl); scholarly teaching; research engagement; dissemination; discipline-based education research engagement with scholarship of teaching and learning (sotl) is known to have many benefits for faculty, staff, and students. studies indicate faculty who conduct sotl inquiry employ more student-centered pedagogical approaches, boast improved student learning outcomes, and develop their own teaching practices (openo et al., 2017; trigwell, 2013). additionally, sotl inquiry provides opportunities for interdisciplinary exchange which aid in the development of innovative teaching practices, as well as critical reflection on disciplinary norms (huijser et al., 2021; marquis, 2015). despite these benefits, there are several known barriers to faculty and staff participation in sotl, including epistemological discomfort, limited time and monetary resources, and lack of perceived value of sotl research by peers (mckinney & jarvis, 2009; miller-young et al., 2018; schroeder, 2007). support for sotl engagement is frequently offered through institutional teaching and learning centers (kim et al., 2021; wuetherick & yu, 2016). however, teaching and learning centers must adapt support practices from the literature to suit their specific institutional contexts. in this paper, we explore the myriad ways that faculty and staff engage with sotl at our medium-size, research-intensive university in southern ontario, canada. we modalities of faculty engagement with the scholarship of teaching and learning 2 journal of effective teaching in higher education, vol. 6, no. 1 investigated how faculty members participated in threefold aspects of sotl: (a) staying current on the literature, (b) conducting sotl inquiry, and (c) publicizing research findings. additionally, we examined barriers and frustrations faculty experienced when trying to engage in sotl activities. we used this information about modalities of sotl engagement to inform future decisions about sotl programming and support activities offered by our teaching and learning center. in this article, we outline how we documented current trends of faculty sotl participation at our institution, and connect themes expressed by our respondents to the broader literature, so that others might be able to conduct similar environmental scans at their own institutions. literature review faculty and staff engage with sotl literature in manifold ways. extant literature demonstrates that, in addition to reading academic papers and books, faculty and staff learn about evidence and trends in sotl by attending presentations, participating in workshops, reading professional blogs, and engaging with other products of knowledge mobilization efforts (miller-young et al., 2017; newton et al., 2019). at times, this can be overwhelming given the considerable growth in avenues for the publication and dissemination of teaching and learning scholarship (vithal, 2018). it can be challenging for both newcomers and more experienced sotl researchers to know where to begin to find relevant literature (kenny & evers, 2011; kim et al., 2021). entry into the sotl literature is further complicated by the broad range of research methods employed, which draw on a variety of disciplinary foundations (divan et al., 2017). navigating and engaging with literature outside of one’s disciplinary expertise can be daunting for newcomers to sotl (kenny & evers, 2011; miller-young et al., 2018; simmons et al., 2013). further, faculty who explore interdisciplinary research, such as sotl, report epistemological friction between the research norms of their disciplinary background and those of other disciplines (miller-young et al., 2018). although these epistemological tensions can lead to growth and self-reflection, it does not make the process of engaging with sotl literature any less challenging. due to this potential for disorientation, faculty and staff often also engaged with literature with the assistance of experts in the field (dalgarno et al., 2020; kim et al., 2021; williams et al., 2013). this might take the form of pre-prepared bibliographies on common teaching and learning subjects or as forms of professional development (dalgarno et al., 2020). rather than approach the literature on teaching and learning systematically, felten (2013) described the process used by most faculty when engaging with sotl literature as “ad hoc.” what this means is that faculty and staff looked to the literature to address problems they faced while teaching (felten, 2013). allen and field (2005) describe these faculty and staff who consult the literature and integrate findings into their teaching practice as “scholarly teachers.” often, faculty and staff look to disciplinary journals, rather than general journals devoted to sotl or teaching and learning in higher education (tierney, 2017). tierney (2017) speculates that this is due in part to the fact that faculty and staff receive little modalities of faculty engagement with the scholarship of teaching and learning 3 journal of effective teaching in higher education, vol. 6, no. 1 formal training on how to engage in sotl research, so they default to their disciplinary training. this lack of formal training, may contribute to frustration on the part of faculty and staff who wish to engage in sotl (billot et al., 2017; harland et al., 2014; brownell & tanner, 2012). this frustration stems from the fact that faculty and staff, who are experts in their respective fields, are novices in sotl but unrealistically, expect a high degree of competence when engaging with sotl literature (harland et al., 2014; simmons et al., 2013). unfortunately, for most faculty and staff, the work involved in keeping up with sotl literature is often described as “off the side of one’s desk,” as it is not part of scholars’ disciplinary research and often “doesn’t count” toward their research portfolio (digregorio et al., 2016; harland et al., 2014). sometimes this labor is supported by the work of student partners, however, the precarity of faculty members engagement with sotl can in turn limit opportunities for student participation (mercer-mapstone et al., 2017). although there is a small but growing number of academics whose primary research focus is sotl, disciplinary inquiry continues to take precedence over sotl engagement for many practitioners (webb & tierney, 2020). as sotl is perceived as less valued than disciplinary scholarship (allen & field, 2005; harland et al., 2014), some faculty and staff may not engage with the literature in this field much, if at all. moreover, not all faculty and staff have equal access to or support for this type of work (brown et al., 2013; vajoczki et al., 2011). according to vander kloet and colleagues (2017), contingent faculty (e.g., part-time sessional or adjunct faculty and those with contractually limited appointments) are marginalized from scholarship as they face myriad institutional barriers from participation therein. these barriers include the length of contracts, the precarity of future contracts, exclusion from holding grants or awards, lack of knowledge about institutional support that might be available, and more. although, these facilitators and barriers vary by institution, with some institutional cultures being more supportive than others (miller-young et al., 2016; schroeder, 2007). therefore, there are several known obstacles to faculty and staff engagement with sotl research. how to best support sotl practitioners has been frequently discussed in the literature. supports include providing research funding, creating communities of practice, and providing protected time for sotl research (cambridge, 2004; mckinney & jarvis, 2009). others have argued that changing institutional culture to value sotl inquiry is key (maheux-pelletier et al., 2019; marquis et al., 2017; mckinney, 2006; wuetherick et al., 2016). whatever the intervention, institutional teaching and learning centers play a vital role in supporting researchers’ engagement in sotl (kim et al., 2021; miller-young et al., 2016). being that institutional context has been identified as an integral component for the design and delivery of sotl supports (kim et al., 2021; kolomitro et al., 2018), how might a teaching and learning center identify which supports would most benefit its faculty and staff sotl practitioners? to that end, the objective of this study was to identify modalities of faculty and staff engagement in sotl. the purpose herein, was to leverage our findings when (re)developing research supports and training in sotl that accounted for our institution’s culture. modalities of faculty engagement with the scholarship of teaching and learning 4 journal of effective teaching in higher education, vol. 6, no. 1 methodology this paper represents one sub-aim of a larger environmental scan and strategic assessment of teaching and learning scholarship at mcmaster university, a research-intensive doctoral-granting university in ontario, canada (harvey et al., 2022). the environmental scan was conducted by the research team on behalf of the paul r. macpherson institute for leadership, innovation & excellence in teaching, the teaching and learning center for mcmaster university, with the goal to examine how faculty and staff members describe teaching and learning research, engage in research activities, and what motivates their engagement. the goal of this larger assessment was to document the current state of teaching and learning research at mcmaster university, allowing for the development of tailored supports and resources. we were guided by the question: how do faculty and staff engage with scholarly literature and research on teaching and learning? from this information, our institutional teaching and learning center would have better knowledge and awareness to make informed decisions about sotl programming and related activities. herein, we share findings from our inquiry into how faculty and staff engage with scholarly research on teaching and learning. research design reflecting on pat hutchings’s (2000) taxonomy of sotl work, we designed our study to follow the paradigms of “what is?”, where sotl projects seek to describe a current state of teaching and learning. our aim was to better understand the landscape of faculty and staff engagement with sotl at our institution. this would allow us to know which forms of engagement are most popular, thus allowing the mcmaster university teaching and learning center to better tailor its supports for sotl. as such, we employed a concurrent, nested, mixed-methods research design, allowing us to quantitatively assess faculty and staff engagement modalities with sotl while incorporating qualitative data to provide situational context (creswell et al., 2003; warfa, 2016). we used an anonymous online survey to gather quantitative data. after completion of the survey, respondents could indicate their interest (on a separate page so as to not link their identity with their survey response) in participating in an optional, follow-up semi-structured interviews to gather qualitative data. data was analyzed concurrently, with integration following analysis to broadly interpret ‘what is’ engagement with sotl by faculty and staff. participants our target population was faculty and staff members at mcmaster university who had participated in sotl research in the past five years. we narrowed our scope of inquiry to those with sotl experience in the past five years to better understand the impact of present sotl support, university policies, and teaching contexts on sotl engagement. we received 56 complete responses to our online survey. the majority of respondents were from stem disciplines (n=35, 63%), with 23% (n=13) having humanities and social science backgrounds (table 1). additionally, 7% (n=4) of respondents indicated multidisciplinary and cross-faculty affiliations (table 1). modalities of faculty engagement with the scholarship of teaching and learning 5 journal of effective teaching in higher education, vol. 6, no. 1 table 1 survey respondent faculty affiliation at mcmaster university faculty respondents (%) respondents (n) science 23.2 13 health sciences 21.4 12 engineering 17.9 10 humanities 12.5 7 social sciences 7.1 4 business 3.6 2 other 7.1 4 prefer not to disclose 7.1 4 note. examples of “other” category affiliations included multidisciplinary appointments and the teaching and learning center. n=56. over half of respondents were research-track faculty members (52%), 14% were full-time teaching-track faculty, and 16% were sessional or contractually limited instructors (table 2). at mcmaster university, sessional instructors are part-time contract workers who facilitate a course offering during a specific academic term, while contractually limited instructors are contracted to instruct multiple courses for one to three years. more junior academic positions, such as postdoctoral fellows (7%), graduate students (5%), and instructional or research assistants (4%) were also represented in our sample (table 2). table 2 survey respondent academic position at mcmaster university academic position respondents (%) respondents (n) research-track faculty member 51.8 29 teaching-track faculty member 14.3 8 contractually limited appointment faculty or sessional instructor 16.1 9 postdoctoral fellow 7.1 4 graduate student 5.4 3 instructional assistant or research assistant 3.6 2 other 8.9 5 note. respondents could select multiple responses. examples of role descriptions in the “other” category include research coordinator, assistant dean, and recently completed postdoctoral fellow. n=56. overall, our survey sample was largely comprised of research-track faculty members from stem disciplines (tables 1, 2). this skew is representative of the known demographics of mcmaster university faculty (mcmaster university, 2022). modalities of faculty engagement with the scholarship of teaching and learning 6 journal of effective teaching in higher education, vol. 6, no. 1 each week while the survey was live, we contacted respondents who expressed an interest in being interviewed to schedule a time to meet. if we did not hear back from a respondent, we sent one reminder email 2-4 weeks after the original email. eighteen survey respondents expressed an interest in being interviewed but ultimately eight elected to participate in a follow-up interview. the aim of these semi-structured interviews was for us to gain a deeper understanding of their varied experiences engaging in sotl activities. while limited in scope due to the small number of interviewees, the participants were diverse in their positionalities and experiences. half identified as men and half as women. faculties represented were science, health sciences, social sciences, business, engineering, humanities, arts & sciences. their ranks ranged from contractually limited appointments to full professor. there was also a balance between faculty with teaching and research-track appointments. data collection the online survey was open from may 10, 2021, to august 31, 2021. it was delivered through limesurvey, an open-source statistical web survey program. the survey took approximately 20-30 minutes to complete. the survey tool consisted of quantitative measures, including multiple choice and yes-no forced-choice questions. all data collected through the survey was anonymized. demographic questions, including faculty affiliation and academic position, had options to indicate if respondents would prefer not to answer. surveys containing blank responses to mandatory questions were treated as participant withdrawal from the study and were not included in analysis. virtual interviews took place over the summer of 2021 and were conducted by members of the research team. interviews were recorded using zoom. members of the research team cleaned and anonymized the ai-generated transcripts. data analysis descriptive statistical analysis was conducted on survey data using spss version 26.0 (ibm corporation, armonk, united states) (pallant, 2020). graphs were designed using graphpad prism 9.3.1 (graphpad software, san diego, usa). thematic analysis was used to examine interview data (braun & clarke, 2006; kara, 2022). first, the researchers familiarized themselves with the data through transcribing and reading each transcript. then, the researchers coded the transcripts using both deductive and inductive coding to identify passages and subthemes related to engagement with scholarly research and literature. results we classified faculty and staff sotl engagement activities into three main categories: staying informed on sotl literature, conducting sotl research, and modalities of faculty engagement with the scholarship of teaching and learning 7 journal of effective teaching in higher education, vol. 6, no. 1 disseminating sotl findings. herein, research participants used a variety of terms to describe their teaching and learning research, such as pedagogical research, education research, and sotl. in this paper, we will describe these research activities as sotl, as they represent “the systematic study of teaching and learning. . . resulting in products that are publicly shared for critique and use” (potter & kustra, 2011, p. 2). staying informed on literature participants reported a variety of activities to stay informed of sotl research. all participants had engaged with at least one activity to stay informed over the past five years. the most common activities included attending a teaching and learning conference (77%) and reviewing the teaching and learning center website for information (77%). many (71%) respondents reported implementing research findings of others into their own teaching (figure 1). although three-quarters of respondents indicated they used the teaching and learning center website to find information on sotl research, only 30% used other mcmaster university websites for similar purposes (figure 1). passive, or one-way, activities used to stay informed of sotl research included subscriptions to newsletters (52%), subscriptions to publication or journal alerts (43%), and reading the literature (57%) (figure 1). active, or two-way, activities used to stay informed included participating in a teaching and learning research network (50%), attending relevant professional development sessions (57%), or attending teaching and learning sessions at disciplinary conferences (64%) (figure 1). echoing the survey, the faculty we interviewed described engaging in myriad one-way and two-way knowledge-sharing scholarly activities to stay informed on sotl literature. one-way activities highlighted in interviews included conducting literature reviews and listening to sotl presentations, while two-way activities included attending sotl workshops and community of practice meetings. overall, there was no distinct preference for one-way or two-way knowledge-sharing activities, with respondents indicating participation in both. this reinforces the need to maintain both styles of activities to help faculty, staff, and students stay informed about sotl research. however, interviewees did lament the struggle of staying informed on scholarly teaching literature, particularly due to lack of time. i must admit that one of the biggest difficulties that i have, and i’ve heard my colleagues have, is just the amount of time it takes to keep up to date with what's going on. that the sort of the lovely-background reading, the absorbing, the thinking about things, that-we just don't have time. (participant c) faculty interviewees also expressed frustrations with the quantity of sotl literature. part of this frustration seemed to stem from researchers’ disciplinary modalities of faculty engagement with the scholarship of teaching and learning 8 journal of effective teaching in higher education, vol. 6, no. 1 traditions and epistemological positions that faculty perceived as ill-suited to research questions being asked in sotl: ...there's a lot of [stem education research] work that's extremely poor primarily because we're applying [stem] ways of thinking to something that should really involve much more social science ways of looking at [the] world. right? (participant h) figure 1 information sources and activities used by faculty and staff to stay informed about research on teaching and learning note. percentage value represents the proportion of respondents who answered ‘yes’ to the listed questions. n=56, error bars display standard error of the mean. additionally, some interviewees from stem disciplines were dissatisfied with the literature they found on teaching and learning when trying to implement evidenceinformed practices. these sentiments seemed particularly linked to reflective essays and a longing for what interviewees perceived as more objective analysis: modalities of faculty engagement with the scholarship of teaching and learning 9 journal of effective teaching in higher education, vol. 6, no. 1 i was frustrated because we would make some change [to our teaching], and i would look in the literature for some evidence and the evidence was bad. i just found it was just arbitrary… [in the literature] somebody was saying [this] was their experience, and i’m not saying that's not a type of data, it's just not a type of data that i find is actionable (participant f) in contrast, some faculty shared that they specifically sought literature that was practical and based on educators’ lived experiences, or as participant g put it “something that comes out of, you know, out of actually from people who are actually teaching.” i want to know from my colleagues who tell me ‘okay, i tried to implement, you know, i tried to do it this way, i tried to tweak it this way…’ you know what i’m saying? that's useful to me... (participant g) thus, survey respondents and interviewees expressed a preference in a variety of knowledge sharing activities to stay informed on the sotl literature. this friction may also reflect the breadth of research methods used in sotl (divan et al., 2017), given the wide variety of disciplinary training and perspectives that faculty and staff bring to sotl inquiry. conducting research next, we asked respondents about activities associated with conducting sotl research. only 54% of respondents indicated they received formal or informal training on conducting sotl research over the past five years (figure 2). this suggests that respondents may be drawing on their disciplinary-specific knowledge when conducting sotl research or relying on past training. however, this may also indicate a gap in availability or opportunities to be trained in teaching and learning specific research methodology. indeed, few faculty we interviewed had received formal teaching or sotl research training during their own educational trajectories. many were self-taught, learned from mentors, or engaged in educational workshops offered by a teaching and learning center. indeed, just over 73% of survey respondents indicated they received support from the teaching and learning center when conducting sotl research (figure 2). some faculty expressed irritation with other academics’ perceptions that sotl research was easier than disciplinary research. this sentiment was shared by participant f: “there’s this idea that anybody can do [education research] right away. . . it’s right to say that anybody could do education research, pretty much. but it’s just not that easy.” this impression that sotl was simpler than disciplinary research led some interviewees to feel that their sotl inquiry was “second-class” compared to disciplinary research. modalities of faculty engagement with the scholarship of teaching and learning 10 journal of effective teaching in higher education, vol. 6, no. 1 figure 2 activities completed by respondents in the past five years to support their research on teaching and learning note. percentage value represents the proportion of respondents who answered ‘yes’ to the listed questions. n=56, error bars display standard error of the mean. curiously, only 43% of survey respondents identified that sotl research was part of their formal role and responsibilities (figure 2). by far the biggest complaint interviewees shared was regarding the lack of time to conduct sotl research. as explained by participant c, “...people who are teaching professors normally do not have time in their pie chart of their duties: there’s 80% teaching and 20% service. where does this scholarship occur?” this suggests many respondents are conducting sotl research in addition to their other professional responsibilities, which highlights the importance of available sotl research supports. another interesting trend was related to how respondents’ research was funded, with 79% indicating they received internal funding and 41% indicating they had external funding support. it is unclear if this is due to higher success rates of internal funding applications or if fewer external funding applications are submitted compared to internal funding applications. one potential explanation for fewer external funding applications is the limited number of respondents who have sotl research as part of their formal role. if this work is not part of a broader program of research, respondents may not consider entering external funding competitions and instead focus on disciplinary-based research funding. nevertheless, this highlights the potential impact of mcmaster university’s internal funding options for sotl, such as the student partners program grants, small teaching and learning exploration grant, the priority areas for learning and modalities of faculty engagement with the scholarship of teaching and learning 11 journal of effective teaching in higher education, vol. 6, no. 1 teaching research (palat) grants, and inclusion, diversity, equity, accessibility and sustainability (ideas) grants. eighty-six percent of respondents reported receiving at least one source of funding for their teaching and learning research. overall, it appears respondents conduct sotl research of their own accord, despite this being outside their formal role and their lack of recent formal training, which is consistent with previous literature (billot et al., 2017; brownell & tanner, 2012). we also asked respondents about the scope of their sotl-related research activities, namely who they work with, what kind of work they do, and if they take on leadership-related activities. the majority of respondents reported working with colleagues at the same institution (73%), followed by student partners (59%), colleagues at different institutions (52%), and lastly teaching and learning center staff (43%) (figure 3). 57% of respondents had research questions related to their own classroom context, and 48% worked on interdisciplinary teams to accomplish their goals (figure 3). much of this sotl research, according to the faculty interviewed, remained disciplinary. for example, participant g shared that they “would never [have] gotten tenure, if [i] only published in [disciplinary] education. so, my tenure case was based on the combination of publications in [discipline] and [disciplinary] education. and for [disciplinary] education, i was told only international publications counted.” thus, interviewees felt that some of the activities in which faculty engaged “did not count” and were “in addition to” their disciplinary duties (participant c). overall, 95% of respondents indicated they had collaborated with at least one other person on their teaching and learning research work, such as a colleague, student partner, or teaching and learning center staff member (figure 3). in the interview data, participants elaborated on their participation in collaborations with scholars within and outside of mcmaster on sotl research. faculty herein expressed appreciation particularly for involving students as partners in the research process, so that students learn to become “generators of information, as opposed to just consumers” (participant e). it was in this manner that conducting research was also an act of educating and teaching students. some faculty considered this act an additional benefit: i mean, there's the inherent value of actually kicking the ball down the field and gaining some knowledge about what things work and don't work, and then there's that secondary gain of teaching other people how to do research so that they can continue to kick the ball down the field, right? to me, that's the thing. there’s the inherent and then there's the secondary gain or the collateral help that you get. (participant f) however, these research collaborations with students were not always equitable partnerships. although faculty participants reveled in sharing the research process with students, some revealed that more time-consuming tasks, like literature review writing, were often a duty they would offload to students. however, it was the faculty member who reaped the rewards of students’ efforts, including formal recognition from publications and presentations. modalities of faculty engagement with the scholarship of teaching and learning 12 journal of effective teaching in higher education, vol. 6, no. 1 figure 3 scope of research on teaching and learning activities reported by respondents note. percentage value represents the proportion of respondents who answered ‘yes’ to the listed questions. n=56, error bars display standard error of the mean. compared to the high percentage of respondents reporting collaborative engagement, far fewer respondents indicated they took on leadership roles related to sotl research. twenty percent of respondents indicated they have tried to influence departmental teaching policies, while 11% took a leadership role in a sotl research organization or journal (figure 3). this suggests that while respondents conduct sotl research with a variety of collaborators, they may be less invested in taking on leadership positions involving sotl. disseminating findings finally, we asked respondents about how they disseminated their research on teaching and learning. when asked where they delivered formal presentations, 64% indicated sotl research conferences outside of mcmaster university, 52% at discipline-based conferences, 46% at a mcmaster university sotl research conference, and 39% at a departmental meeting or event, with 77% indicating they have delivered at least one type of presentation (figure 4). 59% of respondents reported publishing their sotl research findings in a journal or book (figure 4). in the interview data, participants further identified knowledge translation activities, modalities of faculty engagement with the scholarship of teaching and learning 13 journal of effective teaching in higher education, vol. 6, no. 1 conference presentations, and incorporating research evidence into their teaching practice as approaches to disseminating findings. figure 4 research on teaching and learning dissemination methods reported by respondents note. percentage value represents the proportion of respondents who answered ‘yes’ to the listed questions. n=56, error bars display standard error of the mean. while over half of respondents said they shared their findings to influence curriculum development (59%), only 25% made efforts to influence policy (figure 4). interviewed faculty cautioned that implementation of educational practices and policy must be driven by evidence “rather than the way it is now, [in] which [educational change] is just adopted on the basis of novelty” (participant f). meaning, faculty perceived other educators and administrators as adopting some pedagogical practices on the basis of what was new or trending, rather than modalities of faculty engagement with the scholarship of teaching and learning 14 journal of effective teaching in higher education, vol. 6, no. 1 adopting practices with strong empirical support. this highlights a need for further sotl dissemination efforts influencing departmental, faculty, or institutional-level educational policies to encourage the use of evidence-based teaching strategies. when asked about more informal means of dissemination, 79% of respondents indicated they discussed their findings with colleagues at mcmaster university, and 52% shared their research results with students (figure 4). a few respondents used other informal means of research dissemination, such as a website or blog post (21%) or media interview (16%) (figure 4). overall, this data indicates that respondents use a mix of formal and informal research dissemination methods. discussion in this study, we explored how faculty and staff at a medium-sized, researchintensive university engage with sotl, including activities to stay informed about sotl literature, carry out their own sotl inquiry, and disseminate sotl research findings. similar to previous scholarship, we identified a multitude of sotl engagement possibilities (kim et al., 2021; miller-young et al., 2017; newton et al., 2019). we also identified familiar hurdles which may impede sotl participation by faculty and staff members (kolomitro et al., 2018; maheux-pelletier et al., 2019). while these opportunities and challenges for sotl practitioners echo extant literature, our analysis provides further insight into how these phenomena are related to effective teaching practices. when we asked respondents about their preferences when engaging in the literature, we found mixed preferences for more passive forms of participation, such as newsletter and journal subscriptions, as well as active forms of participation, such as professional development sessions and conference attendance. sotl organizations often emphasize the importance of active engagement for students and teachers (mckinney, 2013). however, these types of activities often require a greater time commitment than passive engagement strategies. similar to other groups, our participants identified a lack of protected time as a barrier to engaging with the literature (brownell & tanner, 2012; kim et al., 2021; kolomitro et al., 2018), which would explain the level of passive engagement activities. this highlights the need for teaching and learning centers to provide multiple modes of sotl training and engagement, to provide options for faculty and staff to engage with sotl in ways that match their interest and time availability. moreover, teaching and learning center staff could provide syntheses of sotl research. doing so could alleviate faculty time constraints and would ensure that staff who possess a high degree of expertise in sotl research methods are tasked with translating the literature in a way that would be useful to faculty from a variety of disciplinary backgrounds. a compounding factor reducing time for sotl activities is the absence of sotl from formal job descriptions, a precarity which has been identified by other groups (digregorio et al., 2016; vander kloet et al., 2017). at a structural level, the inclusion of sotl research into formal roles of faculty and staff members would increase individual time for sotl inquiry. additionally, formalizing sotl as part of job descriptions is a way for institutions to indicate they prioritize scholarly teaching modalities of faculty engagement with the scholarship of teaching and learning 15 journal of effective teaching in higher education, vol. 6, no. 1 and learning. above all, the fact that study participants were actively engaged in sotl, despite a lack of training and formal recognition, speaks to the intrinsic motivation driving sotl research at our institution. with regards to the self-disclosed lack of formal sotl training reported by participants, this observation is consistent with previous findings which suggest that sotl practitioners rely frequently on their disciplinary training or self-taught sotl research skills (billot et al., 2017; brownell & tanner, 2012; tierney, 2017). not only did this result in a perception that sotl research was easier than disciplinary inquiry, but also unfamiliarity with the best practices and norms employed in sotl, particularly those originating from social sciences and humanities. many sotl practitioners have reported similar epistemological tension between the norms of their disciplinary training and sotl, leading to a disruption of academic identity (miller-young et al., 2018; simmons et al., 2013). this further emphasizes the potential benefit of sotl workshops and training programs offered through teaching and learning centers, not only to learn practical research skills but also to navigate disciplinary disorientation amongst a group of like-minded individuals. collaborating with other like-minded sotl practitioners was also of importance to faculty and staff participants, given that 95% of survey respondents indicated they worked with at least one other person on their teaching and learning research. sotl as a field has historically emphasized collaborative inquiry as vital (chick & poole, 2013; elton, 2009). working in teams has been shown to decrease feelings of invisibility and isolation reported by sotl practitioners, as well as help scholars acculturate to sotl norms and develop relevant research skills (kolomitro et al., 2018; vander kloet et al., 2017). faculty interviewees also brought up the importance of including students in the research process, thus increasing student confidence by becoming producers of knowledge through engaging in scholarly inquiry. there are several positive outcomes stemming from partnerships between faculty, staff, and students through sotl, including increased student engagement and confidence, student development of research skills, and development of better teaching materials (mercer-mapstone et al., 2017). however, the students-aspartners literature emphasizes the need for reciprocity and equity in studentfaculty-staff partnership for these benefits to occur (marquis et al., 2022; mercermapstone et al., 2017), something that was not always present in the partnerships described by the interviewees. teaching and learning centers could, therefore, play an important role in supporting student engagement in sotl. this could take the form of training and mentoring students in sotl, as well as ensuring students are equally recognized for their sotl contributions (e.g., paying students to work as sotl research assistants and/or student partners) (healey et al., 2014; mercermapstone et al., 2017). when examining sotl dissemination methods, a foundational part of sotl best practices (felten, 2013), a greater proportion of survey respondents indicated using lower-cost dissemination strategies such as talking with colleagues (79%) or delivering presentations (77%), than more cost and time intensive dissemination strategies including journal publications (59%). indeed, others have identified a lack of access to dissemination-oriented funding as a barrier to more formal or costly means of sharing research findings (brown et al., 2013; vander kloet et al., modalities of faculty engagement with the scholarship of teaching and learning 16 journal of effective teaching in higher education, vol. 6, no. 1 2017). though these lower-cost dissemination approaches help with personal and colleagues' scholarly teaching practices (vajoczki et al., 2011), these barriers contribute to gaps within published scholarship. this in turn limits the evidence available for administrators crafting educational policy. in response to this barrier, other sotl programs have created grants specifically for dissemination activities (miller-young et al., 2016). teaching and learning centers can play a valuable role in guiding and funding sotl dissemination, both for scholarly publications and crafting educational policy. limitations one limitation of our study was our use of self-reported survey tools, results of which can be impacted by social desirability bias (larson, 2019). however, we implemented best practices to reduce the potential impact of social desirability bias, including anonymizing our survey, providing options to decline demographic questions, emphasizing respondent confidentiality, and the use of forced-choice yes-no questions (lau & kennedy, 2019). additionally, we must consider the limitations of our sample size (survey n=56, interview n=8) when generalizing about faculty and staff engagement with sotl. additionally, the majority of our respondents were from stem disciplines and were research track faculty members. although our demographic factors support that we had broad representation from faculty and staff across mcmaster university, when seeking to apply our findings to other postsecondary institutions we need to consider how different contextual factors, such as disciplinary affiliation and academic position, may limit generalizability. implications our findings offer insights as to how faculty and staff can engage with sotl literature and research. teaching and learning centers, and other postsecondary units like departments or schools/faculties, can draw several implications from these findings when considering how they can foster a culture of support for sotl inquiry. when reading these suggestions, one must keep in mind that different institutional cultures and viewpoints of sotl will impact what supports are needed and which engagement strategies are most effective. these proposals should be viewed as starting points for conversation, in order to be adapted to each institutional context. first, this work offers an outline of how to conduct similar evaluations of faculty and staff sotl engagement in other institutional contexts. the literature has consistently demonstrated the impact of contextual factors on individual sotl practitioner experiences (kolomitro et al., 2018; wuetherick & yu, 2016). thus, when drawing on inspiration from sotl supports documented in the literature, one should consider their institutional context and the current engagement needs of faculty and staff. the culture at the institution where this study took place values sotl and is proud of its reputation in being a leader in education-related research. therefore, the institution has devoted resources to supporting sotl and this modalities of faculty engagement with the scholarship of teaching and learning 17 journal of effective teaching in higher education, vol. 6, no. 1 support has been championed by some key members of the administration. building on recommendations from kim and colleagues (2021), we believe that intentional, structured sotl support can yield long-term benefits for individual sotl practitioners and their academic organizations. with this in mind, present methods of faculty and staff sotl engagement must be examined to inform how practices could be improved. second, a common barrier to all types of sotl engagement, from reading literature to conducting primary research to disseminating findings, was a lack of time to do sotl work. this lack of time stemmed from sotl not being perceived to be as valuable as disciplinary inquiry and not having sotl incorporated into formal job descriptions. breaking down the false dichotomy between disciplinary research and sotl inquiry can encourage scholars at the individual level to use their time to pursue sotl (billot et al., 2017; webb & tierney, 2020). toward this aim, we recommend embedding recognition of sotl into faculty job descriptions. the findings from this study are currently being institutionally enacted at mcmaster university by first performing a review of criteria for promotion and tenure for faculty and then ensuring sotl is reflected therein. this change in policy largely stems from the advocacy work by faculty members calling for increased recognition and protected time for sotl activities. other options include arguing for the inclusion of sotl in job descriptions, as well as in the tenure and promotion criteria when negotiating new contracts for faculty. these will ensure that faculty have the time to devote to sotl. protected time should not only be extended for faculty engaging in the creation of sotl research, but also to the “scholarly teachers” described by allen and field (2005) who engage by reading the literature and incorporating evidence-based teaching strategies into their classrooms. in short, the synthesis and application of sotl, in addition to the creation of new ideas, must become an activity which is seen as valuable. indeed, for sustainable change to occur, structural changes must be implemented to shift institutional views on the value of sotl. departments, programs, and teaching and learning centers aiming to embed sotl into their institutional cultures need to set up structures that reward sotl excellence and provide protected work hours to pursue sotl inquiry (webb & tierney, 2020). sustained monetary and professional support allows for faculty and staff to have meaningful engagement with sotl, as opposed to doing it “off the side of one’s desk” (webb & tierney, 2020; williams et al., 2013). our institution offers three grant programs dedicated to funding sotl, which helps provide financial resources to faculty undertaking sotl. these funds can be used to hire student partners and research assistants who can assist faculty with their projects, as well as support project dissemination costs like conference and publishing fees. third, this research speaks to the value of creating partnership opportunities for sotl. examples of effective practices from the literature to foster collaborations include targeted communities of practice, cohort-based programs, and informal social networks for scholars (kim et al., 2021; marquis et al., 2017; miller-young et al., 2016; tierney et al., 2020). these strategies cultivate a sense of community amongst individual scholars, allowing for the sharing of resources, ideas, and modalities of faculty engagement with the scholarship of teaching and learning 18 journal of effective teaching in higher education, vol. 6, no. 1 struggles (marquis et al., 2017). moreover, creating equitable partnership opportunities for students interested in sotl can provide faculty and staff with much-needed support while also helping train the next generation of sotl scholars. teaching and learning centers can play a key role in affecting these sorts of collaborations (simmons & taylor, 2019). for example, the teaching and learning center at mcmaster university has dedicated educational developer staff support who foster research circles and writing retreats with faculty who are undertaking sotl research. these educational developers also aid in the dissemination of sotl findings through both active and passive strategies, including weekly newsletters, research summaries, annual research symposia, a teaching and learning month event series, and other professional development opportunities. the aims of having this variety of activities is to aid in the dissemination of sotl findings to colleagues who may not be familiar with the sotl literature, as well as to bring together and foster connections between faculty from various departments who share an interest in sotl. fourth, when designing assistance programs to foster sotl engagement, one must also consider how to stimulate the dissemination of sotl knowledge in addition to programming that aids in the creation of knowledge. public sharing of findings is a foundational aspect of sotl inquiry (felten, 2013), yet the methods and target audience of dissemination of sotl scholarship is frequently an afterthought. teaching and learning centers can aid sotl practitioners in identifying and creating avenues for dissemination (pechenkina, 2020), which can lead to increased uptake and impact of findings. in addition to the aforementioned avenues for dissemination, centers can also support sotl practitioners with addressing reviewer feedback, as faculty may be unfamiliar with the norms and expectations of sotl journals (miller-young et al., 2018). conclusion there are countless ways that faculty and staff members can engage in sotl inquiry. individual sotl practitioners will choose different modalities to read sotl literature, conduct primary research, and share research findings based on their own preferences and goals, as well as structural barriers. by examining trends in how faculty and staff members participate in sotl, institutions can identify common trends and barriers which shape the individual choices made by scholars. this information can be used to design strategies to support faculty and staff who conduct sotl inquiry which fit institutional and disciplinary contexts. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. this research was supported by the paul r. macpherson institute for leadership, innovation and excellence in teaching internal operating grant and summer 2021 student partner program award. modalities of faculty engagement with the scholarship of teaching and learning 19 journal of effective teaching in higher education, vol. 6, no. 1 references allen, m. & field, p. (2005). scholarly teaching and scholarship of teaching: noting the difference. international journal of nursing education scholarship, 2(1). https://doi.org/10.2202/1548-923x.1094 billot, j., rowland, s., carnell, b., amundsen, c., & evans, t. (2017). how experienced sotl researchers develop the credibility of their work. teaching & learning inquiry the issotl journal, 5(1). https://doi.org/10.20343/teachlearninqu.5.1.8 braun, v., & clarke, v. (2006). using thematic analysis in psychology. qualitative research in psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa brownell, s. e., & tanner, k. d. (2012). barriers to faculty pedagogical change: lack of training, time, incentives, and...tensions with professional identity? cbe life sciences education, 11(4), 339–346. https://doi.org/10.1187/cbe.12-090163 brown, n. r., university of tasmania, kelder, j.-a., freeman, b., carr, a. r., university of tasmania, university of melbourne, & university of tasmania. (2013). a message from the chalk face—what casual teaching staff tell us they want to know, access and experience. journal of university teaching & learning practice, 10(3), 102–118. https://doi.org/10.53761/1.10.3.6 cambridge, b. l. (2004). campus progress: supporting the scholarship of teaching and learning. american association for higher education. chick, n., & poole, g. (2013). launching tli: sotl’s purposes, processes, and people. teaching & learning inquiry the issotl journal, 1(1), 1–4. https://doi.org/10.20343/teachlearninqu.1.1.1 creswell, j. w., plano clark, v. l., gutmann, m., & hanson, w. (2003). advanced mixed methods research designs. in a. tashakkori & c. teddlie (eds.), handbook of mixed methods in social and behavioral research (pp. 209–240). sage. dalgarno, n., laverty, c., egan, r., garton, k., soleas, e., babando, j., & van wylick, r. (2020). participant perceptions of the faculty development educational research series. teaching & learning inquiry, 8(1), 221–245. https://doi.org/10.20343/teachlearninqu.8.1.15 digregorio, n., maurer, t. w., & pattanaik, s. (2016). documentation of scholarship of teaching and learning trends at national council on family relations annual conferences 2006-2015. family science review, 21(2). https://doi.org/10.26536/fsr.2016.21.02.03 divan, a., ludwig, l., matthews, k., motley, p., & tomljenovic-berube, a. (2017). modalities of faculty engagement with the scholarship of teaching and learning 20 journal of effective teaching in higher education, vol. 6, no. 1 research approaches in scholarship of teaching and learning publications: a systematic literature review. teaching & learning inquiry, 5(2), 16. https://doi.org/10.20343/teachlearninqu.5.2.3 elton, l. (2009). continuing professional development in higher education. arts and humanities in higher education, 8(3), 247–258. https://doi.org/10.1177/1474022209339955 felten, p. (2013). principles of good practice in sotl. teaching & learning inquiry, 1(1), 121–125. https://doi.org/10.2979/teachlearninqu.1.1.121 harland, t., raja hussain, r. m., & bakar, a. a. (2014). the scholarship of teaching and learning: challenges for malaysian academics. teaching in higher education, 19(1), 38–48. https://doi.org/10.1080/13562517.2013.827654 harvey, k., suart, c., aspenlieder, e., evanovitch, j., karim, f., minhas, a., krone, j., cassidy-neumiller, m., & de bie, a. (2022). results from an environmental scan of research on teaching and learning across mcmaster university. macpherson institute for leadership, innovation, and excellence in teaching. https://mi.mcmaster.ca/app/uploads/2022/09/research-on-teaching-andlearning-at-mcmaster-a-discussion_april2022.pdf healey, m., flint, a., & harrington, k. (2014). engagement through partnership: students as partners in learning and teaching in higher education. the higher education academy. https://www.advance-he.ac.uk/knowledgehub/engagement-through-partnershipstudents-partners-learning-andteaching-higher huijser, h., reis, c., soo, y. j., tan, v., walker, i., & wu, s. m. (2021). sotl enquiries in four disciplines: unlocking the potential of interdisciplinary approaches and methodologies. asian journal of the scholarship of teaching and learning, 11(1), 38–45. https://nus.edu.sg/cdtl/engagement/publications/ajsotl-home/archive-of-pastissues/v11n1/v11n1-huijser-et-al hutchings, p. (2000). approaching the scholarship of teaching and learning. in p. hutchings (ed.), opening lines: approaches to the scholarship of teaching and learning (pp. 1–10). carnegie foundation for the advancement of teaching. kara, h. (2022). qualitative research for quantitative researchers. sage. kenny, n., & evers, f. (2011). 4. responding to the challenging dilemma of faculty engagement in research on teaching and learning and disciplinary research. collected essays on learning and teaching, 3, 21–26. https://doi.org/10.22329/celt.v3i0.3234 kim, a. s. n., popovic, c., farrugia, l., saleh, s. a. f., maheux-pelletier, g., & frake-mistak, m. (2021). on nurturing the emergent sotl researcher: responding to challenges and opportunities. the international journal for modalities of faculty engagement with the scholarship of teaching and learning 21 journal of effective teaching in higher education, vol. 6, no. 1 academic development, 26(2), 163–175. https://doi.org/10.1080/1360144x.2020.1842743 kolomitro, k., laverty, c., & stockley, d. (2018). sparking sotl: triggers and stories from one institution. the canadian journal for the scholarship of teaching and learning, 9(1), 1–14. https://doi.org/10.5206/cjsotlrcacea.2018.1.10 larson, r. b. (2019). controlling social desirability bias. international journal of market research, 61(5), 534–547. https://doi.org/10.1177/1470785318805305 lau, a., & kennedy, c. (2019). when online survey respondents only “select some that apply.” pew research center. http://www.pewresearch.org/methods/2019/05/09/when-online-surveyrespondents-only-select-some-that-apply/ maheux-pelletier, g., marsh, h., & frake-mistak, m. (2019). the benefits of writing retreats revisited. in n. simmons & a. singh (eds.), critical collaborative communities: academic writing partnerships, groups, and retreats (vol. 17, pp. 92–105). brill. https://doi.org/10.1163/9789004410985_008 marquis, e. (2015). developing sotl through organized scholarship institutes. teaching & learning inquiry, 3(2), 19–36. https://doi.org/10.20343/teachlearninqu.3.2.19 marquis, e., carrasco-acosta, e., de bie, a., krishna prasad, s., wadhwani, s., & woolmer, c. (2022). toward redressing inequities through partnership: a critical assessment of an equity-focused partnership initiative. international journal for students as partners, 6(1), 10–29. https://doi.org/10.15173/ijsap.v6i1.4895 marquis, e., holmes, t., apostolou, k., centea, d., cockcroft, r., knorr, k., maclachlan, j. c., monteiro, s. d., & karamanis, t. (2017). sotl research fellows: collaborative pathfinding through uncertain terrain. the canadian journal for the scholarship of teaching and learning, 8(3), 1–19. https://doi.org/10.5206/cjsotl-rcacea.2017.3.9 mckinney, k. (2006). attitudinal and structural factors contributing to challenges in the work of the scholarship of teaching and learning. new directions for institutional research, 2006(129), 37-50. https://doi.org/10.1002/ir.170 mckinney, k. (2013). the scholarship of teaching and learning in and across the disciplines. indiana university press. mckinney, k., & jarvis, p. (2009). beyond lines on the cv: faculty applications of their scholarship of teaching and learning research. international journal for the scholarship of teaching and learning, 3(1). https://doi.org/10.20429/ijsotl.2009.030107 modalities of faculty engagement with the scholarship of teaching and learning 22 journal of effective teaching in higher education, vol. 6, no. 1 mcmaster university (2022). mcmaster university fact book 2021-2022. https://ira.mcmaster.ca/app/uploads/2022/11/fact-book-2022-final.pdf mercer-mapstone, l., dvorakova, s. l., matthews, k. e., abbot, s., cheng, b., felten, p., knorr, k., marquis, e., shammas, r., & swaim, k. (2017). a systematic literature review of students as partners in higher education. international journal for students as partners, 1(1). https://doi.org/10.15173/ijsap.v1i1.3119 miller-young, j., anderson, c., kiceniuk, d., mooney, j., riddell, j., schmidt hanbidge, a., ward, v., wideman, m. a., & chick, n. (2017). leading up in the scholarship of teaching and learning. the canadian journal for the scholarship of teaching and learning, 8(2), 1–17. https://doi.org/10.5206/cjsotlrcacea.2017.2.4 miller-young, j., yeo, m., & manarin, k. (2018). challenges to disciplinary knowing and identity: experiences of scholars in a sotl development program. international journal for the scholarship of teaching and learning, 12(1), 3. https://doi.org/10.20429/ijsotl.2018.120103 miller-young, j., yeo, m., manarin, k., carey, m., & zimmer, j. (2016). sotl2: inquiring into the impact of inquiry. new directions for teaching and learning, 2016(146), 55–62. https://doi.org/10.1002/tl.20187 newton, g., miller-young, j., & sanago, m. (2019). characterizing sotl across canada. the canadian journal for the scholarship of teaching and learning, 10(2). https://doi.org/10.5206/cjsotl-rcacea.2019.2.8174 openo, j. a., laverty, c., kolomitro, k., borin, p., goff, l., stranach, m., & gomaa, n. (2017). bridging the divide: leveraging the scholarship of teaching and learning for quality enhancement. the canadian journal for the scholarship of teaching and learning, 8(2), 1–18. https://doi.org/10.5206/cjsotlrcacea.2017.2.6 pallant, j. (2020). spss survival manual: a step by step guide to data analysis using ibm spss. mcgraw-hill education (uk). pechenkina, e. (2020). chasing impact: the tale of three sotl studies. teaching & learning inquiry, 8(1), 91–107. https://doi.org/10.20343/teachlearninqu.8.1.7 potter, m. k., & kustra, e. (2011). the relationship between scholarly teaching and sotl: models, distinctions, and clarifications. international journal for the scholarship of teaching and learning, 5(1). https://doi.org/10.20429/ijsotl.2011.050123 schroeder, c. (2007). countering sotl marginalization: a model for integrating sotl with institutional initiatives. international journal for the scholarship of teaching and learning, 1(1). https://doi.org/10.20429/ijsotl.2007.010115 modalities of faculty engagement with the scholarship of teaching and learning 23 journal of effective teaching in higher education, vol. 6, no. 1 simmons, n., abrahamson, e., deshler, j., kensington-miller, b., manarin, k., morón-garcía, s., oliver, c., & renc-roe, j. (2013). conflicts and configurations in a liminal space: sotl scholars’ identity development. teaching & learning inquiry, 1(2), 9–21. https://doi.org/10.20343/teachlearninqu.1.2.9 simmons, n., & taylor, k. l. (2019). leadership for the scholarship of teaching and learning: understanding bridges and gaps in practice. the canadian journal for the scholarship of teaching and learning, 10(1). https://doi.org/10.5206/cjsotl-rcacea.2019.1.7995 tierney, a. m. (2017). threshold concepts in academic practice: engagement with the scholarship of teaching and learning. practice and evidence of scholarship of teaching and learning in higher education, 12(2), 65–184. https://www.pestlhe.org/index.php/pestlhe/article/view/167 tierney, a. m., aidulis, d., park, j., & clark, k. (2020). supporting sotl development through communities of practice. teaching & learning inquiry, 8(2), 32–52. https://doi.org/10.20343/teachlearninqu.8.2.4 trigwell, k. (2013). evidence of the impact of scholarship of teaching and learning purposes. teaching & learning inquiry, 1(1), 95–105. https://doi.org/10.20343/teachlearninqu.1.1.95 vajoczki, s., watt, s., vine, m., & liao, r. (2011). inquiry learning: level, discipline, class size, what matters? international journal for the scholarship of teaching and learning, 5(1). https://doi.org/10.20429/ijsotl.2011.050110 vander kloet, m., frake-mistak, m., mcginn, m. k., caldecott, m., aspenlieder, e. d., beres, j. l., fukuzawa, s., cassidy, a., & gill, a. (2017). conditions for contingent instructors engaged in the scholarship of teaching and learning. the canadian journal for the scholarship of teaching and learning, 8(2), 1–20. https://doi.org/10.5206/cjsotl-rcacea.2017.2.9 vithal, r. (2018). growing a scholarship of teaching and learning institutionally. studies in higher education, 43(3), 468–483. https://doi.org/10.1080/03075079.2016.1180350 warfa, a-r. m. (2016). mixed-methods design in biology education research: approach and uses. cbe life sciences education, 15(4). https://doi.org/10.1187/cbe.16-01-0022 webb, a. s., & tierney, a. m. (2020). investigating support for scholarship of teaching and learning; we need sotl educational leaders. innovations in education and teaching international, 57(5), 613–624. https://doi.org/10.1080/14703297.2019.1635905 williams, a., verwoord, r., beery, t., dalton, h., mckinnon, j., strickland, k., pace, j., & poole, g. (2013). the power of social networks: a model for weaving the scholarship of teaching and learning into institutional culture. teaching & modalities of faculty engagement with the scholarship of teaching and learning 24 journal of effective teaching in higher education, vol. 6, no. 1 learning inquiry, 1(2), 49–62. https://doi.org/10.20343/teachlearninqu.1.2.49 wuetherick, b., & yu, s. (2016). the canadian teaching commons: the scholarship of teaching and learning in canadian higher education. new directions for teaching and learning, 2016(146), 23–30. https://doi.org/10.1002/tl.20183 wuetherick, b., yu, s., & greer, j. (2016). exploring the sotl landscape at the university of saskatchewan. new directions for teaching and learning, 2016(146), 63–70. https://doi.org/10.1002/tl.20188 microsoft word dorland_final.docx journal of effective teaching in higher education, vol. 5, no. 1 https://doi.org/10.36021/jethe.v5i1.115 that’s a good question: using design thinking to foster question formulation skill development annemarie dorland, mount royal university, adorland@mtroyal.ca abstract. design thinking is a critical and creative process understood to support innovation and creative idea generation in a wide variety of contexts. increasingly, it is being used as a pedagogical approach by educators interested in supporting divergent thinking skill development. this study evaluated the effects of the use of design thinking practices on student learning within the context of the first-year post-secondary classroom. analysis of student generated reflective statements and of in-class student work demonstrated that the use of design thinking practices may improve creative and critical thought and may enable students to achieve question focused and divergent thinking-focused learning outcomes more easily. the core of all student learning is in asking strong and resonant questions—this study demonstrates that the inclusion of design thinking practices in an undergraduate learning community may foster the skills required to do this critical work. keywords: design thinking; divergent thinking; question formulation; inquiry students are trying to learn how to ask strong and divergent questions every day in each of their classroom engagements, but researchers interested in the scholarship of teaching and learning know well that this is an elusive and difficult skill set to develop in the course of in-class learning (iverson, 2018). instead, undergraduate students across disciplines often rely solely on convergent and solution-oriented question types, a heuristic that may limit their creative and critical thinking skill development (goldschmidt, 2016). instead of “how might we…?”, they ask “how does it work?”. instead of “what would happen….?”, they ask “what have i seen happen before?”. the challenge for post-secondary instructors is to find a new model for fostering a robust question formulation practice: a new way of building a question-asking culture in our classrooms. in this paper, i suggest that a questionasking culture can be in a setting, namely the design studio, where a community of practitioners have developed the ability to generate new, creative, and critical thoughts in a reliable, repeatable manner within tight time and resource constraints using design thinking. here, i propose that the practices of designers—design thinking—may serve as an effective model for instructors seeking to foster divergent and strong question-asking skills as a learning outcome for their students. can design thinking help students learn how to ask stronger questions? to explore this question, i worked with students to examine the ways in which design thinking practices could be used to improve the question formulation skills of undergraduate students enrolled in an interdisciplinary first-year course. i compared the impact of four methods of question formulation used in many higher education classrooms (group discussion, brainstorming, mind mapping, and case study analysis) with four methods of design thinking-based learning (ethnographic futures, shadowing, user experience journey mapping, and informance). my aim that’s a good question: using design thinking 31 journal of effective teaching in higher education, vol. 5, no. 1 was to better understand how the practices of design thinking may impact the development of a question-asking skill set among undergraduate learners and to explore the differences in the student experience of the two learning approaches. studies of the use of design thinking in the undergraduate classroom (matthews & wrigley, 2017), business settings (kelly & kelly, 2013), and the innovation development process (beckman & barry, 2007) have also shown that the use of a “designerly way of knowing and doing” (cross, 2006) can have a strong positive impact on learning (henriksen et al., 2017). in their work on the impact of teaching design thinking in business schools, researchers from boise state university have demonstrated the effectiveness of the use of design thinking as a part of curriculum at the undergraduate level, suggesting that design thinking-based learning may present a powerful framework for dealing with “unstructured problems and for managing the innovation process” (glen et al., 2015). design thinking is becoming an ever more vital part of post-secondary education (charosky et al., 2018) from the classroom to the massive open online course setting (wrigley et al., 2018). my expectation that the use of design thinking practices may lead to the development of stronger or more divergent student questions is informed by design theorist lucy suchman’s (2011) analogy of “navigating” and “wayfinding” as two different approaches toward creating new ideas or new forms of knowledge. we anticipated that students would apply a wayfinding approach to developing questions during the design thinking-informed phase of their work—and that this approach would help further develop their strong or divergent question-asking skills. this study of student learning and the impact of design thinking practices on divergent thinking focused learning outcomes is rooted in the particular local context (huber & hutchings, 2005) of the interdisciplinary, first-year undergraduate seminar classroom, and not of the design studio. as such, it provides a new perspective into the use of a design thinking-based learning approaches in an undergraduate research class. many investigations of the use of design thinking practices examine the use of the five-stage process in corporate brand, product, and strategy development work (ideo, 2019). in this paper, i present findings from research conducted in partnership with student participants (earley, 2014; winn, 1995) that explores what happens when educators use design thinking to help in the first stages of creating new thoughts and questions. specifically, i ask this question: do design thinking practices help students learn to ask better questions? this microlevel examination (williams et al., 2013) of the use of design thinking in the development of “ideamaking” (resnick, 2017) presents evidence which suggests that design thinking-informed approaches, such as ethnographic futures; shadowing and participant observation; user experience journey mapping; and informance may help students develop the skills they need to ask strong and divergent questions during the research process. data collected from reflective statements generated after two phases of in-class work indicates that students can learn to ask stronger and more divergent questions by using design thinkingthat’s a good question: using design thinking 32 journal of effective teaching in higher education, vol. 5, no. 1 oriented learning strategies to support what resnick (2017) defines as a playground, rather than a playpen, classroom environment. overview design thinking the practice of design thinking, a general term used often to describe the work of connecting creativity and innovation within an integrative series of divergent and convergent social practices, has grown exponentially since its conception in the systems engineering world of the 1960s and popularization in the fields of business in the earlier part of this century (brown, 2008). increasingly, design thinking (framed as a practice, protocol, or mindset) is being applied as a problem-solving heuristic in fields as diverse as k-12 education (samberg, 2018), health care (roberts et al., 2016), and industry (wrigley et al., 2020). design thinking is most often understood as an iterative and cyclical series of divergent and convergent stages through which individuals and teams can effectively solve human centered problems (dorst & cross, 2001). as tschimmel and santos (2018) have suggested, effective use of design thinking processes and protocols relies on the designer’s ability to simultaneously consider human needs, the available resources, and the constraints and opportunities of a presented challenge. most commonly, this is achieved through team based and participatory protocol or process exercises grouped into five main categories: empathy, problem definition, ideation, prototyping, and testing (cross, 2011). design thinking is most effectively understood as a tested and established approach to metacognition: a process of designing how one thinks about human-centered challenges and needs and of deliberately engaging in the practice of mental ambidexterity aimed at shifting reasoning and sensemaking practices within a culture of changemakers (dorland, 2018). question asking and design thinking questions form the heart of the research process in any academic environment, but they are also the distinguishing feature of design thinking. however, the development of a robust question-asking skill set remains a challenge in teaching and learning. as rothstein et al. (2015) have identified, learning to ask the right question is critical for student learning in the undergraduate setting—so critical that it has been listed as the foundation of 6 of the 10 “future skills” in the world economic forum’s future of jobs report (2018) and as key components of the majority of high impact practices proposed by kuh (2008). minigan and beer’s (2017) work on the question formulation technique in higher education indicated that developing strong and divergent question forms remains an area of weakness in the undergraduate learning community. hassi and laasko (2011) have suggested that within a learning community where questions are often discouraged, individuals at any level of expertise may rely heavily on solution finding, rather than question asking, activities. the role of evaluation and feedback has also been discussed by the educational development team at ideo, who outlined the impact that’s a good question: using design thinking 33 journal of effective teaching in higher education, vol. 5, no. 1 that a reliance on summative evaluation practices can have on the development of question-asking skills (ideou, 2019). addressing the development of a question-asking skill set using design thinking the use of design thinking-based learning approaches in the undergraduate classroom may provide a new and effective way to support students in the development of their question-asking skills. firstly, design thinking-based learning approaches may be more conducive to question asking than other research processes. as helfand (2016) has illustrated in her work on design communities, the ability to generate divergent and convergent thought while diving ever deeper into the question-asking process is the critical skill of the designer—one which might be translated to the classroom through design thinking-based learning processes. in fact, members of design communities have reported that learning to ask questions about how a thing might be done or what possibilities arise from a unique situation is both the most critical and the most challenging part of their creative and cultural production work (dorland, 2018). the ability to develop strong questions and to remain “curious” (brown, 2008) for extended periods of time enables designers to generate more creative, innovative, and authentic solutions to critical social and cultural challenges (dorland, 2018). research conducted in undergraduate studio courses suggests that design thinking practices such as those employed in this study may be key to establishing the development of creative and critical thought in student communities as well (rashdan, 2017). secondly, the use of design thinking-based learning practices may support the use of play in the development of a meaningful question-asking practice. in his work on divergent thinking processes, resnick (2017) proposes that aspects of creative and critical thought—or the making, rather than the getting, of ideas—are premised on four key elements, each of which can be enhanced in the learning experience. projects (or the iterative development of an idea towards a goal) provide students with a required point of focus; passion (or the alignment of learning activities with learner interest) fuels the collaborative work of a classroom; and peers provide and support reflection and reflection-in practice (schön, 1983). most interestingly, he also suggests that what is most often left unconsidered in the higher education classroom space is play: the deliberate implementation of opportunities to explore ideas and directions of thought. resnick identifies two models that can support the critical and creative thinking fostered by play: the “playpen” and the “playground.” playpens, according to resnick, provide opportunity for limited and bounded creativity and can be supportive and generative while employing clear boundaries for playful thought. the playground, on the other hand, is an analogy for an experimental space where students are able to develop and nurture their creative and critical thinking practices (resnick, 2017). brown (2008) has also argued that design thinking practices, whether deployed in the studio space, the boardroom, or the classroom, are supportive of exactly this type of play-informed creative and experimental learning process. additionally, he suggests that design thinking practices may contribute towards a better understanding of psychologist joy paul guilford’s (1967) foundational work on that’s a good question: using design thinking 34 journal of effective teaching in higher education, vol. 5, no. 1 convergent thinking and divergent thinking. as guillford outlined, creative thought is reliant on both convergent and divergent thinking practices—without the balance between the two students would be unable to make use of a question-asking skill set in their innovation focused work. student’s convergent thinking practices may already be well supported: research suggest that it is incumbent on educators to foster skills in divergent processes of thought instead (briggs, 2014). the study purpose my motivation for undertaking this study of student learning was threefold. first, i am a design practitioner and researcher interested in the application of studiobased creative problem-solving strategies in the higher education classroom. secondly, a review of the literature outlining evidence about the impact of design thinking practices in generating innovative or critical thought surfaced little on the impact of these practices in understandings of student learning. finally, key learning objectives in the course that formed the grounding for this study included fostering and developing an understanding of the research process for students and enhancing the skills required for students to develop a research practice. as such, students were welcomed as collaborators on the data collection and analysis phases of this study whenever possible. this study assesses student experiences of design thinking-focused learning strategies by analyzing their reflections on how design thinking practices might bolster or strengthen the questioning skills that are so critical to the undergraduate learning experience. this work is guided by the following research questions: 1. do participating students ask questions in a different way after engaging with design thinking practice during an interdisciplinary undergraduate course? 2. how is the student’s learning experience affected by the use of design thinking-based learning approaches? this study contributes to addressing a gap in the literature on both student learning and creativity, especially with respect to new options for using design thinking as a tool for research practice and critical thinking at the undergraduate level. though this study examined the use of design thinking in a group learning and undergraduate context, the evidence generated and analyzed in partnership with student learners presents potential new areas of exploration that can be taken up by scholarship of teaching and learning (sotl) researchers interested in individual research practice skill development and the development of creative and critical thought in experiential learning environments. methods in asking how the use of design thinking-based learning approaches affected the student’s question formulation processes, i chose an approach that allowed for the inclusion of student participants into all aspects of the research process (felten et that’s a good question: using design thinking 35 journal of effective teaching in higher education, vol. 5, no. 1 al., 2013; hutchings, 2000), including data generation and analysis. using a collaborative approach that incorporated student learners as research partners allowed us all to focus on considerations of learning as a lived practice engaged within a community that included both instructor and student members (lave & wenger, 1991). this methodological approach in both data collection and analysis was aimed at generating “thick” descriptions based on “being there” (borneman & hammoudi, 2009), a goal that would have been impossible to realize without the collaborative input of student participants. participants for the purposes of this study, i collected data in from two sections of a firstyear, interdisciplinary, and inquiry-based course at a large research-focused canadian university. the sample was made of two relatively small classes (with fewer than 30 enrolled students per section), and 95% of students in each section consented to participate and to include their work as part of the data set (n = 37). students who did not chose to consent to participate in the project (n = 2) were able to expunge their class work from the final data set (including their reflective statements and question samples) and to debrief their experience individually with the course instructor. the classes met twice a week for a 90-minute course focused on developing a personal research practice. the course context that forms the basis for this study was open to any student with less than one year of course credit at the institution (regardless of major, gpa, or faculty designation). students came from eight disciplines of study and were enrolled in the faculties of arts, engineering, science, education, social work, and kinesiology. all participants in this study self-identified as first-year undergraduate students aged 17–23, and the course that provided the context for this study was taken during their first term of study at the university. this is especially important because this meant that student participants were engaging with some of these learning outcomes and learning practices for the first time and were participating in research on their own learning for the first time as well. the instructor for this course taught both sections included in this study and brought a background in qualitative research and design practice to their work as a facilitator and guide for student learning. procedure and data collection data was collected during 12 observation opportunities for each section of the course (with a total of 24 observation opportunities). the data generated by both course sections was combined for the purposes of analysis, and the amalgamated data set is presented and discussed below. student participants in this study were recruited by a research assistant with no connection to class work or course evaluation during the first day of class. they were given the opportunity to review the study protocol, the conjoint faculties research ethics board (cfreb) ethics approval statement, and the study’s alignment with the learning outcomes of the class and discuss the details of participation with the research assistant. informed consent for participation was granted by students after this initial briefing session that’s a good question: using design thinking 36 journal of effective teaching in higher education, vol. 5, no. 1 with the research assistant and re-confirmed at the end of the study prior to the final debriefing session. the process of doing this project in the classroom is outlined in figure 1 below. figure 1 study timeline and in class exercise protocol in this class, students were working on developing a research program in support of innovation. they were tasked with working in groups to research a challenge area (for example, new forms of climate friendly farming) and to use their research to generate an idea for an innovation that would address the challenge area in a meaningful way. the final assignment for this class was a research poster, a written essay, and a public innovation presentation. i began this study by collaborating with students on a baseline definition of a “strong or divergent question.” during the first week of class, students participated in a facilitated exercise where they identified the characteristics of “strong or divergent” and “weak or convergent” questions (as outlined in table 1, below). this guiding taxonomy of strong and weak question types was then made available to that’s a good question: using design thinking 37 journal of effective teaching in higher education, vol. 5, no. 1 students for their use on the course management platform (and was referred to during course work throughout the rest of the term). after this facilitated exercise, students were prompted in their course management platform to write a brief reflective statement (>200 words) about how asking strong and divergent questions may be of benefit in their learning practice. table 1 student generated taxonomy of weak/convergent and strong/divergent question types question type weak/convergent questions strong/divergent questions row 1 easily solved. are difficult to solve. row 2 are answered based on my prior knowledge. require new experiences for me to answer them. row 3 have answers that don’t generate more questions. breed new questions for me to think about. row 4 don’t make me think. make me think about new things. row 5 aren’t very interesting to others. are interesting to others. row 6 lump things together. blow things apart. phase one: playpen learning using traditional question formulation methods students then spent the next four class sessions (two weeks of course time) engaging in facilitated group discussions aimed at generating question statements that they could use in their research on the common course project. this portion of the study was defined as the “playpen” phase. these facilitated discussions employed the following traditional (non-design thinking oriented) strategies of question generation: 1. group discussion 2. brainstorming/mind mapping 3. research using secondary sources of peer reviewed scholarship 4. discussion based case study analysis in each of the sessions, students were grouped in small teams of five team members or less and were tasked with generating questions that they felt would guide their research process moving forward. questions were tracked by writing them on sticky tabs coded numerically for future categorization and all generated questions were collected at the end of the session, transcribed to a shared document posted on the course management platform, and kept for debriefing at that’s a good question: using design thinking 38 journal of effective teaching in higher education, vol. 5, no. 1 the end of the project. at the end of these four sessions, students were prompted in their course management platform to write a brief reflective statement (>200 words) on their experience of generating guiding questions for researching the common class project using the traditional (non-design thinking) strategies deployed in this phase of the course work. phase two: playground learning using design thinking question formulation methods students then spent time in the next four classes working with design thinking methods to generate question statements. this phase of the work required additional work on behalf of the course instructor and the students as many of the design thinking methods engaged in this phase required adapting to new skills and approaches as part of the research process. this portion of the study was defined as the “playground” phase. questions were tracked in the same manner as phase one (with sticky tabs coded using numeric indicators) and transcribed for use during debriefing in a common document. this second set of four facilitated sessions employed the following strategies to help students generate guiding questions for the common class project: 1. ethnographic future studies (wherein students used samples of media representations of the future to conduct desk-based hypothetical ethnographies of future states and to generate new questions about what might be possible in that future). 2. shadowing (whereby students embedded within a social setting of their choice to develop an understanding of a different context and to generate questions from the point of view of a different person with a different perspective than their own). 3. user experience journey mapping (wherein students mapped the experience of a subject involved in a possible solution to the class project to generate questions that the subject might pose at any given intersection on their journey). 4. informance (whereby students studied a social practice and then used performance to share that social practice with team members who asked new kinds of questions about the performance itself). at the end of these four sessions, students were again prompted in their course management platform to write a brief reflective statement (>200 words) on their experience of generating guiding questions for researching the common class project using the strategies deployed in this phase of the work. in order to debrief the study findings and to engage students in further collaboration on this study, student participants did a sorting exercise of all questions generated in phase one (playpen) and phase two (playground) classes, using a matrixed version of the original taxonomy of strong/divergent and weak/convergent question types generated at the beginning of the study. students that’s a good question: using design thinking 39 journal of effective teaching in higher education, vol. 5, no. 1 worked with their peers to place the question sticky tabs on the large wall sized matrix in a way that would indicate their placement on the continuum of strong/divergent to weak/convergent question forms using the particulars of each type identified in the taxonomy. students were not able to see whether the questions had been generated during the first or second phase of the project during this sorting exercise, and the questions used were not sorted into teams or groups (and were rather compiled as one common pool or selection). a class discussion followed which provided an opportunity to contextualize study findings, member check the conclusions drawn by the principal investigator, and identify areas of future research or study limitations in collaboration with the student participants. finally, students met separately with a research assistant who conducted a group discussion focused on the research questions identified at the beginning of the study, and the student’s self-evaluation of their question formulation skill development. at this informal gathering, students were also invited to share their experience with the different learning practices employed during both phase one and phase two of the study. the research assistant also shared results of their sorting exercise with students at this time. data analysis after the collaborative data collection work was finalized, i used qualitative content analysis to derive coding categories directly from the collected and transcribed data (braun & clarke, 2006; hsieh & shannon, 2005). to do so, we—the principal investigator and a research assistant—read the data broadly as a whole to develop a contextual understanding of the student perspective. i then employed thematic content analysis to examine the data for commonalities in order to better understand the student experience of phase one and two of the study and to generate coding categories. initial coding categories developed from this close reading were then shared on the course management platform with student participants as a form of member checking. one coding category was adjusted for clarity based on the feedback shared by students. all other coding categories were found to be well aligned with the responding student’s experience of their learning in class. data collected through the reflective statements and the student-generated question strength continuum was then analyzed by the principal investigator using qualitative data analysis software. to establish intercoder reliability, a blind sample of data was then recoded by a research assistant. the intercoder reliability was 93%, providing additional validation to the coding scheme. findings this study explores student experiences of design thinking-focused learning strategies and their reflections on how the design thinking practices originating in the creative studio might bolster or strengthen their questioning skills. the following research questions formed the foundation for this research: that’s a good question: using design thinking 40 journal of effective teaching in higher education, vol. 5, no. 1 1. do participating students ask questions in a different way after engaging with design thinking practice during an interdisciplinary undergraduate course? 2. how is the student’s learning experience affected by the use of design thinking-based learning approaches? coding of the reflective statements generated after in class work during both the playpen (first) and playground (second) phases of the study (n = 74) generated the themes listed below. there was no indicated difference between common and uncommon themes in the analysis process. themes included 1. producing unexpected or new directions for further research 2. generating possible solutions to the larger class project 3. contextualizing or exploring the larger class project 4. decoding or evaluating thinking processes 5. testing possible solutions to the larger class project 6. uncovering assumptions or gaps in knowledge 7. building on pre-existing knowledge basis 8. drawing from the lived experiences of others 9. identifying unknown particulars of the larger class project phase one: playpen learning using traditional question formulation methods the number of reflective statements generated using “playpen” learning strategies (group discussion, brainstorming or mind mapping, secondary peer reviewed research, and case studies) that included content related to each theme is outlined below in table 2. student-generated reflective statements often corresponded to more than one thematic code, though that was not always the case. in this phase of the study, most students reflected on the facets of the larger class project that were unfamiliar or unknown. eighteen student participants discussed the complexities of identifying unknown particulars of the larger project as the key contribution of their question-asking process. one student participant commented on the theme of identifying unknown particulars of the larger class project: doing questions through the group work helped me understand more about the bigger challenge and now i know what i need to research more about. i didn’t understand the bits involved and now it is coming together. (student 6) other students (68%) focused their reflections on the way that asking questions through group discussion and brainstorming (or mind mapping) helped them build on their preexisting knowledge basis. in particular, students reflected on the that’s a good question: using design thinking 41 journal of effective teaching in higher education, vol. 5, no. 1 commonalities of their experience and knowledge and the complexities of the class project: i asked questions about how this is the same as some of the things we do because the things we do aren’t all that different than what’s happening in the rest of the world. everyone is the same and if we just ask questions about things we can understand then we can find what we have in common and use that to create a good solution. (student 14) a common theme in student reflections generated after the four classes in which they developed questions through group discussion, brainstorming or mind mapping, the use of secondary peer reviewed research, and case studies was the overwhelming nature of questions themselves and the tendency of questions to prompt unexpected or new directions for further research. one of the 23 participants who reflected on this theme noted the following: you ask a good question but you can’t just use that. a question just makes everyone else in the group go off in a new direction and you can’t bring it back to what you are supposed to be thinking about. having a group for this part was hard and having to jump in with more questions everyday just confused us (student 19) table 2 thematic analysis of phase one reflective statement responses variable thematic categories n % row 1 drawing from the lived experiences of others 1 0.03% row 2 decoding or evaluating thinking processes 2 0.5% row 3 uncovering assumptions or gaps in knowledge 4 1.0% row 4 producing unexpected or new directions for further research 5 13.5% row 5 testing possible solutions to the larger class project 13 35.1% row 6 contextualizing or exploring the larger class project 17 45.9% row 7 identifying unknown particulars of the larger class project 23 62.1% row 8 building on pre-existing knowledge basis 25 67.5% row 9 generating possible solutions to the larger class project 29 78.3% phase two: playground learning using design thinking question formulation methods the number of reflective statements using the design thinking-based learning strategies (ethnographic future studies, shadowing and participant observation, user experience journey mapping, and informance) that included content related to that’s a good question: using design thinking 42 journal of effective teaching in higher education, vol. 5, no. 1 each theme is outlined below in table 3. again, student-generated reflective statements often corresponded to more than one thematic code, though that was not always the case. two student-generated reflective statements did not correspond to any of the identified themes. table 3 thematic analysis of phase two reflective statement responses variable theme n % row 1 generating possible solutions to the larger class project 7 18.9% row 2 decoding or evaluating thinking processes 11 29.7% row 3 building on pre-existing knowledge basis 14 37.8% row 4 contextualizing or exploring the larger class project 16 43.2% row 5 testing possible solutions to the larger class project 19 51.3% row 6 producing unexpected or new directions for further research 23 62.1% row 7 uncovering assumptions or gaps in knowledge 24 64.8% row 8 identifying unknown particulars of the larger class project 27 72.9% row 9 drawing from the lived experiences of others 30 81.0% in this second phase of the study, far more reflective statements focused on the importance of drawing on the lived experience of others as key in their question generating process (77%). in these statements, students reflected on the role of moving outside of their own knowledge base and their own context to learn more about the complexities of the larger class project from the point of view of others, and 21 students mentioned “getting outside your own bubble” or “leaving your comfort zone” as part of developing a strong or divergent question. i wasn’t thinking of what questions to ask about how a solution might work before doing this part of the class. i was just thinking about what might work, but i can see more about impact now than i did then. (student 3) there are lots of sides to the problem but we already knew more about that. i didn’t really think of how many people or what kind of people were involved now and i’m asking more interesting questions now that i do. (student 18) additionally, students reflected on the role of their questions in testing possible solutions and in evaluating their own thinking process. fully half of all submitted statements included reflections on the role of using the questions generated at the beginning of the process to better understand the validity of the solution that they would propose at the end. one participant noted the following: that’s a good question: using design thinking 43 journal of effective teaching in higher education, vol. 5, no. 1 without the good questions that came from the interview i wouldn’t have a way to know if my proposal would work or not. now i have something to navigate with. (student 21) debriefing and question sorting exercises some of the most interesting outcomes from this study came from the debriefing exercise that students completed after phases one and two of the collaborative data collection work. as mentioned above, students helped define the collective understanding of the role of design thinking practices in the question formulation process by generating a taxonomy of strong and weak question types (table 1) at the beginning of the study. all enrolled students (n = 39) took part in the generation of this taxonomy during their class time. this taxonomy was used in the debriefing exercise conducted after the collection and analysis of the phase one and phase two data. after a group discussion about the merits of convergent and divergent thinking practices students worked collaboratively to plot all of the questions generated during their in-class work onto a large 2 x 2 matrix, which is shared below. questions generated during phase one of the study (using learning strategies such as group discussion, case studies, secondary peer reviewed research, and brainstorming or mind mapping) are indicated in figure 1 below using a red dot. questions generated using phase two of the study (using design thinking-focused learning strategies such as ethnographic future studies, shadowing and participant observation, user experience journey mapping, and informance) are indicated in figure 1 using a blue dot. students did not know which questions were generated during phase one and phase two of the study and placed questions onto the matrix based on their current interpretations of the question itself. this matrix indicates that students were more likely to categorize questions generated during the “playpen” phase of the study as weak and convergent in nature (58% of total questions generated during phase one) and, conversely, more likely to categorize questions generated during the “playground” phase of the study as strong and divergent in nature (52% of total questions generated in phase two). students debated the placement of each question in the matrix, with many landing in (as one participant described) the “in between zones” of divergent/weak questions (23%) and convergent/strong questions (18%). fully 85% of the questions generated in the “playground” (with the learning strategies generated using design thinking) were categorized as strong, with either a divergent or a convergent focus. as part of the debriefing process, students worked as a group to generate a representative question from each quadrant of the two by two matrix, as indicated in figure 2 below. one participant shared in the final reflective statement for the class that she completely revised her approach to both generating and using questions in her research work as a result of this sorting exercise: that’s a good question: using design thinking 44 journal of effective teaching in higher education, vol. 5, no. 1 i never thought about how i could make the question do the work when it came to setting my direction. i always used the questions that were clearest, but i think the ones i liked best and that i identified with were more the ones that led you in a million directions and made you think a little harder. (student 12) figure 2 student generated matrix of question types c on ve rg en t t hi nk in g d iv er ge nt t hi nk in g strong question weak question that’s a good question: using design thinking 45 journal of effective teaching in higher education, vol. 5, no. 1 figure 3 student generated matrix of question types with representative questions discussion this study examined whether students could learn to ask stronger and more divergent questions by using design thinking-oriented learning strategies to support what resnick (2017) defines as a playground, rather than a playpen, classroom environment. i analyzed the differences and commonalities between reflective statements generated by student participants after engagement with what i defined as “playpen” learning strategies (including group discussion, case studies, secondary peer reviewed research, and brainstorming or mind mapping) and “playground” or design thinking-oriented learning strategies (including ethnographic future studies, shadowing and participant observation, user experience journey mapping, and informance). i asked how students defined strong/divergent versus weak/convergent question forms, how they experienced generating questions using different types of learning strategies, and how they understood their own work in relationship to divergent and convergent thinking paradigms after having reflected that’s a good question: using design thinking 46 journal of effective teaching in higher education, vol. 5, no. 1 upon different learning practices. the intent of this research was to better understand whether a “playground,” or design thinking-oriented environment truly supported the development of creative and critical thinking skills in first year undergraduate learners. the key findings about the impact of design thinking on this particular aspect of student learning can be grouped into two larger categories which align with the literature in the field of innovation and creativity (most notably the work of guilford, 1967): convergent processes and divergent processes. first, the evidence suggests that the use of design thinking as a learning strategy supports and nurtures the development of divergent thinking approaches with respect to the activities of cognition, production, and evaluation. secondly, the evidence suggests that the use of playpen (or limited creativity) learning strategies can still support the development of strong/divergent question forms but also formulates more convergent thinking practices among undergraduate students—increasing the solution determinacy of student efforts and decreasing the function of play as a form of exploration in the making of new ideas. the majority of participating students reflected at least once (68%) upon the positive difference in their individual learning approaches after engaging with design thinking practices, which exemplifies the positive role such learning activities may play in the classroom. this suggests that after engaging with design thinking practices in their learning strategies, students might be asking questions in a different way—one that is perhaps more aligned with the high impact practices identified in the literature on student learning and creative practice (kuh, 2008). one third of student reflections about their improved learning approaches also made reference to aspects of what dweck (2008) has defined as a growth mindset: specifically, that they were developing knowledge over time and that they were learning more from failed question forms than from successful ones. i interpret this to mean that design thinking practices, when engaged as learning strategies, may have a positive effect on the development of a growth mindset in students. what becomes evident in the examination of the matrix of question types and of the sample representative questions generated by students in the debriefing session is that students find divergent and strong/divergent questions to be especially generative for creative and critical thought—and that the use of design thinking learning strategies enabled the creation of those question forms. as resnick (2017) identified, the critical and creative thinking generated through playground form play (in contrast to the limited opportunities for creativity found in playpen style play) supported more resonant and deeper expressions of student learning. the local context for this study was well grounded in the creative learning foundations identified by resnick (2017): project, passion, and peer supports were strong/divergent and aligned with the group and inquiry-based learning approaches to the work students completed as a learning community. with that in mind, this findings from this study suggest that the integration of design thinking practices may be a critical variable in the development of strong and divergent question forms: using design thinking in the class appears to increase the playground, and decrease the playpen, aspects of play in creative learning. that’s a good question: using design thinking 47 journal of effective teaching in higher education, vol. 5, no. 1 how effective did students think design thinking methods were as a method of improving their question-asking abilities? in final reflective statement data, nearly two-thirds of the students reported that their strongest and most divergent questions came from the design thinking process, and many indicated the applicability of this process to other course work. my interpretation of this finding is that the learning strategies introduced through design thinking practices are transferable, useable, and well aligned with all forms of student inquiry, not just creative practice. seventy-five percent of students reflected on the playful nature of the design thinking practices they encountered during the second phase of the study and on the freedom that these processes engendered. additionally, 59% shared that their experience of the design thinking (or playground) practices used as learning strategies allowed them to take on additional points of view or diverse perspectives in order to generate new ideas or approaches to a problem. these two findings indicate to us that incorporating design thinking practices as learning strategies in the undergraduate classroom is both a positive student experience and a potential high impact practice in the development of citizenship and global perspective development. perhaps the most interesting aspect of the data that emerged after thematic analysis was the differentiation students made between the uses of questions in their future work. design theorist suchman (2011) refers to the use of “navigating” and “wayfinding” as helpful analogies to understand the differences identified in the students’ reflections, and they are useful when examining the data presented here as well. in this study, students shared that a “navigating” approach to creating new ideas through question development required them to presuppose a solution and to plan a series of inquiries that will help them arrive at the designated result. on the other hand, students using a “wayfinding” approach to developing new thoughts or making new ideas instead relied on an iterative process of trial and error—on exploring the territory of the ideas in development rather than developing ideas in service of a solution. in this study, i saw students applying a wayfinding approach to developing questions during the “playground” phase of their work—an approach that, if nurtured, may be of benefit to students exploring the development of new ideas in more traditional learning environments. limitations this study of whether the use of design thinking methods can help students ask better questions was limited in several ways. my hope is that with consideration of these limitations in mind instructors and faculty members may be able to find new ways to further examine this function of student learning in higher education. first, student participants in this study came into their role with a preconceived awareness of the value of design thinking and the value of divergent thinking practices. phrases like “thinking outside the box” and “thinking differently” formed a second level theme during the coding of the first round of reflective statements, indicating that students were prepared to value divergent practices more highly than convergent thinking methods. additionally, the students’ enthusiasm for the that’s a good question: using design thinking 48 journal of effective teaching in higher education, vol. 5, no. 1 learning strategies involved in phase two was markedly higher, which may have resulted in students placing a higher value on the questions formed using these tools. this is difficult to mitigate, as the active and experiential strategies associated with design thinking are, in this author’s opinion, just more fun. with that in mind, it may be worthwhile to experiment with different pairings of learning strategies in future in order to work around this limitation in the data collected. secondly, due to the structure of the study, students may have been improving the strength of their questions in ways that could not be explained by the use of the design thinking practices. for example, students had already been familiarizing themselves with the larger class project for several weeks before tackling the second phase of learning strategies. this is also made clear through the overlap of coding themes in the second phase of the research work: there were markedly more reflective statements that coded in more than one thematic category in the second phase of the research work. these themes have not been combined in this analysis in order to preserve this distinction in the data. additionally, students had developed a familiarity with the class dynamic, with their section peers, and with their role in the larger university community before beginning phase two of the study. the steep learning curve present in all first-year student experiences may have been a factor in the type of work generated by students for analysis. instructors interested in considering the role of design thinking within the development of critical thinking and reflective practice in first-year programs may consider reordering the introduction of learning strategies or running two different classes in parallel in order to assuage this effect. finally, many student participants indicated in their reflective statements and during the debriefing sessions that they were not comfortable with group work and group discussion. this is a critical issue for many undergraduate students and a notable one for first-year students in general. this discomfort with group work may have caused some contributions to go unheard during class time or even have caused some students to silence themselves during the creation of questions in phase one and phase two of the study. in the future, this research could be replicated in a grade-free context in order to better understand whether it is the group work or the assessment structure that has caused this discomfort. it is important to note that though students reflected on their discomfort with group work, none of the participants indicated that this discomfort extended to nonparticipation in the class learning. conclusion this research demonstrates that design thinking practices, when integrated into a playful, peer directed, project-oriented, and passion-fueled learning engagement, may improve creative and critical thought and may enable students to achieve question-focused and divergent thinking-focused learning outcomes more easily. the core of all student learning is in asking strong, divergent, and resonant questions—this study demonstrates that the inclusion of design thinking practices in an undergraduate learning community may foster the skills required to do this critical work. that’s a good question: using design thinking 49 journal of effective teaching in higher education, vol. 5, no. 1 this research helps outline why and how design thinking can be an effective learning strategy and how it can be reimagined as a formative research methodology to be used in any learning engagement. the data generated in collaboration with students as part of this study indicates that the skills developed through the use of design thinking practices can be transferred to other learning engagements and that students find this form of learning engaging and well aligned with a growth mentality. by demonstrating the ways in which design thinking practices can support divergent thinking, creative practice, critical thought, and student learning this study makes it clear that design thinking is not just for designers anymore. conflict of interest the author declares that there is no conflict of interest regarding the publication of this article. references beckman, s. l., & barry, m. (2007). innovation as a learning process: embedded design thinking. california management review, 50, 25–56. borneman, j., & hammoudi, a. (2009). the fieldwork encounter, experience, and the making of truth: an introduction. in j. borneman & a. hammoudi (eds.), being there: the fieldwork encounter and the making of truth (pp. 1–24). university of california press. braun, v., & clarke, v. (2006). using thematic analysis in psychology. qualitative research in psychology, 3(2), 77–101. briggs, s. (2014). 30 ways to inspire divergent thinking. [blog posting] https://www.opencolleges.edu.au/informed/features/divergent-thinking/ brown, t. (2008, june). design thinking. harvard business review, 84–92. charosky, g., hassi, l., leveratto, l., papageorgiou, k., ramos, j., & bragos, r. (2018, october). education for innovation: engineering, management and design multidisciplinary teams of students tackling complex societal problems through design thinking. in 4th international conference on higher education advances (head'18) (pp. 1081–1087). editorial universitat politècnica de valència. cross, n. (2006). designerly ways of knowing. springer. cross, n. (2011). design thinking: understanding how designers think and work. berg. that’s a good question: using design thinking 50 journal of effective teaching in higher education, vol. 5, no. 1 dorland, am. (2018). doing design thinking: an ethnography of the digital graphic design studio [doctoral dissertation, university of calgary]. the vault: electronic theses and dissertations. http://dx.doi.org/10.11575/prism/31949 dorst, k., & cross, n. (2001). creativity in the design process: co-evolution of problem-solution. design studies, 22, 425–437. dweck, c. (2008). mindset: the new psychology of success. random house. earley, m. a. (2014). a synthesis of the literature on research methods education. teaching in higher education, 19(3), 242–253. felten, p., bagg, j., bumbry, m., hill, j., hornsby, k., pratt, m., & weller, s. (2013). a call for expanding inclusive student engagement in sotl. teaching & learning inquiry: the issotl journal, 1(2). 63–74. glen, r., suciu, c., baughn, c. c., & anson, r. (2015). teaching design thinking in business schools. the international journal of management education, 13(2), 182–192. goldschmidt, g. (2016). linkographic evidence for concurrent divergent and convergent thinking in creative design. creativity research journal, 28(2), 115–22. guilford, j. p. (1967). the nature of human intelligence. mcgraw-hill. hassi, l., & laasko, m. (2011). conceptions of design thinking in the management discourse. proceedings of the 9th european academy of design conference. helfand, j. (2016). design: the invention of desire. yale university press. henriksen, d., richardson, c., & mehta, r. (2017). design thinking: a creative approach to educational problems of practice. thinking skills and creativity, 26, 140–153. hsieh, h., & shannon, s. e. (2005). three approaches to qualitative content analysis. qualitative health research, 15(9), 1277–1288. huber, m. t., & hutchings, p. (2005). the advancement of learning: building the teaching commons. jossey-bass. hutchings, p. (2000). opening lines: approaches to the scholarship of teaching and learning. carnegie foundation for the advancement of teaching. ideou. (2019). ideou. solve anything creatively. [web page]. retrieved from https://www.ideou.com/ that’s a good question: using design thinking 51 journal of effective teaching in higher education, vol. 5, no. 1 iverson, b. (2018). the importance of learning to ask the right questions. https://news.utexas.edu/2018/03/05/the-importance-of-learning-to-ask-theright-questions/ kelly, t., & kelly, d. (2013). creative confidence: unleashing the creative potential within us all. crown publishing. kuh, g. (2008). high-impact educational practices: what they are, who has access to them, and why they matter. association of american colleges and universities. lave, j., & wenger, e. (1991). situated learning: legitimate peripheral participation. cambridge university press. matthews, j., & wrigley, c. (2017). design and design thinking in business and management higher education. journal of learning design, 10, 41–54. minigan, a., & beer, j. (2017). inquiring minds: using the question formulation technique to activate student curiosity. the new england journal of history, 74(1). 114–136. rashdan, w. (2017). impact of design thinking on conceptual design in interior design studio courses. people: international journal of social sciences, 3, 85– 99. resnick, m. (2017). lifelong kindergarten. cultivating creativity through projects, passion, peers, and play. mit press. roberts, j. p., fisher, t. r., trowbridge, m. j., & bent, c. (2016). a design thinking framework for healthcare management and innovation. healthcare (amsterdam, netherlands), 4(1), 11–14. https://doi.org/10.1016/j.hjdsi.2015.12.002 rothstein, d., santana, l., & minigan, a. (2015). making questions flow. educational leadership, 73(1). 70–75. samberg, m. j. (2018). problem solving in the digital age: bringing design and computational thinking to the k-12 classroom [doctoral dissertation, east carolina university]. the scholarship: east carolina university’s institutional repository. https://thescholarship.ecu.edu/handle/10342/6714 schön, d. (1983). the reflective practitioner: how professionals think in action. basic books. suchman, l. (2011). anthropological relocations and the limits of design. annual review of anthropology, 40(1), 1–18. that’s a good question: using design thinking 52 journal of effective teaching in higher education, vol. 5, no. 1 tschimmel, k., & santos, j. (2018). how designers can contribute to education. advances in intelligent systems and computing. williams, a., verwoord, r., beery, t., dalton, h., mckinnon, j., strickland, k., pace., j., & poole, g. (2013). the power of social networks: a model for weaving the scholarship of teaching and learning into institutional culture. teaching & learning inquiry: the issotl journal, 1(2). 49–62. winn, s. (1995). learning by doing: teaching research methods through student participation in a commissioned research project. studies in higher education, 20(2), 203–214. wrigley, c., mosely, g., & tomitsch, m. (2018). design thinking education: a comparison of massive open online courses. she ji: the journal of design, economics, and innovation, 4(3), 275–292. wrigley, c., nusem, e., & straker, k. (2020). implementing design thinking: understanding organizational conditions. california management review, 62(2), 125–143. world economic forum. (2018). the future of jobs report. centre for world economy and society, world economic forum. microsoft word love_final.docx journal of effective teaching in higher education, vol. 5, no. 2 https://doi.org/10.36021/jethe.v5i2.324 video feedback and instructor social presence in an asynchronous online course savanna love, randolph-macon college, savannalove@rmc.edu david marshall, auburn university abstract: understanding best practices in online learning environments has become an important area of research in recent years, both before and during the pandemic. video feedback has been studied as a way to enhance instructor social presence and create classroom community, though more research is needed to understand these constructs in various contexts. the current study sought to understand the extent to which video feedback enhanced instructor social presence in an asynchronous online course. participants included five sections of an advanced educational psychology course for pre-service teachers. a qualitative, descriptive design using both surveys (n = 63) and interviews (n = 10) was employed to explore student perceptions of video feedback. findings demonstrate that video feedback enhanced instructor social presence and that weekly videos improved students’ overall experience in the course. the video-based feedback led students to develop a better relationship with the course instructor and improved their overall experience in the course. implications for student outcomes and instructor planning in online courses are discussed. keywords: video feedback; writing feedback; asynchronous; graduate students; instructor social presence an increased interest in online learning environments has led to discussions about the effectiveness of various forms of feedback students receive throughout an online course. while traditionally, feedback has come in the form of written comments, scholars have begun to focus on alternative methods that provide more opportunities for personalized, high-quality feedback messages. specifically, research has investigated the effectiveness of video feedback in establishing positive learning environments and connections between students and instructors (anson et al., 2016; borup et al., 2014). recent literature has demonstrated the ways in which asynchronous, one-to-one video feedback is advantageous for building student-teacher relationships, with students claiming that watching video feedback felt personal (darby & lang, 2019; marshall et al., 2020) and made them feel like the instructor knew them (parton et al., 2010), cared about them, (henderson & phillips, 2015), and valued them (harper et al., 2012). students have also reported that video feedback made asynchronous instructors feel more real, which motivated them to complete assignments (borup et al., 2014). instructors reported that it was easier to give encouragement and communicate authentically with their students using video feedback, compared to text-based comments (harper et al., 2012). though studies exploring student perceptions of video feedback have largely yielded positive findings, students have also reported potential drawbacks, including video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 44 feeling anxious to watch their video feedback and finding it difficult to contextualize their video feedback comments within their written projects (henderson & phillips, 2015). additionally, some students were found to be more likely to respond to text feedback due to convenience (borup et al., 2014) while others reported that they preferred video to text feedback but found it time consuming to download (mccarthy, 2015). understanding the influence asynchronous video feedback has on student perceptions of student-teacher relationships may help instructors facilitate online and blended learning environments that support a sense of classroom community and encourage student engagement and cognitive presence (collins et al., 2019). the current study sought to understand student perceptions of video feedback in a graduate online asynchronous educational psychology course offered to pre-service teachers. our goal was to answer the following research question: to what extent did the video feedback enhance instructor social presence in an asynchronous course? literature review instructor social presence social presence has been considered in a variety of contexts, though scholars have recently been interested in how social presence impacts online spaces. according to gunawardena (1995), social presence can be considered in part a product of behavior and therefore could “be cultivated” by participants (p.162). this idea became widely accepted and has been the foundation for much of the recent research in this area. garrison et al. (1999) built on gunawardena’s ideas when they created the community of inquiry (coi) framework to examine text-based learning interactions in an online learning environment. according to garrison et al.’s (1999) coi framework, the online educational experience consists of three presences: social presence, cognitive presence, and teaching presence (oyarzun et al., 2018). this framework posits that these three presences overlap to create an effective learning experience and specifically discusses the role of social presence, defined as the learner’s ability to present himself or herself as a “real person” (garrison et al., 1999, p. 89). this can be a prerequisite to cognitive presence, or the extent to which students are able to construct knowledge from their interactions with others in the online environment (borup et al., 2014). teaching presence is defined as the design, facilitation of discourse, and direct instruction of the cognitive and social processes of learning to achieve higher order thinking (anderson et al., 2001), and the intersection of teaching and social presence is known as instructor presence or instructor social presence (lowenthal, 2015). garrison and colleagues (1999) go on to explain that social presence’s dimensions include affective expression, open communication, and group cohesion (garrison et al., 1999; rourke et al., 2001). although social presence largely focuses on student social presence within the coi framework, researchers have acknowledged that a teacher’s responsibility to facilitate discourse overlaps with the behaviors identified in the coi’s framework for video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 45 social presence (anderson et al., 2001). specifically, instructor social presence has been defined as instructors establishing their presence in terms of frequency of communication and interaction with students as well as supporting students through the learning process (lowenthal, 2015). in their study on the nature of social presence in online course discussions, swan and shih (2005) found that instructors’ social presence had a larger impact on student outcomes than students’ social presence. more recently, pollard et al. (2014) found instructor social presence to be a significant contributor to positive learning communities. the covid-19 pandemic forced all instruction to temporarily move online (benito et al., 2021) and demand for online courses, especially at the graduate level, is likely to persist. as such, finding ways to improve instructor social presence is of great importance. video feedback an important way in which instructor social presence can be established is through various forms of feedback students receive throughout an online course. traditionally, feedback has come in the form of written comments on assignments. however, recent literature has established that audio and video feedback have the ability to convey the non-verbal immediacy cues necessary to create closeness and increase social presence (anson et al., 2016; borup et al., 2014). collins et al. (2019) also argue that asynchronous video may improve student engagement by increasing students’ perceptions of instructor social presence in online courses. in a recent review of the literature on video feedback, bahula and kay (2021) found that a majority of higher education students preferred video feedback over text-based feedback. specifically, they found that video feedback offered more detail, was easier to understand, supported higher-level thinking, kept students engaged, was perceived as more personal and authentic, increased social connections and their connection to the instructor, and increased student interaction with the course (bahula & kay, 2021). research has also demonstrated some challenges associated with video feedback. accessibility problems have proven to be barriers for many students, including students not knowing how to access the video-based feedback (thompson & lee, 2012), files incompatible with devices (deeley, 2018), slow download speeds or internet connections (mccarthy, 2015), poor audio quality (ali, 2016), or the absence of speakers or headphones (hyde, 2013). additionally, studies have reported that students found the linear nature of video feedback to be problematic as they were unable to scan the feedback (as one would text comments), requiring them to review the videos multiple times and slowing down the revision process (i.e., borup et al., 2015; thompson & lee, 2012). while some studies have found video feedback to soften the impact of receiving challenging feedback (e.g., marshall et al., 2020), other studies have reported that students experienced negative feelings when receiving video-based feedback, such as anxiety, nervousness, discomfort, awkwardness, or a hesitancy to watch the feedback (ali, 2016; edwards et al., 2012, hyde, 2013; lamey, 2015). therefore, it is important to continue to investigate student perceptions of video feedback to fully understand the impacts it may have on student engagement and instructor social presence. understanding the influence asynchronous video feedback has on student video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 46 perceptions of instructor social presence may help online and blended format instructors facilitate student-teacher relationships that support a sense of classroom community and encourage student engagement and cognitive presence (collins et al., 2019). methods this study aimed to understand student perceptions of video-based writing feedback in an asynchronous online course offered to pre-service teachers. this study’s participants were students enrolled in a master’s level educational psychology course. the study took place at a large public university located in a mid-sized city in the mid-atlantic region of the united states that enrolls over 30,000 students, one-third of which are from underrepresented groups. advanced educational psychology is a required course for students seeking a master’s degree in teaching. the course focuses on theories and research about educational psychology principles that influence effective learning and teaching. five sections of the same course were offered in the fall 2019 (one section), spring 2020 (two sections), and spring 2021 (two sections) semesters; all were taught by the same instructor. a qualitative descriptive design was employed to explore student perceptions of the video feedback they received in the course. the primary deliverable in the course was the review of research and application project (rrap). the rrap was a multiple-part assignment that offers students the option of choosing a topic they feel best meets their individual learning needs and preferences. the focus of the assignment was a review and critique of current educational psychology research and the application of both research and theories of learning and motivation to instructional practice. the assignment was broken into four parts, all of which received feedback. in part i, students were asked to identify three potential topics for their research, and the instructor provided written feedback on the quality of these topics. part ii was dedicated to finding empirical research studies related to the students’ topics. students were asked to submit apa citations of five articles for the instructor to review, and three of these were expected to be used in the final product. the instructor provided written feedback on the articles submitted for this assignment. in part iii, students submitted an outline of their rrap and received video feedback from the instructor. students were instructed to provide as much detail as possible, since more detailed outlines received more feedback. the final part of the paper, part iv, received extensive video feedback as students were given the opportunity to revise their final papers based on the feedback they received. minimal written comments were included in the paper to be used as markers for discussion in the feedback video. a rubric was used to score the final paper, and the video feedback addressed the scores on the rubric. the process of recording and sharing video feedback changed over the course of the study due to differences in learning management systems. the first two semesters of the course, fall 2019 and spring 2020, used google classroom, and video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 47 the third semester of the course, spring 2021, used canvas1. within google classroom, the instructor used zoom to screencast each student’s paper and verbally walk through the feedback provided. videos were downloaded from zoom and uploaded to youtube as a private video. links were then generated for individual students to be able to view their video feedback and were posted in the assignment comments in google classroom. this process averaged about 20-25 minutes for each paper. videos ranged from 5 to 10 minutes in length, with the additional time spent on reading and scoring the paper. in canvas, the instructor used the video feedback tool embedded in the platform to provide feedback. while this did not allow the instructor to screencast the paper, it reduced the amount of time required to download, upload, and share a link to a video. videos ranged from three to six minutes in length, and the entire process of reading and scoring the paper and recording the video took approximately 20–25 minutes, with some additional time spent on written comments compared to previous semesters since screencasting was not an option in canvas. participants participants were selected using purposive sampling. the students enrolled in this course were pre-service teachers, all of whom were pursuing a master’s degree. approximately half of these students were a part of a teacher residency program that sought to train teachers for urban and high poverty school contexts in the metropolitan region in which the university resides. all students enrolled in the instructor’s graduate educational psychology courses from the fall 2019, spring 2020, and spring 2021 semesters were invited to participate in the study. overall, 63 students, representing a response rate of 72.4%, agreed to participate in the study. demographic information was intentionally not collected from individual survey participants, since doing so would have “outed” some students. see table 1 for descriptive statistics for the students enrolled in the course. two types of data were collected. first, all students were invited to complete a survey at the end of the semester about their experiences in the course, including their perceptions of the feedback received on their rrap project. the survey was a routine part of the course structure, serving as a way for the instructor to collect formative feedback about the students’ experiences in the course. specific items were added to this survey that were relevant to the study’s research, and other items were considered as appropriate based on student responses. for example, students were asked to comment on their perception of video versus written feedback and how connected they felt to their instructor and classmates. additionally, students were asked to describe the most effective aspect of the course, and responses related to video feedback were included in this study. at the conclusion of the survey, students were invited to leave their email address if they were willing to participate in an interview about their experiences in the course. they were informed that we were specifically interested in their experiences with the video feedback they received on their writing assignments in the course. 1the instructor used google classroom until the university adopted canvas in the 2020–2021 academic year. video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 48 table 1 demographics of students enrolled in the course variable race/ethnicity fall 2019 n (%) spring 2020 n (%) spring 2021 n (%) total n(%) african american or black 2 (9.1) 6 (18.8) 10 (30.3) 18 (20.7) asian american 0 (0.0) 1 (3.1) 2 (6.1) 3 (3.4) hispanic or latina/o 1 (4.5) 1 (3.1) 0 (0.0) 2 (2.3) white or caucasian 19 (86.4) 22 (68.8) 20 (60.6) 61 (70.1) other 0 (0.) 2 (6.3) 1 (3.0) 3 (3.4) gender female 17 (77.3) 27 (84.4) 29 (87.9) 73 (83.9) male 5 (22.7) 5 (15.6) 4 (12.1) 14 (16.1) teacher education program elementary 17 (77.3) 15 (46.9) 11 (33.3) 43 (49.4) special education 4 (18.2) 15 (46.9) 19 (57.6) 38 (43.7) secondary 0 (0.0) 2 (6.3) 0 (0.0) 2 (2.3) teacher residencya 0 (0.0) 13 (40.6) 8 (24.2) 21 (24.1) other total 1 (4.5) 22 0 (0.0) 32 3 (9.1) 33 4 (4.6) 87 note. n = 87; a student was enrolled in the university’s teacher residency program ten of the 63 students who completed a survey participated in an interview. interviews followed a semi-structured protocol and lasted between 15 and 30 minutes. questions probed for participants’ previous online learning experiences, how connected they felt with peers and the instructor in this course, and their perceptions of the video feedback they received in the class, including on the rrap assignment. the interview protocol used for this study can be found in appendix a. all interviews were audio recorded and transcribed verbatim. several considerations were made to ensure the protection of the participants in the study. this was especially important given the power dynamics that exist between instructors and students. a member of the research team who worked at a different university and had no prior interactions with the students was introduced to the class two weeks prior to the conclusion of each semester. this team member independently contacted students to solicit their participation in the project. this team member consented, interviewed participants, and transcribed the interviews thereafter. no interviews were completed until final grades were posted for the course. audio recordings were deleted following the verification of transcripts. race/ethnicity, gender, and other demographic variables were not collected in the survey participation as part of an effort to protect the identity of the participants. video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 49 collecting demographic data would have identified some individuals enrolled in the course. transcripts did not contain any personally identifiable information and transcript files were named using a unique code. personally identifiable information that was shared during the course of an interview, such as the university the student attended for their undergraduate education, was redacted. the instructor of the course did not interact with any of the transcripts until they were in deidentified form. surveys were administered at the conclusion of the semester; responses were anonymous. an information letter was used for both in lieu of an informed consent form since a signed consent form would have been the only documentation linking an individual with participation in this study. this work was approved by both universities’ institutional review boards. positionality2 our team of researchers consisted of two individuals: one with expertise in educational psychology, who taught the course in which the study took place. the other member of the team was not connected with the course under study but had also previously used video-based feedback for writing deliverables assigned in their online coursework. both members of this research team have studied and published on writing feedback previously (marshall et al., 2020). similar to the study’s participants, both researchers have been involved with teacher preparation programs previously and both were formerly secondary public education teachers. our collective expertise allowed for different perspectives on the data during analysis. had only the instructor of the course conducted the study, it would have posed ethical issues related to power dynamics discussed previously, and it may have been difficult for them to separate a desire for positive outcomes in their teaching from the findings in this study. data analysis after the interviews were conducted and transcribed, we followed a member checking protocol which afforded participants the opportunity to review the transcripts of their interviews to ensure that we accurately captured their thoughts and perspectives (maxwell, 2013). a hybrid coding approach was used for this analysis (saldaña, 2015), and interview transcripts were coded using atlas.ti version 9. survey data were also imported into atlas.ti for analysis. a priori codes were initially created to correspond to aspects of writing feedback, video-based feedback, and instructor social presence. additional codes were added as needed based on the transcripts. a total of 48 codes were initially created during the first round of coding. team members met to discuss coding and created a final list of 23 codes that were used to analyze the qualitative data. a second round of coding was conducted thereafter. emerging themes were identified individually. the team met a final time to discuss discrepancies and reach consensus on findings. 2positionality statements are important to include when reporting on qualitative research (foote & bartell, 2011). they allow the researchers to identify the experiences they bring into the work that may influence the research process. video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 50 results this study sought to understand the extent to which using video-based feedback for student writing deliverables enhanced instructor social presence in an asynchronous online graduate-level course. these findings reflect what we learned from the survey and interview data we collected. overall, four themes emerged in the data. three themes correspond with aspects of instructor social presence, and the fourth represents an unexpected factor in our findings: (1) participants perceived the instructor as a “real person”; (2) video feedback added emotional expression; (3) participants reported feeling connected to the instructor; and (4) weekly videos improved student experience. participants perceived the instructor as a “real person” the students we surveyed and interviewed shared that the video feedback they received helped them to perceive their instructor as a “real person” that they got to know over the course of the semester. students in the course described this especially in terms of the writing feedback they received. one student shared that videos made the course more personable, explaining, “the professor made everything super personable. she made herself very human to us and reached out to us as often as she could.” another student shared that seeing the instructor’s face made a difference: a lot of my professors in the past would take a pen and write on my paper or just put a grade on my paper or attach a rubric to it and circle where i fell on the rubric. but this was different because i could see her face…i think it was as if i was sitting down in front of her in person and she was going through my paper with me, which was incredibly helpful. a third student similarly shared, “she gave me video feedback for one of them, but she zoomed with me about the other one, and she was actually going through my paper with me. and she was sitting on her front porch with her dog outside with her and i was like, ‘wow, she’s actually just another human being.’” as basic as this first finding may seem, it can be easy for those engaged in asynchronous online teaching and learning to be faceless names on a screen. the addition of videos in the writing process added a dimension of instructor social presence in the course. video feedback added emotional expression participants described the added benefit of having tone, expression, and nonverbal communication cues added to the feedback that they received on their writing, all of which are missing when feedback is only provided in written form. in open-ended survey responses, only three percent of participants indicated that they preferred to receive written feedback only. alternatively, 86% of students specifically commented on aspects of video feedback that enhanced their experience in the course. one student shared the video feedback allowed them to see the instructor’s body language, explaining “…when she was providing feedback over the video, it’s like you can read her body language. you can see something and listen versus video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 51 [only] reading the words.” another student discussed the importance of hearing the instructor’s tone of voice, “…i know [the instructor is] invested in our class. but hearing how she speaks to us and hearing in her voice how she cares about our assignments kind of transforms that a little bit.” overall, participants shared that the emotional expression and nonverbal communication that was present in videobased feedback helped humanize the writing experience. participants reported feeling connected to the instructor the video feedback embedded in the course also helped to develop instructorstudent relationships. in responses to open-ended survey items asking students about their experiences in the class, 87% of participants shared that they felt more connected with their instructor because of the video feedback. in an interview, one participant shared that the videos made them feel more connected to the course instructor in this asynchronous course than they did to professors who taught some of their face-to-face courses. they shared, “i felt pretty connected to her just because she was providing very regular feedback on all of my content. so, weird on some level, i felt like i was more connected to her than some of my professors…” another student shared that seeing the instructor’s face made it feel like an inperson experience, “this was different because i could see her face. …it was as if i was sitting down in front of her in person.” this feeling of connection also extended to the course in general. one student explained, “i felt more connected to the course, because i felt like with the personalized video feedback, in a way, it made me feel like i was held a little bit more accountable, because i was getting such personalized feedback, that i was more motivated to expand more on my work or my thoughts.” overall, participants felt that having the ability to see and hear the course instructor when receiving feedback on their writing enhanced the rapport they had with them. weekly videos improved student experience our study sought to understand the impact that video feedback had on students’ writing. a factor that complicated our understanding of this was the instructor’s weekly video posts. each week, she posted a weekly video that was shared with the whole class, which discussed course content, major points in the current week’s content, and an overview of upcoming assignments. some of these assignments involved components of the writing project, which also received personalized video feedback; however, other assignments were discussed as well. students reported difficulty in untangling the impact of the personalized videos they received on their writing from the impact of the weekly videos posted for the entire class to view. one student shared, “i felt pretty connected to the professor due to the videos she uploaded each week, explaining our assignments and what was going on in the class.” another student shared, “she made videos regarding our assignments, and it was the most personable of any online class i’ve ever taken. it was different but so much nicer to be able to see her face and hear her voice when telling us about the upcoming week. it was very helpful.” a third student shared, “[in] most online video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 52 courses i have taken, i didn’t even know what the professor looked like. i enjoyed being able to see the professor through the weekly videos and through [the writing feedback]. this created more of a relationship with the professor and made the experience of this course more comfortable.” students overwhelmingly described the weekly videos the instructor made for the entire class’s consumption to be helpful, and they described them as developing instructor social presence. thus, separating the students’ experience with individualized video feedback and wholegroup class videos was somewhat difficult as both seemed to have added to instructor social presence. discussion this study’s findings are consistent with much of the previous literature on video feedback. they also provide added insight on the specific context of writing feedback in a graduate-level asynchronous online course. specifically, the finding that participants perceived the instructor as a “real person” is consistent with garrison and colleagues’ (1999) conceptualization of social presence. this, along with the finding that participants reported feeling connected to the instructor, confirms the findings that video feedback is able to create closeness and increase social presence (anson et al., 2016; borup et al., 2014; lamey, 2015). our findings offer an important addition to the research in this area given the context of a fully online asynchronous course compared to blended courses (borup et al., 2014) and in-person courses with video feedback on virtually submitted assignments (anson et al., 2016; lamey, 2015). given the increase we have seen in courses offered online, especially since the covid-19 pandemic, it is important to understand how students perceive instructional methods in an online environment compared to a traditional or blended environment. additionally, our finding that video feedback added emotional expression is consistent with garrison’s (1999) multidimensional understanding of social presence, which includes affective expression as one of the three dimensions. thus, our findings clearly demonstrate a connection between video feedback and an increased social presence. previous research has demonstrated the positive outcomes related to increased instructor social presence, including student performance (swan & shih, 2005) and positive learning communities (pollard et al., 2014). understanding the impact of improving instructor social presence in a fully online learning environment could therefore have important implications for student experiences in online courses. one unexpected finding from this study was the impact weekly videos had on students’ experience. while we set out to explore the impact of video feedback, specifically, we found that students were often unable to separate the video feedback they received from the weekly videos they viewed to hear course updates, announcements, and content from the instructor. given that both forms of videos were an important way for students to “hear” from the instructor each week, it is not surprising that the weekly videos served as an alternative way to build instructor social presence. this may be a particularly relevant finding for instructors seeking to increase their social presence in an asynchronous online course. while video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 53 video feedback can be time intensive, weekly videos tend to be less so and serve multiple purposes for the entire class. implications for practice online learning is attractive for several reasons, including its ability to reach learners for whom work schedules and geographic distance might have previously been barriers (alexander et al., 2012; mcisaac & gunawardena, 1996). however, asynchronous courses have the potential to be impersonal and “dehumanizing” (miller & mazur, 2009). it can be easy for learners and instructors alike to view each other as names on a screen, not as real people. this study’s findings echo those found in previous studies suggesting that the use of video-based feedback helps to humanize the instructor and foster greater instructor social presence (darby & lang, 2019; marshall et al., 2020). research suggests that feedback is both important for student learning and that it often fails to meet student needs (nilsen, 2016). as such, feedback that is well received by students deserves attention. additionally, understanding the practicality and time commitment for instructors is also important. instructors must have a clear plan and time set aside to interact with their students asynchronously, similar to the time they would spend in class for a traditional in-person course. throughout this study, the researchers learned to use different digital tools and learning management systems, and each came with its own set of strengths and weaknesses. exploring these options and understanding how they will support course objectives is another element to consider when planning for video feedback. given the research findings related to video feedback and the increasingly userfriendly applications available to provide it in various ways, it may be that video feedback or video engagement becomes an accepted, perhaps expected, best practice for online courses. as such, it will be important for instructors to become knowledgeable and trained in implementing video tools appropriate for their context. limitations and future research there are some limitations that are worth noting. while students shared that the individual video feedback they received on their writing was helpful, they also shared that the weekly videos that were posted on the course learning management system were also helpful. we cannot state with certainty that the video feedback received on writing was solely responsible for enhancing instructor social presence; it is likely the case that all forms of video-based communication contributed to this. although we did receive at least survey responses from over 70% of the students enrolled in the course and conducted in depth interviews with 10 additional students, it is always possible that those who agreed to participate in the study had experiences with the video feedback that differed from those who elected not to participate. we believe that the very act of conducting this study led to improved learning outcomes for the students enrolled in this asynchronous online graduate-level educational psychology course. future work should continue to explore the importance of video feedback in graduate settings and should seek to video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 54 continue to find ways to use video and/or other digital tools to improve the feedback that students receive on their writing and improve instructor-student relationships. conclusion this paper has discussed the extent to which video feedback enhanced instructor social presence in an online asynchronous course. overall, findings highlighted the positive outcomes associated with video feedback, including that it helped students perceive the instructor as a real person, added emotional expression to the feedback they received, and made students feel more connected to the course and instructor. additionally, findings pointed to the improved student experience associated with weekly videos. these findings add to the research on video feedback by exploring the specific context of a fully online, asynchronous graduate course for pre-service teachers. there are also important implications for instructors who wish to design and implement effective online courses. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. references alexander, m. w., truell, a. d., & zhao, j. j. (2012). expected advantages and disadvantages of online learning: perceptions from college students who have not taken online courses. issues in information systems, 13(2), 193– 200. https://doi.org/10.48009/2_iis_2012_193-200 ali, a. d. (2016). effectiveness of using screencast feedback on efl students’ writing and perception. english language teaching, 9(8), 106–121. anderson, t., rouke, l., garrison, d. r., & archer, w. (2001). assessing teaching presence in a computer conferencing context. journal of asynchronous learning networks, 5(2), 1–17. https://olj.onlinelearningconsortium.org/index.php/olj/article/view/1875 anson, c. m., dannels, d. p., laboy, j. i., & carneiro, l. (2016). students’ perceptions of oral screencast responses to their writing: exploring digitally mediated identities. journal of business and technical communication, 30(3), 378–411. https://doi.org/10.1177/1050651916636424 bahula, t., & kay, r. (2021). exploring student perceptions of video-based feedback in higher education: a systematic review of the literature. journal of higher education theory and practice, 21(4), 248–258. https://doi.org/10.33423/jhetp.v21i4.4224 video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 55 benito, á., dogan yenisey, k., khanna, k., masis, m. f., monge, r. m., tugtan, m. a., ...& vig, r. (2021). changes that should remain in higher education post covid-19: a mixed-methods analysis of the experiences at three universities. higher learning research communications, 11, 4. https://doi.org/10.18870/hlrc.v11i0.1195 borup, j., west, r. e., thomas, r., & graham, c. (2014). examining the impact of video feedback on instructor social presence in blended courses. international review of research in open & distance, 15(3), 1–15. https://doi.org/10.19173/irrodl.v15i3.1821 borup, j., west, r. e., & thomas, r. (2015). the impact of text versus video communication on instructor feedback in blended courses. educational technology research and development, 63(2), 161–184. https://doi.org/10.1007/s11423-015-9367-8 collins, k., groff, s., mathena, c., & kupczynski, l. (2019). asynchronous video and the development of instructor social presence and student engagement. turkish online journal of distance education, 20(1), 53–70. darby, f., & lang, j. m. (2019). small teaching online: applying learning science in online classes. jossey-bass. deeley, s. j. (2018). using technology to facilitate effective assessment for learning and feedback in higher education. assessment & evaluation in higher education, 43(3), 439–448. https://doi.org/10.1080/02602938.2017.1356906 edwards, k., dujardin, a. f., & williams, n. (2012). screencast feedback for essays on a distance learning ma in professional communication. journal of academic writing, 2(1), 95–126. https://doi.org/10.18552/joaw.v2i1.62 foote, m. q., & bartell, t. g. (2011). pathways to equity in mathematics education: how life experiences impact researcher positionality. educational studies in mathematics, 78, 45–68. https://doi.org/10.1007/s10649-011-9309-2 garrison, d. r., anderson, t., & archer, w. (1999). critical inquiry in a text-based environment: computer conferencing in higher education. the internet and higher education, 2(2–3), 87–105. https://doi.org/10.1016/s10967516(00)00016-6 gunawardena, c. n. (1995). social presence theory and implications for interaction and collaborative learning in computer conferences. international journal of educational telecommunications, 1(2), 147–166. https://www.learntechlib.org/primary/p/15156/ video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 56 harper, f., green, h., & fernandez-toro, m. (2012, september). evaluating the integration of jing screencasts in feedback on written assignments. in 2012 15th international conference on interactive collaborative learning (icl) (pp. 1–7). ieee. https://doi.org/10.1109/icl.2012.6402092 henderson, m., & phillips, m. (2015). video-based feedback on student assessment: scarily personal. australasian journal of educational technology, 31(1). https://doi.org/10.14742/ajet.1878 hyde, e. (2013). talking results: trailing an audio-visual feedback method for esubmissions. innovative practice in higher education, 1(3). lamey, a. (2015). video feedback in philosophy. metaphilosophy, 46(4–5), 691– 702. https://doi.org/10.1111/meta.12155 lowenthal, p. r. (2015). a mixed methods examination of instructor social presence in accelerated online courses. handbook of research on strategic management of interaction, presence, and participation in online courses, 147. https://doi.org/10.4018/978-1-4666-9582-5.ch006 marshall, d. t., love, s. m., & scott, l. (2020). “it’s not like he was being a robot”: student perceptions of video-based writing feedback in online graduate coursework. international journal for the scholarship of teaching and learning, 14(1), article 10. https://doi.org/10.20429/ijsotl.2020.140110 maxwell, j. a. (2013). qualitative research design: an interactive approach. sage publications. mccarthy, j. (2015). evaluating written, audio and video feedback in higher education summative assessment tasks. issues in educational research, 25(2), 153–169. mcisaac, m. s., & gunawardena, c. n. (1996). distance education. in d. h. jonassen (ed.), handbook of research for educational communications and technology (pp. 403–437). simon & schuster macmillan. https://doi.org/10.4324/9781410609519 miller, c., & mazur, j. m. (2009). toward a person-centered model of instruction: can an emphasis on the personal enhance instruction in cyberspace? in a. orellana, t l. hudgins, & m. simonson (eds.), the perfect online course: best practices for designing and teaching (pp. 275–296). information age publishing. nilsen, l. b. (2016). teaching at its best: a research-based resource for college instructors (4th ed.). jossey-bass. video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 57 oyarzun, b., barreto, d., & conklin, s. (2018). instructor social presence effects on learner social presence, achievement, and satisfaction. techtrends, 62(6), 625–634. doi:10.1007/s11528-018-0299-0 parton, j., west, r. e., thomas, r. a., & graham, c. r. (2010). examining the impact of video feedback on instructor social presence in blended courses. the international review of research in open and distance learning, 15(3), 232–256. https://files.eric.ed.gov/fulltext/ej1033092.pdf pollard, h., minor, m., & swanson, a. (2014). instructor social presence within the community of inquiry framework and its impact on classroom community and the learning environment. online journal of distance learning administration, 17(2). rourke, l., anderson, t., garrison, d. r., & archer, w. (1999). assessing social presence in asynchronous text-based computer conferencing. the journal of distance education/revue de l'ducation distance, 14(2), 50–71. https://www.learntechlib.org/p/92000/ saldaña, j. (2015). the coding manual for qualitative researchers. sage, london. swan, k., & shih, l. f. (2005) on the nature and development of social presence in online course discussions. journal of asynchronous learning networks, 9(3), 115–136. http:// doi.org/10.24059/olj.v9i3.1788 thompson, r., & lee., m. j. (2012) talking with students through screencasting: experimentations with video feedback in higher education. the journal of interactive technology and pedagogy, 1(1). https://digitalcommons.tacoma.uw.edu/ias_pub/229 video feedback and instructor social presence journal of effective teaching in higher education, vol. 5, no. 2 58 appendix a interview questions 1. how many online classes have you taken prior to this semester? a. what has been your experience in previous online courses? b. in terms of feedback on assignments? c. if you have not taken an online course, what were your expectations coming into this course? 2. describe your online course experience for edus 607. a. to what extent did you feel connected to your classmates? b. to what extent did you feel connected to your instructor? c. what elements do you believe contributed to your online class community? i. what elements did not contribute to a positive class community? 3. describe the process you went through in writing your review of research and application project 4. what was your perception of the feedback that you received throughout the rrap process? a. what was your general perception of the feedback you received b. how was the feedback you received related to the revision process 5. to what extent was your experience receiving feedback on this project similar or different from previous writing experiences in higher education? a. elaborate on the type of feedback you have received in previous courses i. if they have online experience, focus on that b. elaborate on the type of feedback you received in this course i. to what extent did the feedback you received help you see you instructor as a “real person”? ii. probe about tone and non-verbal cues (emotional expression) associated with video-based feedback iii. to what extent did the feedback you received help you build a relationship with your instructor? thank you for your time and willingness to participate in our research. this data will be used to advise and inform our knowledge of how feedback impacts student writing. microsoft word gullo_final.docx journal of effective teaching in higher education, vol. 5, no. 2 https://doi.org/10.36021/jethe.v5i2.312 supporting students with disabilities to be successful in an online learning environment dana gullo, wilmington university, dgullo@cecil.edu abstract. this study examined the factors that contributed to students with disabilities’ positive and challenging experiences transitioning to remote learning in the spring 2020 semester due to the covid-19 pandemic at a small, two-year community college located in the midwestern united states. the study administered a confidential survey to students with disabilities to learn about their positive and challenging experiences and concluded with an interview with the vice president for academic affairs to gain additional insight into the college’s timeline and transition procedures to remote learning. this study discovered both positive and challenging experiences with remote learning for students with disabilities. the positive experiences were comfort with the learning management software blackboard, ease of access to grades, instructor helpfulness with accommodations, and images and video instructional materials utilized in their courses. the challenges discovered were related to faculty and peer communication, testing services, confidence with remote learning, and lack of knowledge about student accessibility services. the data concluded that students diagnosed with a mental health disability were less aware of student accessibility services than students with other disabilities. the recommendations outlined in this study will help higher education institutions, faculty, and student accessibility services offices design effective approaches in supporting students with disabilities’ accommodation needs when taking online courses. keywords: disability; universal design for learning (udl); accommodations; online learning; covid-19; mental health; instructional design online learning in higher education has grown exponentially since the start of the millennium. according to seaman et al. (2018), during the years 2015 to 2016, enrollment of the nation’s students in online programs was 14.9%. in the same period, another 16.7% of students took a combination of face-to-face and online courses (p. 3). combined, 31.6% of all higher education students were taking at least one class online. the growth of online learning is due in part to the changing demographics of traditional students in higher education. today, college students are older; approximately 35% of students currently enrolled in higher education are 25 years of age and older. additionally, 64% of students work during their college careers (lumina foundation, 2019). students today are older, more diverse, have more family responsibilities, and require the flexibility of online programs. community colleges offer an array of online programs and certificates for their diverse student body. perry and pilati (2011) asserted that “…online instruction has been most fully accepted by community colleges that seek to provide educational opportunities for an extremely diverse student population, requiring sensitivity to issues of differential connectivity” (p. 97). supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 60 students with disabilities (swd) make up part of the diverse student population mentioned above. for this study, swd are individuals evaluated as having or being vision impairment, deaf or hard of hearing, mental health conditions, intellectual disabilities, autism spectrum disorder, physical disability, chronic pain, attention deficit disorder (add), attention deficit hyperactivity disorder (adhd), dyslexia, and other medical disabilities. according to the national center for education statistics (nces), during the years 2015 to 2016, 19% of swd were enrolled as undergraduates in postsecondary education (nces, 2019). during this same period, distance education increased by 5.6% in enrollment while on-campus enrollment dropped 11% between 2012 and 2016 (snyder et al., 2018). the drop in oncampus enrollment is partially due to decreased overall enrollment in higher education and online learning growth. online learning provides flexibility and continued education for swd; however, challenges do exist. in late december of 2019, in hebei, wuhan city in china, chinese scientists identified a severe acute respiratory syndrome that affected its citizens. on march 15, 2020, the world health organization called the respiratory syndrome a covid19 pandemic that infected 150,000 persons in 154 countries (chahrour et al., 2020), resulting in 150 school closings. the pandemic affected 80% of students worldwide during the spring of 2020 (sahu, 2020, para. 2). the sudden closure of colleges and universities required a shift from face-to-face instruction to emergency remote online learning. the rapid change to remote learning forced college and university faculty members to create instructional content quickly for students to access on the institution’s learning management system. while online instruction is not a new modality, many college and university faculty have minimal experience teaching online. a study conducted by bay view analytics found that 76% of all faculty moved some of their courses online in spring 2020, but 65% of them had no prior online teaching experience (lederman, 2020). the faculty’s lack of online pedagogy training affects the accessibility of course materials and accommodations needed for students with disabilities (behling & linder, 2017). literature review in response to the increasing infection rate of covid-19 in early march of 2020, colleges and universities quickly ceased all in-person lectures and required all students to transition to remote learning for the remainder of the spring semester. the university of washington, stanford university, and harvard university were some of the first colleges to require students to leave campus and transition to remote learning (bacow, 2020; burke, 2020). since this is the first health crisis of this magnitude in u.s. history, minimal information exists about the impact of students’ remote transition, especially for swd. depietro (2020) noted, “faculty and staff transitioned all learning to online and virtual in a very short period of time, and this will undoubtedly impact the success and retention of students” (para. 11). the remote transitional impact on swd’s learning experiences requires further research. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 61 this section details the review of the literature that was conducted prior to this study to determine prior positive and challenging experiences swd faced in online learning, the effects of swd transitioning involuntarily to remote learning due to a crisis, and the historical context of swd’s graduation success rates in online programs. retention and graduation rates a search of the literature found both positive and negative influences concerning successful completion and graduation rates for swd in traditional and online learning. stewart et al. (2010) stated that swd typically enroll in more courses than students without disabilities (swod) and experience similar retention rates as swod. according to stewart et al. (2010) study of a mid-sized historically black college located in the east coast of the united states, “swd did not score any different from swod” (p. 35) in either online or face-to-face undergraduate or graduate courses concerning final grades. students with disabilities may encounter similar retention and completion rates, but the time to graduation differs for swd from swod. knight et al. (2016) posited that having a disability does not negatively affect eventual graduation, but having a disability influences the amount of time to graduation. the authors noted, “regarding graduation rates, at year 4 swd had statistically significant lower graduation rates than swod, but by years 5 and 6, swd graduation rates had equaled, and even surpassed, swod” (p. 370). credit hour penalties that occur at some institutions present one negative implication resulting from increased time to graduation for swd. this penalty is due to state funding dependent upon students who achieve retention and graduation rates within four years for a bachelor’s degree. the impact on graduation rates due to institutional credit hour penalties and college instituted disincentives needs additional research. for some students with disabilities, the flexibility of online learning is a real draw, especially for students with orthopedic impairments. alamri and tyler-wood (2017) found that swd participate in online courses at higher rates than other student populations, and the enrollment of swd in online learning has increased in recent years. swd may increasingly participate in online learning due to the flexibility, but the perception of success in online learning is low. in a study performed by roberts et al. (2011), the researchers asked swd if their disability would affect their ability to succeed in an online course. even without participation in an online course, 27% of students reported their disability “negatively impacted their ability to succeed in online courses” (p. 245). this negative perception of online learning formed by swd leads one to question what factors, previous barriers, or challenges swd have experienced. the current literature on swd’s negative perception of online learning is minimal, and further research is needed on the topic. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 62 barriers and success strategies common barriers emerged in the literature for swd in traditional and online instruction. the lack of faculty knowledge on accessibility, the understanding of the different needs of students with multiple disabilities, and the perceived stigma of swd’s disclosure to the instructor and the institution were common threads. bissonnette (2006) asserted that 81% of faculty reported not considering the needs of swd, and 12% of faculty indicated that they partially considered the needs of swd. administrators need to assist faculty members concerning the needs of students with disabilities; this begins with education (bissonnette, 2006). faculty training should focus on the types of disabilities common at their institution and the universal design for learning (udl; cast, 2018) strategies needed to enable the academic success for swd (black et al., 2015). learning disabilities, such as autism spectrum disorder (asd), attentiondeficit/hyperactivity disorder (adhd), learning-related anxiety, and dyslexia reported by the u.s. department of education as “specific learning disability[ies]” accounted for 34.8% of children 3 to 21 years old served under the individuals with disabilities education act of 2004 (snyder et al., 2018, p. 117). the prevalence could be even higher since many students with learning disabilities “…would not consider using the disabilities service because they perceived this to be for students with greater needs” (couzens et al., 2015, p. 35). understanding the accommodations needed by the learning disability population entering higher education institutions will foster their academic success. griful-freixenet et al. (2017) found in their study, made up mostly of students with learning disabilities and dyslexia, that using udl was beneficial to their learning. training faculty on the udl principles and strategies previously mentioned would support the needs of students with dyslexia and swod, which is the primary foundation of udl: removing barriers to support all types of learners. however, a study by griful-freixenet et al. (2017) indicated that meeting the needs of students with dyslexia created barriers for other learning disabilities, such as autism. a onesize-fits-all approach to accommodations is not sufficient for all learners. a theme that emerged in the reviewed literature was that instructor interaction was paramount to the academic success and retention for swd. in a survey conducted by alamri and tyler-wood (2017), the researchers studied the factors contributing to swd’s success in online courses. the authors identified two main factors that lead to their success: “(1) the teaching and social presences, and (2) the facilitating and supporting of individual communication related to learners with disabilities and their instructors in online courses” (p. 67). swd perceived that satisfaction with online learning is dependent on the instructor-demonstrated presence and substantive and regular feedback. in some online situations, swd may experience learning-related anxiety when engaging in online courses with low studentinstructor interaction. for example, students with visual impairment may encounter assistive technology issues when accessing course materials. oh and lee (2016) supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 63 asserted that attention, concentration, and motivational and organizational difficulties cause higher-level anxiety for swd than their swod counterparts. stigma a significant barrier revealed in the literature was the stigma associated with swd disclosing their disability to the institution and their instructors. roberts et al. (2011) found, “…71% of those students who indicated that they have a documented disability also reported that they never requested any accommodation” (p. 247). swd, especially those who have a non-apparent disability, choose not to disclose their disability; other students are simply unaware of the services available to them. in a semi-structured interview by barnar-brak et al. (2010) of swd concerning disclosing their disability status with their instructors, a graduate student with cerebral palsy commented, “…there is a stigma attached with receiving accommodations because there is a stigma about being disabled” (p. 421). in a similar non-apparent or invisible disability, kruse and oswal (2018) found that post-secondary students with mental health disorders often do not disclose to the office of disability services or a similar mental health center at their institution for two reasons: the stigma surrounding mental health disorders and the lengthy process of proving they are disabled “enough” for assistance. students with mental health disorders stated that their instructors and peers often see their invisible disability as an excuse to receive extra help with assignments, providing them with an unfair advantage over their peers. additionally, students with mental health disorders stated that they felt they are perceived as violent, unstable, or lying about their condition to receive accommodations from their instructors (kruse & oswal, 2018). in contrast to the nondisclosure of their disability, those who did disclose their disability had the highest course completion rates. moisey (2004) focused on the course completion of swd in distance learning. the research found that swd who requested accommodations from their professors had higher course completion rates than those who did not. however, students with multiple or mental health disabilities receiving accommodations had the lowest course completion rates. higher education will need to focus on student mental health issues to provide swd the tools for academic success (moisey, 2004). crisis impact on learning the literature identified instances where colleges and universities decided to quickly transition to remote learning in response to a national crisis, such as hurricane florence, hurricane katrina, and the h1n1 pandemic (field, 2020; meyer & wilson, 2011). due to the 2020 covid-19 health crisis during this paper’s writing, minimal peer-reviewed research was available about the 2020 pandemic’s effects on swd learning. the majority of the extant research focused on how colleges decided to transition to remote learning, structure new academic calendars to increase social distancing on supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 64 campus, and implement new health and safety protocols on campus. the priority for most colleges was the lack of disruption for instruction; there was little mention of how the transition to remote learning affected students with disabilities academically. a few studies related to the covid-19 transition to remote learning suggested potential benefits and identified challenges that swd faced during the transition. zhang et al. (2020) posited that the remote transition could benefit swd due to the flexibility of asynchronous learning and reducing barriers for students with physical mobility impairments. even with the few potential benefits of online learning, zhang et al. (2020) explain that “more than 50% of students with disabilities/health concerns were very concerned about grades in both winter and spring quarters [2020]” (p. 7) when compared to students without disabilities. financial difficulties, discrimination, household tensions, and isolation related to covid-19 affected swd more than their non-disabled counterparts (zhang et al., 2020). a similar study similar by horgos et al. (2020) found that isolation and stress related to the pandemic increased mental health disorders in students attending college in spring 2020 compared to 2019. this study was conducted between may and july of 2020 at nine research universities and found that graduate and professional students with major depressive disorders were two times higher in 2020 than in 2019, and diagnosed generalized anxiety disorders in students were 1.5 times higher than in 2019. to combat isolation and the lack of connection between students and their instructors, some colleges tried to limit the asynchronous online courses and utilized synchronous technologies for instruction. during the quick transition to remote learning due to covid-19, many colleges embraced synchronous (real-time) web conferencing software to simulate in-person lectures. even though synchronous instruction creates a sense of community and connection with faculty and peers, swd struggle with live web conferencing software that does not contain live closed captioning. custodio (2020) asserted that live web conferencing software “…proves especially problematic for deaf students, who would need a sign-language interpreter or live captioning to understand lectures. automated captioning tends to be riddled with errors” (para, 33). offering swd a recording of the synchronous lecture with closed captioning is not an adequate replacement for the live lecture experience. methodology research questions a comparative mixed methods design was utilized to gather quantitative and qualitative data on swd’s positive and challenging experiences during the transition to remote learning due to covid-19. additionally, causal-comparative research was used to compare swd experiences forced to remote learning versus swd who chose to take online courses in spring 2020. according to salkind (2010), causalcomparative design attempts to identify a cause-effect relationship between two or more groups of individuals after an event transpired. the causal-comparative research design was used to identify the potential differences in students’ supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 65 experiences and feelings of success between swd forced to remote learning versus those voluntarily enrolling in online courses. wanting to know more about how the remote transition affected students with disabilities led to this research study and the following research questions: 1. in the transition to remote learning in the spring semester of 2020, what types of support(s) did the college offer to help swd complete their courses? 2. what were the positive experiences of remote learning for swd? 3. what were the challenges of remote learning for swd? 4. what factors helped swd to cope during the transition from face-to-face to remote learning? 5. did the lms present challenges or help swd with remote learning? 6. are the experiences different for swd who were forced to take remote courses in comparison to those swd who voluntarily enrolled in online courses? participants this study’s population was all 1,746 students who attended the community college (a pseudonym) in spring 2020 and enrolled in face-to-face and/or online courses. the participants who were invited to participate in the study included swd who disclosed their disability to the college and swd who did not disclose their disability. eighty percent of students who attend the community college are 24 years old or younger, and 20% of students are 25 years old or older. the student body is over 60% part-time, over 68% female, 87% white, and approximately 3% of the student population has a documented disability with the office of accessibility services at the community college. fifteen documented swd attended college during the spring 2020 semester. to obtain the data to answer the research questions, a survey (see appendix a) was sent to the entire population of the two-year community college located in the midwest with an invitation to participate in the study. the entire student population received a reminder email one week later. due to the low participation of students with disabilities completing the survey within two weeks, a second reminder email was sent only to students who disclosed a disability to the office of student accessibility services. the survey closed on november 8, 2020. a decision was made in the research design to send the survey to all students in hopes of identifying as many students with disabilities as possible. student accessibility services stated that only 15 students who had disclosed their disability to the college attended classes in the spring 2020 semester, so knowing the disclosed swd population was small, the researcher had the community college send the entire college student population the survey invitation. the researcher hoped to attract students with disabilities to participate in the study who did not disclose to the office of accessibility services but who had an individualized education program (iep) or 504 plan in high school or a diagnosed disability. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 66 thirty-eight students with and without disabilities participated in the survey; however, only 10 students of the 38 respondents had a diagnosed disability, and, therefore, only 10 students matched the criteria to participate in the study. these 10 students made up the population for the study; however, all students declined interviews, so the present paper only includes analyses of the survey data (see appendix b). participant demographics the study’s participants consisted of 10 swd who attended the community college and transitioned to remote learning during the covid-19 pandemic in the spring 2020 semester. all participants indicated they were caucasian, 70% female, and 30% male. 30% (n = 3) of the participants (swd) were between the ages of 18 to 20, 20% (n = 2) were between 20 and 29, 20% (n = 2) were between 30 to 39, and 30% (n = 3) were between 40 to 49 years. 40% (n = 4) of swd had taken four or more online courses, while 10% (n = 1) took four online courses, 20% (n = 2) took two online courses, and 20% (n = 2) took one online course at the community college previous to the spring 2020 semester. there were 10% (n = 1) of swd that did not take any online courses. half of the participants indicated having a diagnosed mental health disorder. the complete range of diagnoses is as follows: 20% (n = 2) reported they had a mental health disability, 30% (n = 3) reported mental health and other disabilities, 10% (n = 1) reported a learning disability, 10% (n = 1) reported having adhd and autism disorder, 10% (n = 1) reported a chronic pain disorder, 10% (n = 1) reported a chronic post-traumatic stress disorder, and 10% (n = 1) reported multiple disabilities. four swd disclosed their disability to the office of student accessibility services (sas), and three of those four students required accommodations for their courses. the accommodations the swd received were extra time on tests and extra time on assignments. one student who disclosed used a quiet room for testing in addition to receiving extra time on tests and assignments. one student who contacted sas, but did not finish the application process for disclosing, used extra time on tests and assignments and taking an exam in alternative formats. results swd responded to a survey question about the level of services offered during the transition to remote learning on a likert-type scale of one to five, with one representing no support was offered and five being just in time support. overall, 37% of swd reported not needing support from the college. when swd needed college support, 31% preferred just in time assistance, .075% responded that they did not receive support “until i [the student] reached out,” 10% responded that no support was offered, .075% indicated late support was offered, and 21.85% did not respond to the survey question. as shown in figure 1, of the college services offered to all students, financial aid, technical, chat function for immediate help, student support services, and academic advising were the top five just-in-time supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 67 services swd utilized during the transition. the data revealed the timeliness of the chat function for immediate help positively correlated with both the timeliness of blackboard support (rs[10] = .79, p < .05) and faculty support (rs[10] = .88, p < .05). figure 1 timeliness of support(s) offered to swd (n = 10) note. the graph above shows the overall timeliness of the support(s) the college offered to swd during the remote transition. conversely, when swd were asked about the timeliness of testing services, they ranked testing services 30% late support and 10% no support was offered, academic advising 13% late support and 13% no support was offered, and student accessibility services office support 11.2% late support and 11.2% no support was offered. the data identified a statistically significant negative correlation between the notice of transitioning time to remote learning and both the helpfulness of sas (rs[10] = -.70, p < .05, two-tailed) and the helpfulness of tutoring support (rs[10] = -.68, p < .05, two-tailed). the data identified three areas in which swd experienced positive aspects of the transition to remote learning: ease of access with the blackboard lms, instructors’ use of varied media resources in courses, and instructor helpfulness with 0 1 2 3 4 5 6 7 fin an cia l a id te ch nic al bla ck bo ar d ch at fu nc tio n f or im m ed iat e h elp fa cu lty lib ra ry ac ad em ic ad vis ing stu de nt su pp or t s er vic es m en to rin g tu to rin g stu de nt a cc es sib ilit y s er vic es te sti ng se rvi ce s no support was offered didn't need support i did not receive support until i reached out late support just in time supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 68 accommodations. the positive experiences for swd of remote learning are detailed below. ease of access the survey asked respondents to rate the response using a likert scale of one to five, with 1 (extremely difficult) and 5 (extremely easy) to access when considering the ease of access to the blackboard lms that was used for remote learning. the survey looked at specific components of the blackboard lms, such as grades and the use of various technology devices. overall, the ease of access to grades resulted in 70% (n = 7) of respondents rating the lms as extremely easy. the ability to access the lms on any technology device resulted in 70% of respondents describing access as extremely easy (n = 6) or somewhat easy (n = 1). when asked about students’ comfort with using the blackboard lms, 80% indicated it was extremely easy (n = 6) or somewhat easy (n = 2) to use in their remote courses. the data identified a statistically significant (rs[10] = .89, p < .001, two-tailed) positive correlation between the ease of accessing course grades and comfort using blackboard. lastly, swd indicated that 40% of them were extremely satisfied, 30% slightly satisfied, and 30% extremely dissatisfied with the college’s technology support services. varied media resources swd responded to a survey question about the helpfulness of media resources used in their remote courses with a likert scale of zero to five, with zero representing did not use and five being media resources were very helpful. figure 6 illustrates the most helpful media resources for swd including incorporating images in course content where 89% of respondents found very helpful to slightly helpful, using websites, that 89% of students indicated were very helpful or slightly helpful, and incorporating instructor created videos that 89% of responding student rated as very helpful or slightly helpful. within the top three media resources most helpful to swd in remote learning, each shows a statistically significant negative correlation between the media resource and the community college’s communication to students about the transition to remote learning: images (rs[9] = -.72, p <.05, two-tailed), instructor created videos (rs[9] = -.86, p <.05, two-tailed), and use of websites (rs[9] = -.67, p <.05, two-tailed). supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 69 figure 2 ease of access of blackboard lms components (n = 10) note. the graph displays the swd’s ease of access of blackboard lms components. instructor helpfulness with accommodations swd responded to a survey question about the degree to which their instructors helped provide accommodations in the remote courses. students responded using a likert-type scale of one to five, with one representing not helpful at all and five being very helpful. of the swd who disclosed and used accommodations (n = 4) in spring 2020 remote classes, 75% found their instructors very helpful, and 25% found them helpful in providing accommodations. of the four swd who used accommodations in their remote courses, 80% strongly agreed, and 20% agreed that accommodations helped them complete their remote courses. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 70 figure 3 helpfulness of media resources (n = 10) note. the graph illustrates the media resources that were helpful to swd in their remote courses. the data identified five key issues that swd experienced during the transition to remote learning, shown in figure 6: difficulty with testing services, lack of awareness of student accessibility services, lack of counseling and psychological services, lack of confidence in remote learning, and difficulty in communication and interaction. these five issues will be described in the next paragraphs. testing services swd indicated that 60% found it somewhat difficult or extremely difficult to take exams in blackboard. additionally, swd ranked their satisfaction with testing services in their remote courses using a likert scale of one to five, with one representing extremely dissatisfied and five being extremely satisfied. swd stated they were 20% extremely satisfied, 30% neither satisfied nor dissatisfied, supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 71 20% somewhat dissatisfied, 30% extremely dissatisfied with the college’s testing services, as shown in figure 6. out of the 50% of swd who were somewhat dissatisfied to extremely dissatisfied with testing services, 30% of them indicated that they had a mental health disorder. lastly, the data showed a statistically significant (rs[10] = .77, p < .05, two-tailed) positive correlation between satisfaction with tutoring services and satisfaction with testing services. over 50% of swd indicated not needing tutoring services during the transition to remote learning. awareness of student accessibility services overall, 40% of the responding swd disclosed their disability to the student accessibility services’ office. the disability types disclosed to sas were: neurocognitive difference (adhd and autism), chronic pain disorder, learning disability, and mental health and other disabilities. swd gauged their awareness of the services that sas offered to swd on a likert scale of one to five, with one representing not aware at all and five being extremely aware. as illustrated in figure 4, half of the participants were slightly aware or not aware at all of sas’ services for swd. swd responded to a survey question about disclosing their disability to sas; 60% of swd did not disclose their disability to the college. of the swd who did not disclose their disability, 83% had a mental health disorder. figure 4 awareness of student accessibility services note. the pie chart depicts the percentage of swds’ awareness of student accessibility services. exremely aware 40% very aware 10% slightly aware 10% moderately aware 10% not aware at all 30% supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 72 lack of counseling and psychological services swd responded to a survey question about the helpfulness of people/services they contacted or used during the transition to remote learning using a likert scale of zero to five, with zero representing did not use and five being very helpful. swd ranked the counseling and psychological services as 10% very helpful, 10% helpful, 20% undecided, 10% not helpful at all, 50% did not use. lack of confidence in remote learning swd responded to a survey question about their confidence level learning remotely using a likert scale of one to five, with one representing no confidence at all and five being complete confidence. overall, the top three challenges swd experienced were little confidence or no confidence at all in the following areas: organizing my online coursework (50%), taking notes during class instruction (30%), and finishing online homework assignments by deadlines (30%). these are depicted in figure 5. the spearman’s rho revealed a statistically significant relationship (rs[10] = .91, p < .001) between confidence in organizing online course work and confidence in finishing online homework assignments by deadlines. figure 5 confidence in remote learning note. the graph indicates swd’s level of confidence in remote learning. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 73 communication and interaction difficulty the survey included two questions concerning swd level of difficulty (as shown in figure 2) and degree of satisfaction of instructor and peer interaction (as shown in figure 6). as depicted in figure 2, 40% of swd had somewhat difficult or extremely difficult experiences communicating with faculty and classmates in their remote courses. swd ranked their satisfaction with faculty and peer interaction in their remote courses on a likert scale of one to five, with one representing extremely dissatisfied and five being extremely satisfied. fifty percent of swd identified as neither satisfied nor dissatisfied or extremely dissatisfied with faculty interaction. as noted in figure 6, swd’s degree of satisfaction with interacting with peers, fifty percent of swd rated their satisfaction as slightly dissatisfied (20%) or extremely dissatisfied (30%). figure 6 swd satisfaction with college services note. the graph identifies swd’s level of satisfaction with college services offered during the remote transition. the data found a statistically significant positive correlation between swd’s satisfaction with faculty interaction (rs[10] = .80, p <.05, two-tailed) and student supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 74 interaction (rs[10] = .64, p <.05, two-tailed) and satisfaction with technology services (blackboard collaborate, microsoft teams, and zoom) during the transition to remote learning. concurrently, the data identified a statistically significant (rs[10] = -.68, p <.05, two-tailed) negative correlation between the notice of transitioning to remote learning and satisfaction with faculty interaction. figure 7 satisfaction of technology services by disability type note. the graph depicts the satisfaction of technology services by disability type. a statistically significant positive correlation was found between type of disability and satisfaction with technology services (rs[10] = .99, p < .01). the data also showed a statistically significant (rs[10] = -.67, p < .05, two-tailed) negative correlation between the type of disability and satisfaction of tutoring support, as shown in figures 7 and 8. students with mental health disorders or a combination of mental health and other disorders reported 20% extremely dissatisfied with technology services and 10% extremely dissatisfied with tutoring services than other disability types. the data revealed five top support services and people that swd found very helpful to helpful during the transition to remote learning: 80% classmates, 70% friends, 70% faculty, 70% academic advising, and 60% financial aid services, as shown in figure 8. 0 0.5 1 1.5 2 2.5 3 3.5 mental health disorder learning disability neuro-cognitive difference (adhd and autism) chronic pain disorder chronic post traumatic stress disorder multiple disabilities mental health and other disorders extremely satisfied somewhat satisfied extremely dissatisfied supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 75 figure 8 satisfaction of tutoring support by disability type note. the graph presents the satisfaction of tutoring support by disability type. it is interesting to note the people and services that were ranked by respondents as not helpful at all or did not use [these] services offered by the community college included: student accessibility services (50% of swd), tutoring services (50%), student support services (40%), and counseling and psychological services (60%). swd responded to a survey question about their ease of access with blackboard’s course components. overall, swd’s five main challenges with using the blackboard lms were the submission of assignments, the organization within the course, communication with faculty and classmates, and taking online exams, as depicted in figure 10. the areas in blackboard that swd found helpful included their comfort level using the lms, ease of access to course grades, and the ability to access blackboard on any technology device. sixty percent of students ranked their comfort using the lms as extremely easy and 20% somewhat easy; 70% found it extremely easy to access the course grades, and 60% found it extremely easy to access blackboard on any technology device with 10% ranking it somewhat easy to do the same. what about the interview with the vice president for academic affairs mentioned in the abstract? if there is value in including the questions asked, then there is also value in at least summarizing the results/responses. alternately, if it is simply an supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 76 additional/follow-up sort of thing in the discussion, then maybe remove it as a focus in the abstract. figure 9 helpfulness of people/services the college offered during the transition to remote learning note. the graph displays the percentage of swd’s level of helpfulness for the people/services the college offered during the transition to remote learning. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 77 figure 10 five challenges within the lms note. the chart illustrates the five challenges swd experienced within the lms. discussion quantitative and qualitative data were collected to determine what factors contributed to swd’s positive and challenging experiences transitioning to remote learning in the spring of 2020 due to covid-19. the results of this research identified three areas in which swd experienced positive aspects of the transition to remote learning: ease of access with the blackboard lms, instructor use of varied media resources in courses, and instructor helpfulness with accommodations. swd’s ability to access blackboard with multiple technology devices and overall comfort with the lms was positively rated. before the remote transition occurred, the community college’s instructors provided technology training materials and tutorials to all students. the additional technology training offered to students might have attributed to swd’s comfort with the lms and online tools. in addition to faculty supporting students with blackboard and online tools, the community college made wi-fi and technology available to students who needed access during the remote transition. while swd did have positive experiences with blackboard, the research did identify five challenges that swd experienced during the transition to remote learning: awareness of student accessibility services, communication and interaction 0 2 4 6 8 10 12 submission of assignments organization of the course communication with faculty communication with classmates taking exams extremely easy somewhat easy neither easy nor difficult somewhat difficult extremely difficult supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 78 difficulty with instructors and classmates, testing services, lack of confidence in remote learning, and lack of counseling and psychological services. sas awareness according to stebleton et al. (2014), swd, especially students with mental health disorders, are not aware or have not heard of the disability support services that are offered by colleges. the research by stebleton et al. (2014) supports this study’s findings wherein the majority of swd who did not disclose to sas had a mental health disorder. the data also reported that half of swd at the community college did not use sas services. however, in the open-ended question asking swd to describe their experience with the office of accessibility services, most students who used sas services found their experience helpful and commented that “they [sas] have been wonderful to work with.” this researcher believes the sas awareness issue centers around the lack of communication about the services they offer and constitutes an area for further exploration and planned improvement. communication and interaction difficulty the data of this study indicate that communication and interaction with faculty and students was a challenge for swd in the transition to remote learning. the community college should expand its communication efforts about sas’s services through faculty and staff development, hiring an instructional designer, and promoting sas services through multiple media channels. testing services challenges swd expressed difficulty with taking blackboard (lms) tests, and half of the swd were dissatisfied with the college’s testing services. in support of this study’s findings, scott and aquino (2020) explained that more than half of swd experienced difficulty receiving online test accommodations at public colleges (2year and 4-year). in this researcher’s experience as an instructional designer, the transition to remote learning due to the pandemic created additional online testing barriers. during the transition to remote learning, instructors’ use of remote proctoring tools, such as proctorio, honorlock, and proctoru, for online tests increased significantly. remote proctoring services for online tests commonly record the computer screen and the student's webcam video while they take the exam. certain disability types may require exam accommodations where remote proctoring is not possible and the student requires a paper exam. lack of confidence in remote learning overall, swd experienced a lack of confidence in their learning remotely in organizing their online coursework, taking notes during class instruction, and finishing online homework assignments by deadlines. this lack of confidence could be reduced if instructors use udl principles to help swd with online learning organizational skills. faculty providing students with recorded videos, access to supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 79 powerpoint slides before synchronous lectures, and frequent communication through lms announcements may help the swd organization. lack of counseling and psychological services according to the literature, more than half of undergraduate and graduate students who stated having difficulty and a lack of confidence with the transition to remote learning screened positive for major depressive disorder and generalized anxiety disorder (horgos et al., 2020). this difficulty in transitioning and confidence in learning remotely was evident in the swd responses who had a diagnosed mental health disorder in this current study. the literature showed that 59% of swd experienced difficulty accessing counseling/mental health services during the transition to remote learning (scott & aquino, 2020). in an interview, the community college’s vice president for academic affairs (see appendix c) stated that some students, who had medical insurance, did use the counseling and psychological services they offered for a brief period until the covid-19 pandemic closed the campus. unfortunately, some students could not use mental health services due to a lack of medical insurance. this obstacle could be removed if future counseling services are free and offered via web conferencing and on campus to expand the community college’s outreach to students with mental health disorders. the results of this research identified that swd who had taken more than four courses before the remote transition were slightly more confident in finishing assignments by deadlines, organizing online coursework, and using the library to get information for assignments. the data showed no statistically significant differences between students forced to take online courses versus students currently enrolled in online classes. limitations the research included only one small rural two-year higher education institution. at the time of the transition to remote learning, only 15 students who had disclosed their disability to the college attended classes in the spring 2020 semester. knowing the disclosed swd population was small, this researcher had the community college send the entire student population the survey invitation. the researcher hoped to attract students with disabilities who did not disclose to the office of accessibility services who had either an individualized education program (iep) or a 504 plan in high school or a diagnosed disability. as a result, after two survey invitation reminders, 10 swd participated in the study. the small sample size of the research limits the study's overall generalizability. the swd who participated in the survey all declined to participate in a one-on-one interview with the researcher. while the open-ended questions in the survey helped elaborate swd’s responses, one-on-one interviews would have added a more indepth explanation of the remote transition’s effect on swd learning and added rich qualitative data to support quantitative results. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 80 this study’s last limitation was not interviewing the faculty who taught the remote courses and the student accessibility services office staff. interviewing both faculty and sas staff would provide a more holistic understanding of the college’s remote transition. recommendations for future practice at all higher education institutions in a study conducted by horgos et al. (2020), undergraduate, graduate, and professional students experienced substantial mental health difficulties during the transition to remote learning. based on this study’s findings, all higher education institutions should offer free counseling and psychological services on campus and online through web conferencing to expand their mental health awareness outreach. faculty could add statements about student mental health resources in their online courses and syllabi. additionally, this researcher suggests a faculty mentorship program be developed by sas and faculty supporting students with mental health disorders. faculty demonstrating support for swd needs will help “… de-stigmatize mental health disorders and increase students' comfort when reaching out for help” (horgos et al., 2020, p. 9). this researcher recommends that all institutions develop an instructional design and technology team (idt). idt can assist faculty in online course design, ensuring the accessibility of media resources, training lms tools for faculty and students, assisting faculty with designing testing alternatives to meet the needs of swd with testing accommodations, online pedagogy, and udl instruction techniques. lastly, this study found that all students agreed that accommodations helped them complete their remote courses. to that end, this researcher recommends that the office of disability services and instructional designers should partner to offer universal design for learning and accommodation laws workshops for faculty. it is recommended that the delivery of udl and accommodations training offered to the faculty are in multiple formats: asynchronous just-in-time training, synchronous web conferencing sessions, and face-to-face sessions. conclusion the pandemic affected all education globally; challenges about students learning remotely began to surface in the spring and summer months of 2020. students had to leave college campuses quickly and, for some, moving back home meant additional family responsibilities, inadequate internet connection and technology, and poor study environments. a challenge that affected most individuals during the pandemic is the feeling of isolation when social distancing. studies have shown that major depressive disorder diagnoses doubled in the summer of 2020 in graduate students (horgos et al., 2020). students no longer interacted with peers and instructors in class or extracurricular activities as they once did before the pandemic. as a result of the pandemic, the need for interaction and connection now is even more vital. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 81 this study discovered that swd found the connection of their friends, classmates, and instructors were a strong support system for learning remotely. the data indicate the need for all higher education institutions to support their faculty with professional development on mental health awareness, udl pedagogy, and building a collaborative and inclusive community in the classroom. lastly, this researcher also iinvites other scholars to contribute to the body of literature about the academic needs of swd in online learning; the technology to support online learning is ever-changing and affects swd in both challenging and supportive ways. conflicts of interest the author declares that there is no conflict of interest regarding the publication of this article. references alamri, a., & tyler-wood, t. (2017). factors affecting learners with disabilities– instructor interaction in online learning. journal of special education technology, 32(2), 59–69. https://doi.org/10.1177/0162643416681497 bacow, l. (2020, march 10). covid-19—moving classes online, other updates. harvard university. https://www.harvard.edu/covid-19-moving-classesonline-other-updates barnar-brak, l., lechtenberger, d., & lan, w. y. (2010). accommodation strategies of college students with disabilities. qualitative report, 15(2), 411–429. https://doi.org/10.46743/2160-3715/2010.1158 behling, k., & linder, k. e. (2017). collaborations between centers for teaching and learning and offices of disability services: current partnerships and perceived challenges. journal of postsecondary education and disability, 30(1), 5–15. https://eric.ed.gov/?id=ej1144608 bissonnette, l. a. (2006, april). teaching and learning at concordia university: meeting the evolving education needs of faculty in providing access for university students with disabilities (publication no. le 3 c66e38p 2006 b58) [doctoral dissertation, concordia university‒quebec, canada]. https://spectrum.library.concordia.ca/8741/ black, r. d., weinberg, l. a., & brodwin, m. g. (2015). universal design for learning and instruction: perspectives of students with disabilities in higher education. exceptionality education international, 25(2), 1‒26. https://doi.org/10.5206/eei.v25i2.7723 burke, l. (2020, march 9). colleges move online amid virus fears. inside higher ed news. https://www.insidehighered.com/news/2020/03/09/colleges-moveclasses-online-coronavirus-infects-more supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 82 chahrour, m., assi, s., bejjani, m., nasrallah, a. a., salhab, h., fares, m., & khachfe, h. h. (2020). a bibliometric analysis of covid-19 research activity: a call for increased output. cureus, 12(3), e7357. doi:10.7759/cureus.7357 couzens, d., poed, s., kataoka, m., brandon, a., hartley, j., & keen, d. (2015). support for students with hidden disabilities in universities: a case study. international journal of disability, development and education, 62(1), 24‒41. https://doi.org/10.1080/1034912x.2014.984592 custodio, j. (2020, april 7). disabled students already faced learning barriers. then coronavirus forced an abrupt shift to online classes. chronicle of higher education online 2.0. https://www.chronicle.com/article/disabled-studentsalready/248444 depietro, a. (2020, april 30). here’s a look at the impact of coronavirus (covid19) on colleges and universities in the u.s. forbes news. https://www.forbes.com/sites/andrewdepietro/2020/04/30/impactcoronavirus-covid-19-colleges-universities/#54aeee4461a6 field, k. (2020, march 26). 5 lessons from campuses that closed after natural disasters. chronicle of higher education online 2.0. https://www.chronicle.com/article/5-lessons-from-campuses-that/248346 griful-freixenet, j., struyven, k., verstichele, m., & andries, c. (2017). higher education students with disabilities speaking out: perceived barriers and opportunities of the universal design for learning framework. disability and society, 32(3), 1–23. https://doi.org/10.1080/09687599.2017.1365695 horgos, b., soria, k. m., chirikov, i., & jones-white, d. (2020, august 18). the impact of the covid-19 pandemic on students with emotional or mental health concerns or conditions. seru consortium, university of californiaberkley and university of minnesota. https://escholarship.org/uc/item/83m75056 individuals with disabilities education act of 2004, pub. l. no. 108-446, 20 u.s.c. §§ 1400 et seq. (2005). retrieved from http://www.wrightslaw.com/idea/idea.2004.all.pdf knight, w., wessel, r. d., & markle, l. (2016). persistence to graduation for students with disabilities: implications for performance-based outcomes. journal of college student retention: research, theory and practice, 19(4), 362–380. https://doi.org/10.1177/1521025116632534 kruse, a. k., & oswal, s. k. (2018). barriers to higher education for students with bipolar disorder: a critical social model perspective. social inclusion, 6(4), 194–206. https://doi.org/10.17645/si.v6i4.1682 supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 83 lederman, d. (2020, april 22). how teaching changed in the (forced) shift to remote learning. inside higher education news online. https://onlinelearningconsortium.org/news_item/how-teaching-changed-inthe-forced-shift-to-remote-learning/ lumina foundation. (2019). who is today’s student? https://www.luminafoundation.org/files/resources/todays-studentsummary.pdf meyer, k. a., & wilson, j. l. (2011). the role of online learning in the emergency plans of flagship institutions. online journal of distance learning administration, 14(1). https://ojdla.com/archive/spring141/meyer_wilson141.pdf moisey, s. d. (2004). students with disabilities in distance education: characteristics, course enrollment and completion, and support services. journal of distance education, 19(1), 73‒91. https://files.eric.ed.gov/fulltext/ej807840.pdf national center for education research (nces). (2019, december). postsecondary education (nces 2020-009, chapter 3). digest of education statistics: 2018. https://nces.ed.gov/programs/digest/d18/ch_3.asp oh, y., & lee, s. m. (2016). the effects of online interactions on the relationship between learning-related anxiety and intention to persist among e-learning students with visual impairment. international review of research in open and distributed learning, 17(6), 1–19. https://doi.org/10.19173/irrodl.v17i6.2581 perry, e. h., & pilati, m. l. (2011). online learning. new directions for teaching and learning, 2011(128), 95–104. https://doi.org/10.1002/tl.472 roberts, j. b., crittenden, l. a., & crittenden, j. c. (2011). students with disabilities and online learning: a cross-institutional study of perceived satisfaction with accessibility compliance and services. internet and higher education, 14(4), 242–250. https://doi.org/10.1016/j.iheduc.2011.05.004 sahu, p. (2020). closure of universities due to coronavirus disease 2019 (covid19): impact on education and mental health of students and academic staff. cureus, 12(4), e7541. doi: 10.7759/cureus.7541 salkind, n. j. (2010). encyclopedia of research design (1st ed.). sage. scott, s., & aquino, k. (2020). covid-19 transitions: higher education professionals’ perspectives on access barriers, services, and solutions for students with disabilities. https://higherlogicdownload.s3.amazonaws.com/ahead/38b602f4-ec53 supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 84 451c-9be0-5c0bf5d27c0a/uploadedimages/covid19_/ahead_covid_survey_report_barriers_and_resource_needs.pdf seaman, j. e., allen, i. e., & seaman, j. (2018). grade increase: tracking distance education in the united states. babson survey research group. https://www.onlinelearningsurvey.com/reports/gradeincrease.pdf snyder, t. d., de brey, c., & dillow, s. a. (2018, february). digest of education statistics: 2016 (nces 2017-094, 52nd ed.). national center for education statistics, institute of education sciences, u.s. department of education. https://nces.ed.gov/pubs2017/2017094.pdf stebleton, m. j., soria, k. m., & huesman, r. l. (2014). first-generation students’ sense of belonging, mental health, and use of counseling services at public research universities. journal of college counseling, 17(1), 6–20. https://doi.org/10.1002/j.2161-1882.2014.00044.x stewart, j. f., ii, mallery, c., & choi, j. (2010). a multilevel analysis of distance learning achievement: are college students with disabilities making the grade? journal of rehabilitation, 76(2), 27–39. https://doi.org/10.2190/cs.15.3.d zhang, w., wang, y., yang, l., & wang, c. (2020). suspending classes without stopping learning: china’s education emergency management policy in the covid-19 outbreak. journal of risk and financial management, 13(3), 55– 61. https://doi.org/10.3390/jrfm13030055 supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 85 appendix a invite letter to participants in survey dear prospective survey participant, my name is dana gullo. i am researching students with disabilities' (swd) experiences during the transition to remote learning during the spring 2020 semester due to covid-19 as part of my doctoral program at wilmington university. i intend to use the resulting data to help college professionals better understand swd experiences in online courses and would really like to hear your story. if you are a student who has a disability and/or had an individualized education program (iep) or 504 plan in high school and is interested in participating in the study, please complete the survey that will take approximately 10-15 minutes. your answers will be confidential. you will not be identified in any way during the research study. participation is voluntary, and you may decline from answering specific questions or withdraw from the study at any time without any penalty. below is a link to a survey to help me gather data for the study. as a gift for the first twenty participants who complete the survey will receive a fifteen-dollar amazon gift card. by completing the survey, you are providing informed consent to be a part of my research and consent for me to use your data and comments in my research report. additionally, if you wish to elaborate on your open-ended responses from the survey, please indicate in the survey that you are willing to be interviewed. lastly, in appreciation of the first five participants who volunteer to interview for the study will receive an amazon gift card of ten dollars. if you have any questions about this research, please contact dana gullo at dgullo002@my.wilmu.edu. the chair of my research project is dr. joanne damminger, who can be reached at joanne.k.damminger@wilmu.edu. this survey and study have been approved by wilmington university’s human subjects review committee that can be reached at humansubjectsresearch@wilmu.edu. thank you for considering your participation in this study. sincerely, dana gullo link to the survey: https://ycp.co1.qualtrics.com/jfe/form/sv_0bpef4pzhny251j supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 86 appendix b students with disabilities experiences during the transition to remote learning in spring 2020 survey thank you in advance for completing the survey. your responses to the survey will help us better understand students with disabilities’ experiences during the transition from face-to-face instruction to online-only classes during the spring 2020 semester due to covid-19. your responses to the survey will be confidential and no one will be identified in the study. you may withdraw from the study at any time. throughout this survey, the term "remote learning" will describe any face-toface class that transitioned to an online course taken during the spring 2020 semester. if you have any questions or concerns, please feel free to contact dana gullo at dgullo002@my.wilmu.edu or my dissertation chair, dr. joanne damminger joanne.k.damminger@wilmu.edu. this survey will take approximately 15-20 minutes to complete. by clicking to the next screen, you are providing consent to participate in this research. q1) i was given the following amount of notice that my courses would be transitioning to remote learning in the spring 2020 semester. 0-6 days 1 week 2 weeks 3 weeks other q2) the communication from the college about the transition to remote learning in the spring 2020 semester was (check all that apply) timely helpful clearly understood not helpful no communication q3) please rate the timeliness of the support(s) the college offered to help students with disabilities complete their courses during the transition to remote learning in the spring 2020 semester. just-in-time support (5), late support (4), i did not receive support until i reached out (3), didn’t need support (2), no support was offered (1) technical support in the use of blackboard chat function for immediate help academic advising support supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 87 faculty library student support services tutoring accessibility services testing services mentoring support financial aid information other q4) please indicate the number of online class(es) you had taken prior to the spring 2020 semester. (check all that apply) zero one two three four more than four enrolled in a fully online program q5) please rate your satisfaction with the following services during remote learning in the spring 2020 semester? extremely satisfied (5), somewhat satisfied (4), neither satisfied nor dissatisfied (3), somewhat dissatisfied (2), extremely dissatisfied (1) technology (blackboard collaborate, microsoft teams, zoom) faculty interaction library peer interaction blackboard student support academic advising accessibility services tutoring testing services financial aid mentoring other q6) please rate your level of confidence with the following during the remote learning in the spring 2020 semester: complete confidence (5), much confidence (4), some confidence (3), very little confidence (2), no confidence at all (1) i could finish online homework assignments by deadlines. use the library to get information for online class assignments. organize my online coursework. participate in class discussions. supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 88 take notes during class instruction. q7) please elaborate on your reasons for your selection in the previous question. q8) how helpful were the following people/services to you during the transition to remote learning in the spring 2020 semester? very helpful (5), helpful (4), undecided (3), slightly helpful (2), not helpful at all (1), did not use (0) friends classmates faculty significant other parents siblings other family members student support services accessibility services tutoring support financial aid department admissions academic advising counseling and psychological services special support program(s) (please name in “other” below) other q9) please rate the ease of your access to each of the following course components in blackboard during remote learning in the spring 2020: extremely easy (5), somewhat easy (4), neither easy nor difficult (3), somewhat difficult (2), extremely difficult (1) organization of the course communication with faculty submission of assignments communication with classmates access to course grades comfort using blackboard access on any technology devices (laptop, tablet, smartphone, ipad) taking exams other/comment q10) did you experience any difficulty reading any of the following file types? (check all that apply) word document powerpoint presentation google document excel spreadsheet pdf file supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 89 other q11) please elaborate on your reason for your selection(s) in the previous question. q12) please indicate any of the following media resources you used in your spring 2020 courses. (check all that apply) websites (links to academic articles, tedtalks, etc.) youtube videos instructor recorded videos podcasts simulations voicethread padlet other (explain) none q13) please rate your experience using the media resources you selected in the previous question. very good (5), good (4), acceptable (3), poor (2), very poor (1), did not use (0) youtube videos instructor recorded videos podcasts simulations voicethread padlet websites (links to academic articles, tedtalks, etc.) q14) please rate the helpfulness of the following materials to your remote learning in the spring 2020 semester. very helpful (5), helpful (4), undecided (3), slightly helpful (2), not helpful at all (1), did not use (0) youtube videos images instructor created videos podcasts websites (links to academic articles, tedtalks, etc.) textbook readings q15) did you have an individualized education program (iep) or 504 plan in high school? yes no q16) have you been diagnosed with any disability or impairment? yes no prefer not to respond supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 90 q17) how do you describe your disability/ability status? (check all that apply) i do not have any disability or impairment a sensory impairment (vision or hearing) a mobility impairment a learning disability (eg. dyslexia) neuro-cognitive difference (adhd and autism) a mental health disorder a chronic pain disorder a disability or impairment not listed here other prefer not to respond q18) how aware are you of the services offered to students with disabilities at the office of accessibility services? extremely aware (5), very aware (4), moderately aware (3), slightly aware (2), not aware at all (1) awareness of the office of accessibility services? q19) did you disclose a disability to the office of accessibility services? yes no prefer not to respond q20) do you require accommodations for your classes? yes no q21) why did you not disclose your disability to the office of accessibility services? (please explain) q22) describe your experience with the office of accessibility services. q23) did you use any of the following accommodations in your remote courses in the spring 2020 semester. (check all that apply) extra time on tests extra time on assignments taking an exam in alternative formats assistive listening devices a quiet room for test taking screen reader sign language interpreters note taker tape recorder other (explain) did not use accommodations supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 91 q24) please indicate the level of helpfulness of your instructor providing accommodations to your remote course. very helpful (5), helpful (4), undecided (3), slightly helpful (2), not helpful at all (1) in the spring 2020 semester, please rate the helpfulness of your instructor(s) in providing accommodations in your remote classes. q25) please indicate your level of agreement or disagreement with the following statement strongly agree (5), agree (4), undecided (3), disagree (2), strongly disagree (1) in the spring 2020 semester, the accommodations i used in my online class(es) helped me to complete my courses. q26) what is your gender? male female other q27) what is your age? 18-20 21-29 30-39 40-49 50-59 60+ q28) what is your race/ethnicity? african american asian bi-racial caucasian hispanic latin american native american other q29) are you interested in a zoom interview to discuss your online experiences in more detail? yes no q30) if you are interested in a zoom interview to discuss your online experiences in more detail, please leave your name and email address in the fields below. name email address supporting students with disabilities journal of effective teaching in higher education, vol. 5, no. 2 92 appendix c interview of the vice president for academic affairs at the community college hello, thank you for your willingness to meet with me today. i would like to ask you a few questions to get more detail about the timeline and transition to remote learning at the community college. i will use the information to learn how colleges can better support students, including those with disabilities, to succeed in online courses. you don’t have to answer any question if you would prefer not to, and we can stop the interview at any time if you request to do so. do you have any questions before we start? interview questions 1. can you talk about the decision to move to remote learning and describe the timeline to go fully online? 2. how did you communicate to students and faculty about the remote transition? what tools did you use to communicate? which works best, in your opinion? 3. how did the college prepare the students for online learning? 4. how did the college prepare the faculty for online learning? is there a required training for online learning? 5. how many faculty members previously taught online? 6. name three key steps the college implemented to make the transition to remote learning successful? 7. what would you do differently in preparing for the next crisis? microsoft word dandotkar_final.docx journal of effective teaching in higher education, vol. 5, no. 2 https://doi.org/10.36021/jethe.v5i2.323 knowing, thinking, and learning: fostering critical thinking in undergraduate psychology classes srikanth dandotkar, university of southern indiana, sri.dandotkar@usi.edu laura cruz, the pennsylvania state university m. anne britt, northern illinois university abstract. we examined the relationship between the levels of sophistication (highsophisticated and low-sophisticated) of students’ domain general epistemic beliefs and an important component of students’ critical thinking skills—their ability to evaluate arguments. participants evaluated arguments and took an epistemic belief survey before recalling arguments in a surprise recall task. our findings suggest that students’ general beliefs about the speed of knowledge acquisition predicted how well they evaluated arguments and their memory for critical argument elements (i.e., claim-predicates). implications of this connection between argument analysis and epistemic beliefs in the context of improving students’ critical thinking skills are discussed. keywords: critical thinking; epistemic beliefs; argument evaluation; argumentation; psychology class introduction critical thinking remains one of the primary learning outcomes ascribed to higher education; a position the skill has held for decades. as the basic argument goes, the content knowledge students may gain in college is likely to either dissipate or become surpassed in a rapidly evolving post-graduation world. critical thinking, on the other hand, is frequently framed by both academics and employers as a persistent skill that can be applied across multiple contexts to solve a multitude of future problems (bellaera, 2021). while the preceding statement about the value of critical thinking is shared by many, the articulation of precisely which skills, dispositions, and other attributes comprise the skill of “critical thinking” is highly contested and the subject of a robust line of scholarly inquiry (davies, 2015). similarly, communication scholars have identified a number of frameworks for argument analysis and development that, when provided to students, may influence their beliefs about the complexity of knowledge (gaipa, 2004; seiter & gas, 2007; walton, 2006; weston, 2018). the shaky construct validity of the term “critical thinking” has contributed to assessment challenges that have constrained the development of shared, evidencebased pedagogical practices designed to turn college students into proficient critical thinkers (behar-horenstein & niu, 2011). despite these limitations, some progress has been made in developing consensus (bellaera, 2021). in 2009, for example, the american association of colleges and universities (aac&u) issued a crossinstitutional rubric for critical thinking as one of sixteen fundamental learning outcomes in higher education. this widely used value rubric identified five essential components of critical thinking, including the evaluation of evidence and knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 124 the analysis of argument(s) and position(s) (aac&u, 2009). similarly, in 2021, british scholar lauren bellaera interviewed 176 faculty in the humanities and social sciences and determined that analysis and evaluation were consistently identified as prominent components of critical thinking (bellaera et al., 2021). while the considerable scholarly debates surrounding critical thinking have produced limited consensus, with areas of shared agreement, such as valuing “the skill of recognizing and constructing arguments—i.e., critical thinking as reasoning skills,” and a host of others that are more disputed (davies, 2015). the present study seeks to gain insight into factors that contribute to students’ abilities to engage in critical thinking, with particular emphasis on the mitigating role of epistemic beliefs. literature review as the example in the previous paragraph illustrates, many of the areas of consensus related to critical thinking focus on the construct as a cognitive skill or set of skills. others have noted that cognitive processes do not occur in vacuum, and successful critical thinkers also possess an identifiable set of dispositions, or frames of mind, about a subject, including attributes such as curiosity, respect for the viewpoints of others, open-mindedness, fairness (ethical), and the desire to engage in critical thinking (davies, 2015). a 2010 australian study comparing students (n = 26) and faculty (n = 21) members’ beliefs indicated high levels of agreement on the significance of both skills and dispositions in critical thinking, though there was less agreement on which specific skills and dispositions were the most salient (bahr, 2010). a review of the broader literature leads to similar conclusions. many of the previously mentioned studies and practices operate under the assumption that critical thinking is a generalized, even generalizable, set of skills and dispositions, which can be taught both within and independent of the disciplinary context. that said, there remains considerable scholarly debate related to the influence of disciplinary context and the degree to which specific disciplinary domains may produce stronger critical thinkers than others (moore, 2004; moore 2011; wang, 2017). part of the argument in favor of discipline-specific critical thinking points to differences in epistemology (e.g., theories of knowing) across disciplinary domains, but this distinction is complicated by research on student epistemic beliefs (e.g., ways of knowing). a 2005 study by schommer-aikins et al. (2005), for example, found that higher levels of academic performance could be similarly predicted by both domain-general and domain-specific epistemic beliefs. in other words, what students believe about how knowledge is acquired and constructed (i.e., epistemic beliefs) is a significant factor in determining their ability to engage in critical thinking, regardless of discipline. schommer et al. (1997) further advanced the concept of epistemic beliefs to identify distinct domains (e.g., structure of knowledge, speed of knowledge acquisition) and levels (sophisticated and unsophisticated). educational psychologists have recognized that dispositions and epistemic beliefs, while certainly not the same, often have a symbiotic relationship, and that they demonstrate “similar patterns of relationships to other constructs” (kardash & knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 125 sinatra, 2003; schommer-aikins & easter, 2006). those “other constructs” include critical thinking. several seminal studies established an early link between sophisticated, or advanced, epistemic beliefs and success with critical thinking (kurfiss, 1988), though the exact nature of that link remains disputed across disciplinary contexts (jones & merritt, 1999). it even seems likely that these linkages persist across cultural contexts which affect epistemic beliefs. in 2011, for example, two linked experiments affirmed that both epistemic belief and cognitive disposition strongly influenced the ability of chinese students (n = 111 and 138, respectively) to engage broadly in critical thinking (chan et al., 2011). the first study established a base line for this population using pre-existing surveys, but for the second study, the researchers used an argument evaluation task, in this case, participants were given a passage describing a controversial subject, as a proxy for critical thinking. psychologists have studied this skill independently from critical thinking, noting that a student’s ability to analyze an argument may be sensitive to differences in reading ability, vocabulary, and a host of other factors (britt et al., 2016). to date, however, there has been comparatively little research that has explored the relationship between students’ epistemic beliefs and argument evaluation, irrespective of whether the latter stands as a proxy for critical thinking. given that the link between critical thinking and epistemic belief has been well-established in prior literature, there may be a similar link between epistemic belief and argument evaluation. if so, then the existence of such a link may provide deeper insights into how instructors can strengthen the critical thinking of their students. the present study was designed to test this hypothesis. before providing the details of the current study, we would like to define important constructs specific to our study, namely, epistemic beliefs and argument evaluation. epistemic beliefs refer to one’s beliefs about knowledge (hofer & pintrich, 1997; schommer, 1990) and knowing (schommer, 1990). while epistemic beliefs could be about any topic (e.g., climate change) or domain (e.g., science), we are particularly interested in students’ domain-general epistemic beliefs as conceptualized by schommer (1990). according to schommer (1990), epistemic belief is a system of relatively independent beliefs (of an individual) pertinent to knowledge and how it is acquired. schommer has identified five such independent beliefs (listed in table 1) that comprise the epistemic belief system, including one’s knowledge and knowingspecific beliefs about speed (i.e., how quickly or slowly one acquires knowledge), structure (i.e., how complex, or simple is the structure of knowledge), construction and modification (i.e., how one creates new knowledge or modifies the existing one), successful student (what makes one successful in acquiring knowledge and if such characteristics can be acquired or not), and objectivity (whether the truth value of the knowledge is certain or probable). students’ beliefs specific to each of these components, according to schommer, exist on a continuum of sophistication with unsophisticated views on one end and sophisticated views on the other. table 1 shows what constitutes these extreme views specific to five independent beliefs as conceptualized by schommer (see table 1, which is copied from dandotkar et al., 2022). knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 126 table 1 general epistemic beliefs on a continuum of sophistication beliefs unsophisticated view sophisticated view b1. speed of knowledgeacquisition-ksp quick learning slow learning b2. structure of knowledge-kst simple knowledge complex-knowledge b3. knowledge construction & modification-kcm passive-learning active learning by questioning b4. meaning of successful students-ss innate fixed ability acquired incremental ability b5. obtaining objective truth-ot certainty-knowledge probabilistic knowledge note. based on schommer (1990) as presented in dandotkar et al. (2022). schommer (1990) has developed a questionnaire, the epistemic belief survey, to measure individuals’ levels of sophistication pertinent to each of these five epistemic belief components. in our study, we have utilized a condensed version of this epistemic belief survey (wood & kardash, 2002), the details of which are provided in the method section. in this study, we have conceptualized the ability of students to engage in argument evaluation as a proxy for critical thinking skill (bellaera et al., 2021). argument evaluation is an individual’s ability to evaluate simple two-clause arguments (see arguments 1 and 2 below labeled as a1 and a2). simple arguments (e.g., arguments 1 and 2) have a claim (i.e., banks shouldn’t charge atm fees) and at least a reason (i.e., because the fees make their customers unhappy in a1) that is relevant and minimally sufficient in supporting the claim for the argument to be acceptable or warranted (johnson & blair, 1977; toulmin, 1958; voss & means, 1991). whether or not a reason is relevant to the claim depends on how the main verb or verb-phrase (i.e., predicate) of the claim (e.g., “shouldn’t charge atm fees” in a1 & a2) is related to the reason. the predicate for a1 and a2 is the same, “shouldn’t charge atm fees,” and it is related to the reason in a1 (because the fees make their customers unhappy) but not the one in a2 (because banks are financial institutions). therefore, a1 is a minimally acceptable argument whereas a2 is not. a1. banks shouldn’t charge atm fees because the fees make their customers unhappy. a2. banks shouldn’t charge atm fees because banks are financial institutions. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 127 evaluating arguments such as a1 and a2 requires students to examine whether the reasons are relevant and minimally sufficient to support the claim (angell, 1964; johnson & blair, 1977; freeman, 1991; salmon, 1984; toulmin, 1958; voss & means, 1991). this ability is contingent on students’ memory for the precise nature of the predicate (e.g., “shouldn’t charge atm fees” in a1 & a2). in fact, research has found that students who remember the predicates of argument claims also evaluate arguments better than those whose memory for the predicates is poor. this was found both when students recalled the argument immediately after they evaluated it (britt et al., 2007) and after a delay (dandotkar et al., 2016). our study the present study examined the relationship between students’ epistemic beliefs and their ability to evaluate simple arguments. for the study, participants completed three sequential tasks: an on-line argument evaluation task (i.e., flawed judgment task), in which students identified flawed arguments; an electronic survey related to their epistemic beliefs; and a surprise recall task, where participants recalled the arguments they had evaluated earlier. the students’ scores on the first task (argument evaluation as measured by the flawed judgement task) served as the dependent measure while their scores on the third task (surprise recall task of the argument predicates) served as a covariate. the level of participants’ epistemic sophistication, drawn from the second task (epistemic belief survey), served as the critical independent variable. of the five epistemic dimensions that the epistemic belief survey is designed to capture, some dimensions are considered to be knowledge-related and some knowing-related (hofer & pintrich, 1997; schommer et al., 1997). for instance, epistemic dimension pertinent to the speed of knowledge acquisition-ksp can be viewed as knowing-related while the dimension specific to the structure of knowledge-kst can be viewed as knowledge-related. as per the knowledge-only hypothesis, we predicted that students with high-sophisticated views about the structure of knowledge-kst beliefs would evaluate arguments more accurately than students with low-sophisticated structure of knowledge beliefs. as per the knowingonly hypothesis, on the other hand, we predicted that students with highsophisticated views about the speed of knowledge acquisition-ksp beliefs would evaluate arguments more accurately than those with low-sophisticated beliefs. these hypotheses are based on the previous findings pertinent to ksp and kst beliefs (schommer, 1993; schommer et al., 1997). methods participants after receiving approval from the institutional review board, 127 undergraduate students were recruited from lower (intro: n = 75) and upper level (cognition: n = 44) psychology classes at a midwestern university. students received course credit for their voluntary participation. as per the irb rules, students were not penalized for their non-participation or for refraining from continuing their participation. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 128 because the data were collected online, students were given clear instructions about exiting the study. data from 12 participants were dropped because of incomplete responses, making data from 115 participants included in the final analysis. as requested in the approved irb application, students’ demographic information was only used to present aggregates, without identifying individuals. participants included 71 females and 44 males with an average age of 20.14 years. students’ year in school was as follows: 51 first-year, 24 second-year, 30 thirdyear, and 10 fourth-year. tasks and instruments flawed judgment task (fj task). this task involved students evaluating simple two-clause arguments (e.g., a1 and a2) (britt et al., 2007). to measure students’ argument evaluation skills, we deployed a widely used argumentation task, the flawed judgment task (britt et al., 2007; britt et al., 2016; dandotkar et al., 2016; larson et al., 2009). in this task, participants read 36 arguments presented one-ata-time on the computer and rated whether each argument was flawed (f) or acceptable (ok) based only on its structure. students’ performance on the flawedjudgment task, as measured by argument evaluation scores, served as the dependent measure. surprise recall task. to control for memory for claim-predicates specific to the arguments that students had evaluated as part of the flawed-judgment task, a surprise recall task was devised where students—after taking the epistemic belief survey that we will elucidate next—were asked to recall the arguments they had evaluated before based on a topic prompt (e.g., “banks” for a1 and a2). epistemic belief survey (ebs). as stated earlier, we measured participants’ level of epistemic sophistication pertinent to schommer’s (1990) five components with the help of a condensed version of schommer’s (1990) epistemic belief survey (wood & kardash, 2002). please see dandotkar et al. (2022) for a basic but elaborative account of the survey, how it is measured, and how a dimension-reducing technique (i.e., principal component analysis) is used to identify items from the survey that sufficiently measure the epistemic dimensions unique to the sample. the condensed version of the epistemic belief survey itself is widely viewed as reliable, and it has consistently demonstrated sound psychometric properties (wood & kardash, 2002). the 38 statements about knowledge and its acquisition (e.g., “you can believe most things you read”) in the survey that students read had agreement rating options. students rated their agreement with each statement on a 5-point likert-type scale (1: strongly disagree; 5: strongly agree). these survey items captured students’ beliefs about knowledge and knowing along five dimensions listed in table 1. a complete list of the survey items is presented in appendix a. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 129 procedure the study was conducted online using the qualtrics platform. participants received a link when they signed up for the study (either through sona-platform or from their instructors) and performed the following tasks in that order after they voluntarily consented using the online consent form that they saw first when they clicked on the study link. participants first read the 36 arguments (as a part of the flawed judgment task) one-at-a-time on the computer and rated whether each argument was flawed (f) or acceptable (ok) based only on its structure and not on what they believed about it. then, participants completed the domain-general epistemic belief survey. as stated earlier, participants indicated the extent to which they agreed with each item on a 5-point likert-scale (1: strongly disagree; 5: strongly agree). finally, participants received cues from the first task (flawed judgment task) and were asked to recall the arguments they had evaluated earlier as a part of the surprise task; they were not informed about the need to remember the arguments ahead of time. participants were instructed to recall the arguments as completely and accurately as possible. furthermore, participants were asked not to leave any item blank, essentially forcing them to make the best possible guess. measures, preliminary data analyses, and design dependent measure. students’ performance on the flawed judgment task was used to compute this measure. for each participant and condition, average proportion of accurately evaluated arguments was computed, which were arcsine transformed to get participants’ flawed-judgment scores (kirk, 1982) that served as the critical dependent measure. covariate measure. participants’ responses from the surprise recall task were used to calculate this measure. towards the end of the study, participants were prompted with key words from each argument (e.g., “banks” for a1 and a2) that they had evaluated in the flawed judgment task and were asked to recall the arguments in as close to the exact wording as possible. participants’ responses were coded for accuracy of claim-predicates. only an exact matching predicate response (e.g., “should be abolished”) or its synonym (e.g., “must be abolished” or “should be removed” or “must be removed”) was coded as accurate, and the rest were coded as inaccurate. for each participant and condition, proportions of accurately recalled predicates were computed and were arcsine transformed (kirk, 1982) to obtain each participant’s predicate-recall score, which served as a covariate in the analyses reported in the primary findings part of the results section. it may appear as if the recall measure used does not directly address the relationship between epistemic beliefs and argument evaluation. it is important to note that the relationship between memory for claim-predicates and an individual’s argument evaluation ability has been established (britt et al., 2007; dandotkar et al., 2016). the current study attempts to control the mediating role that participants’ memory for claim-predicates plays in how they evaluate arguments. accordingly, participants’ performance specific to memory for claim-predicates, knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 130 which is elucidated in the methods section, is used as a covariate. using participants’ memory performance pertinent to claim-predicates allows us to isolate its effects and test the unique effect that epistemic belief has on a participant’s argument evaluation performance, over and above the established effect of memory for claim-predicates on argument evaluation. independent measure. students’ responses to items on the epistemic belief survey were utilized to identify participants as high or low sophisticated students pertinent to each of the identified epistemic dimensions. in other words, each participant was identified either as having a high-sophisticated or low-sophisticated view related to each epistemic dimension. although the survey itself was meant to capture five epistemic dimensions (i.e., structure of knowledge-kst, speed of knowledge acquisition-ksp, knowledge construction and modification-kcm, meaning of successful students-ss, & attainability of objective truth-ot), preliminary analysis usually yields fewer dimensions than the original number (i.e., five). this is because we used stringent measures to validate the items in the survey by conducting a preliminary principal component analysis (pca). using such dimension reducing analysis as pca allows researchers to systematically identify the survey items that uniquely measure a particular epistemic dimension. further, pca analysis allows researchers to weed out the items that do not uniquely contribute to measuring an epistemic dimension. consequently, only those epistemic dimensions that are unique to the sample are captured by the survey. in other words, pca usually results in capturing fewer than expected epistemic dimensions, which makes the selected dimensions (because of the selected items) uniquely reliable and valid to the study sample. details of this preliminary data analysis are presented next. for each of the identified epistemic dimensions, participants would be grouped (or identified) into sophisticated or unsophisticated believers using a tertiary split. only those participants’ data for a given epistemic dimension that fell in the upper and lower one-third of the epistemic sophistication score would be included in the final analysis (reported in the results section). participants’ level of sophistication (high or low) specific to each of the identified epistemic-dimensions would serve as the between-subjects independent measure. preliminary data analysis. the epistemic belief survey had 38 statements about knowledge and knowing that students rated their agreement with on a 5-point likert scale. out of these 38 items, we removed items that were unrelated (<1.5) or negatively related to other items in a given dimension, as recommended by ferguson et al. (2013). this resulted in 20 remaining items. an initial principal component analysis with oblique rotation yielded three factors—speed of knowledge acquisition-ksp, structure of knowledge-kst, and knowledge construction and modification-kcm—that met the kaisser-guttman retention criteria of eigenvalues greater than unity. further, we removed one item because it loaded on two factors significantly. the final 12 items loaded on to three factors (speed of knowledge acquisition-ksp, structure of knowledge-kst, & knowledge construction & modification-kcm) which met the kaisser-guttman retention criteria of eigenvalues greater than unity and explained 48.49% of sample-variation. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 131 because the pca yielded only three epistemic dimensions that are unique to our sample, the current study is restricted to examining the relationship between students’ domain-general epistemic beliefs pertinent only to these three epistemic dimensions (i.e., speed of knowledge acquisition-ksp, structure of knowledgekst, & knowledge construction & modification-kcm) with students’ critical thinking skills as measured by the flawed-judgment task. design. we adopted a between-participants design with the level of epistemic sophistication (high-sophisticated or low-sophisticated) as the between-subjects factor and predicate-recall score as the between-subjects covariate. results primary findings three ancovas—one each for the speed of knowledge acquisition-ksp, structure of knowledge-kst, & knowledge construction and modification-kcm—were conducted on participants’ flawed-judgment scores with epistemic-sophistication level (high or low) as a between-subject factor and predicate-recall score as a covariate. for each dimension, high and low scored participants were identified based on a tertiary split. there was a main effect of speed of knowledge acquisition-ksp, f(1,77) = 5.9, p = .017, ƞp2 = .07. no other findings were significant (all fs<2). table 2 shows participants’ argument evaluation scores for each of the three identified dimensions as a function of participants’ level of epistemic sophistication (high-sophisticated or low-sophisticated). these results suggest that students who had high-sophisticated views about the speed of knowledge acquisition-ksp evaluated arguments significantly better than those who had low-sophisticated views about it. this effect was found after controlling for students’ memory for claim-predicates. table 2 proportion of accurately evaluated arguments (arscined) with standard errors as a function of epistemic belief dimension and its level (low or high) level of epistemic sophistication epistemic belief dimension speed of knowledge acquisition structure of knowledge knowledge construction & modification high-sophisticated .77 (.02) .75 (.03) .71 (.02) knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 132 low-sophisticated) .66 (.02) .70 (.02) .73 (.03) additional findings to explore which epistemic dimension predicted students’ memory for claimpredicates, we ran three separate independent-samples t-tests on participants’ predicate-recall scores, one for each of the identified epistemic dimensions (i.e., speed of knowledge acquisition-ksp, structure of knowledge-kst, and knowledge construction and modification-kcm). the effect of epistemic sophistication on participants’ memory for claim-predicates was found for the speed of knowledge acquisition-ksp dimension alone, t(78) = 2.78, p = .007. students who had highsophisticated views about the speed of knowledge acquisition also recalled claimpredicates more often than those who had low-sophisticated views. this finding is important because it corroborates the importance of students’ views about speed of knowledge acquisition. students’ views about the speed of knowledge acquisitionksp not only predicted students’ argument-evaluation performance (as reported in the main results), but it also significantly predicted students’ memory for claimpredicates (as reported here). this shows the potentially important role students’ beliefs about speed of knowledge acquisition-ksp plays in determining students’ critical thinking skills. epistemic sophistication was not significant in predicting participants’ memory for claim-predicate when it comes to the structure of knowledge-kst or knowledge construction and modification-kcm (t<2) dimensions. table 3 shows average predicate-recall scores as a function of epistemic sophistication related to three epistemic dimensions. table 3 predicate recall score (arscined) with standard errors as a function of epistemic belief dimension and epistemic sophistication level of epistemic sophistication epistemic belief dimension speed of knowledge acquisition structure of knowledge knowledge construction & modification high-sophisticated .82 (.02) .84 (.04) .78 (.03) low-sophisticated .68 (.05) .72 (.04) .83 (.03) knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 133 discussion in this study, we examined the effects of domain-general epistemic beliefs—speed of knowledge acquisition-ksp, knowledge construction and modification-kcm, and structure of knowledge-kst—on students’ evaluation of informal arguments. knowledge-only hypothesis predicted that students with a sophisticated belief about the structure of knowledge-kst as complex would also evaluate arguments more accurately than those held the more basic belief that it is a simple list of facts. in contrast, the knowing-only hypothesis predicted that students with complex belief about the speed of knowledge acquisition-ksp as a slow, time-consuming process would evaluate arguments more accurately than those who held a simpler belief that acquiring knowledge is a quick process. the current study findings support the knowing-only hypothesis. as predicted, students with complex views about speed of knowledge acquisition-ksp as a slow and time-consuming process also evaluated arguments more accurately than their counterparts who held simpler views. this was found after controlling for participants’ predicate recall skill. on the other hand, the knowledge-only hypothesis was not supported. there was no significant difference in the first task performance between students with complex and simpler views about the structure of knowledge. even though no specific predictions were made pertinent to construction and modification of knowledge-kcm beliefs, we examined its effect and found that students with simpler and complex views about construction and modification of knowledge-kcm did not differ in their argument-evaluation performance. the current study findings extend previous findings (schommer-aikins et al., 2005; schommer et al., 1997) about the importance of the speed of knowledge acquisition-ksp beliefs in academic performance. schommer et al. (1997), for instance, found that the gpas of students with complex beliefs about the speed of knowledge acquisition-ksp was higher than those who had simpler views. the current study found that having complex views about the speed of knowledge acquisition-ksp also facilitates students’ evaluation of arguments. in fact, these findings confirm the notion that individuals’ beliefs about knowing play an important role in argument evaluative tasks and, by extension, critical thinking. implications for teaching and learning practice the present study does not assess the impact of a particular teaching intervention. in other words, we do not seek to answer a “what works?” question from pat hutchings’s well-known taxonomy of research questions in the scholarship of teaching and learning (2000). rather, we ask a “what is” question, intended to shed insight into a known bottleneck in student learning, in this case the development of critical thinking skills, so that future researchers and practitioners may develop effective interventions to address it. our findings suggest that epistemic beliefs, especially about the complexity of knowledge and the speed by which it is acquired, may have a strong influence on how students engage in key parts of critical thinking, especially argument analysis. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 134 given the insights gained from this study, instructors may wish to pay increased attention to epistemic beliefs as a critical component of how their students engage (or do not engage) in critical thinking. there is some evidence that relatively shortterm interventions, such as directed reflective writing, may bolster the sophistication of students’ epistemic beliefs, at least in the social sciences (dandotkar et al., 2022). further, emerging research on undergraduate research practices suggests that participation in the creation of knowledge through mentored research, may also positively influence, and be influenced by, epistemic beliefs, especially in a stem context (lopatto et al., 2022; reed, 2022). while comparatively less work has been conducted on epistemic beliefs in other domains, ongoing psychology research suggests that epistemic beliefs may be sensitive to both disciplinary and broader, culturally-generated beliefs about knowledge (merk et al., 2018). in other words, instructors will need to recognize that students will hold a variety of epistemic beliefs, which necessitates teaching strategies that are inclusive of this range. from a transdisciplinary perspective, scholars posit epistemic beliefs as a companion to the larger process of metacognitive awareness, or how students navigate the pathway towards becoming self-directed learners (mason & bromme, 2010; muis, 2007) and for which a wide range of validated interventions have been identified. these may include practices such as exam or assignment wrappers (e.g., structured reflections on performance), concept mapping as a study strategy (e.g., connecting course concepts visually), or teach-backs (e.g., paired discussions of how problems are solved). these practices may influence beliefs about how knowledge is acquired, perhaps especially if students are prompted to make these connections (ku & ho, 2010). the effects of those beliefs may not be limited to the students. a recent line of inquiry focuses on the epistemic beliefs of instructors, for example, and how these beliefs may influence how their students learn skills such as critical thinking (fuesting et al., 2019; maggioni & parkinson, 2008; muenks et al., 2021). these insights suggest that instructors may wish to engage in reflective and reflexive practice regarding their own epistemic beliefs and the influence these may have on the learning that takes place in their classrooms. it is possible that short-term interventions, whether a single assignment or even a single semester, may be insufficient to change some of the fundamental beliefs about knowledge held by students. scholars of critical thinking have suggested that the associated skills and dispositions take multiple iterations of practice, over considerable periods of time, to promote measurable changes in a student's abilities. epistemic beliefs may take even longer to develop, which means that potential interventions would not be undertaken at the level of the single classroom but across the curriculum, perhaps even extending to encompass primary and secondary instruction, in one direction, and graduate education, the other. this longer time period may be necessary to hone the integration of an increasing number of components that scholars keep identifying as integral to the critical knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 135 thinking process. research on critical thinking has suggested that the process includes first skills, then dispositions, and finally actions. this study suggests that there may even be a fourth component to take into consideration i.e., epistemic beliefs. if a student believes that you can learn everything there is to learn about a subject in a single setting, that student will not see the value in taking the time to learn critical thinking skills. if a student believes in singular truths, that student will not see the value in being open to multiple points of view, a key disposition in the critical thinking process. indeed, if a reader is not willing to believe that the acquisition of learning is a super-complex process to be illuminated rather than solved, then that reader may not be willing to consider the multitude of ways in which what we know, as both learners and teachers, how we think, and why we learn are intertwined. conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references angell, r. b. (1964). reasoning and logic. appleton-century-crofts. association of american colleges and universities (aac&u). (2009). critical thinking value rubric. https://www.aacu.org/initiatives/valueinitiative/value-rubrics/value-rubrics-critical-thinking bahr, n. (2010). thinking critically about critical thinking in higher education. international journal for the scholarship of teaching and learning, 4(2). https://doi.org/10.20429/ijsotl.2010.040209 behar-horenstein, l. s., & niu, l. (2011). teaching critical thinking skills in higher education: a review of the literature. journal of college teaching & learning, 8(2). https://doi.org/10.19030/tlc.v8i2.3554 bellaera, l. (2021, june 18). making connections that matter. liberal education. https://www.aacu.org/liberaleducation/articles/making-connections-thatmatter-critical-thinking-in-theory-and-practice bellaera, l., weinstein-jones, y., ilie, s., & baker, s. t. (2021). critical thinking in practice: the priorities and practices of instructors teaching in higher education. thinking skills and creativity, 41, 100856. https://doi.org/10.1016/j.tsc.2021.100856 britt, m. a., kopp, k. j., durik, a. m., blaum, d., & hastings, p. (2016). identifying general cognitive abilities involved in argument comprehension and evaluation. zeitschrift für pädagogische psychologie, 30(2–3), 79–95. https://doi.org/10.1024/1010-0652/a000173 knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 136 britt, m. a., kurby, c. a., dandotkar, s., & wolfe, c. r. (2007). i agreed with what? memory for simple argument claims. discourse processes, 45(1), 52–84. https://doi.org/10.1080/01638530701739207 chan, n. m., ho, i. t., & ku, k. y. (2011). epistemic beliefs and critical thinking of chinese students. learning and individual differences, 21(1), 67–77. https://doi.org/10.1016/j.lindif.2010.11.001 dandotkar, s., cruz, l. e., stowell, j. r., & britt, m. a. (2022). knowing about knowing: assessing how reflective writing influences undergraduate students’ epistemic beliefs. journal of effective teaching in higher education, 5(1), 1– 17. https://doi.org/10.36021/jethe.v5i1.251 dandotkar, s., magliano, j. p., & britt, m. a. (2016). effect logical relatedness and semantic overlap on argument evaluation. discourse processes, 53(7), 581– 602. https://doi.org/10.1080/0163853x.2015.1087295 davies, m. (2015). a model of critical thinking in higher education. in paulsen, m. (eds.), higher education: handbook of theory and research ( vol. 30). springer. https://doi.org/10.1007/978-3-319-12835-1_2 ferguson, l. e., bråten, i., strømsø, h. i., & anmarkrud, ø. (2013). epistemic beliefs and comprehension in the context of reading multiple documents: examining the role of conflict. international journal of educational research, 62, 100–114. https://doi.org/10.1016/j.ijer.2013.07.001 freeman, j. b. (1991). dialectics and the macrostructure of arguments: a theory of argument structure. foris. fuesting, m. a., diekman, a. b., boucher, k. l., murphy, m. c., manson, d. l., & safer, b. l. (2019). growing stem: perceived faculty mindset as an indicator of communal affordances in stem. journal of personality and social psychology, 117(2), 260–281. https://doi.org/10.1037/pspa0000154 gaipa, m. (2004). breaking into the conversation: how students can acquire authority for their writing. pedagogy, 4(3), 419–437. hofer, b. k., & pintrich, p. r. (1997). the development of epistemic theories: beliefs about knowledge and knowing and their relation to learning. review of educational research, 67(1), 88–140. https://doi.org/10.3102/00346543067001088 hutchings, p. (2000). approaching the scholarship of teaching and learning. in huber and hutchings, ed. opening lines: approaches to the scholarship of teaching and learning, 1–10. carnegie foundation for the advancement of learning. https://doi.org/10.3102/00346543067001088 johnson, r. h., & blair, j. a. (1977). logical self-defense. mcgraw-hill ryerson. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 137 jones, p. c., & merritt, j. q. (1999). critical thinking and interdisciplinarity in environmental higher education: the case for epistemological and values awareness. journal of geography in higher education, 23(3), 349–357. https://doi.org/10.1080/03098269985290 kardash, c. m., & sinatra, g. m. (2003). epistemological beliefs and dispositions: are we measuring the same construct? poster paper presented at the annual meeting of the american educational research association, chicago, il. kirk, r. (1982). experimental design: procedures for the behavioral sciences (2nd ed). brooks/cole. ku, k. y., & ho, i. t. (2010). metacognitive strategies that enhance critical thinking. metacognition and learning, 5(3), 251–267. https://doi.org/10.1007/s11409-010-9060-6 kurfiss, j. g. (1988). critical thinking: theory, research, practice, and possibilities. ashe-eric higher education report no. 2. ashe-eric higher education reports, the george washington university, one dupont circle, suite 630, dept. rc, washington, dc 20036–1183. larson, a. a., britt, m. a., & kurby, c. a. (2009). improving students’ evaluation of informal arguments. the journal of experimental education, 77(4), 339–366. https://doi.org/10.3200/jexe.77.4.339-366 lopatto, d., rosenwald, a. g., burgess, r. c., silver key, c., van stry, m., wawersik, m., ... & reed, l. k. (2022). student attitudes contribute to the effectiveness of a genomics cure. journal of microbiology & biology education, e00208–21. https://doi.org/10.1128/jmbe.00208-21 maggioni, l., & parkinson, m. m. (2008). the role of teacher epistemic cognition, epistemic beliefs, and calibration in instruction. educational psychology review, 20(4), 445–461. https://doi.org/10.1007/s10648-008-9081-8 mason, l., & bromme, r. (2010). situating and relating epistemological beliefs into metacognition: studies on beliefs about knowledge and knowing. metacognition and learning, 5(1), 1–6. https://doi.org/10.1007/s11409-0099050-8 merk, s., rosman, t., muis, k. r., kelava, a., & bohl, t. (2018). topic specific epistemic beliefs: extending the theory of integrated domains in personal epistemology. learning and instruction, 56, 84–97. https://doi.org/10.1016/j.learninstruc.2018.04.008 moore, t. (2004). the critical thinking debate: how general are general thinking skills? higher education research & development, 23(1), 3–18. https://doi.org/10.1080/0729436032000168469 knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 138 moore, t. j. (2011). critical thinking and disciplinary thinking: a continuing debate. higher education research & development, 30(3), 261–274. https://doi.org/10.1080/07294360.2010.501328 muenks, k., yan, v. x., woodward, n. r., & frey, s. e. (2021). elaborative learning practices are associated with perceived faculty growth mindset in undergraduate science classrooms. learning and individual differences, 92, 102088. https://doi.org/10.1016/j.lindif.2021.102088 muis, k. r. (2007). the role of epistemic beliefs in self-regulated learning. educational psychologist, 42(3), 173–190. https://doi.org/10.1080/00461520701416306 reed, l. k. (2022). student attitudes contribute to the effectiveness of a genomics cure. journal of microbiology & biology education, e00208–21. https://doi.org/10.1128/jmbe.00208-21 salmon, m. h. (1984). logic and critical thinking. harcourt brace jovanovich. schommer-aikins, m., duell, o. k., & hutter, r. (2005). epistemological beliefs, mathematical problem-solving beliefs, and academic performance of medical school students. the elementary school journal, 105 (3), 89–304. https://doi.org/10.1086/428745 schommer-aikins, m., & easter, m. (2006). ways of knowing and epistemological beliefs: combined effect on academic performance. educational psychology, 26(3), 411–423. https://doi.org/10.1080/01443410500341304 schommer, m. (1990). effects of beliefs about the nature of knowledge on comprehension. journal of educational psychology, 82(3), 498–504. https://doi.org/10.1037/0022-0663.82.3.498 schommer, m. (1994). an emerging conceptualization of epistemological beliefs and their role in learning. in r. garner & p. a. alexander (eds.), beliefs about text and instruction with text (pp. 25–40). lawrence erlbaum associates. schommer, m., calvert, c., gariglietti, g., & bajaj, a. (1997). the development of epistemological beliefs among secondary students: a longitudinal study. journal of educational psychology, 89(1), 37. https://doi.org/10.1037/0022-0663.89.1.37 seiter, j. s., & gass, r. h. (2007). teaching students how to analyze and adapt to audiences. communication teacher, 21(2), 45–48. https://doi.org/10.1080/17404620701529456 toulmin, s. e. (1958). the uses of argument. cambridge university press. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 139 voss, j. f., & means, m. l. (1991). learning to reason via instruction in argumentation. learning and instruction, 1, 337–350. https://doi.org/10.1016/0959-4752(91)90013-x walton, d. (2006). examination dialogue: an argumentation framework for critically questioning an expert opinion. journal of pragmatics, 38(5), 745–777. https://doi.org/10.1016/j.pragma.2005.01.016 wang, s. (2017). an exploration into research on critical thinking and its cultivation: an overview. theory & practice in language studies, 7(12), 1266–1280. http://dx.doi.org/10.17507/tpls.0712.14 weston, a. (2018). a rulebook for arguments. hackett publishing. wood, p. k., & kardash, c. a. (2002). critical elements in the design and analysis of studies of epistemology. in b. k. hofer & p. r. pintrich (eds.) personal epistemology: the psychology of beliefs about knowledge and knowing (pp. 231–260). erlbaum. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 140 appendix a items from the epistemic belief survey listed under the intended epistemic belief category epistemic belief (latent factor) item # item speed of knowledge acquisition (ksp) 3 if something can be learned, it will be learned immediately. 7 almost all the information you can understand form a textbook you will get during the first reading. 11 you will just get confused, if you try to integrate new ideas in a textbook with knowledge you already have about a topic. 16 working on a difficult problem for an extended period of time only pays off for really smart students. 18 usually, if you are ever going to understand something, it will make sense to you the first time. 24 if i can't understand something quickly, it usually means i will never understand it. 34 most words have one clear meaning. 38 the information we learn in school is certain and unchanging. structure of knowledge (kst) 4 i like information to be presented in a straightforward fashion; i don't like having to read between the lines. 5 it is difficult to learn from textbook unless you start at the beginning and master one section at a time. 12 when i study, i look for specific facts. 13 if professors would stick more to the facts and do less theorizing, one could get more out of college. 21 i really appreciate instructors who organize their lectures carefully and then stick to their plan. 26 i don't like movies that don't have a clear-cut ending. 28 it's waste of time to work on problems that have no possibility of coming out with a clear-cut answer. 30 it is annoying to listen to lecturers who cannot seem to make their mind up as to what they really believe. 31 a good teacher's job is to keep students from wandering from the right track. knowing, thinking, and learning journal of effective teaching in higher education, vol. 5, no. 2 141 33 the best thing about science courses is that most problems have only one right answer. 36 when i learn, i prefer to make things, as simple as possible. knowledge construction & modification (kcm) 2 the only thing that is certain is uncertainty itself. 6 forming you own ideas is more important than learning what the textbooks say. 8 a really good way to understand a textbook is to reorganize the information according to your own personal scheme. 10 you should evaluate the accuracy of information in textbooks if you are familiar with the topic. 15 wisdom is not knowing the answers but knowing how to find answers. 20 today's facts may be tomorrow's fiction. 22 the most important part of scientific work is original thinking. 23 even advice from experts should be questioned. 25 i try my best to combine information across chapters or even classes. 32 a sentence has little meaning unless you know the situation in which it was spoken. 37 i find it refreshing to think about issues that experts can't agree on. characteristi cs of successful students (ss) 14 being a good student generally involves memorizing a lot of facts. 17 some people are born good learners; others are just stuck with a limited ability. 19 successful students understand things quickly. 29 understanding main ideas is easy for good students. 35 the really smart students don't have to work hard to do well in school. attainment of objective truth (ot) 1 you can believe most things you read. 9 if scientists try hard enough, they can find the answer to almost every question. 27 scientists can ultimately get to the truth. microsoft word gastreichmilakovic_jethe_final.docx journal of effective teaching in higher education, vol. 4, no. 3 https://doi.org/10.36021/jethe.v4i3.131 science, poetics, and immersive field ecology: generating new knowledge through interdisciplinary inquiry karin r. gastreich, avila university, karin.gastreich@avila.edu amy e. milakovic, avila university abstract. complex global challenges and declining scientific literacy demand novel approaches to engaging students with science and the natural world. while evidence supports integrating creative and scientific modes of inquiry, these approaches are often separated in undergraduate education. we designed ecology through the writer’s lens (etwl) to allow students to explore an ecosystem of critical importance, the tall grass prairie, through an interdisciplinary field experience. co-taught by biology and english professors and open to students of all majors, etwl leverages classroom activities to prepare for and process the immersive field experience over the course of one semester. field-based exercises include natural history observations, hypothesis building, experimental design, analysis of the literature, and creative/reflective writing. learning was assessed through multiple assignments, including a final creative project that spanned diverse writing genres. students met and exceeded expectations with respect to course objectives. non-science majors learned how scientific knowledge is generated; science majors learned how creative approaches can open new pathways for exploration. many students overcame fear of natural spaces. several students independently engaged with tall grass prairie in post-course activities. we conclude that interdisciplinary approaches to field-based inquiry can generate transformative experiences, even when the immersive component is short-term and close to home. etwl provides one model by which different modes of inquiry can be blended to enhance student appreciation of science, literature, and the environment. keywords: human-nature relations; stem education; steam education; interdisciplinary education; transdisciplinary education; experiential education in a time of global challenges such as climate change, large scale habitat destruction, and emerging infectious diseases, we face a critical need to expand the understanding of science while strengthening connections to nature. quantitative scientific knowledge provides the foundation for informed decision-making, but perspectives from the arts and humanities often allow a broader, more effective context for engaging with the environment (jacobson et al., 2007; jacobson et al., 2016; swanson et al., 2008). fostering a creative, interdisciplinary mindset in the context of science education can produce visionary innovators needed to solve complex problems (dehaan, 2011; madden et al., 2013; mueller et al., 2014; scheffer et al., 2015; you, 2017). at the same time, introducing non-science students to scientific disciplines through the creative lens can cultivate a deeper understanding of scientific concepts (gray & thomson, 2016; jacobson et al., 2016). we set out to integrate scientific and literary modes of inquiry in a semester-long undergraduate course, ecology through the writer’s lens (etwl). science, poetics, and immersive field ecology 72 journal of effective teaching in higher education, vol. 4, no. 3 etwl allowed students from diverse backgrounds to collaborate in a focused, interdisciplinary exploration of the natural world. this model helped students develop an appreciation for the scientific process, creative literature, and wild spaces. implemented close to our home campus with minimal resources, etwl offers a basic structure for interdisciplinary education in science and creative literature that could be adapted to different institutional and student body needs. creative and scientific modes of inquiry form complementary aspects of the relationship between humans and their environment. yet often in undergraduate education, these are separated by discipline and not allowed to meet on common ground. integrating science with the creative arts, often referred to as “steam” education, has been shown to enhance learning and strengthen scientific creativity in multiple contexts (brown, 2015; dehaan, 2011; gray & thomson, 2016; jacobson et al., 2016; madden et al., 2013; scheffer et al., 2015). at the same time, use of creative arts to teach science can facilitate a greater appreciation of natural systems and scientific modes of inquiry (jacobson et al., 2016; swanson et al., 2008). much of the conversation around steam education focuses on the arts as a path for inspiring creativity within the sciences (e.g., brown, 2015; dehaan, 2011; liao, 2016; madden et. al, 2013; pollack & korol, 2013; taylor & taylor, 2019). it is important, however, to respect the creative arts as a discipline rather than as a mere set of heuristic tools in service of science education (bequette & bequette, 2012; liao, 2016). with respect to etwl, we sought to give two subject areas, ecology and creative literature, equal footing throughout the course, integrating them through direct exploration of an ecosystem to enhance and enrich learning across both disciplines. this model followed a more classic definition of interdisciplinary education (cone et al., 1998, as cited in you, 2017) that has been successfully implemented in other contexts (gray & thomson, 2016; jacobson et. al, 2016). in addition to on-campus activities, etwl included an immersive field component that served as both a laboratory and a writer’s workshop. working collaboratively, students collected data, generated hypotheses, and created new meanings through direct observation and the written word. they also crafted a creative project based on their field experience. etwl was inspired by the h. j. andrews experimental forest long term ecological reflections program (swanson et al., 2008). designed to complement the national science foundation’s long-term ecological research (lter) program, andrews forest reflections seeks to advance understanding of the natural world by enhancing collaboration between the sciences and the humanities. the program sponsors immersive workshops designed to engage ecologists, creative writers, and philosophers in the task of building and communicating ecological knowledge (swanson et al., 2008). in the summer of 2011, one of the authors participated in an andrews forest writers residency. based on that experience, we developed etwl to offer our students a similar interdisciplinary experience as part of their undergraduate degree program. from the course inception, we wanted to make etwl available to students from all majors and professional tracks. to achieve this, we took advantage of an existing framework within the core curriculum of our home institution: interdisciplinary science, poetics, and immersive field ecology 73 journal of effective teaching in higher education, vol. 4, no. 3 studies (is) courses. taught by faculty from two or more disciplines, is courses challenge students to consider the strengths and limits of disciplinary knowledge as well as integrate, synthesize, and communicate interdisciplinary knowledge (avila university, 2020). is courses are designed to illustrate why certain challenges warrant interdisciplinary examination and how each discipline contributes to the process of understanding and resolution. our university has a decades-long history of is courses with the basic model persisting through multiple revisions of the core curriculum. this provided an excellent context within which to implement etwl. to facilitate direct engagement with the natural world, we took the students to konza prairie biological station, about a two-and-a-half-hour drive from our midwestern campus. although our campus occupies a region characterized by the interface between deciduous forest to the east and prairie to the west, many of our students start the course with minimal knowledge of prairie as an ecosystem of local and global significance. prairie dominates the great plains, with a natural range that extends over 3.5 million square kilometers from central texas into southern canada (savage, 2011). it is estimated that 96% of north america’s original prairie has been altered or destroyed, much of it for agricultural use (national park service, 2020). konza prairie sits in the kansas flint hills and represents one of the best-preserved tall grass prairie remnants in the united states. the station is a long term ecological research (lter) site maintained by kansas state university and is equipped to support overnight stays by researchers and small groups. one of our questions going into etwl was whether a field site close to home could generate a transformative field experience in a short amount of time. a body of literature supports the transformative nature of international field experiences (gastreich, 2002; mclaughlin & johnson, 2006). however, many students cannot take advantage of long-distance travel for a variety of reasons. as designed, etwl could be adapted to a variety of wilderness sites worldwide, and in the planning stages, we did consider international destinations. however, we hoped that by choosing a location and ecosystem close to home, we could provide a valuable transformative experience for a broad range of students. etwl had three primary learning objectives that we sought to achieve by giving students first-hand experience in both the scientific and creative processes while immersed in the prairie ecosystem: (1) explore how scientific knowledge is generated and use that knowledge to inspire creative expression, (2) articulate how creativity contributes to the process of science, and (3) describe how both paths of exploration allow us to better understand and interact with our natural environment. within our primary learning objectives, we had several goals pertinent to scientific and creative/literary aspects of the course (see table 1 below). science, poetics, and immersive field ecology 74 journal of effective teaching in higher education, vol. 4, no. 3 table 1 ecology through the writer’s lens learning goals scientific interpretation creative and literary interpretation o acquire a foundation of knowledge about the tall grass prairie ecosystem. o generate hypotheses based on detailed quantitative and qualitative observations. o develop and strengthen writing skills. o explore the central role of metaphor in understanding our world and ourselves. note: learning goals were embedded within the larger course learning objectives, as discussed in the introduction. course structure we implemented etwl in three stages. pre-trip preparation included class meetings to provide critical information and set the stage for our field experience. experiential learning in the field consisted of intensive activities in residence at konza prairie field station. finally, post-trip integration included follow-up class meetings and critical assignments to help students process the experience. here we describe each of these stages in detail. throughout the semester, the biology professor led science-related activities and the english professor led creative and literary activities. however, both professors were present for all course events. moreover, both professors participated in discussion of all course material. in this way, each professor brought the lens of their own discipline to bear on material presented by the other professor. by actively engaging each other and the students in cross-disciplinary discussion, we strived to show by example how professionals from different backgrounds can, through mutual respect and open discussion, find common ground. pre-requisite student knowledge as with all is courses at our institution, students needed to have completed their foundation courses (one semester each of english composition, communications, and college-level math) to enroll. advanced study of biology or english was not required. students enrolled for a variety of reasons. some had enjoyed previous courses with the same professors or had an interest in creative writing or natural history. others enrolled out of a simple need to fill the is requirement for their degree. every semester attracted a new and fresh mix of disciplinary backgrounds and interests. because of the diversity of student backgrounds, well-conceived prescience, poetics, and immersive field ecology 75 journal of effective teaching in higher education, vol. 4, no. 3 trip activities that accommodated a wide range of skills and expertise were necessary to prepare them for the best outcomes. pre-trip preparation we met three times before the immersive trip. our first classroom event was an organizational meeting. we followed up with two formal lecture periods, one that focused on scientific interpretation of the prairie and a second that focused on creative and literary interpretation. organizational meeting as part of the course overview, we provided basic logistical information for the field trip. this included confirming dates of travel, clothing, and equipment needs, and organizing the group into cooking and cleaning crews. we also provided time for individual introductions and questions. this first meeting concluded with a low-stakes, in-class writing assignment. one of the most informative prompts we have used was “what scares you about the field trip?” their often-surprising answers were invaluable. some students had lived only in the urban core with little exposure to the natural world beyond a city park and held fears for their physical safety. in addition, many students, especially science majors, admitted fear of creative writing. knowing their comfort level early in the course helped us build a supportive environment for everyone. this first writing assignment also benefitted students by encouraging them to think and write reflectively, an important component of interdisciplinary study. finally, reviewing their responses allowed us to assess student writing skills and anticipate where additional help might be needed. scientific interpretation the primary biology textbook for this course was prairie: a natural history by candace savage (2011). savage’s narrative was well-suited because the author (1) collates a tremendous amount of scientific information about the prairie in a single volume and (2) relates this information in language accessible to both science and non-science majors. thus, the textbook achieved a dual purpose of providing a scientific foundation while illustrating one modality—natural history writing—that can result from creative interpretation of the prairie. prior to class, students read the first three chapters of savage (2011). these chapters cover the geological history of the great plains, current climate patterns, and the unique adaptations of grasses (family poaceae). take-home points from the textbook were reviewed through formal lecture and group discussion. in addition, we introduced the unique qualities of our destination, konza prairie in the flint hills, within the larger context of the great plains. science, poetics, and immersive field ecology 76 journal of effective teaching in higher education, vol. 4, no. 3 creative and literary interpretation the second lecture period focused on students’ interactions with creative literature—a collection of short essays and both canonical and contemporary poems. the essays we selected, spanning a 40-year period, were chosen in part for their range of tones from the contemplative to the playful. this broad cross-section (which we expanded with odes in the field) served several purposes. for one, students benefited from realizing that creative literature can engage nature and science in a variety of forms; thus, they need not panic for fear that they would be required to write sonnets if they were more comfortable with essays. additionally, given many students’ limited exposure to quality literature, providing an array of different genres and styles increased the possibility that each student would discover a piece that resonated with them, even if they were not already familiar with that particular form. we opted not to include novels, as the course format was more conducive to studying shorter pieces, and we wanted to concentrate on forms in which students could successfully write at semester’s end. some texts were chosen that would tie directly to students' upcoming field experiences. we did not share this information with students during the classroom meeting as we knew that allowing “aha” moments to emerge organically in the field creates more powerful connections for students than having them predicted beforehand. for example, during a night walk at konza prairie, our students had the opportunity to strike flint together and create sparks. this usually resulted in at least one student excitedly recalling rossetti’s last line, “but a flint holds fire!” students clearly enjoyed discovering connections between literary readings and field experiences; such moments also helped them appreciate that, although interdisciplinary, this course was as thoughtfully constructed as the disciplinespecific classes to which they were accustomed. students completed the literary readings and responded to writing prompts before the second lecture period. these prompts asked students to consider ideas such as the prairie as metaphor, prairie as community, and life lessons from the prairie. professors guided students through textual explication in class, followed by a group discussion in which they built upon their previous written responses in light of what they had learned in the lecture period. thus, this assignment created layers of learning: students first encountered the texts personally, where meaning is directly negotiated between the author and the reader. responding to the instructor’s prompts encouraged a different type of learning as students used writing to explore aspects of the texts perhaps unnoticed on their first reading. their knowledge was further expanded through classroom literary instruction. finally, open discussion with peers enriched students’ understanding of the power and possibilities of nature-based creative literature. the course syllabus contains a complete list of all literary readings (see appendix a). experiential learning in the field during our four-day stay at konza prairie, we engaged the students in a series of intensive field exercises designed to build detailed observation skills, practice science, poetics, and immersive field ecology 77 journal of effective teaching in higher education, vol. 4, no. 3 creative interpretation, and explore the scientific method. creative and scientific activities were interwoven throughout the field experience. in this section, however, we separate activities by theme (creative and literary vs. scientific) in the interest of clarity for the reader. to take maximum advantage of our time in the field, students were expected to complete all course readings before starting the immersive experience. while in the field, we focused on multiple short assignments rather than single, long works. assignments were designed to encourage high levels of creative engagement, imagination, and open-ended responses, an approach not often taken in the context of science education (pollack & korol, 2013). evidence indicates that limiting the length of writing assignments to short “microthemes” in which students distill ideas without emphasis on grammar or mechanics is an effective practice (collins, 2004; leahy, 1994; mcmillan, 2014). in the field, our primary goal was to get our students comfortable with the environment and with writing. this, in turn, served to enhance skills needed for post-trip integration of their experience. creative and literary interpretation two writing activities took place on the first day of the trip. the first focused on detailed observations from the creative/literary perspective. while this exercise in some ways mimicked the close attention to detail practiced by scientists, it did so through a creative, sensory perspective. in the first phase, we led students on a short hike then settled them on field trails at intervals which allowed them to work in solitude. each selected one specimen from the field: forb, leaf, grass, etc. their assignment was to write detailed observation of this one specimen in their journals, employing every sense: the specimen’s texture(s), sound produced when it is manipulated, appearance, smell, taste (if safe to sample). to encourage sustained, in-depth observation, we required one full page that completely described every aspect of their chosen specimen. once this detailed observation was complete, we turned students’ attention from close inspection to a wide, encompassing view. in this second phase, the class hiked 1.3 miles through lowland galley forest and over limestone ledges to reach a plateau in upland tallgrass prairie. this view overlooked the prairie on three sides, including the flint hills and the kansas river valley. in the fourth view lay a small highway and farmsteads. once again, students were asked to write one full page of exactly what they experienced, but this time from a god’s-eye view: colors, vegetation, animal presence, sounds, sensations of sun and wind, and the like. although the size of the hilltop did not allow complete student separation, students were generally so exhilarated by the unexpected view after a tiring climb that they readily immersed themselves in the experience. still, we asked them to sit and record observations for at least 20 minutes. this requirement helped ensure that they fully inhabited the time and space and thus captured impressions lost without a slower, contemplative pace. science, poetics, and immersive field ecology 78 journal of effective teaching in higher education, vol. 4, no. 3 throughout the remainder of the four-day trip, we interspersed other creative and literary writing activities with scientific ones. in one, we introduced students to haiku, a three-line, unrhymed japanese poetry form which focuses on images from nature. haikus emphasize simplicity, directness, and the evocation of a singular moment. learning and writing haikus requires students to distill a profusion of sensations and information into a clear, disciplined structure. as pollack and korol (2013) report, assigning haikus in their science classroom helps students synthesize specific scientific concepts. while our interdisciplinary course focused on ends other than the “scientific fluency” sought in theirs, our experience validates pollack and korol’s (2013) findings that creative assignments tap distinct cognitive processes compared to other written assignments. we also added field readings which draw on a western literary tradition by introducing students to classic odes befitting particular moments in the field; examples include john keats’ “to autumn” and percy shelley’s “ode to the west wind.” hearing poetry aloud is important for both enjoyment and literary instruction, so we gathered on the open prairie and an instructor (or student volunteer skilled in recitation) performed a reading of these odes to the assembled group. odes often struck a deep chord with students, with several choosing to write their own in admiration of one specimen or prairie experience as their final creative project. in an open-enrollment course such as this, many students had had little experience with the performance of literature, just as many were new to scientific observations. consequently, both instruction and modeling were important elements of helping students develop and integrate these skills. scientific interpretation science activities focused on how to build hypotheses based on observation and how the scientific method can be used to test hypotheses about the prairie. examples included completion of a field notebook, a group activity called “20 questions,” and a guided scientific tour of konza prairie research station. to build observational skills, students kept a field notebook that included a directed natural history exercise. students located, identified, and provided detailed descriptions of ten species of grasses or forbs (non-woody, flowering plants) as well as ten species of other plants or animals. to award full credit, we required that each notebook entry include date, time, and location of the observation as well as a full description of the organism’s habitat. the first field-based writing exercise (described in the previous section) had provided students an example of the level of detail expected. the biology professor gave instructions and supporting materials regarding characteristics for identifying common prairie organisms. for example, in the case of plants, we discussed growth form, leaf shape, position of the leaves on the stem, size, and the shape and structure of the flower. field guides tailored to the tall grass prairie were provided, and professors were on hand to assist the identification process. the level of identification varied depending on the organism in question. bison, for example, could be confidently recorded by their species name, bison bison. most science, poetics, and immersive field ecology 79 journal of effective teaching in higher education, vol. 4, no. 3 insects, on the other hand, were identified to order or family. the objective was not to convert students into skilled taxonomists but to engage them directly with the environment and the process of recording detailed qualitative data. a handful of organisms, such as wild turkey and big blue stem, appeared in all the student notebooks. at the same time, students explored the environment on their own and made unique observations. in addition to narrative descriptions, students often included illustrations in their entries. to explore the process of creating and testing hypotheses, students completed a “20 questions” exercise, based on a field activity developed by the organization for tropical studies for graduate and undergraduate field courses (organization for tropical studies, unpublished). in the konza prairie adaptation, each student generated 20 questions about the prairie based on course readings, written exercises, and natural history observations. students then worked in small groups to select one question around which they developed a testable hypothesis. under the guidance of the biology professor, each group then designed an appropriate experiment to test their hypothesis, taking into account independent and dependent variables, sample size, and other important considerations. proposed experimental designs were then shared with the entire class in an informal seminar. to observe real-world applications of the scientific method, we completed a halfday scientific tour of konza prairie research station, including the opportunity to see bison. every semester, one of the station scientists generously contributed her time for this activity, explaining to students how large-scale, landscape-level questions are being addressed at the lter site. students traversed konza watersheds to see how they are divided into experimental parcels to examine the impacts of variables such as grazing, fire, and drought on biodiversity. they also learned about practical aspects of prairie management, such as the logistics of controlled burning and maintaining a viable population of grazing bison. post-trip integration following the konza prairie field experience, we engaged with the students in processing the experience through three class meetings and additional follow-up assignments. these assignments included the final creative project and comprehensive essay. approximately one week after the field trip, students also completed an exam that covered both scientific and literary content of the course. final creative project students completed a final written project based on their experiences during the field portion of the course. they could write in any genre: natural history, creative nonfiction, poetry, or any type of fiction. students wrote short stories, screenplays, and created illustrated books for first graders. they crafted odes to natural elements and detailed travelogues. while the final project was creative in nature, students had to incorporate a scientific foundation as part of their work. in other words, scientific knowledge of the prairie had to be accurately reflected in the creative product. a sample rubric is provided in appendix b. science, poetics, and immersive field ecology 80 journal of effective teaching in higher education, vol. 4, no. 3 getting good final projects required solid groundwork. the process we found successful began with meeting individually with each student while still at konza prairie. on the afternoon of the third day, when most of their field work had been completed, each student signed up for a 15-minute session with both professors to discuss their ideas for the final project. a prospectus was due three weeks after returning from the field trip. both professors reviewed these and either approved as-is or recommended changes we believed would help students be more successful. first complete drafts were submitted two weeks later. again, both professors reviewed these drafts and offered written critiques on content, form, and writing mechanics. students submitted their final revised pieces four weeks later. the class met one last time for students to orally present their creative projects. while some students initially seemed reticent to present, our experience was that they became more confident as they listened to their peers and relaxed in the supportive atmosphere. comprehensive essay after submitting their final creative project, students composed a 500–750-word essay that addressed the main theme of the course. building on their course experience, they discussed how both scientific and creative modes of inquiry contribute to our understanding of the prairie. they also articulated the advantages and challenges of integrating these modes of inquiry, describing how creative inquiry enhances scientific understanding and how scientific knowledge informs creative inquiry. a sample rubric is provided in appendix c. findings from the first semester we ran this course, qualitative and quantitative measures of student success have met and exceeded our expectations. in this section, we discuss how students achieved learning objectives as illustrated by the final creative project and comprehensive essay. we also highlight more general responses to the experience as recorded in student evaluations. final creative project we have been impressed by the number and variety of ways our students applied their prairie experience to the final creative project. they utilized a range of written formats, including natural history essays, short stories of all genres, poems, memoirs, children’s stories, and screenplays. one student well advanced in writing skills even developed their own genre of expression uniquely suited to communicating their understanding of the flint hills. here we describe a sampling of projects generated by students. students from urban settings often confronted fears about nature. in many cases, these students used the creative project to illustrate their personal journey in a metaphorical fashion. their stories often featured characters that went from being fearful of nature to experiencing comfort and joy in the natural world. one student science, poetics, and immersive field ecology 81 journal of effective teaching in higher education, vol. 4, no. 3 who wrote about this theme developed a children’s story with a special target audience: their niece. after going from fear of nature to a deep appreciation of the prairie, this student wanted to help a beloved family member make the same transition through storytelling. several biology majors enrolled in etwl. often coming from a cell or molecular biology background, they discovered a passion for ecology as a result of their prairie experience. our science students have produced diverse creative works for their final creative projects, including poetry, humorous pieces, and fantasy short stories. inspired by etwl, several have gone on to complete capstone research projects on some aspect of prairie ecology. for example, one student wrote a poem in the form of an ode to the monarch butterfly. because of her own history, she found a companion spirit in this beautiful species that undertakes a multigenerational migration across north america. in the semesters that followed, this student decided to dig deeper into the science of monarch migration. for their capstone project in the biology major, this student conducted an in-depth literature review on monarchs in the americas, comparing their migratory habits to painted lady butterflies in europe and africa. no matter our students’ backgrounds, they have found numerous opportunities to relate on a personal level to the tall grass prairie. even our international students, who face an additional challenge with english as a second language, have benefitted from this course. for example, we had a student from central america who was anxious at the beginning of the semester because of their uncertainty with the language as well as an overall sense of being “out of place.” we met with this student individually to encourage them to stay, emphasizing the unique and valuable perspective the student could bring to the group, given their prior experience with the neotropics. for the student’s final creative project, they crafted a skillfully written poem that compared the starry nights of the prairie with the starry nights of the tropical beaches in their home country. comprehensive essay as described earlier, the comprehensive essay asked students to articulate their understanding of the primary theme of the course: the benefits and challenges of integrating scientific and creative/literary modes of inquiry. we graded the essay on a 10-point scale. all participants earned at least a 7 (c or passing) grade with an average score of 8.3 (b). overall, we were pleased with the thoughtful and engaged responses of our students. representative excerpts are provided in figure 1. course evaluations a total of 32 students have participated in etwl across three semesters (2014, 2016, and 2018), of which 29 responded to course evaluations (10 in 2014, 9 in 2016, and 10 in 2018, with one student not responding each semester). on student evaluations, the vast majority have ranked course quality and teaching as “excellent” or “above average.” on a scale of 1 to 5 (1=excellent, 5=poor) the average student ranking for the course has remained at or near 1.2 overall. student comments on evaluations reflected what we heard informally in the field science, poetics, and immersive field ecology 82 journal of effective teaching in higher education, vol. 4, no. 3 and classroom. they consistently identified the course as not only fun but “life changing.” they expressed appreciation for the connection they developed with the prairie as well as the opportunity for simultaneous intellectual growth in more than one academic field. figure 1 selected excerpts from three final comprehensive essays “the beauty of both [scientific and creative inquiry] is that they compliment [sic] each other well. for example, when describing an organism, it is natural for a proper description to include details of shape and color but when that description is written with the intent of holding a reader’s attention . . . writing and science can genuinely blend and flourish . . . [a]pproaching an ecosystem with both modes allows one to genuinely convey the setting, appearance, and feel of the world in an appealing manner while simultaneously understanding . . . why its existence came to be and how it is still functioning.” “. . . [creative] writing tells a story that reaches the reader in a way that scientific writing often can’t. the facts that are being stated become interesting, and the reader cares about them. instead of skimming over calculations and statements, the reader creates a personal experience . . . this new world of writing creates a personal connection between the author and the reader.” “reading the scientific literature can be very boring . . . given the current situations with climate change and people not vaccinating their children, science needs a new public relations plan . . . people do however understand story telling . . . science can use this personal connection a reader can have to a literary work to tell the truths about the world . . . using creative means to get people interested in nature can help make a domino effect of people wanting change in the world.” note: in this assignment, students described how creative inquiry enhances scientific understanding and how scientific knowledge informs creative inquiry. discussion over the three semesters of etwl, students from diverse disciplinary backgrounds have consistently achieved course objectives. by engaging in detailed natural history observations, generating original hypotheses, designing experiments, and observing how science is applied to real-world situations, etwl students came away with a deeper understanding of how scientific knowledge is generated. students also engaged in critical exploration of creative literature and writing, and science, poetics, and immersive field ecology 83 journal of effective teaching in higher education, vol. 4, no. 3 they successfully used scientific knowledge of prairie ecosystems to inspire creative expression as evidenced by short field assignments and the final creative project. in the final comprehensive essay, students articulated how creativity contributes to the scientific process and how science informs creative inquiry. through the comprehensive essay, students demonstrated an understanding of the importance of both scientific and creative paths of exploration when interacting with our natural environment. throughout the course, students developed and strengthened their writing skills while exploring the central role of metaphor in understanding our world and ourselves. the results we observed when integrating literary and scientific modes of inquiry are supported by other published studies of interdisciplinary and transdisciplinary approaches to science education (brown, 2015; dehaan, 2011; mueller et al., 2014; pollack & korol, 2013; scheffer et al., 2015; you, 2017). we also found that many students underwent a transformative experience even though the immersive portion of the course was short and close to home. this was evidenced by recurrent themes in class discussion and creative projects, such as the challenge of overcoming fear of natural spaces. students succeeded in converting trepidation into excitement through direct engagement with, and a greater understanding of, the natural environment. every student finished the course producing creative work that reflected deep respect for, and foundational knowledge of, the tall grass prairie ecosystem. that brief immersive experiences close to home can be transformative has also been reported by other authors, for example, in the context of place-based science education (gray & thomson, 2016; jacobson et al., 2016; mueller et al., 2014). in traditional academics, there is sometimes a desire to broaden students’ experiences in giant leaps. we may believe we are showing students a larger world only if we take them to a distant part of the country or to another country entirely. as instructors, we started etwl with this preconception. for example, we had considered implementing the course in a tropical environment such as the forests of costa rica. however, because many of our students are non-traditional or come from financially stressed situations, choosing a nearby site made the course more accessible in important ways. student fees for konza prairie totaled one-tenth the budget necessary for an international course. students with family or work obligations who could not be away for extended international travel were able make arrangements for a short field experience in the flint hills. an unforeseen benefit of the immersive field component was the sense of community that emerged among students. this bonding was engendered in part by the field setup, where students shared the same cabin, coordinated cooking and cleaning, shared field guides, and helped with assigned tasks, such as identifications. this communal experience created personal bonds that encouraged them to learn from and support each other. bonding was also facilitated through integrative activities. for example, every evening all course members took part in a story circle where we shared something from our journal or the day’s experience. often these included a moment of surprise or an appreciation of an expanding sense of self as members of the natural community. in their post-trip assignments, science, poetics, and immersive field ecology 84 journal of effective teaching in higher education, vol. 4, no. 3 students recognized this process and drew parallels between their experience and what they observed in the tall grass prairie community (figure 2). figure 2 experiences of connection and community “when i observed the prairie, i did not see just land, i saw a community of all types of life that had a common goal of surviving. . .” “. . . the writing allowed me to see and feel the connections that my classmates made to the prairie. i could see how their views had changed as they read their pieces. even if i was not there when they had their experience, i could feel what they went through.” “we became more of a community who now cared and learned about something bigger than us. . .” note: these observations represent excerpts from different integrative assignments and course evaluations following the immersive field experience. implications complex global challenges demand novel approaches to enhancing scientific literacy. this process cannot be divorced from engaging each other and the environment as part of the same community. based on our experience, the semester-long undergraduate course ecology through the writer’s lens is one model by which scientific understanding and creative expression can be integrated to better connect students with each other and the natural world. we observed that the course benefited both science and non-science majors in important, complementary ways. non-science majors developed a greater understanding of the power of science as well as a deeper appreciation of the importance of verified facts in the pursuit of knowledge. science majors learned how creative approaches to observation can open new pathways for exploration, adding a different kind of scaffolding on which to build scientific inquiry. particular strengths of this course included its accessibility to a broad range of students, many of whom had not previously traveled far from home. while this paper has detailed our experience merging ecology and writing, many disciplines may be fruitfully combined. as noted earlier, our institution has a long history of offering interdisciplinary courses. for example, faculty from visual design and rhetorical studies teach the “art” of war, which explores differences between rhetoric and propaganda. like etwl, the “art” of war includes an immersive travel component in the form of a four-day trip to washington, d.c. likewise, faculty from biology and history teach the course plagues. conducted entirely on campus, plagues considers the transnational causes and effects of epidemics, including their science, poetics, and immersive field ecology 85 journal of effective teaching in higher education, vol. 4, no. 3 origin, techniques of control, and contemporary ethical debates concerning the nature of disease and treatment of patients. through the combined experience of avila faculty, three factors have been revealed as essential for interdisciplinary course success, regardless of the disciplines being combined: faculty collegiality, effective communication between co-instructors, and equal treatment of the disciplines represented in each course. as a final note, we want to mention the special challenge of implementing a fieldbased course during covid-19. because the pandemic precluded our class from sharing living quarters in the field, during fall 2020 we brought etwl even closer to home. instead of an extended field trip to the flint hills, we conducted our field exercises in a series of day trips to prairie, woodland, and wetland sites close to campus. due to the pandemic surge in november, students shared their final creative pieces during a virtual meeting. this was the first semester we ran the course under this protocol. for this reason, it is difficult to draw conclusions as any differences between our fall 2020 experience and previous semesters may be due to idiosyncrasies of this group as well as the larger social stresses affecting campus and academic life. nonetheless, we found the major course objectives were successfully met for most students under this new format. at the same time, communal aspects, such as the group bonding that came with the visit to konza prairie, were missed. while our preference continues to be for a multi-day field immersion experience as part of the course, our experience this semester indicated the basic practices developed for etwl can be adapted under different circumstances and needs. we also recognize, in hindsight, the benefits we received as instructors by adapting this course so that it could be offered despite covid-19. even in a scaled-down field environment, students responded to their scientific and literary discoveries with enthusiasm. at a time when the regular rhythms of academic life had been totally disrupted, the choice not to cancel the course but to find a creative way to carry it forward helped carry us forward as well. acknowledgements we would like to thank the avila university administration and faculty for continued support of the interdisciplinary studies model. konza prairie was an excellent site for hosting our course. we are especially grateful to dr. eva horne of kansas state university for working with our students in the field. andrews forest long term ecological reflections project inspired this course and made the experience possible for our students. editor and reviewer comments on earlier versions of the manuscript were very helpful in shaping the final submission. author contributions both authors contributed equally to the design, implementation, and assessment of ecology through the writer’s lens. we also contributed equally to the submitted manuscript. science, poetics, and immersive field ecology 86 journal of effective teaching in higher education, vol. 4, no. 3 conflicts of interest the authors declare that there are no conflicts of interest regarding the publication of this article. references avila university (2020). academic course catalogs. https://catalog.avila.edu/ bequette, j. w., & bequette, m. b. (2012). a place for art and design education in the stem conversation. art education, 65(2), 40–47. brown, s. (2015). creative expression of science through poetry and other media can enrich medical and science education. frontiers in neurology, 6(3). https://doi.org/10.3389/fneur.2015.00003 collins, m. a. j. (2004). using short pieces of writing (microthemes) to improve student learning. in m. druger, e. d. seibert, & l. w. crow (eds.), teaching tips: innovations in undergraduate science instruction (pp. 7–8). national science teacher association press. dehaan, r. (2011). teaching creative science thinking. science, 334, 1499– 1500. https://doi.org/10.1126/science.1207918 gastreich, k. r. (2002). student perceptions of culture and environment in an international context: a case study of educational camps in costa rica. canadian journal of environmental education, 7(1), 167–182. https://cjee.lakeheadu.ca/article/view/281 gray, t., & thomson, c. (2016). transforming environmental awareness of students through the arts and place-based pedagogies. learning landscapes journal, 9(2), 239–260. https://doi.org/10.36510/learnland.v9i2.774 jacobson, s., mcduff, m., & monroe, m. (2007). promoting conservation through the arts: outreach for hearts and minds. conservation biology, 21(1), 7– 10. https://doi.org/10.1111/j.1523-1739.2006.00596.x jacobson, s., seavey, j., & mueller, r. (2016). integrated science and art education for creative climate change communication. ecology and society, 21(3), 30. http://dx.doi.org/10.5751/es-08626-210330 leahy, r. (1994). microthemes: an experiment with very short writings. college teaching, 42, 15–18. https://doi.org/10.1080/87567555.1994.9926808 liao, c. (2016). from interdisciplinary to transdisciplinary: an arts integrated approach to steam education. art education, 69(6), 44–49. madden, m. e., baxter, m., beauchamp, h., bouchard, k., habermas, d., huff, m., ladd, b., pearon, j., & plague, g. (2013). rethinking stem education: an science, poetics, and immersive field ecology 87 journal of effective teaching in higher education, vol. 4, no. 3 interdisciplinary steam curriculum. procedia computer science, 20, 541– 546. https://doi.org/10.1016/j.procs.2013.09.316 mclaughlin, j. s., & johnson, d. k. (2006). assessing the field course experiential learning model: transforming collegiate short-term study abroad experiences into rich learning environments. frontiers: the interdisciplinary journal of study abroad, 13(1), 66–85. https://doi.org/10.36366/frontiers.v13i1.174 mcmillan, s. l. (2014). bravo for brevity: using short paper assignments in international relations classes. international studies perspectives, 15(1), 109–120. https://doi.org/10.1111/insp.12003 mueller, a., s.j. juris, k. willermet, e. drake, s. upadhaya, & p. chhetri (2014). assessing interdisciplinary learning and student activism in a water issues course. journal of the scholarship of teaching and learning 14(2):111–132 https://doi.org/10.14434/josotl.v14i2.4052 national park service. (2020, may 11). tall grass prairie: national preserve kansas. https://www.nps.gov/tapr/index.htm pollack, a. e., & korol, d. l. (2013). the use of haiku to convey complex concepts in neuroscience. june, 12(1), a42–a48. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc3852870/ rossetti, c. (1862). precious stones. in l. untermeyer (ed.) the golden treasury of poetry. (p. 306). golden press. savage, c. (2011). prairie: a natural history. greystone books. scheffer, m., bascompte, j., bjordam, t. k., carpenter, s. r., clarke, l. b., folke, c., marquet, p., mazzeo, n., meerhoff, m., sala, o., & westley, f. r. (2015). dual thinking for scientists. ecology and society, 20(2), 3. http://dx.doi.org/10.5751/es-07434-200203 swanson, f., goodrich, c., & moore, k. d. (2008). bridging boundaries: scientists, creative writers, and the long view of the forest. frontiers in ecology, 6(9), 499–504. https://doi.org/10.1890/070076 taylor, p. c. s., & taylor, e. (2019). transformative steam education for sustainable development. in y. rhamawati & p. c. taylor (eds.), empowering science and mathematics for global competitiveness (pp. 3–8). taylor and francis. you, h. s. (2017). why teach science with an interdisciplinary approach: history, trends, and conceptual frameworks. journal of education and learning, 6(4), 66–77. http://doi.org/10.5539/jel.v6n4p66 science, poetics, and immersive field ecology 88 journal of effective teaching in higher education, vol. 4, no. 3 appendix a sample course syllabus is 365 ecology through the writer’s lens prerequisites completion of core foundations textbooks savage, candace. 2011. prairie: a natural history. greystone books, vancouver, canada. literary selections distributed in class and on canvas. (see “bibliography of literary readings” at the end of the syllabus.) course description students will integrate literary and scientific modes of inquiry while engaging in an in-depth study of the natural world. each semester will focus on a particular ecosystem such as tropical forest, temperate grasslands, temperate rainforest or montane ecosystem. the course will include a multi-day field trip, providing a firsthand experience of the ecosystem of interest. this ecosystem will serve as both a laboratory and a writer’s workshop where students will generate hypotheses and create new meanings. before the trip, students will study the particular ecosystem and read representative works in nature writing, including scientific writing and creative writing. during the trip, students will write journals and narratives, do scene painting with words, and conduct detailed natural history observations geared toward generating hypotheses about species interactions and ecosystem dynamics. for their final writing project, students may focus on natural history, creative nonfiction, or fantasy. students will share their work with the class at the final course meeting. please note that is 365 is blended course that includes coursework online, in the classroom, and in the field. students are responsible for keeping track of assignments, schedule changes, and other announcements on canvas. course learning objectives • acquire a foundation of knowledge about the focal ecosystem. • develop and strengthen writing skills through a variety of writing experiences. • explore the central role that metaphor plays in understanding our world and ourselves. science, poetics, and immersive field ecology 89 journal of effective teaching in higher education, vol. 4, no. 3 • generate scientific hypothesis on the basis of detailed quantitative and qualitative observations of the natural world. • use scientific knowledge to inform other types of writing: nature writing, creative nonfiction, poetry, and fiction. evaluation 10% participation 10% field exercises 15% classroom exercises, homework, quizzes 20% field journal 15% exam 10% comprehensive essay 20% final project participation this course is based on the principle of building a strong academic and interdisciplinary dialogue among the participants. to have a positive learning experience, active participation—coming prepared to class, engaging in discussions, listening to the perspectives of other students, critically examining the readings, and so forth—is essential. attendance is mandatory. field exercises during field exercises, students will record observations, generate questions and hypotheses, and discuss experimental design. field exercises will, for the most part, be completed during the field trip. classroom exercises, homework, and quizzes this course includes classroom exercises where students will develop and clarify concepts relevant to prairie ecology. homework may be assigned, and periodic quizzes given over readings and lecture material. exam one exam will be administered mid-semester, shortly after the field trip. the exam will be of mixed format, including multiple choice and short answer questions. writing journal/field notebook students will keep a field notebook in which they will record daily natural history as well as personal and scientific observations. each entry should be dated, and notes must be as detailed as possible. notes may include a variety of observations, such as descriptions of plants and animals, quantitative or qualitative data gathered during field exercises, sketches, bird lists, poems, and so forth. science, poetics, and immersive field ecology 90 journal of effective teaching in higher education, vol. 4, no. 3 in addition, students will write a reflective piece at the end of each day. these entries should record thoughts about what you have experienced, such as your emotional/intellectual/spiritual response to something you saw, learned, or took part in. additional detail on expectations for writing journal/field notebooks will be provided by your instructors. final project students will submit a final written project based on their experiences during the field portion of the course. written projects may be natural history writing, creative nonfiction, or any genre of fiction. students will work closely with instructors and other students, including the submission of drafts, to complete their projects. the final written project must be an original work completed by each individual student. projects that do not meet this requirement will receive an “f.” please see avila’s policy on plagiarism. comprehensive essay toward the end of the semester, students will submit an essay discussing the theme of the course: the integration of scientific and creative modes of inquiry. detailed expectations regarding this comprehensive essay will be provided by instructors. tentative course schedule the instructors reserve the right to make adjustments during the semester if deemed necessary. the schedule of classroom meetings does not always reflect online work and assignments. this is a blended course, so you must check canvas for updates on course work and assignments. students must complete the assignments on the due dates given. date topic readings aug 28 course overview, logistics sep 4 prairie ecosystems savage chapters 1-4 sep 18 natural history and creative writing assigned canvas readings sep 27sep 30 field trip to konza prairie (detailed daily schedule is below.) savage chapters 5-9 assigned canvas readings oct 9 no class meeting, but there will be a canvas quiz oct 23 exam final project prospectus due nov 6 final project due for review and revision self-assessment due nov 20 final presentations revised project due dec 4 comprehensive essay due science, poetics, and immersive field ecology 91 journal of effective teaching in higher education, vol. 4, no. 3 konza prairie schedule note: we will not have access to the internet at konza prairie, so canvas readings must be completed ahead of time. thursday, september 27 8:00am meet on campus to depart. 11:00am arrival at konza prairie biological station. settling in. 12:00pm lunch 1:00pm orientation and walk through the facilities. 2:00pm visit to nature trail. plant identification; observation writing exercises. remember to take your field notebooks, field guides, water, sunscreen, and snacks. 5:00pm return to cabins. 6:30pm dinner after dinner: group reflection, reading, journaling friday, september 28 9:00am field exercise: 20 questions, hypothesis building and experimental design. 12:00pm lunch 1:00 pm writing exercises in the field (haikus and odes); work on field journals. 6:30pm dinner 7:30pm night walk after walk: group reflection, reading, journaling saturday, september 29 9:00am tour of reserve and visit to buffalo area with dr. eva horne. conservation, management, and research at konza prairie. 12:00pm lunch 1:00pm project planning/reflection/journaling. individual meetings with course instructors. 6:30pm dinner 7:30pm discussion: prairie conservation. reading, journaling. sunday, september 30 7:30am cleanup cabins and pack. remember that if you use konza prairie linens, you must wash them before departing. 9:00am departure 11: 30am arrival at home campus science, poetics, and immersive field ecology 92 journal of effective teaching in higher education, vol. 4, no. 3 bibliography of literary readings blake, w. (1803). auguries of innocence. https://www.poetryfoundation.org/poems/43650/auguries-of-innocence crosby, c. (2003). pulling weeds. in j. t. price (ed.), the tallgrass prairie reader (pp. 311-318). university of iowa press. dillard, a. (2007). living like weasels. teaching a stone to talk: expedition and encounters. harpercollins. gruchow, p. (1995). what the prairie teaches us. in j.t. price (ed.), the tallgrass prairie reader (pp. 311-318). university of iowa press. hasselstrom, l. (2003). coffee cup café. land circle: writings collected from the land. fulcrum press. jordan, t. (1994). playing god on the lawns of the lord. in j. t. price (ed.), the tallgrass prairie reader (pp. 231-237). university of iowa press. keats, j. (1820). to autumn. https://www.poetryfoundation.org/poems/44484/toautumn least-heat moon, w. (1999). prairyerth: a deep map. mariner books (reissue edition). price, j. t. (2008). man killed by pheasant. in j.t. price (ed.), the tallgrass prairie reader (pp. 311-318). university of iowa press. rossetti, c. (1862). precious stones. in louis untermeyer (ed.), the golden treasury of poetry (p. 306). golden press. shelley, p. (1820). ode to the west wind. https://www.poetryfoundation.org/poems/45134/ode-to-the-west-wind van etten, w. m. (1978). three worlds. in j.t. price (ed.), the tallgrass prairie reader (pp. 168-183). university of iowa press. science, poetics, and immersive field ecology 93 journal of effective teaching in higher education, vol. 4, no. 3 appendix b grading rubric for final creative project criteria point range engaging: the piece is interesting, capturing and holding attention throughout. language level is appropriate for the target audience. observes conventions for the specific genre (poetry, short story, natural history essay, etc.). 5-0 scientific accuracy: reflects lived experience in the field. creative work shows clear evidence of grounding in scientific knowledge. with the exception of creative license that serves the objectives of the work (e.g., allowing animals to talk in children’s books), all scientific information is correct. 5-0 language command: reflects mastery of standard american english. 3-0 process: reflects active engagement with professors’ written feedback and comments. 2-0 appendix c grading rubric for comprehensive essay criteria point range modes of inquiry: articulates a clear understanding of both scientific and creative modes of inquiry, including the strengths and limitations of each. 3-0 integration: thoughtfully discusses the value of integrating scientific and creative modes of inquiry in understanding a natural system such as the prairie. 3-0 cites specific examples from readings and course activities that appropriately illustrate essay contentions. 2-0 language command: reflects mastery of standard american english. 1-0 adheres to page and format instructions. 1-0 microsoft word cogswell et al final version.docx journal of effective teaching in higher education, vol. 2, no. 2 agile and active: sustaining pedagogical change in a large-enrollment calculus course cynthia a. cogswell, phd, ohio university, cogswell@ohio.edu scott pauls, phd, dartmouth college adrienne gauthier, dartmouth college erin desilva, dartmouth college abstract. it is well documented that the use of active learning strategies increases student learning (freeman et al., 2014; prince, 2004; springer, stanne, & donovan, 1999). a key difficulty in innovating college mathematics is identifying and sustaining what works for both students and the faculty. this study discusses efforts to innovate and sustain curricular change in introductory calculus at a private, elite institution. to examine if incorporating active learning strategies made a difference in student performance, student grades in the redesigned course and performance in subsequent courses were analyzed. using austin’s 2011 framework to understand the context in which the course redesign took place, individual faculty and contextual barriers and “levers” to sustain change are discussed. findings are applicable to other stem disciplines and to colleges and universities in general. next steps in this research include identifying how to scale change, including, perhaps, networks of faculty to implement and spread the reform on campus. keywords: stem retention, sotl, active learning introductory courses play a critical role in introducing students to the content of the discipline and the potential of the field. it is well documented that the use of active learning strategies increases student learning (freeman et al., 2014; prince, 2004; springer, stanne, & donovan, 1999), retention of underrepresented populations in the course, and indirectly affects student departure decisions (braxton et al., 2008; braxton, milem, and sullivan, 2000). students report greater engagement in courses where instructors signal openness to student questions, discussion, and peer collaboration (gasiewski et al., 2012; mazur, 2009). peer instruction and group work, especially when structured to increase diversity, lead to lower dropout rates among women and racial and ethnic minorities (blickenstaff, 2005; cole, 2007; panitz, 1999; toppings, 2005; watkins & mazur, 2013). drawing on individual students’ experiences and knowledge in class discussions and providing opportunities to challenge the professor’s ideas better engage a diverse student body (hooks, 1994). in a study of over 2000 classes, taught by more than 500 stem faculty, at 25 institutions, stains et al. (2018) found that although the methods of and practices in lecturing vary, the lecture is still a very prominent method of instruction. of the observations, 55% featured lecture 80% of the time. organizations are looking for ways to create and sustain broad reform, but not much is currently generalizable agile and active 2 journal of effective teaching in higher education, vol. 2, no. 2 for how to sustain curricular change once the redesign is complete (association of public & land-grant universities, n.d.; bressoud, mesa, & rasmussen, 2015). innovating college mathematics is not novel; the difficulty is identifying and sustaining what works for each campus context, students, and faculty. common efforts to integrate active learning into college mathematics have included flipping courses, using adaptive technology, utilizing active learning spaces, and integrating peer learning (association of public & land-grant universities, n.d.; bressoud, mesa, & rasmussen, 2015; fain, 2015; najmabadi, 2017). the gateway initiative dartmouth college is a mid-sized, private, liberal arts institution. a member of the ivy league, dartmouth is known to emphasize teaching through a “scholar-teacher” model. in 2014, dartmouth college undertook its own program to support curricular change. the dartmouth center for the advancement of learning (dcal) unveiled an incentive program, named gateway, that would provide resources to faculty to redesign existing introductory courses. gateway is designed to address several scarcities in teaching: a flexible and robust budget for teaching assistants, media services, and technology; partnership opportunities with non-faculty educators providing instructional design and project management support; evaluation and assessment services; and finally, status, reward and acknowledgement. all faculty were invited to apply to gateway, and among courses selected in 2014 was mathematics 3, an introductory calculus course. the course annually enrolls ~300 students, with multiple sections offered in both fall and winter terms. mathematics 3 is a prerequisite to courses in the mathematics, physics, economics, chemistry departments, and more. unlike other gateway courses, where faculty teach alone, rotate teaching, or teach in partnerships, mathematics 3 traditionally has a “teaching team” model, with a single course coordinator who oversees the curriculum and 3-4 new instructors teach sections of the course during the year. has incorporating active learning strategies made a difference in student performance? what is the context in which this change occurred, and what were the barriers and levers for sustaining change? this study explores these research questions for mathematics 3. mathematics 3 the course coordinator, pauls, submitted the mathematics 3 application for gateway. once selected as a gateway awardee, pauls worked with an instructional designer to revise the course. the revised course uses a mixture of online and written homework sets, where the online platform provides real-time feedback to instructors on student progress. the feedback is aggregated at multiple levels, allowing instructors to adapt class content appropriately. students also completed problem sets in class in fixed groups of 4-6 members. over the instances of the course, learning fellows (lf) worked in the course to mentor the groups, and each lf was responsible for 3 or 4 groups (dartmouth learning fellows, n.d.). lf’s are paid undergraduate teaching assistants recruited to facilitate group work and agile and active 3 journal of effective teaching in higher education, vol. 2, no. 2 interaction. the lf model is based on the work of mchenry, martin, castaldo and ziegenfuss (2010). additionally, graduate teaching assistants run drop-in problem sessions and tutorials. this multi-level and multi-modal instructional approach aims at acknowledging and mitigating student heterogeneity upon entry to the course. this structure echoes a principle elucidated by the dana center at university of texas, austin: “the aim isn’t to water down math requirements but to provide ‘the right math for the right student at the right time’” (najmabadi, 2017). students at this institution are unusually well prepared in mathematics. the average sat math score is 727 out of 800 for the class of 2020, and typically over half the incoming class has placement out of one or more courses in the calculus sequences via advanced placement, a-level, international baccalaureate, or local placement exams. historically, enrollees are a mixture of students who have generally solid pre-calculus preparation, most of whom have seen some or all of the material in the course before, and students with some deficiencies in their precalculus training. this heterogeneity presents challenges in aim and scope for mathematics 3, as well as for placement. the course redesign sought to address these difficulties. as sections of mathematics 3 are often taught by a mixture of permanent and temporary faculty, as well as graduate students, the redesign focused significant effort on creating a course infrastructure which minimizes startup time for new instructors. it also provides a degree of uniformity across instances of the course. since the initial course revision in 2014, all sections of mathematics 3 share the same syllabus, learning outcomes, a core set of homework sets that are common between the sections, common exams, and final grades that are set by the same rubric. mathematics 3 sections use small groups and learning fellows (lfs) to facilitate in class discussion and problem solving. further, faculty designed modular curricular with a menu of active learning components to complement content delivery. subsequent to each instance, instructors revised and added to content reflecting course experiences. norms for the course as well as revisions are managed by one course coordinator teaching the course regularly throughout the evaluation period, providing continuity as well as training and guidance for instructors new to the course. now, instructors, instructional designers, and lfs have weekly meetings to assess student progress, resolve problems, and engage in training around issues arising in the active learning components of the course. this “just-in-time” component augments baseline training by providing instructors and lfs with practical, timely tools to respond directly to course challenges. these efforts in sameness across section and instructor are aimed at creating environments where students from all sections finish the course with similar mastery of the content and preparedness for what they do next, whether that is in mathematics or some other field. agile and active 4 journal of effective teaching in higher education, vol. 2, no. 2 theoretical/conceptual framework this research uses austin’s (2011) model for understanding faculty members’ teaching-related decisions (figure 1). we use this model to analyze faculty work and choices made regarding teaching. this model is a natural choice as it places strong emphasis on the context, it also has the potential to shed insight, tease out unique characteristics, and identify barriers or levers that have promoted change in faculty behavior. methodology this research took an intrinsic case study approach. in this, the researcher chooses what is to be studied, emphasizing that the case is of primary interest. stake (2003) wrote, “i call a study an intrinsic case study if it is undertaken because, first and last, the researcher wants better understanding of this particular case.” the case is not selected because it represents all other cases or because it emphasizes a particular problem, but because of “its particularity and ordinariness” (p. 136). case study research seeks to understand human phenomena in all of its bounded complexity. the mathematics 3 course in the gateway context is of particular interest because of the way change occurred and has been sustained over three years. this research seeks to understand the particularity and ordinariness of the mathematics 3 course in the gateway context. figure 1
austin’s original model case study lends itself to reflective research, multiple data sources, and teasing out the context in which the case exist. this study uses interview data, course artifacts, agile and active 5 journal of effective teaching in higher education, vol. 2, no. 2 student grades, and instructor and instructional designer reflections. these artifacts informed the case as they materialized as data sources. course artifacts, focus groups, and interviews became available at essentially the same time; course grades followed much later; followed even later by this co-constructed paper, making meaning of the case together: the outside-researcher and the insideparticipants. intrinsic case study enables the researcher to interact with the case, and to converse with key players about their lived experiences; which is precisely how this paper was composed. data data collection began after obtaining irb approval for this study. to understand the impact of the redesign, student focus groups were conducted (three one-hour focus groups) and faculty were interviewed (three one-hour interviews with the course coordinator). the focus groups asked students about teaching and interactions in class, preparations for class, giving and receiving feedback, and quality and frequency of assessments. faculty interviews included questions on gateway, course change, and teaching. we utilized institutional student data from mathematics 3 between 2009 and 2016, which reflects five years of data before the revision and three years of data with the revision. additionally, narratives from the course coordinator and instructional designers were collected. the research questions are examined separately followed by a discussion. measuring impact: student performance to answer our first research question (has incorporating active learning strategies made a difference in student performance?), we compared student performance in the revised course to a cohort of students from a five-year window before the revision, utilizing descriptive statistics as well as two-sample tand kolmogorovsmirnoff (ks) tests. the ks-test is less well known than the t-test; it compares two samples and evaluates whether they have been drawn from the same unknown distribution (corder & foreman, 2014). overall grade distributions differed slightly. quartiles for the revised course were (c, b-, b+) while those for the five-year comparison window were (c, b, b+) but the distributions cannot be distinguished statistically by either the tor ks-test. we see a rise in the number of withdrawals students who choose to drop the course with penalty in the revision where 8.3% of students withdraw versus 5.8% in the historical instances. withdrawal rates give us our initial measure of persistence as it shows the percentage of students who exit the course before completion. table 1 shows withdrawal rates across several demographic categories while table 2 shows the raw counts. while in all but one category asian/asian american/pacific islander withdrawals rose in the revised course, the magnitudes were unevenly distributed. women withdrew at lower rates than men. among ethnic categories, hispanic and latinx students fared the best in the revised course with only a slight uptick in withdrawal rates. white (nonhispanic), black (non-hispanic), and multi-racial students’ withdrawal rate increases are almost identical to the overall increase, while native american students fared worse. agile and active 6 journal of effective teaching in higher education, vol. 2, no. 2 we examined longer term persistence by looking at outcomes of students who took the sequel calculus course in the term after completing the target course. historically, these persistence rates are 14.9% while the revised course saw a rate of 15.3%. student grade outcomes in the sequel calculus course to the revised course were significantly better than those of comparable students historically (see table 3). the two-sample t-test is significant (p<0.01) as is the two-sample ks-test (p<0.05) for the sequel course grades when restricted to students who got less than an a in the target course. over the entire range of grades the p-value is approximately 0.06 (t-test) and 0.15 (ks-test). table 1 withdrawal rates for students in revised and unrevised versions of the target course course

 all female male aaapi bnh hl mr na wnh unrevised target 5.8% 4.6% 6.9% 2.3% 7.3% 8.4% 10.0% 16.2% 4.7% revised target 8.3% 6.8% 9.8% 1.8% 10.2% 8.9% 12.8% 20.3% 7.4% percentages of students withdrawing from the target course, broken down by gender and ethnic/racial categories. aaapi=asian/asian american/pacific islander; bnh=black, nonhispanic; hl=hispanic/latinx; mr=multi-racial; na=native american; wnh= white, nonhispanic. table 2 withdrawal totals for students in revised and unrevised versions of the target course course all female male aaapi bnh hl mr na wnh unrevised target 83 (1428) 31 (671) 52 (755) 4 (172) 17 (231) 13 (155) 2 (20) 11 (68) 31 (656) revised target 80 (958) 32 (468) 48 (489) 2 (113) 17 (166) 10 (112) 5 (39) 15 (74) 31 (418) raw counts of students withdrawing from the target course (total enrollment in parentheses), broken down by gender and ethnic/racial categories. aaapi=asian/asian american/pacific islander; bnh=black, non-hispanic; hl=hispanic/latinx; mr=multi-racial; na=native american; wnh= white, non-hispanic. table 3 grades in a sequel course conditioned on grades in the target course letter grade target course grade quartiles
sequel course
(after unrevised target) grade quartiles
sequel course (after revised target) d (f,d,d) (f,c-,c) c(f,f,d) (f,c+,c+) agile and active 7 journal of effective teaching in higher education, vol. 2, no. 2 c (f,d,c) (c,c+,c+) c+ (f,d,c+) (c-,c+,b) b(d,c,b-) (c,c+,b) b (c,b-,b+) (b-,b,b+) b+ (b-,b,b+) (b-,b+,b+) a(b,b+,a-) (b,b+,a-) a (b+,a,a) (b+,a,a) we show the median grade in the sequel for students coming from the target with a specific grade. we see statistically significantly higher grades in sequel for students with lower grades (less than a) in the revised target than similar students from the unrevised version (two sample t-test, p<0.01, two sample ks-test, p<0.05) analyzing the case in this section we use austin’s (2011) framework to examine the context in which this case took place, addressing the second research question, what is the context in which this change occurred, and what were the barriers and levers for sustaining change? austin’s model, figure 1, involves labeled concentric circles, layering individual background characteristics, values, and training, within the organizational environment and all its many potential barriers or ‘levers’ to promote change in faculty behavior. the analysis is organized in three sections: (1) relevant individual characteristics, (2) contexts affecting faculty practice, and (3) barriers and/or levers impacting faculty teaching practices. intrinsic case study lends itself to reflective research, multiple data sources, and teasing out the context in which the case exist. the following sections do just this. to examine the layers within austin’s framework, cogswell and pauls wrote the analysis of the case together. cogswell drafted the initial analysis, pauls, reviewed, edited and provided feedback. after multiple exchanges and conversations, they arrived at a shared description of the case. the paragraphs below reflect their work; pauls drawing on his experience and cogswell drawing on interview data, course artifacts, student focus groups, and end of term student evaluations. relevant individual characteristics prior experience. pauls is a tenured faculty member. he earned his phd in mathematics from university of pennsylvania and bachelors in mathematics from columbia university. during his graduate coursework he received recognition for teaching several times over (5 terms). at the outset of the redesign, pauls had over 15 years of teaching experience in mathematics. calculus is a core mathematics course the “bread and butter” of most departments as it draws the highest enrollments. as with most mathematicians, he has taught calculus at several levels regularly throughout his career. prior to gateway, pauls taught math 3 and all of his courses using predominately lecture and or interrupted lecture formats, and the latter only in the last few years before the gateway redesign as he began to experiment with different approaches. when applying to be a part of gateway, he mentioned wanting to integrate video agile and active 8 journal of effective teaching in higher education, vol. 2, no. 2 instruction into orientation and placement materials for the course. the proposal pauls submitted details that student and faculty evaluation of the course indicate that “the traditional lecture format hinders progress of a substantial subset of students,” which he stated, “contributes to both frustration and poor learning outcomes.” at dartmouth he has served as vice chair and then chair of the department of mathematics. in addition, he has served on a number of campus committees, including a first-generation enrichment program steering committee, advisor to first year students for mathematics, mathematics recruiting committee chair, and more. pauls seems to prioritize teaching and improving access to the mathematics field for all individuals. doctoral socialization. graduate programs oriented towards research careers do not typically have extensive pedagogical training. while programs view competence in the classroom as necessary for successful students, they do not necessarily invest resources in this direction. pauls’ pedagogical training consisted of a series of seminars where advanced graduate students mentored newer students and introduced them to teaching at the institution. the last seminar, shortly before classes started, included several faculty members who brought their perspectives. methodologically, the seminars presented only lecture oriented teaching. consequently, pauls’ teaching was oriented solely towards lecture for the majority of his career. however, noteworthy from pauls’ doctoral socialization is that while teaching in graduate school he received the moez alimohamed graduate student teaching award, as well as four departmental teaching awards. discipline. austin (2011) wrote, “disciplines have distinct cultures, including values and criteria about what constitutes excellent work and norms for the behavior of members of the field (austin, 1994, 1996; becher, 1987)” (p. 7). the mathematics discipline at dartmouth college approaches teaching in alignment with the field broadly. it neither encourages nor dissuades attention from teaching. instructors and faculty have conversations about teaching, just like they do about their research. career stage and appointment type. austin’s framework states that “early career faculty members and doctoral students show that they are often eager to share their passion for their discipline and fields with novice learners,” but pauls is not an early career faculty member. pauls is a tenured full professor with many years of classroom experience. with tenure and promotion in the past, pauls has the flexibility and experience to devote time and energy to this curricular project. yet, pauls still has enthusiasm to “share [his] passion for [the] discipline and fields with novice learners,” as evidenced in his voluntarily undertaking this project. pauls shared that he took on this project to improve the curriculum of the course and to create a better alignment between student readiness and course content. pauls believes that if the mathematical community wishes to bring more people into mathematics (and stem more broadly) and address the problems of uneven representation, then we have no choice but to rethink how we educate and train agile and active 9 journal of effective teaching in higher education, vol. 2, no. 2 students. such solutions will not be quick or easy, but rest on sustained effort, evaluation, and revision. faculty motivation. an historical look at student outcomes prompted the beginning of this work. pauls’ started the work on improving mathematics 3 before he knew of the gateway initiative. analyzing student outcomes both formally and anecdotally showed the unsurprising result that calculus instruction, including his own, was not terribly effective. particularly troubling were the results for students from groups underrepresented in mathematics, who disproportionately left the field. these analyses prompted the revision. moreover, other efforts to incorporate new pedagogical methods particularly those under the umbrella of active learning at dartmouth provided examples and templates for work in mathematics. support from dcal multiplied the benefits of the examples by providing support and research tools. to some degree, one could question how much the support from the gateway initiative influenced what happened. if the gateway initiative did not exist, would as much work as has been done taken place? probably not. contexts affecting faculty practice in this section the next layers of austin’s framework, the institutional, department and external contexts, are discussed. institutional context. as stated earlier, dartmouth college is a mid-sized, private, liberal arts institution. a member of the ivy league, dartmouth is nearly 250 years old. sometimes the age of the institution is used in jest to explain why change is slow on campus. as pauls stated, scholarship, teaching, and service are considered in tenure review in the order listed. how teaching is reviewed varies but it is through a combination of a self-authored essay, seeking comments from past students, and a review of end of term student evaluations of teaching. beyond tenure review, excellent teaching is recognized on campus formally and informally. formally, the dean of faculty annually recognizes about ten faculty for instruction and there are five named teaching awards. the awardees are selected by either the dean of faculty, deans of the college, or the dartmouth center for the advancement of learning. these awards do not include significant reward (e.g. a course buy-out), but some include a stipend. informally, faculty learn of excellent teaching through conversation. they learn from conversations amongst themselves, within and across departments. campus news and blog posts feature teaching innovations and those who have received awards. also, faculty learn of and about teaching from conversations and events with instructional designers. departmental context. mathematics faculty value strong and effective pedagogy. dartmouth faculty reflect this commitment, aiming to excel both in their agile and active 10 journal of effective teaching in higher education, vol. 2, no. 2 research and pedagogical endeavors. this context provided both motivation and support for the curricular and pedagogical revision. many faculty experiment with novel teaching approaches to fine tune their instruction and, by mid-career, typically settle upon a collection of techniques they find most effective for the course they teach. the department chair assigns courses each year based on faculty requests balanced across curricular needs. the chair also assigns experienced permanent faculty as course supervisors for courses where less experienced instructors are teaching. junior faculty have teaching mentors (both formally and informally) who visit classes, review materials, and give advice. course supervisors mentor new post-doctoral instructors and other visitors as they join the department and the teaching faculty. dartmouth college’s graduate program is something of an exception, where students are required to work through a rigorous theoretical and practical pedagogical curriculum before entering the classroom at teachers. many introductory courses use advanced undergraduates as graders and often have graduate students assigned as teaching assistants. the learning fellows program extends pedagogical support teams which, in particular, makes it feasible to run intensive active learning in larger classes. creating the new structure for calculus and refining the active learning components to fit dartmouth college’s students and curriculum does present a barrier for broader adoption. some members of the mathematics faculty are still skeptical of using these approaches in their own classrooms. two factors have helped in broadening interest: our analysis of outcomes and demonstration of success in later parts of the curriculum, and our construction of a library of materials and methods for bringing active learning into the classroom. the first helps convince skeptics of the usefulness of the methods, particularly in the context of courses in their own department while the second lowers the time barrier for adoption. the latter has also been very helpful when bringing in new and/or less experienced instructors to the class. of course, this has meant more work for pauls and his team. external contexts. within the external contexts of government, the federal government leaves much of the review of educational quality to accreditors. dartmouth college is accredited and in good standing with its regional accreditor, the new england commission of higher education (neche) (2019). among accreditors, neche is known for allowing institutions more flexible expectations for student learning outcomes and measures. in compliance with neche, dartmouth degree programs have learning outcomes posted online. the mathematics department is no exception, and has degree outcomes and expectations clearly articulated. the oversight from the state government is minimal, and mostly directed at initial review of institutions seeking establishment, as opposed to ongoing review (new hampshire department of education, 2012). for these reasons, both the accreditor and the government are barriers to change. on behalf of the mathematical association of america, bressoud, mesa and rasmussen (2015) edited a volume of recommendations for content and instruction agile and active 11 journal of effective teaching in higher education, vol. 2, no. 2 of college calculus. foremost, their work calls attention to the need to make calculus more accessible to all. in addition, saxe and braddy (2015) looked at seven curricular guides published by five mathematical professional associations to reconcile their recommendations and outline a course or reform, whose fundamental finding for mathematics curriculum and instruction was that “the status quo is unacceptable” (p. 1). additional authors have had similar recommendations (association of public & land-grant universities, n.d.; fain, 2015; najmabadi, 2017). from this review and the existing research and support from these groups, we determine the scholarly associations to be levers for change. the last external context in austin’s model are employers. the national task force (2012) scanned the literature, held a series of roundtables, and vetted their findings with employers on what they desire from collegiate graduates. they concluded with five recommendations, all demanding and valuing greater emphasis on a range of student learning outcomes and competencies. for this reason, we concluded that employers are levers for change. barriers and/or levers impacting faculty teaching practices the next step in applying austin’s theory to this case is to identify if the rewards system, work allocation, professional development and leadership are barriers or levers to change. while some of these have already been described, in the paragraphs that follow they will be discussed and labeled as either barriers to innovation or levers facilitating innovation. please refer to figure 2 as a guide to reading the following paragraphs. figure 2
 austin’s model in the mathematics 3 case agile and active 12 journal of effective teaching in higher education, vol. 2, no. 2 reward systems. the rewards systems in place at dartmouth college include: tenure and promotion, teaching awards, grants to fund teaching innovations, and informal recognition. austin (2011) writes, “higher education institutions are sending strong messages about the relative value of time spent on research versus time spent on teaching” (p. 11). there is a lot of pressure on faculty to do research. there are not published, clear guidelines or expectations for scholarship, teaching and service in the tenure and promotion process, thus clouding what the emphasis should be on each part, and leaving it to each individual to determine how much time to allocate to each effort. how faculty respond to these institutional signals and messages varies. broadly, tenured faculty mentor tenure track faculty, and tend to give them less burdensome courses so that they can devote more time to research and publications. teaching awards, discussed earlier, do not have significant resources attached to them and they do not carry enough provenance to be a reason in of themselves to teach well. as stated earlier, while dartmouth expects a high level of pedagogical excellence, the tenure and promotion system does not set research, teaching, and service on equal footing. one could reason that pauls was able to execute this revision only after tenure work on such a project earlier would likely have hurt the tenure decision. in short, the rewards systems in place, purposeful or not, are barriers to curricular change. the lack of clarity on tenure and review fails to promote the importance of teaching and does not ascribe clear expectations, or values, for what is considered bad, good, or exceptional teaching. work allocation. faculty teaching expectations are decided on a departmental basis. faculty within a single department have uniform teaching loads set by the dean of their division, mitigated only by course buy-out. the number of courses taught per year by individual is 5, 4 or 3. broadly, the expectations are the same for tenure-track and tenured faculty. in contrast to the national context, dartmouth has very few adjunct faculty and rarely permits graduate students to teach. in mathematics, advanced graduate students do teach sections of mathematics 3 but with required training, and now because of gateway, with out-of-class and in-class support. additionally, dartmouth is on the quarter system so faculty teaching loads are divided across three terms, leaving faculty with the fourth term “off,” which is commonly devoted to research. for these reasons work allocation is at best a secondary condition for change. if teaching loads were to be reduced, systematically or a one-time recognition of work for innovative teaching, one could easily reason that the extra time would go to research. professional development. faculty have access to professional development for their teaching. on campus, the dartmouth center for the advancement of learning (dcal), the library, and educational technologies offer teaching resources. faculty can sign up for a one-time workshop on a topic, join a reoccurring book agile and active 13 journal of effective teaching in higher education, vol. 2, no. 2 club, and speak one-on-one about teaching related topics. dcal also offers modest funding for faculty to present on teaching at conferences away from campus. recently, dcal had an influx of funding which has been distributed through the experiential learning initiative and learning fellows (lf) program— efforts that followed after gateway. similar to gateway, faculty and staff can receive funding to support teaching innovations, and to have lfs in their classroom. these professional development opportunities have offered a faculty community, team approaches to teaching, and access to funds supporting instruction (e.g. to record videos). additional support for faculty development are the instructional designers on campus. as stated earlier, instructional designers work with faculty, supporting them in any pedagogical changes they wish to undertake, from experimenting with one assignment to an entire course redesign. the instructional designers are levers to change. another teaching opportunity, the dean of faculty offers funding for two faculty from differing departments to create a cross-division course. if faculty want to coteach or team teach a course, they submit a course proposal to the dean of faculty. this initiative is one of the only ways to co-teach a course while still receiving full teaching credit. related, if a faculty member wants to create a new course, the course must first be reviewed and approved by its aligning department or program, then the registrar, then be reviewed by the divisional council, and finally, reviewed and approved by the faculty council on instruction. from our analysis, professional development is a lever to change but only if the faculty member is interested in it themselves. it is a small lever. there are resources available if a faculty member seeks them out and values interacting with others about their teaching. leadership. austin (2011) said of leadership as a barrier or lever that “leaders at each level of the institution are important in creating a culture that encourages, supports, and rewards teaching innovations that support student learning” (p. 14). she wrote that provosts can signal the importance of teaching through their leadership, and can provide guidelines for tenure and review, “emphasizing the importance of commitment to teaching excellence as part of the review criteria and the expectations for success” (p. 14). likewise, deans and department chairs influence what is valued and can provide support and incentives for professional development. institutional leaders have not clearly defined the role of teaching in tenure review, other than to indicate its importance. the faculty handbook states amidst the expectations for promotion to associate professor, “it is not possible to enumerate specific qualifications for tenure so precisely and objectively that the need for judgement is obliviated” and, “it is difficult to define outstanding teaching in specific terms” (dartmouth college, 2016, p. 34). however, the language for evaluating scholarship is specific, stating, “the qualitative assessment of books and articles and of artistic and other professional accomplishments or contributions to agile and active 14 journal of effective teaching in higher education, vol. 2, no. 2 the larger scholarly community will be more consequential than the quality of the work,” defining a hierarchy of values. neither have the leaders illuminated the expectations for teaching beyond tenure. austin argued that a key element that fosters leadership as lever is “the presence of an institutional leader who serves as a champion, is committed to the overall [scholarship of teaching and learning] goals, and has sufficient institutional seniority to allocate institutional resources and time” (p. 14). deans and chairs do not oppose innovations in teaching, but they have limited (if any) discretionary funding to support the cost of teaching innovation or to reward faculty for exceptional work. however, as a lever, the dartmouth center for the advancement of learning does have the positioning and the funds to award faculty opportunity to innovate teaching. these funds have come from institutional leadership, so one could argue that there is executive leadership, it is merely diffused. dartmouth currently has some incentives to facilitate this type of innovation. the gateway initiative is one of them providing pauls with design and pedagogical support, financial support for learning fellows, and a like-minded community of faculty. further incentives, as mentioned earlier, the number of instructional designers on campus has doubled in recent years. however, this support is peripheral in austin’s “reward levers” – which emphasizes tenure, promotion, and salary. a second incentive, there is an institutional team that meets regularly and works together effectively. there is a stem retention committee, which includes stem faculty and institutional leaders. the group meets regularly to discuss current research, analyzes student retention behaviors, and discusses interventions to better support undergraduate stem students. beyond this committee, there is not a clear institutional vision for the goals to be accomplished. other than a general charge to continue excellent research and teaching, there is not clear direction on where energies should be directed. we view the administrative processes as barriers to change but the current leadership as levers. from our analysis, leadership is both a barrier and a lever. discussion this paper examined the impact of the redesign by examining student grades, followed by an analysis of the context in which the redesign took place. in this section we discuss the fit of austin’s framework to this case study, followed by a discussion of the intervention’s impact on student grades and academic performance. austin’s model as a framework for this case austin (2011) challenges the assumption that change resides solely with the faculty member. earlier research by fairweather (1996) examined faculty attitudes and “found that faculty perceived their rewards to be dependent on research, not teaching, including faculty from institutions with a strong emphasis historically on agile and active 15 journal of effective teaching in higher education, vol. 2, no. 2 teaching” (p. 46). individual faculty members make teaching decisions through and within their values, backgrounds, abilities, and aspirations. institutional change often does not happen because of barriers that are insurmountable, lack of incentives, recognition, or reward. why did pauls’ contribute so extensively to undergraduate instruction? pauls has tenure. so, further, why did someone with tenure undertake a project such as this? the individual faculty member, and perhaps also the departmental context, which were not evaluated as levers or barriers, seem to have tipped the scales as course change did take place despite the counting of barriers versus levers. in this analysis of mathematics 3, there were more barriers to innovation than levers (see figure 2). there was no single layer of the institution that was entirely a lever or entirely a barrier; these permeated the often “top-down” or “bottom-up” approaches to institutional change. further, it is up to our interpretation to define to what extent different layers within the model should be weighed. some factors encouraged adaptation (e.g. professional development, scholarly associations, and employers), other factors discouraged innovation (e.g. the reward system, accrediting organizations, and the government). while this analysis concluded that there were more barriers to change than levers for change, the framework is not prescriptive. austin outlined the multiple contexts, and in this analysis we filled in our understanding of the influences and interactions of various systems. next steps in this research include identifying how to scale this change, including, perhaps, networks of faculty to implement and spread the reform at dartmouth college. student intervention and impact on grades and performance over the four instances of the course, instructors tested several different combinations of in-class work ranging from almost entirely active learning techniques to an equal balance of lecture and active learning activities. student feedback both in midterm and end-of-term surveys indicate roughly equal preferences for the mixed approach. student focus groups and anecdotal follow-up shows that students with high levels of competence in calculus topics prior to taking the course hold the second preference. consequently, we conjecture that preparation and confidence in the material differentiates between the two preferences. the overall increase in withdrawal rates is potentially evidence of the failure of the revision to decrease persistence but could also be due to other factors, such as failure to adequately advise and place incoming students. however, the impact on different demographic subgroups of students provides evidence that the revision is having an impact on persistence and success. in particular, the revised course looks particularly effective among the groups of asian/asian american/pacific islander and hispanic and latinx students. moreover women, black (non-hispanic), and agile and active 16 journal of effective teaching in higher education, vol. 2, no. 2 multi-racial students seem to benefit from the revision more than others. we consider these results suggestive but not yet compelling. as advising and placement align more completely with the revised course, we expect future instances of this course to more fully answer the question of student persistence and the effectiveness of the methods utilized in the revision. student grades in the sequel calculus course compared to historical instances provide the most compelling evidence that the course revision provides students with better tools for learning and retaining mathematics. students who receive lower grades in the target course have increased performance in the sequel course compared to the historical cohort, suggesting that the success of the redesigned course lies in overcoming under-preparedness in students. given the heterogeneity of student preparation in the course, instructors found that an approach balancing active learning with lecture in roughly equal proportions proved most effective in terms of student satisfaction. outcomes, as measured by levels of success in sequel courses, are not significantly different between the instances with different delivery modes in the redesigned course, but the lack of significance may be due to small sample sizes. as results from mid-term questionnaires and end-of-term evaluations are mixed, we still have the opportunity for further gains. a central approach focuses on better initial student placement. at the beginning of the course redesign, the institution offered an alternate, two-term calculus sequence which replaced the target course for students who needed review of algebra and trigonometry alongside an introduction to calculus. after two instances of the redesign, we realized that the heterogeneity of preparation among students in the course was detrimental to overall positive outcomes, reflected in grades and end of term evaluations. consequently, we redesigned the sequence in the department to have a single course aimed at students who had seen no calculus before. in the following two instances of the redesigned course, student preparation heterogeneity significantly decreased resulting in more uniform course outcomes. we expect continued positive gains from this change as campus-wide advising on mathematics placement permeates the collective student awareness. a revolving instruction model brings challenges to sustaining curricular change (jones & harris, 2012; society for industrial and applied mathematics, 2012). consistency issues in curriculum, and what students learn across different sections of the same course, arise when sections are taught by multiple instructors. one of the limitations in college mathematics instructional models is that large sections of entry-level courses are taught by graduate students, postdocs, or visiting professors. it is the same model at this institution, where new hires are asked to teach mathematics 3 their first term on campus and teach it at most 2 or 3 more times before they leave the institution or switch to another courses. with pauls’ as the course coordinator, the changed curriculum is both sustained and extended to the revolving door of instructors—extending the investment in the course to more instructors and thus more students. agile and active 17 journal of effective teaching in higher education, vol. 2, no. 2 agile, active, and vulnerable from this examination, it is evident that incorporating active learning strategies made a difference in student performance, and through applying austin’s (2011) framework, a new understanding of the context in which change occurred, and the barriers and levers for sustaining change explored. prior to listening to how faculty change, in our experience, audiences want to know if the student learning improved. in this course, it did. pedagogical innovation is messy and matching learning design to student needs, course outcomes, and preparedness takes time. the mathematics 3 course described here is of particular interest because change has been sustained over three years. deslauriers et al. (2019) set out to study if the students and instructors who felt they learned less in active learning environments than their lecture-based peers’ feelings were true and found they were not. they discussed how a superb lecturer could create a greater feeling of learning than the more effective active learning. students feeling that lecture is more informative rails against the push towards active learning, and reinforces the vulnerability of the mathematics 3 revisions. the learning fellows, regular teaching huddles, and continued investment of pauls’ time all support the changes to the course. absent strong, stable levers, the pedagogical innovations are vulnerable—easily lost should pauls leave his role and equally lost should the funding for learning fellows or instructional designers be unsustained. while gateway is likely a temporary blip in inspiring innovation, this research has documented and shed light on why it was effective in order to ‘save’ the learning for future pedagogical innovation programs or levers. institutions are complex organizations. change is hard, value laden, and not a linear trajectory. in this case pauls is the singular continuous thread across the narrative and across the changes in mathematics 3. cogswell has considerable concern over what would happen should pauls leave his position. but the purpose of using austin’s framework was to shed light on why the change even happened in the first place. when pauls began this work, pedagogical innovation at dartmouth was not commonplace. the addition of the gateway program added a cohort of others that are engaged in similar work, dissolving some existential angst. limitations although there are some limitations to this study, including the heterogeneity of the population, and the assumption that grades are an indicator of learning and course revision success, this study contributes to the higher education research on revising mathematics curricula. while the course is still being taught, it is still evolving, and this research provides a case account of course change, impact, and how change has been sustained. agile and active 18 journal of effective teaching in higher education, vol. 2, no. 2 implications and future research research on the process and impact of active learning at ivy league institutions is limited. through this study we documented how an entry-level course changed its curriculum, sustained it over time, and what impact the changes have had on student performance. this study serves to add a deeper understanding of sustaining course change in introductory mathematics courses by examining the context in which change occurred. expanding on this work, implications for future research include direct observation of classes, interviews with other mathematics 3 instructors, and direct gains measurement of student learning in the course. additionally, because of the emphasis on small group work in mathematics 3, examining student confidence in mathematics could be insightful. conclusion higher education practitioners and scholars have been talking about change and reform in science, technology, engineering and mathematics education for three decades (kezar, gehrke, & elrod, 2015). often, the topics of tenure and pedagogical change are discussed together (tagg, 2012). stains et al. (2018) recommend that institutions revise tenure, promotion, and merit-recognition policies to align with evidence-based instruction. the reward of tenure is the oftenblamed crux of tension between research and teaching as it is, in the words of fairweather (1996), “the very structure which ensures maintenance of the disciplines [and] works against faculty involvement in teaching and learning and against developing a more successful undergraduate curriculum” (p. 105). the department is the preserver of the discipline and individual space, with tenure rewarding contribution to the discipline and teaching as a tax to the institution. this paper contributes to a research void, with few accounts of institutional change efforts (kezar, gehrke, & elrod, 2015). if we’re going to consider how to reform stem education, reform can be sparked by efforts from national, state, institution, departmental contexts, but it will not be sustained if the faculty member is not at the center. changing pedagogy and thus student learning in science, technology, engineering and mathematics has been a call to action for decades. yet the issue persists. we studied mathematics 3 to better understand the impact curricular change has had and to better understand how and why the change occurred. it is the authors’ hope that by documenting this case and using austin’s (2011) framework we contribute to the work and research on teaching interventions. conflicts of interest the authors declare that there is no conflict of interest regarding the publication of this article. agile and active 19 journal of effective teaching in higher education, vol. 2, no. 2 references austin, a. e. (2011). promoting evidence-based change in undergraduate science education. national academies national research council. retrieved from tidemarkinstitute.org association for the advancement of science [aaas]. (2011). vision and change in undergraduate education: a call to action. retrieved from http://visionandchange.org/files/2011/03/revised-vision-and-change-finalreport.pdf association of public & land-grant universities. (n.d.). student engagement in mathematics through a network for active learning (seminal). association
of public & land-grant universities. retrieved from
 http://www.aplu.org/projects-and-initiatives/stem-education/seminal/ adelman, c. (2015). to imagine a verb: the language and syntax of learning outcomes statements. national institute for learning outcomes assessment, february 2015. champaign-urbana, il: national institute for learning outcomes assessment. blickenstaff, j. c. (2005). women and science careers: leaky pipeline or gender filter? gender and education, 17(4), pp. 369-386. braxton, j. m, jones, w. a., hirschy, a. s., & hartley, h. v., iii. (2008). the role of active learning in college student persistence. new directions for teaching and learning, 115, fall 2008, pp. 71-83. braxton, j., milem, j., & sullivan, a. (2000). the influence of active learning on the college student departure process: toward a revision of tinto's theory. the journal of higher education, 71(5), pp. 569-590. doi:10.2307/2649260 bressoud, d. (2015). insights from the maa national study of college calculus. mathematics
teacher, 109(3). retrieved from
 http://www.nctm.org/publications/mathematicsteacher/2015/vol109/issue3/insights-from-the-maa-national-study-ofcollege-calculus/

 bressoud, d., mesa, v., & rasmussen. c. (2015). insights and recommendations from the maa
national study of college calculus. maa press. retrieved from:
 http://www.maa.org/sites/default/files/pdf/cspcc/insightsandrecommendatio ns.pdf brown, m. b. (2012, may 5). class flipping: good pedagogy or online hype? (blog post). retrieved from http://markbbrown.com/2012/05/05/class-flippinggood-pedagogy-or-online-hype/ agile and active 20 journal of effective teaching in higher education, vol. 2, no. 2 cole, d. (2007). do interracial interactions matter? an examination of studentfaculty contact and intellectual self-concept. journal of higher education, 78(3), 248–272. corder, g. w. & foreman, d. i. (2014). nonparametric statistics: a step-by-step approach. 288pp. wiley. dartmouth college (2016). handbook of the faculty of arts & sciences. hanover, nh. retrieved from https://faculty.dartmouth.edu/dean/sites/faculty_dean.prod/files/dean_facult y/dartmouth_fac_handbook.pdf dartmouth learning fellows (n.d.). home. retrieved from https://sites.dartmouth.edu/learningfellows/ deslauriers, l, mccarty, l. s., miller, k., callaghan, k. & kestin, g. (2019). measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. proceedings of the national academy of sciences of the united states [pnas], september 24, 2019, 116(39), pp. 19251-19257. retrieved from https://www.pnas.org/content/116/39/19251 fain, p. (2015 august 28). learning to adapt. inside higher ed. retrieved from: https://www.insidehighered.com/ fairweather, j. s. (1996). faculty work and public trust: restoring the value of teaching and public service in american academic life. allyn and bacon: boston, ma. freeman, s., eddy, s., mcdonough, m., smith, m. k., okoroafor, n., jordt, h. & wenderoth, m. p. (2014). active learning increases student performance in science, engineering, and mathematics. proceedings of the national academy of sciences of the united states [pnas], 111(23), pp. 8410-8415. gasiewski, j. a., eagan, m. k., garcia, g. a., hurtado, s., & chang, m. j. (2012). from gatekeeping to engagement: a multicontextual, mixed method study of student academic engagement in introductory stem courses. research higher education, 53, pp. 229-261. hooks, b. (1994). teaching to transgress: education as the practice of freedom. new york, ny: routledge. jones, f. & harris, s. (2012). benefits and drawbacks of using multiple instructors to teach a single course. college teaching, 60, pp. 132-139. routledge. kezar, a., gehrke, s., & elrod, s. (2015). implicit theories of change as a barrier to change on college campuses: an examination of stem reform. the review of agile and active 21 journal of effective teaching in higher education, vol. 2, no. 2 higher education, 38(4), pp. 479-506. association for the study of higher education. mazur, e. (2009). farewell, lecture? science, 323, 50-51. retrieved from http://science.sciencemag.org/content/323/5910/50 mchenry, n., martin, a., castaldo, a., & ziegenfuss, d. (2010) learning assistants program: faculty development for conceptual change. international journal of teaching and learning in higher education, 22(3), pp. 258-268. najmabadi,s. (2017 april 16). math gets a makeover. the chronicle of higher education. retrieved from: http://www.chronicle.com/ the national task force. (2012). a crucible moment: college learning & democracy’s future. association of american colleges and universities. washington, d.c. retrieved from https://www.aacu.org/sites/default/files/files/crucible/crucible_508f.pdf new england commission of higher education. (2019). colleges and universities. retrieved from https://www.neasc.org/colleges-and-universities panitz, t. (1999). benefits of cooperative learning in relation to student motivation, in theall, m. (ed.) motivation from within: approaches for encouraging faculty and students to excel. new directions for teaching and learning. san francisco, ca: jossey-bass publishing. prince, m. (2004). does active learning work? a review of the research. journal of engineering education, 93, pp. 223-232. saxe, k., & braddy, l. (2015). a common vision for undergraduate mathematical sciences programs in 2025. maa: mathematical association of america and common vision: for undergraduate mathematical sciences programs in 2025. retrieved from https://www.maa.org/sites/default/files/pdf/commonvisionfinal.pdf society for industrial and applied mathematics. (2012). modeling across the curriculum. philadelphia, pa: society for industrial and applied mathematics. springer, l., stanne, m. e. & donovan, s. s. (1999). effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: a meta-analysis. review of educational research, 69(1), pp. 21-51. stains, m., harshman, j, barker, m. k., chasteen, s. v., cole, r., dechenne-peters, s. e., eagan jr., m. k., esson, j. m., knight, j. k., laski, f. a., levisfitzgerald, m., lee, c. j., lo, s. m., mcdonnell, l. m., mckay, t. a., michelotti, n., musgrove, a., palmer, m. s., plank, k. m., rodela, t. m., sanders, e. r., agile and active 22 journal of effective teaching in higher education, vol. 2, no. 2 schimpf, n. g., schulte, p. m., smith, m. k., stetzer, m., van valkenburgh, b., vinson, e., weir, l. k., wendel, p. j., wheeler, l. b., & young, a. m. (2018). anatomy of stem teaching in north american universities. science, 30 mar 2018, pp. 1468-1470. retrieved from https://science.sciencemag.org/content/sci/359/6383/1468.full.pdf stake, r. e. (1995). the art of case study research. thousand oaks, ca: sage publications. stake r. e. (2003). chapter 5: case studies in norman k. denzin and yvonna s. lincoln’s strategies of qualitative inquiry (second edition). pp. 134-164. thousand oaks, ca: sage publications suskie, l. (2009). assessing student learning (2nd edition). san francisco, ca: jossey-bass. tagg, j. (2012). why does the faculty resist change? change: the magazine of higher learning, 44(1), pp. 6-15. retrieved from changemag.org toppings, k. j. (2005). trends in peer learning. educational psychology, 25(6), december 2005, pp. 631-645. routledge. watkins, j., & mazur, e. (2013). retaining students in science, technology, engineering and mathematics (stem) majors. journal of college science teaching, 42(5), 36–41.