Microsoft Word - 91-Manuscript-1445-1-11-20210712.docx Street et al. Advancements in Agricultural Development Volume 2, Issue 2, 2021 agdevresearch.org 1. Ashton Street, Agriculture Teacher, Harmony High School, 3601 Arthur J Gallagher Blvd., Harmony, FL 34771, ashton.street@osceolaschools.net, https://orcid.org/0000--0002-4900-0824 2. Christopher T. Stripling, Associate Professor & Interim Department Head, University of Tennessee, 320 Morgan Hall, 2621 Morgan Circle Drive, Knoxville, TN 379966, cstripli@utk.edu, https://orcid.org/0000--0002-5045-3492 3. John C. Ricketts, Professor, Tennessee State University, 3500 John A. Merritt Boulevard, Nashville, TN 37209, jricket1@tnstate.edu, https://orcid.org/0000--0002-0559-5803 4. Nathan W. Conner, Associate Professor, University of Nebraska-Lincoln, 236 Filley Hall, Lincoln, NE 68583, nconner2@unl.edu, https://orcid.org/0000--0003-0063-4110 5. Christopher Boyer, Associate Professor, University of Tennessee, 320 Morgan Hall, 2621 Morgan Circle Drive, Knoxville, TN 37996, cboyer3@utk.edu, https://orcid.org/0000--0002-1393-8589 86 Identifying Tennessee School-based Agricultural Education Student Growth and Program Accountability Metrics A. Street1, C. Stripling2, J. Ricketts3, N. Conner4, C. Boyer5 Abstract Over the years, accountability in education has transformed from the primary focus being the school as a whole to the individual teacher. The purpose of this study was to determine the metrics Tennessee school-based agricultural education teachers perceive as indicators of excellent total programs (classroom instruction, FFA, SAE), and a modified Delphi study was used to seek a consensus. The following nine metrics were retained: (a) pesticide certification, (b) program of activities, (c) number of students participating in CDEs, (d) chapter community service hours, (e) total number of FFA activities, (f) number of CDEs coached, (g) at least one proficiency at regional level, (h) one American degree every 3 years, and (i) percentage of students with SAE. Overall, the metrics agreed upon are narrow in focus and all but one is a record of activity and not direct measures of students’ knowledge or skills. As a result, the measures do not include student growth or value- added scores or authentic assessments of 21st century skills. Additional research is needed to further investigate the metrics that should be used to measure a school-based agricultural education program’s success in Tennessee and across the nation. Keywords Delphi method, total program, performance evaluation, SBAE Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 87 Introduction and Problem Statement Accountability in education transformed from the primary focus being the whole school to the individual teacher (Lavigne, 2014). A focus on teachers being accountable for student success has altered education in the United States (Buchanan, 2014). According to the Reform Support Network (2013), value-added measures “convey how much individual teachers contribute to student learning in a particular subject in a particular year” ( p. 2). However, researchers and policymakers found value-added measures alone are not an adequate replacement for traditional teacher evaluation (Harris et al., 2014). Tennessee has been a frontrunner for value-added assessment and was one of the first states to apply a statewide evaluation system (Lavigne, 2014). In the 2011-2012 school year, the Tennessee Educator Acceleration Model (TEAM) was implemented statewide (Lavigne, 2014). TEAM outlines that 50% of a teacher’s evaluation is comprised of student achievement data and within that percentage, 35% is calculated from student growth using students’ value-added scores, and the other 15% is made up of other student achievement measures as decided by the State Board of Education and agreed upon by the teacher and the evaluator (Lavigne, 2014). The other 50% is comprised of personal conferences, observation data, and a review of prior teaching evaluations (Lavigne, 2014). Glover et al. (2016) recommended using multiple types of formative assessment techniques for teacher and student evaluation. School-based agricultural education (SBAE) is one of the disciplines included in Career and Technical Education (CTE). CTE programs are heavily influenced by the Carl D. Perkins legislation, which was designed to enhance the academic and technical skill development of secondary CTE students (Johnson & Mitchell, 2014). The core indicators for secondary CTE programs focus on academics, technical skills, graduation, post-high school education, and career trajectory (Johnson & Mitchell, 2014). Due to the emphasis on value-added measures in Tennessee and the need to consider core indicators of CTE programs, this study seeks to capture the voice of Tennessee SBAE teachers regarding the metrics they perceive the Tennessee Department of Education (TDE) should use to measure student growth and total SBAE program success. Theoretical and Conceptual Framework SBAE programs are designed for students to study and investigate solutions for real problems in the agriculture industry; therefore, the content being studied is flexible and changing (Phipps et al., 2008). For this study, SBAE’s three-component model served as the conceptual framework. The model includes three main instructional components: (a) classroom/laboratory instruction, (b) student leadership development through the National FFA Organization, and (c) supervised agricultural experience (SAE) programs (National FFA Organization, 2019). Classroom/laboratory instruction provides students with the opportunity to discuss and study current and future challenges that pertain to particular areas of the agriculture industry, and Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 88 “as a result, students gain knowledge and discover principles that allow them to arrive at potential solutions to agricultural problems” (Phipps et al., 2008, p. 5.) Through participation in the National FFA Organization, “students develop excellent skills in leading and working with people of all ages” (Phipps et al., 2008, p. 7). The FFA is an intra-curricular program that offers students challenging activities and programs for both the individual student and the whole school chapter (Phipps et al., 2008). The final component, SAE, provides students with the opportunity to apply the knowledge and skills they learn within school to real-world situations through their own SAE program outside of the classroom (Phipps et al., 2008). Students will work on this program throughout their high school career and gain experiences in a real-life setting within an agriculture industry (Phipps et al., 2008). Example SAE’s include livestock production and working in an agricultural business (Phipps et al., 2008). Each component of the three-component model was used to develop an open-ended question that would be used in round one of this study. Purpose The purpose of this study was to determine the metrics Tennessee SBAE teachers perceived as indicators of excellent total programs. Additionally, the following research question guided this study: What metrics should TDE use to measure SBAE student growth and the total program? Methods A modified Delphi study was used to seek a consensus on metrics SBAE teachers perceived as indicators of excellent total programs. The modified Delphi method has been frequently used by researchers in agricultural education (Lundry et al., 2015; Terry & Osborne, 2015). Three rounds of data collection were used to solicit the opinions of an expert panel of SBAE teachers. The criteria for membership on the panel were twofold: (a) currently working as a teacher in SBAE in Tennessee and (b) nominated by members of Tennessee team agricultural education. Team agricultural education is an advisory committee for TDE consisting of teacher educators, SBAE teachers, a community college agriculture instructor, a National FFA staff member, and representatives of the Tennessee Association of Agricultural Educators. Twenty-two teachers were nominated, and 21 teachers were invited to participate in this study. We decided to invite seven teachers from each of the three regions of Tennessee and eliminate the teacher with the fewest years of experience. An email was sent to the 21 teachers informing them of the study, their selection, and the study procedures. The data collection process consisted of three rounds. Qualtrics was used to distribute a survey for each round in order to reach a general consensus. Round one consisted of the following open-ended prompts that were developed based on the three-component model of agricultural education: (a) Please list all of the possible agriculture classroom measures (i.e., TEAM scores, end of course test, certifications) that you think ought to be used to provide student achievement/program data to TDE; (b) Please list all of the Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 89 possible FFA measures (i.e., % FFA membership, CDE participation) that you think ought to be used to provide student achievement/program data to TDE; and (c) Please list all of the possible SAE measures (i.e., percentage with an SAE, dollars earned or hours worked) that you think ought to be used to provide student achievement/program data to TDE. After the initial survey was distributed and two reminders were sent, we obtained responses from 17 of the 21 teachers (81.0%) in round one. A five-point rating scale (1= strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree) was used in round two and allowed the teachers to rate their level of agreement or disagreement for each of the 59 items. We established a priori a 66.7% or 2/3 level of agreement for an item to be retained for round three based on prior works (Shinn et al., 2008; Touchstone, 2015). In addition to the 59 items identified in round one, we included the following open-ended prompt in round two: Please list additional measures you believe should be included that were not included in Round 2. The additional open-ended prompt was asked to give participants the opportunity to share any additional measures that came to mind during round two as a result of seeing the measures identified in round one. We used an open- ended prompt in round two based on prior research (Conner et al., 2017; Warner et al., 2016). After three reminders to complete round two, 18 of the 21 teachers (85.7%) responded. Therefore, the third and final round was comprised of 19 items that were rated by the teachers using the five-point rating scale. As in round two, a 66.7% or 2/3 level of agreement was used to retain an item as a possible metric for measuring a component of the total SBAE program. In round three, responses were obtained from 19 of the 21 teachers (90.5%). Findings The following categories are used to group together the findings: classroom, FFA, and SAE. Round 1 generated 59 unique responses from the expert panel of teachers (see Table 1). Fourteen items met the predetermined 66.7% level of agreement. Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 90 Table 1 Metrics Identified as Indicators of Excellent Agricultural Education Total Programs Metrics Agree/Strongly Agree % 1. Classroom 2. Master Livestock certification (beef, meat, goat, advanced, etc.) 72.3 a 3. Pesticide certification 66.7 a 4. Beef Quality Assurance certification 66.7 a 5. Hunter Education certification 66.7 a 6. Welding certification 66.7 a 7. Number of students who enroll in post-secondary studies 66.7 a 8. Master Gardner certification 55.6 9. Number of students who enroll in postsecondary agricultural studies 55.6 10. Skill attainment rubrics 55.6 11. Tractor Safety certification 55.5 12. Dual credit 50.0 13. Dual enrollment 50.0 14. End of course exam 44.5 15. Boater Education certification 38.9 16. Tennessee Educator Acceleration Model (TEAM) scores 22.2 FFA 17. Program of activities 94.5 a 18. Number of students participating in CDEs 77.7 a 19. Chapter community service hours 72.2 a 20. Average number of volunteer hours per student 66.7 a 21. Number of CDEs coached 66.7 a 22. Number of Team CDEs coached 55.6 23. Number of Individual CDEs coached 55.6 24. Number of students attending State FFA Convention 55.5 25. Regional officers elected 52.9 26. FFA membership percentage of agricultural education enrollment 50.0 27. State officers nominated 50.0 28. Plow awards program 50.0 29. Regional officers nominated 50.0 30. State officers elected 44.4 31. Number of students 38.9 32. Average number of students attending local meetings 38.9 33. Number of students receiving FFA scholarships 38.9 34. Number of students attending 212/360 Conference 27.8 35. National officer candidate 22.2 36. National officer elected 17.6 Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 91 Metrics Agree/Strongly Agree % 37. Number of students receiving FFA grants 11.1 38. Number of FFA chapter grants awarded 11.1 SAE 39. Percentage of students with SAE 77.2 a 40. One American degree every 3 years 76.5 a 41. At least one proficiency at regional level 72.2 a 42. Number of students with an SAE record book 55.5 43. Number of State FFA degree recipients each year 55.5 44. Average dollars earned per student 44.5 45. Number of American degree recipients each year 44.5 46. Average hours worked or dollars earned per student 38.9 47. Number of proficiency applications 38.9 48. Number of students exhibiting livestock 38.9 49. Average dollars expended per student 38.9 50. Average hours worked per student 35.3 51. Number of Regional Star candidates 33.3 52. Number of Regional Star recipients 27.8 53. Number of TN Star candidates 27.8 54. Number of TN Star recipients 27.8 55. Number of National FFA star candidates 22.2 56. Number of regional proficiencies in first place 22.0 57. Number of regional proficiencies in second place 22.0 58. Number of regional proficiencies in third place 22.0 59. Number of proficiencies not in top 3 16.7 60. Number of National FFA Star recipients 16.7 a Retained for Round 3. Five additional metrics were obtained from Round 2. Therefore, the remaining 19 items were retained for Round 3 (see Table 2). The metrics added were: (a) teacher is CASE certified, (b) CASE online student assessments, (c) CASE authentic assessment rubrics, (d) Pork Quality Assurance certification, and (e) total number of FFA activities. Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 92 Table 2 Metrics Suggested in Round 2 as Indicators of Excellent Agricultural Education Programs Metrics Agree/Strongly Agree % Classroom 1. Pesticide certification 68.4 a 2. Welding certification 63.2 3. Beef Quality Assurance certification 63.1 4. Pork Quality Assurance certification 57.9 5. Hunter Education certification 52.6 6. Number of students who enroll in postsecondary education 52.6 7. Master Livestock certification 52.6 8. Teacher is CASE certified 36.8 9. CASE online student assessments 36.8 10. CASE authentic assessment rubrics 36.9 FFA 11. Program of Activities 94.7 a 12. Number of students participating in CDEs 84.2 a 13. Chapter community service hours 78.9 a 14. Total number of FFA activities 78.9 a 15. Number of CDEs coached 68.4 a 16. Average number of volunteer hours per student 36.8 SAE 17. At least one proficiency at regional level 84.2 a 18. One American Degree every 3 years 78.9 a 19. Percentage of students with SAE 77.7 a a Retained as Indicators of Excellent Agricultural Education Total Programs The third and final round began with 19 metrics and resulted in the removal of 10 metrics due to lack of consensus: (a) Beef Quality Assurance certification, (b) Master Livestock certification, (c) Hunter Education certification, (d) Welding certification, (e) number of students who enroll in postsecondary education, (f) teacher is CASE certified, (g) CASE online student assessments, (h) CASE authentic assessment rubrics, (i) Pork Quality Assurance certification, and (j) average number of volunteer hours per student. The reduction of metrics left a general consensus that nine metrics (see Table 3) generated from previous rounds should be used in measuring SBAE total program success. Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 93 Table 3 Metrics Retained as Indicators of Excellent Agricultural Education Total Programs Metrics Classroom 1. Pesticide certification FFA 2. Program of Activities 3. Number of students participating in CDEs 4. Chapter community service hours 5. Total number of FFA activities 6. Number of CDEs coached SAE 7. At least one proficiency at regional level 8. One American Degree every 3 years 9. Percentage of students with SAE Conclusions, Discussion, and Recommendations The SBAE teachers ultimately agreed upon nine metrics, one metric for classroom instruction, five metrics for FFA, and three metrics for SAE as measures of total program success. Overall, the metrics agreed upon are narrow in focus, and all but one is a record of activity and not direct measures of students’ knowledge or skills. As a result, the measures do not include student growth or value-added scores (Lavigne, 2014) or authentic assessments of 21st-century skills (i.e., leadership, communication, critical thinking, and teamwork). The focus on measuring success based on activity within FFA and SAE may be due to a culture that promotes competition rather than academic learning. Therefore, we recommend research be conducted to investigate Tennessee agriculture education teachers’ perceptions of the role of the classroom instructional component of the SBAE model related to the total SBAE program. These insights may shed light on the narrow focus of the metrics identified and the lack of consensus regarding classroom instruction metrics. The one metric the panel agreed upon for classroom instruction was pesticide certification. The core indicators for secondary education included technical skill attainment, including achievement on technical assessments aligned with industry-recognized standards (Johnson & Mitchell, 2014). Based on Johnson and Mitchell (2014), pesticide certification aligns with the current requirements of Perkins IV but may not be practical in nature due to the fact an individual must be 18 years old to receive certification. Furthermore, this metric of student learning is not applicable to all SBAE programs of study in Tennessee. Additional certifications were recognized but did not pass the consensus to be retained. This may be due to the various programs of study and courses offered within Tennessee. Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 94 Due to the lack of agreement on numerous metrics in the classroom setting, TDE should pursue further consensus among SBAE educators on other classroom metrics before implementing statewide classroom accountability metrics. To that end, future research is needed to determine and, if needed, develop appropriate classroom metrics. TDE should look to other state departments of education, CTE and agricultural education consultants, national CTE and agricultural education leaders, industry partners, and SBAE teachers to further explore classroom metrics. We recommend metrics such as TEAM evaluations, meaningful industry certifications, and cognitive skills be considered. According to Lavigne (2014), the TEAM evaluations will take into consideration multiple data points, including student achievement data, personal conferences, observation data, and the review of the teacher’s prior evaluations (Lavigne, 2014). The collection and evaluation of teacher data will help administrators provide constructive feedback to educators (Firestone, 2014). The use of student achievement data both from an academic perspective and from industry certifications will help CTE programs meet the requirements set forth by the Perkins ACT (Johnson & Mitchell, 2014). In addition, student skill assessments appropriate for each SBAE program of study should be considered. In regards to FFA, metrics were agreed upon, which support the Perkins IV core indicator of having a career and technical student organization as a support for the instructional program (Johnson & Mitchell, 2014). The five metrics can be used as a starting point for accountability and should be reviewed by TDE for appropriateness. In addition, the department should review the purpose and goals of FFA as these metrics may not provide data on TDE’s goals for FFA. The panel identified three metrics focused on SAE measures. Percentage of students with an SAE supports the state’s criteria that a majority of the students enrolled in SBAE courses have in place an approved supervised agricultural experience program. At least one proficiency at the regional level is like the state’s current criteria of submitting one or more proficiency awards for state consideration. However, the state’s criteria stipulated the proficiency is submitted at the state level, which is a higher level as compared to the measure agreed upon by the SBAE teachers. The final SAE metric agreed upon, one American degree every three years, is similar to the criteria of directing and submitting one or more State FFA Degree applicants based on the student’s SAE program for state consideration. Unlike the previous metric, this agreed upon metric is a higher degree than the indicator used by TDE. The three metrics can be used as a starting point for accountability and should be reviewed by TDE for appropriateness. Furthermore, if assessing student skills is not appropriate or cost efficient in the classroom instructional component, the SAE component may be able to be used for this evaluation. Teachers, employers, and/or placement coordinators may be able to verify students are able to perform desired skills in a more cost-effective manner. Future research should explore the use of teachers, employers, and/or placement coordinators in conducting skill assessments. Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 95 References Buchanan, R. (2015) Teacher identity and agency in an era of accountability. Teachers and Teaching, 21(6),700–719. https://doi.org/10.1080/13540602.2015.1044329 Conner, N. W., Gates, H., & Stripling, C. T. (2017). Identifying international agricultural concepts for secondary agricultural education curriculum. Journal of Agricultural Education, 58(1), 118–130. https://doi.org/10.5032/jae.2017.01118 Firestone, W. A. (2014). Teacher evaluation policy and conflicting theories of motivation. Educational Researcher, 43(2), 100-107. https://doi.org/10.3102/0013189X14521864 Glover, T. A., Reddy, L. A., Kettler, R. J., Kurz, A., & Lekwa, A. J. (2016). Improving high-stakes decisions via formative assessment, professional development, and comprehensive educator evaluation: The school system improvement project. Teachers College Record, 118(14), 1–26. https://doi.org/10.1177/1534508418811589 Johnson, T. D., & Mitchell, A. (2014). Guidance for Carl D. Perkins grant recipient [Policy Manual]. Division of Post-Secondary and Career Education. https://jnc.dc.gov/sites/default/files/dc/sites/osse/publication/attachments/Guidance% 20MASTER%20POLICY%20MANUAL%20%20FINAL%20Rev%209.16.13.pdf Lavigne, A. L. (2014). Exploring the intended and unintended consequences of high-stakes teacher evaluation on schools, teachers, and students. Teachers College Record, 116(1), 1–29. https://www.researchgate.net/publication/284184049_Exploring_the_Intended_and_U nintended_Consequences_of_High- Stakes_Teacher_Evaluation_on_Schools_Teachers_and_Students Lundry, J., Ramsey, J. W., Edwards, M. C., & Robinson, J. S. (2015). Benefits of career development events as perceived by school-based, agricultural education teachers. Journal of Agricultural Education, 56(1), 43–57. https://doi.org/10.5032/jae.2015.01043 National FFA Organization. (2019). Agricultural Education–About FFA. The National FFA Organization: https://www.ffa.org/agricultural-education/ Phipps, L. J., Osborne, E. W., Dyer, J. E., & Ball, A. (2008). Handbook on Agricultural Education in Public Schools (6th ed.). Cengage Learning. Reform Support Network. (2013). Measuring teacher matters: What different ways of looking at student results tell us about teacher effectiveness. https://www2.ed.gov/about/inits/ed/implementation-support-unit/tech- assist/measuring-teaching-matters.pdf Street et al. Advancements in Agricultural Development https://doi.org/10.37433/aad.v2i2.91 96 Shinn, G. C., Briers, G., & Baker, M. (2008). Forecasting doctoral-level content in agricultural education: Viewpoints of engaged scholars in the United States. Journal of Agricultural Education, 49(1), 121–131. https://doi.org/10.5032/jae.2008.01121 Terry, R. D., & Osborne, E. (2015). Fundamental dimensions and essential elements of exemplary local extension units. Journal of Agricultural Education, 56(2), 43-63. https://doi.org/10.5032/jae.2015.02043 Touchstone, A. J. L. (2015). Professional development needs of beginning agricultural education teachers in Idaho. Journal of Agricultural Education, 56(2), 170–187. https://doi.org/0.5032/jae.2015.02170 Warner, L. A., Stubbs, E. Murphrey, T. P., & Huynh, P. (2016). Identification of the competencies needed to apply social marketing to extension programing: Results of a Delphi study. Journal of Agricultural Education, 57(2), 14–32. https://doi.org/10.5032/jae.2016.02014 © 2021 by authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).