































   Advancements in Agricultural Development 
  Volume 5, Issue 3, 2024 
  agdevresearch.org 

 

1. Rose Judd-Murray, Assistant Professor, Utah State University, 6000 Old Main Hill, Logan, UT 84322-6000, 

rose.juddmurray@usu.edu,  https://orcid.org/0000-0002-2045-1015  
2. Brian K. Warnick, Department Head and Professor, Utah State University, 6000 Old Main Hill, Logan, UT 84322-6000, 

brian.warnick@usu.edu,  https://orcid.org/0009-0001-8608-1897  
3. Daniel C. Coster, Professor, Utah State University, 3900 Old Main Hill, Logan, UT 84322-3900, dan.coster@usu.edu,          

 https://orcid.org/0000-0002-4801-5637  
4. Max L. Longhurst, Professional Practice Associate Professor, Utah State University, Old Main Hill, Logan, UT 84322-2805, 

max.longhurst@usu.edu,  https://orcid.org/0000-0003-3227-800X2805  
91 

 

Development and Validation of a High School Agricultural 
Literacy Assessment 

 
R. Judd-Murray1, B. K. Warnick2, D. C. Coster3, M. L. Longhurst4 

 
 

Article History 
Received: October 12, 2023 
Accepted: May 20, 2024 
Published: June 18, 2024 
 
 
Keywords 
National Agricultural Literacy 
Outcomes; agricultural education; 
agricultural knowledge; high school; 
secondary education   

Abstract 
The National Agricultural Literacy Outcomes (NALOs) are knowledge 
benchmarks for school-aged youth and are used to improve agricultural 
literacy (National Agriculture in the Classroom, 2014; National Center for 
Agricultural Literacy, 2017). Despite educational efforts, prior research 
indicated that high school populations remained at low or deficient 
literacy levels. Additionally, no agricultural literacy assessment 
instruments using the NALOs as a standardization benchmark have been 
developed for the 9-12th grades. The purpose of the study was to validate 
a summative NALO-centered assessment that could provide baseline data 
on agricultural literacy following the completion of secondary education 
(12th grade). The study followed the framework established by Longhurst 
et al. (2020) for similar assessments in elementary grades. A Delphi team 
produced 45 items for validation that were reviewed using a convenience 
sample of Utah State University undergraduate students. Those items 
were evaluated using factor, item, and discriminant analysis. Results 
finalized two 15-item assessments and determined both had acceptable 
reliability, were adequate for model fit, and were valid for the NALOs and 
the three proficiency levels. The instruments are critical tools for 
providing a standardized approach to evaluation efforts. Researchers and 
educators should use these instruments to provide comparable 
agricultural literacy data across populations to better identify trends, 
program needs, and meaningful inferences. 

 

mailto:rose.juddmurray@usu.edu
https://orcid.org/0000-0002-2045-1015
mailto:brian.warnick@usu.edu
https://orcid.org/0009-0001-8608-1897
mailto:dan.coster@usu.edu
https://orcid.org/0000-0002-4801-5637
mailto:max.longhurst@usu.edu
https://orcid.org/0000-0003-3227-800X2805


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  92 
 

Introduction and Problem Statement 
 
Agricultural literacy efforts prepare K-12 students to recognize and interpret information 
relevant for determining adult decisions regarding their health, global environment, public 
policy, and economic benefits (Hess & Trexler, 2011; Lawson & Weser, 1990; Redmond & 
Griffith, 2003). Agricultural literacy also influences positive perceptions and attitudes about 
agriculture (Specht et al., 2014). Due to its importance, literacy assessments were developed 
using a variety of benchmarks and methods for K-12 students (Frick, 1993; Leising et al., 1998, 
2000; Powell et al., 2008). While these instruments provided relevant data, Brandt (2016) and 
Longhurst et al. (2020) noted that older frameworks and definitions did not meet current 
needs. The lack of consistency in instrumentation contributed to a lack of replication and the 
ability to compare results with other populations nationwide. Both suggested that using the 
National Agricultural Literacy Outcomes (NALOs) (Spielmaker & Leising, 2013) K-12 grade-level-
banded benchmarks and the National Agricultural Literacy Logic Model (Spielmaker et al., 
2014), a validated educational framework, could provide the consistency and uniformity 
necessary for a benchmark-based agricultural literacy assessment instrument. Moving toward 
uniformity can provide a pathway for agricultural literacy assessment that is more reliable, 
comparable, and applicable across different studies, disciplines, and contexts. 

The absence of a standardized tool to measure agricultural literacy during and after high school 
limits the understanding of agricultural literacy levels in different populations. Without a 
consistent assessment, it is difficult to identify national knowledge gaps, design targeted 
education interventions, and track progress in educational efforts. Consequently, this study 
aimed to develop a summative high school (12th grade) agricultural literacy assessment using 
the NALOs as a foundational tool for future research. 

Theoretical and Conceptual Framework 
 
The framework for this study is based on Longhurst et al. (2020) who developed and validated a 
3-5th grade NALO-based agricultural literacy assessment. Their work provided a replication 
model for us; their success was centered upon two essential frameworks. 

National Agricultural Literacy Outcomes Framework 
Cosby et al. (2022) stated that the NALOs “provide the most comprehensive learning 
framework across the globe against which to measure student agricultural literacy…and they 
provide benchmarks to increase uniformity across the national education system in the USA” 
(p. 10). The NALO benchmarks were developed via Delphi using a rigorous integration of 
national grade level benchmarks and national standards for science, social studies, and health—
organized through the lens of agricultural literacy (Spielmaker & Leising, 2013). The NALOs 
reflect prior research and five cross-disciplinary themes: (a) Agriculture and the environment, 
(b) Plants and animals for food, fiber & energy, (c) Food, health & lifestyle, (d) STEM, and (e) 
Culture, society, economy & geography (National Center for Agricultural Literacy [NCAL], 2017). 
They were designed as a logic model component for K-20 assessment and program evaluation 
for National Agriculture in the Classroom programs and the National Center for Agricultural 

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  93 
 

Literacy (Spielmaker et al., 2014). The NALOs align with the AAAE Research Values (American 
Association for Agricultural Education [AAAE], 2023), providing a measure for determining the 
impacts of educational outreach. Within this study, the NALOs guided instrument item 
development through the question: What must students know or be able to do with the 
information they have learned to be proficient in the NALO standards? 

Programme for International Student Assessment (PISA) Framework 
Generally, summative assessments are cumulative to determine what students do or do not 
know. A significant limitation is determining only a pass or failing score, where a failing score 
may convey that the student lacks any understanding (Boud & Falchikov, 2006). The National 
Research Council (2009) suggested using assessments that showed a progression sequence 
because it identifies what a person can do within stages of development. Therefore, the PISA 
framework served as a guide because it “assesses students and uses the outcomes of that 
assessment to produce estimates of students’ proficiency in relation to the skills and knowledge 
being assessed in each domain” (OECD: Programme for International Student Assessment 
[OECD: PISA], 2016, p. 276). The PISA framework has well-defined parameters. The domains 
used within this framework were the five NALO themes. The domain skills (assessment items) 
were developed from very low levels of proficiency to very high levels. Following the structure, 
the easiest items focused on content knowledge and the relation to agricultural phenomena. 
The most difficult items drew upon interrelated ideas and concepts that required “an 
understanding of events, consequences, or processes” (OECD: PISA, 2016, p. 282). Within this 
context, a student’s agricultural literacy level determined their place on a sliding proficiency 
scale, ranked by how frequently they answered questions correctly. This premise followed the 
central dogma of the PISA assessment: “If a student’s proficiency level exceeds the item’s 
difficulty, the probability that the student can complete that item is high, and if the student’s 
proficiency is lower than what is required by the item, the probability for student success on 
that item is low” (OECD: PISA, 2016, p. 279). Most importantly, the NALOs were constructed in 
grade-banded levels that interrelated and overlapped, ensuring that students advanced from 
primary to advanced content, practice, and examples of complexities as they moved toward 
more sophisticated curricula. The integration of both frameworks established a well-defined 
model for (a) developing questions that represented an increase in skill and ability for better 
understanding student proficiency and (b) providing data that were representative of 
progression toward literacy. 

Purpose 
 
Guided by the conceptual frameworks, the study aimed to develop and validate a summative 
high school (12th grade) NALO-centered instrument that could assess proficiency levels of 
agricultural literacy. 

The study addressed the following research question: Is the instrument a valid and reliable 
measure of the five Grade 12 NALO themes and the proficiency stages of agricultural literacy 
(i.e., exposure, factual literacy, and applied proficiency)? 

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  94 
 

Methods 
 
There were three phases for defining the quantitative development and validation of the 
agricultural literacy assessment instrument. 

Phase One: Instrument Construction 
Longhurst et al. (2020) showed effectiveness in determining instrument items via a Delphi 
model because of the complexity of the content. Goodman (1987) noted that if the experts 
participating in the development process were representative of the area of knowledge, then 
content validity and reliability could be assumed. Messick (1995) and Sireci (1998) clarified that 
content validation added verification and critical mechanisms of construct validity. Literature 
also indicated that committee selection was an essential part of the process because it 
determined the quality of the items (Jacobs, 1996; Judd, 1972; Taylor & Judd, 1989). Therefore, 
the consideration of experts who participated in the Delphi construction of items was 
paramount. Individuals were direct experts in secondary agricultural education, curriculum 
development, agricultural policy, communications, cooperative extension and outreach, 
agribusiness, and STEM education; they were selected from multiple states and possessed 
advanced degrees or teaching certificates. In all, twelve members participated in item 
construction. Delbecq et al. (1975) suggested that ten to fifteen members were sufficient if the 
background of the Delphi subjects were homogenous. We promoted homogeneity in the 
subject selection to best represent the processing capability. 

Phase Two: Data Collection 
The convenience sample population for validation was N = 600 Utah State University students. 
Convenience sampling of college students is a prevalent approach for data collection in 
educational research (Hanel & Vione, 2016). Undergraduate student samples can be a 
legitimate solution when strongly justified, and problems can be minimized through 
conscientious research design and execution (Bello et al., 2009; Winton & Sabol, 2022). To 
carefully address these parameters, we incentivized college students with extra credit because 
obtaining data from end-of-senior-year high school students was extremely difficult due to our 
state ethics review opt-in-only rules for minors and the willingness of that population to 
participate in a survey at that specific time. We prioritized recruiting first-year, first-semester 
students but allowed older students to participate to ensure that the sample size could 
accurately accommodate the factor analysis that Comrey and Lee (1992) estimated for factor 
analysis (N = 500+) to be very good or excellent. Survey items were accessed via Qualtrics. We 
monitored the survey for three weeks; email reminders were sent to students weekly 
throughout the collection period.  

Phase Three: Instrument Validation 
We analyzed the data following procedures and processes outlined by Longhurst et al. (2020). 
First, data were organized, cleaned for non-response, and dummy coded. The highest and 
partial scores were calculated, and then an Exploratory Factor Analysis (EFA) was conducted in 
SAS (Version 9.4). The frequencies of the relationships between the proficiency stages 
determined the latent constructs. Following EFA, item analyses were conducted on items with 

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  95 
 

varying frequencies. Ultimately, the best items were identified and analyzed using Confirmatory 
Factor Analysis (CFA) and Discriminant Analysis (DA). We concluded our analysis by identifying 
the final items to construct two separate instruments. 

Limitations 
College samples tend to exhibit homogeneity toward diversity, and students may fall within the 
higher spectrum of cognitive skills (Stevens, 2011). We underscored the importance of lived 
experiences that could contribute to agricultural literacy proficiency over time. Additionally, the 
survey items were directly associated with the NALO benchmarks, resulting in correlation, lack 
of independence, and multicollinearity risks. Measures of covariance among the latent variables 
were analyzed, but CFA results should be treated with caution. Finally, using DA enabled 
determining whether differences existed between the proficiency stages. The use of DA defined 
the degree to which the instrument differentiated between the constructs. 

Findings 
 
Phase One: Instrument Construction 
The Delphi team developed survey items by integrating item content, relevance to the NALO 
demands, and effectiveness guidelines for summative assessment. Each team member was 
asked to create between three and five questions (including answers) for each NALO theme. 
The questions had to be identified by one of the three proficiency levels. The first three rounds 
identified which of the 64 questions best represented the NALO theme and the appropriate 
proficiency level. Table 1 shows an example of how a construct analysis clarified the 
requirements of the NALO benchmarks and the parameters for each proficiency level. It is an 
example of how the team combined the defined measures of the proficiency scale parameters 
from PISA, and the proficiency level descriptors. The process provided a point-by-point 
evaluation for each determinant factor required for a valid summative assessment of the 12th-
grade NALOs. From there, the fourth round eliminated questions with the lowest rankings and 
sent the remaining items to be refined by the group for the final two iterations. 

  

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  96 
 

Table 1 
 
Construct Analysis: Examples of Theme Two Items 

Item number 
& Proficiency 
level 

Assessment item 
content NALO demands Proficiency level identifiers 

2.12 
Exposure 
level 

Identify examples of 
organic nutrients. 

Lifecycles of plants and animals; 
distinguish between renewable 
and non-renewable resources; the 
importance of soil nutrients; 
compare natural cycles in 
comparison to managed lifecycles 
within agriculture; how organic 
and inorganic nutrients affect 
plant growth and development. 

Students can recognize terms; 
recall singular facts, especially 
ones that draw upon their 
personal or familiar experiences; 
recognize simple cause-and-effect 
relationships; select simple 
explanations with relevant or 
cueing support. 
 

2.12 
Factual 
literacy level 

Identify the factors 
(including cost, 
culture, 
convenience, access, 
and taste) that affect 
the population’s 
food choices. 

The variety of year-round food 
choices; food distribution and 
transportation systems; major 
factors in food choices for people 
and animals are cost, culture, 
convenience, and access; 
viewpoints on production methods 
and practices; impacts of 
transporting food due to location, 
climate, and geography; consumer 
demand influences production, 
processing, and marketing; 
consumer choices influence food 
production systems. 

Students can order, sort, analyze, 
and move/transfer knowledge 
from one area of application to 
another; draw upon moderately 
complex facts and ideas to 
construct explanations; make 
simple predictions; identify the 
relevancy of facts in context. 
 

2.12 
Applied 
proficiency 
level 

Determine 
agricultural practices 
that balance 
production and 
conservation (e.g., 
using modern 
science and 
technology). 

Importance and stewardship of 
natural resources in delivering 
agricultural products and 
maintaining the environment; 
understand the concept of 
stewardship for soil, water, plants, 
and animals; examine viewpoints 
on production methods and 
practices. 
 

Students can recognize, 
articulate, and evaluate what they 
have learned; can use abstract 
ideas or concepts to explain a 
complex phenomenon; 
demonstrate competency in the 
information that may be 
unfamiliar or novel; draw on a 
range of inter-related ideas; can 
construct complex predictions; 
internalize the significance of 
facts about ‘real-world’ 
application. 

Note. Proficiency levels adapted from the works of Joplin (1981), Roberts (2006), and the PISA 
Technical Report (OECD: PISA, 2016). 
 

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  97 
 

Thus, the team finalized 45 items (three questions for each proficiency level in each NALO 
theme). Longhurst et al. (2020) showed that 15 questions were sufficient for the final 
instrument, but more questions were reviewed to increase the probability of a valid question in 
each theme and proficiency level. 

Based on the commitment to high-quality Delphi development, replication of successful 
methods established in prior research, and connection to the best practices for summative 
evaluation, we showed that the items summatively assessed the Grade-12 NALO benchmarks 
and provided content and construct validity for each survey item. 

Phase Two: Data Collection 
The undergraduate sampling resulted in 71% of participants having completed less than one 
year of college and 89% less than two years (n = 468), with only 11% (n = 47) having completed 
three to four years but being younger than 23 years old. Qualtrics reported that 580 students 
accessed the survey, N = 515 completed the survey, and 48 did not complete the survey (89% 
response rate). We proceeded toward validation based on Comrey and Lee (1992) and 
MacCallum et al. (2001) who determined an acceptable level of N was dependent upon (a) the 
commonality of the variables, (b) the degree of overdetermination of the factor, (c) the size of 
the loading, and (d) model fit (f). These boundaries provided a conservative measure for our 
sample size, with priority given to the requirements of the factor analysis due to its importance 
in the study. 
 
Phase Three: Instrument Validation 
We coded 1 or 0 for correct or non-correct responses. Each possible response option was also 
scored as correct (1) or non-correct (0) for items with more than one correct response. This 
allowed for the allocation of partial scores for each overall item based on the percentages of 
correct responses selected by a respondent. The 45 survey items were first measured for total 
correct response (max = 34, min = 4, M = 21.34, SD = 5.44, N = 515). A maximum score was used 
to determine initial participant proficiency stages based on PISA literature (OECD: PISA, 2016, 
pp. 280–281), testing parameters, and statistical best practices. Partial total correct scoring was 
used to determine if a survey item was too difficult or if there were only poor or too difficult 
portions. An item analysis, difficulty index, and correct partial percentages were critical 
indicators for establishing the baseline measures before factor and item analysis. 

Factor and Item Analysis 
We used a structural linear equation model for EFA and CFA. Three latent factors representing 
the proficiency stages were analyzed against the items from each NALO theme. The factor 
loadings determined the influence of the proficiency groups on the scores associated with each 
survey item. The EFA measured the strength of the relationships between the proficiency 
stages (factors) and items using the percentage of correct and incorrect responses as indicator 
variables representing the NALO themes. Items that targeted EFA loading and frequency correct 
ranges were then examined with item analysis. This resulted in the construction of two 15-item 
assessment instruments. Each instrument contained three questions for each of the five NALO 
themes. Based on the allocated proficiency level, those three questions were staged from the 

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  98 
 

easiest to the most challenging item. Following EFA, we determined questions that were too 
easy, difficult, or poor and eliminated them based on frequency results. Item analyses were 
then conducted on items with varying frequencies, which were also used to determine if the 
EFA frequencies improved when specific poor answer choices were removed. We carefully 
ensured that option changes did not affect the question context. 

Separately, we conducted a CFA for each of the two 15-item assessment instruments to 
determine if the model fit was adequate. For each instrument, each item was loaded on its 
assigned factor (proficiency level) to determine if the underlying correlational structure of the 
independent variables (the five NALO themes) represented each latent factor. Table 2 shows 
that the linear structural equation estimation indicated that both instruments fit adequately. 

Table 2 
 
Confirmatory Factor Analysis Fit Summary Based on Total Correct Items 

Fit Summary  Instrument 1 Instrument 2 
Chi-square χ2 131.80 124.26 
Chi-square df 87 87 
Variance estimate χ2/df 1.51 1.43 
Adjusted Goodness-of-Fit (GFI) .95 .96 
RMSEA estimate .03 .03 
RMSEA lower 90% confidence limit .02 .02 
RMSEA upper 90% confidence limit .04 .04 
Bentler Comparative Fit Index .94 .93 
Bentler-Bonett Non-normed Index .93 .92 

Note. Root mean square error of approximation (RMSEA) 
 

Additionally, the CFA analysis determined that indicator variables significantly loaded on their 
respective proficiency stage factor (all p-values below .001), indicating that differences 
between loadings and zero were significant. Collectively, it identified an almost non-existent 
shared variance among the variables—or a considerable amount of unique variance was seen 
among them. 

The Cronbach’s coefficient across proficiency stages was measured for Instrument I (N = 515): 
Exposure (Total α = .46, Partial α = .55); Literacy (Total α = .58; Partial α = .62); Proficiency (Total 
α = .37; Partial α = .65) and Instrument II (N = 515): Exposure (Total α = .48, Partial α = .50); 
Literacy (Total α = .47; Partial α = .54); Proficiency (Total α = .29; Partial α = .38). The reliability 
coefficients are low; however, Taber (2018) noted that alpha values vary greatly by discipline. 
Additionally, high reliability may indicate that items are redundant, and the length of the 
instrument (less than 20 items) limits the alpha and complicates the process of unpacking 
internal reliability. The partial scores have higher alpha measures because they have a greater 

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  99 
 

range of possible responses. They are relevant because they identify that when questions are 
not scored strictly right or wrong, they lead to a greater understanding of where respondent 
understanding is. The alpha numbers are likely low due to multiple themes for each factor. We 
corroborated these results with Pearson’s product-moment correlation and Difficulty Indices. 
Results indicated an acceptable internal consistency and reliability level for both instruments 
because our goal was to produce non-redundant instrument items that could discriminate skill 
levels. Ultimately, the CFA showed enough evidence to substantiate the model as fitting 
adequately with a small or weak relationship between the proficiency stages. 

 
Discriminant Analysis & Summary 
DA was used to clarify the CFA results. Table 3 indicates that the cross-validation percentages 
for both instruments were extremely accurate and well within the range of p < .05. Equally 
strong re-substitution percentages were as good or better than the cross-validation results. The 
DA was the most definitive conclusion that the items aligned correctly for the five NALO 
themes, indicating that users can accurately administer either assessment to determine 
students’ proficiency levels in agricultural literacy. Users, however, should not “mix and match” 
questions between instruments because both have been independently validated in this study. 
We concluded our analysis by finalizing items to construct two separate assessment 
instruments. 

Table 3 
 
Discriminant Analysis: Cross-validation Summary Using Linear Discriminant Functions 

Proficiency Stage n Cross-validation 
% 

Cross-validation error 
estimation 

Instrument 1    
Exposure 74 97.37 .03 
Factual Literacy 261 98.86 .01 
Applicable Proficiency 175 100.0 .00 
Total 515 100.0 .009* 
    
Instrument 2    
Exposure 90 91.84 .08 
Factual Literacy 317 97.24 .03 
Applicable Proficiency 91 93.41 .00 
Total 515 100.0 .04** 
Note. *p < .01, df = 514; **p < .05, df = 514  

 
Findings Summary 
The findings showed that both instruments were valid and reliable for measuring the 12th-
grade NALO theme benchmarks and determining an agricultural literacy proficiency level. 
Future users need to understand how to use the instruments effectively. 

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  100 
 

Using the Instruments 
Determining the proficiency level of a participant is an essential part of assessment analysis. 
Practitioners can identify the proficiency stages of the two instruments by listing participants 
with a score ≥ 12 (out of 15) as applicably proficient, those with a score of 8 ≥ 11 as factually 
literate, and those ≤ 7 at the exposure level. 

Scores can be interpreted individually or using a group's mean, median, or mode. Total correct 
scores are as helpful as partial correct scores. Partial correct scores can be obtained by 
examining individual assessments to determine which NALO items were incorrect, then using 
that information to identify gaps in thematic content, misinformation, or analysis related to 
experience or agricultural exposure. If the NALO themes are used for program achievement 
goals, and students do not show consistent growth across all five themes, the score can indicate 
curricula or instructional gaps. These instruments were designed to show cumulative 
assessment for K-12 agricultural literacy development. Ideally, students who have been 
instructed throughout their primary and secondary education should be applicably proficient at 
the end of twelfth grade. Proficiency levels that are less than ideal for high school graduates 
give educators and agricultural stakeholders information that can be used to understand where 
adult consumers may need additional information to make informed agricultural decisions. 
Furthermore, although this study sought to provide summative assessment, educators' use of 
the tools as a formative measurement is encouraged. Using the instruments formatively, in 
combination with a qualitative interview, could be the most exact way to determine how 
participants perceived or misperceived a correct answer. 

Conclusions, Discussion, and Recommendations 
 
This study provided two standardized instruments that can measure agricultural literacy 
nationwide. There are now NALO-based assessments for elementary, middle, and high school 
students. By addressing the absence of a standardized tool for high school students and high 
school graduates, we fill a gap in existing literature and enhance the reliability, comparability, 
and applicability of future research in agricultural literacy. We recommend using these 
instruments to unify efforts to identify national knowledge gaps, better target educational 
initiatives, and increase study replication using consistent instrumentation. While not a 
comprehensive assessment, these instruments can impact how we implement and evaluate 
formal and nonformal agricultural education. Additionally, program planners and evaluators 
should use data from these assessments to determine the efficacy of their programs, hopefully 
leading to initiatives driven by program impacts rather than program outputs. Doerfert (2003) 
maintained that the true implications of agricultural literacy could only be seen as we study 
populations and programs over time. Instrument use within the same or similar programs can 
provide a roadmap of program efficacy that showcases which areas of agricultural literacy have 
improved over time and through which methods of instruction. 

Practitioners should work with researchers to identify populations beyond K-12 students or 
formal classroom settings (Warnick, 2022). These instruments can improve learning 
opportunities for youth and adults in community-driven events associated with agritourism, 4-

https://doi.org/10.37433/aad.v5i3.407


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  101 
 

H, community gardening, farmer’s markets, and career awareness fairs. The length of the 
assessment makes it digitally accessible in a variety of environments via a smartphone and may 
open opportunities for greater discussions on agricultural topics with event participants. 
Additionally, Minkler and Salvatore (2012) outlined that collaborative research and evaluation 
processes contributed to greater success within community-engaged programs. Researchers 
working in tandem with communities can assess agricultural literacy and tailor interventions 
based on feedback and data from community members. Land grant institutions and 
Cooperative Extension can fulfill pivotal roles in enhancing their communities by leveraging 
their resources, expertise, and outreach capabilities toward agricultural literacy assessment. 
Using these tools at local levels provides data on knowledge and fosters relationships that can 
promote trust, limit misinformation about agriculture, and encourage informed consumer 
choices. By bridging the gap between research and communities, assessment data can 
contribute to the prosperity of agricultural sectors through programs that empower individuals 
to make informed decisions that positively impact their well-being and the broader community. 

Acknowledgments 
 

R. Judd-Murray - formal analysis, investigation, writing- original draft and editing; B. Warnick - 
investigation, writing- review and editing; D. Coster - formal analysis, investigation, writing- 
review and editing; M. Longhurst - investigation, writing- review and editing.  

References 
 
American Association for Agricultural Education (AAAE). (2023). Research values of the 

American Association for Agricultural Education. 
https://aaaeonline.org/resources/Documents/FOR%20ONLINE%20(8.5%20x%2011)%20-
%20AAAE%20Research%20Values.pdf  

Bello, D., Leung, K., Radebaugh, L., Tung, R. L., & van Witteloostuijn, A. (2009). From the editors: 
Student samples in international business research. Journal of International Business 
Studies, 40(3), 361–364. https://doi.org/10.1057/jibs.2008.101  

Boud, D., & Falchikov, N. (2006). Aligning assessment with long-term learning. Assessment & 
Evaluation in Higher Education, 31(4), 399–413. 
https://doi.org/10.1080/02602930600679050  

Brandt, M. R. (2016). Exploring elementary students’ agricultural and scientific knowledge using 
evidence-centered design [Thesis]. University of Nebraska Lincoln. 
https://digitalcommons.unl.edu/natresdiss/131/ 

Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). Lawrence Erlbaum. 

https://doi.org/10.37433/aad.v5i3.407
https://doi.org/10.1057/jibs.2008.101
https://doi.org/10.1080/02602930600679050


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  102 
 

Cosby, A., Manning, J., Power, D., & Harreveld, B. (2022). New decade, same concerns: A 
systematic review of agricultural literacy of school students. Education Sciences, 12(4), 
Article 4. https://doi.org/10.3390/educsci12040235  

Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program 
planning. Scott, Foresman, and Co. 

Doerfert, D. L. (2003). Agricultural literacy: An assessment of research studies published within 
the agricultural education profession. Proceedings of the 22nd Annual Western Region 
Agricultural Education Research Conference.  

Frick, M. J. (1993). Developing a national framework for a middle school agricultural education 
curriculum. Journal of Agricultural Education, 34(2), 77–84. 
https://doi.org/10.5032/jae.1993.02077  

Goodman, C. M. (1987). The Delphi technique: A critique. Journal of Advanced Nursing, 12(6), 
726–734. https://doi.org/10.1111/j.1365-2648.1987.tb01376.x  

Hanel, P. H. P., & Vione, K. C. (2016). Do student samples provide an accurate estimate of the 
general public? PLoS ONE, 11(12), 1–10. https://doi.org/10.1371/journal.pone.0168354  

Hess, A. J., & Trexler, C. J. (2011). A qualitative study of agricultural literacy in urban youth: 
Understanding for democratic participation in renewing the agri-food system. Journal of 
Agricultural Education, 52(2), 151–162. https://doi.org/10.5032/jae.2011.02151  

Jacobs, J. M. (1996). Essential assessment criteria for physical education teacher education 
programs: A Delphi study [Doctoral dissertation, West Virginia University]. West Virginia 
University The Research Repository. 
https://researchrepository.wvu.edu/cgi/viewcontent.cgi?article=10089&context=etd  

Joplin, L. (1981). On defining experiential education. Journal of Experiential Education, 4(1), 17–
20. https://doi.org/10.1177/105382598100400104  

Judd, R. C. (1972). The use of Delphi methods in higher education. Technological Forecasting 
and Social Change, 4(2), 173–186. https://doi.org/10.1016/0040-1625(72)90013-3  

Lawson, A. E., & Weser, J. (1990). The rejection of nonscientific beliefs about life: Effects of 
instruction and reasoning skills. Journal of Research in Science Teaching, 27(6), 589–606. 
https://doi.org/10.1002/tea.3660270608  

Leising, J. G., Igo, C. G., Heald, A., Hubert, D., & Yamamoto, J. (1998). A guide to food and fiber 
systems literacy. W. K. Kellogg Foundation and Oklahoma State University. 

 

https://doi.org/10.37433/aad.v5i3.407
https://doi.org/10.3390/educsci12040235
https://doi.org/10.5032/jae.1993.02077
https://doi.org/10.1111/j.1365-2648.1987.tb01376.x
https://doi.org/10.1371/journal.pone.0168354
https://doi.org/10.5032/jae.2011.02151
https://researchrepository.wvu.edu/cgi/viewcontent.cgi?article=10089&context=etd
https://doi.org/10.1177/105382598100400104
https://doi.org/10.1016/0040-1625(72)90013-3
https://doi.org/10.1002/tea.3660270608


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  103 
 

Leising, J. G., Pense, S. L., & Igo, C. (2000). An assessment of student agricultural literacy 
knowledge based on the food and fiber systems literacy framework. Journal of Southern 
Agricultural Education Research, 50(1), 146–151. http://jsaer.org/category/journal/vol-
50/  

Longhurst, M. L., Judd-Murray, R., Coster, D. C., & Spielmaker, D. M. (2020). Measuring 
agricultural literacy: Grade 3-5 instrument development and validation. Journal of 
Agricultural Education, 61(2), 173–192. https://doi.org/10.5032/jae.2020.02173  

MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor 
analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637. 
https://doi.org/10.1207/S15327906MBR3604_06  

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ 
responses and performances as scientific inquiry into score meaning. American 
Psychologist, 50(9), 741. https://doi.org/10.1037/0003-066X.50.9.741  

Minkler, I. M., & Salvatore, A. L. (2012). Study design and analysis in dissemination and 
implementation research (2nd Edition). Oxford University Press. 

National Agriculture in the Classroom. (2014). Agricultural literacy. National Agriculture in the 
Classroom. https://agclassroom.org/get/literacy/ 

National Center for Agricultural Literacy. (2017). Multistate research: W3006. Utah State 
University. https://www.agliteracy.org/research/multistate/ 

National Research Council. (2009). Transforming agricultural education for a changing world. 
The National Academies Press. https://doi.org/10.17226/12602  

OECD: Programme for International Student Assessment. (2016). PISA 2015: Technical report. 
Organization for Economic Co-operation and Development (OECD). 
http://www.oecd.org/pisa/sitedocument/PISA-2015-technical-report-final.pdf  

Powell, D., Agnew, D., & Trexler, C. (2008). Agricultural literacy: Clarifying a vision for practical 
application. Journal of Agricultural Education, 49(1), 85–98. 
https://doi.org/10.5032/jae.2008.01085  

Redmond, E. C., & Griffith, C. J. (2003). Consumer food handling in the home: A review of food 
safety studies. Journal of Food Protection, 66(1), 130–161. 
https://doi.org/10.4315/0362-028X-66.1.130  

Roberts, T. G. (2006). A philosophical examination of experiential learning theory for 
agricultural educators. Journal of Agricultural Education, 47(1), 17–29. 
https://doi.org/10.5032/jae.2006.01017  

https://doi.org/10.37433/aad.v5i3.407
http://jsaer.org/category/journal/vol-50/
http://jsaer.org/category/journal/vol-50/
https://doi.org/10.5032/jae.2020.02173
https://doi.org/10.1207/S15327906MBR3604_06
https://doi.org/10.1037/0003-066X.50.9.741
https://agclassroom.org/get/literacy/
https://www.agliteracy.org/research/multistate/
https://doi.org/10.17226/12602
http://www.oecd.org/pisa/sitedocument/PISA-2015-technical-report-final.pdf
https://doi.org/10.5032/jae.2008.01085
https://doi.org/10.4315/0362-028X-66.1.130
https://doi.org/10.5032/jae.2006.01017


Judd-Murray et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.407  104 
 

Sireci, S. G. (1998). The construct of content validity. Social Indicators Research, 45(1), 83–117. 
https://doi.org/10.1023/A:1006985528729  

Specht, A. R., McKim, B. R., & Rutherford, T. (2014). A little learning is dangerous: The influence 
of agricultural literacy and experience on young people’s perceptions of agricultural 
imagery. Journal of Applied Communications, 98(3), 63–74. 
https://doi.org/10.4148/1051-0834.1086  

Spielmaker, D. M., & Leising, J. G. (2013). National agricultural literacy outcomes. Utah State 
University, School of Applied Sciences & Technology. 
https://cdn.agclassroom.org/nat/data/get/NALObooklet.pdf  

Spielmaker, D. M., Pastor, M., & Stewardson, D. M. (2014). A logic model for agricultural 
literacy programming. Proceedings of the 41st annual meeting of the American 
Association for Agricultural Education. National Center for Agricultural Literacy. 
https://www.agliteracy.org/research/logic/  

Stevens, C. K. (2011). Questions to consider when selecting student samples. Journal of Supply 
Chain Management, 47(3), 19–21. https://doi.org/10.1111/j.1745-493X.2011.03233.x  

Taber, K. S. (2018). The use of Cronbach's alpha when developing and reporting research 
instruments in science education. Research in Science Education, 48(6), 1273-1296. 
https://doi.org/10.1007/s11165-016-9602-2 

Taylor, R. E., & Judd, L. L. (1989). Delphi method applied to tourism. In S. Witt & L. Moutinho 
(Eds.), Gazing into the oracle: The Delphi method and its application to social policy and 
public health (pp. 56–88). Jessica Kingsley Publishers. 

Warnick, B. K. (2022). W3006: Multistate agricultural literacy research [2020 Annual Report]. 
Utah State University. https://www.nimss.org/projects/view/mrp/outline/18611  

Winton, B. G., & Sabol, M. A. (2022). A multi-group analysis of convenience samples: Free, 
cheap, friendly, and fancy sources. International Journal of Social Research 
Methodology, 25(6), 861–876. https://doi.org/10.1080/13645579.2021.1961187  

 

 
© 2024 by authors. This article is an open access article distributed under the terms and conditions of 
the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/). 
 

https://doi.org/10.37433/aad.v5i3.407
https://doi.org/10.1023/A:1006985528729
https://doi.org/10.4148/1051-0834.1086
https://cdn.agclassroom.org/nat/data/get/NALObooklet.pdf
https://www.agliteracy.org/research/logic/
https://doi.org/10.1111/j.1745-493X.2011.03233.x
https://doi.org/10.1007/s11165-016-9602-2
https://www.nimss.org/projects/view/mrp/outline/18611
https://doi.org/10.1080/13645579.2021.1961187

