































   Advancements in Agricultural Development 
  Volume 5, Issue 3, 2024 
  agdevresearch.org 

 

1. Lendel Narine, Extension Associate Professor, Utah State University, 4800 Old Main Hill, Logan, UT 84322, 
lendel.narine@usu.edu,  https://orcid.org/0000-0001-6962-2770 

2. Arlene Enderton, Program Specialist, Iowa State University Extension and Outreach, 2625 N Loop Drive, Suite 2430, Ames, 
Iowa 50011, arlene@iastate.edu,  https://orcid.org/0009-0004-7416-6203 

3. Matt Benge, Director, Program Development and Evaluation Center (PDEC), University of Florida, PO Box 112060, 
Gainesville, FL 32610, mattbenge@ufl.edu,  https://orcid.org/0000-0002-5358-3233 

4. Elizabeth Bihn, Director, Produce Safety Alliance, Cornell University, 665 W. North Street, Food Research Lab – NYSAES, 
Geneva, NY 14456, eab38@cornell.edu,  https://orcid.org/0000-0001-5806-3491 

5. Stephanie Brown, Food Safety Specialist, Oregon State University, 1207 NW Naito Parkway, Portland, OR 97209, 
stephanie.brown@oregonstate.edu,  https://orcid.org/0000-0001-8682-9984 

6. Jovana Kovacevic, Associate Professor and Extension Food Safety Specialist, Oregon State University, 1207 NW Naito 
Parkway, Portland, OR 97209, jovana.kovacevic@oregonstate.edu,  https://orcid.org/0000-0003-1254-4294 

7. Elizabeth Newbold, Assistant Director, NE Center to Advance Food Safety, University of Vermont, 310 Main St, PO Box 559, 
Bennington, VT 05201-0559, Elizabeth.Newbold@uvm.edu,  https://orcid.org/0000-0002-0862-4593 

8. Keith Schneider, Professor, University of Florida, 359 FSHN Bldg., Newell Dr., Gainesville, FL 32611, keiths29@ufl.edu,  
 https://orcid.org/0000-0003-0145-3418 

9. Angela Shaw, Professor, Texas Tech University, Box 42141, Lubbock, TX 79409-2141, Angela.Shaw@ttu.edu,  
 https://orcid.org/0000-0001-5908-0429 

13 

 

Produce Safety Alliance Grower Training Knowledge 
Assessment Results 

 
L. Narine1, A. Enderton2, M. Benge3, E. Bihn4, S. Brown5, J. Kovacevic6, E. Newbold7, K. Schneider8,  

A. Shaw9 
 

 
Article History 
Received: February 16, 2024 
Accepted: March 19, 2024 
Published: April 16, 2024 
 
 
Keywords 
evaluation; fruits; vegetables; 
knowledge; assessment; grower 
training; Food Safety Modernization 
Act; produce safety; SDG 3: good 
health and wellbeing   

Abstract 
The Produce Safety Alliance (PSA) Grower Training (GT) curriculum was 
developed to provide produce growers with training that meets 
requirements in §112.22(c) of the Food Safety Modernization Act (FSMA) 
Produce Safety Rule (PSR). Four regional food safety centers evaluated 
course attendees’ knowledge change over four years (2019 to 2022) using 
a pre- and post-test quiz. Knowledge assessment results showed (a) 
respondents gained knowledge on each of the seven modules presented 
in the curriculum; (b) the curriculum content had a large effect on 
knowledge gain; (c) knowledge gain differed significantly between years, 
but the differences were not of practical importance; (d) remote 
participants learned significantly more than in-person participants, but 
differences were not of practical importance; and (e) the quiz consisted 
of low and moderate difficulty questions (no questions were high 
difficulty) and had generally acceptable discriminant properties. 
Implementation of the FSMA PSR has progressed from an educational to 
a regulatory phase, and the authors recommend replacing the knowledge 
assessment with a tool that measures how the PSA course prepares 
growers for compliance. 
 

mailto:lendel.narine@usu.edu
https://orcid.org/0000--0000-0000-0000
mailto:arlene@iastate.edu
https://orcid.org/0000--0000-0000-0000
mailto:mattbenge@ufl.edu
https://orcid.org/0000--0000-0000-0000
mailto:eab38@cornell.edu
https://orcid.org/0000--0000-0000-0000
mailto:stephanie.brown@oregonstate.edu
https://orcid.org/0000-0001-8682-9984
mailto:jovana.kovacevic@oregonstate.edu
https://orcid.org/0000-0003-1254-4294
mailto:Elizabeth.Newbold@uvm.edu
https://orcid.org/0000-0002-0862-4593
mailto:keiths29@ufl.edu
https://orcid.org/0000-0003-0145-3418
mailto:Angela.Shaw@ttu.edu
https://orcid.org/0000-0001-5908-0429


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   14 
 

Introduction and Problem Statement 
 
The Produce Safety Alliance (PSA) Grower Training (GT) curriculum was developed to provide 
produce growers with training that meets requirements in §112.22(c) of the Food Safety 
Modernization Act (FSMA) Produce Safety Rule (PSR). Concurrently, four regional centers, 
funded by United States Department of Agriculture (USDA) and United States Food and Drug 
Administration (FDA), were established to help develop networks of food safety professionals 
who could deliver the PSA GT courses to ensure training was widely available to produce 
growers, regulatory personnel, educators, and others. As of December 31, 2023, PSA Trainers 
have delivered 2,859 PSA GTs domestically since 2016, with 53,992 participants. Prior to the 
COVID-19 pandemic, PSA GTs were exclusively delivered in person. During the pandemic, real-
time remote courses delivered by PSA Trainers using remote conference technologies was 
allowed. This remote delivery policy was temporary, but due to the popularity of the option, an 
assessment of training evaluations was conducted to ensure it was as effective a modality as in-
person courses (Bugingo et al., 2023). In 2023, the remote course delivery policy became 
permanent, allowing PSA Trainers to teach PSA GTs in either course format. 
 
To help evaluate the effectiveness of PSA GTs, pre-training (pre-test) and post-training (post-
test) knowledge assessment tests were developed by the Southern Center (SC) for Food Safety 
Training, Outreach, and Technical Assistance to measure the immediate knowledge change of 
participants. The four regional centers have been collating these data from within their regions 
to gauge effectiveness, impact, and remaining training needs.  
 
Voluntarily reported knowledge assessment data collected by the four regional centers from 
January 2019 to June 2022 are described herein. Data from in-person and remote delivery 
courses were analyzed to assess whether the PSA GTs resulted in short-term knowledge gain. 
 

Theoretical and Conceptual Framework 
 
Since the regional centers were launched, the Targeting Outcomes of Programs (TOP) model 
has been used to assess program performance. The TOP model, an expansion of Bennett’s 
hierarchy (Bennett 1975; Bennett 1976), was first developed in 1994 to evaluate program 
outcomes in planning, implementation, and evaluation (Harder, 2009; Rockwell & Bennett, 
2004). This model includes a two-sided hierarchy (program development and program 
performance), with seven levels shared between the two sides. These include (a) resources; (b) 
activities; (c) participation; (d) reactions; (e) the knowledge, attitudes, skills, and aspirations of 
participants (KASA); (f) practices; and (g) social, economic, and environmental conditions (SEE; 
Harder, 2009; Rockwell & Bennett, 2004). Notably, the model allows for multiple evaluation 
strategies (process and outcomes evaluation) to measure programmatic performance (one side 
of the hierarchy) (Harder, 2009; Rockwell & Bennett, 2004).  
 
In an initial effort to evaluate knowledge gain from programs delivering standardized FSMA 
trainings (e.g., PSA GTs), a plan for sharing basic training information (e.g., average knowledge 

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   15 
 

assessment scores) at a national level was developed in collaboration with the Lead Regional 
Coordination Center and other regional centers. A common set of quantitative indicators (i.e., 
pre- and post-tests) were also created and used across trainings delivered by regional center 
partners. Although the same set of indicators were used, each center approached knowledge 
assessment and additional training-related data collection (for the KASA-related portion of the 
TOP model) differently. To start understanding the short-term training impacts of produce 
safety trainings on a national scale, this manuscript focuses on the evaluation of knowledge 
assessment data collected during trainings delivered from January 2019 to June 2022. Moving 
forward, these data will serve as guiding points to further the standardization of national 
evaluation efforts, including the assessment of medium- and long-term impacts, and re-
evaluation of current quantitative indicators for PSA GTs. This framework will also serve as a 
guide for evaluating other standardized FSMA trainings provided by food safety professionals 
within the regional center networks (e.g., the Food Safety Preventive Controls Alliance 
Preventive Controls for Human Food participant course). 
 

Purpose 
 
The purpose of this study was to assess the short-term knowledge outcomes of the PSA GT 
course over a four-year period. The objectives were to (a) assess knowledge gain for each 
module of the PSA GT course, (b) assess overall knowledge gain by participants, (c) compare net 
knowledge gain across each year of the course, (d) examine differences in knowledge gain by 
delivery modality, and (e) assess the quality of each item in the knowledge assessment via the 
difficulty index and discrimination index. The study’s results will be used to inform strategies for 
improving program implementation and optimizing program evaluation instruments. 
 

Methods 
 
This study examined primary data through quantitative methodology. Quantitative research is 
used to test theory through numerical evaluation by observing the relationships among 
variables (Ary et al., 2006; Creswell, 2014) and generates knowledge by examining phenomena 
affecting individuals (Allen, 2017). Secondary data is information that existed prior to a study, 
and it was not collected by the researcher solely for the purpose of their study (Stewart & 
Kamis, 1992). Zimmerman and Kahl (2018) explained that the collection of preexisting data can 
inform Extension programs and provide an increased understanding of the community by 
“putting individual Extension program impacts into a broader perspective.”  Preexisting PSA 
data were collected from the four regional centers for this study. 
 
The PSA GT knowledge test contained 25 questions within seven curricular modules. Each test 
question consisted of four multiple-choice options, and PSA GT participants took the test before 
and after the training. Each PSA trainer either scored the tests and shared the pre- and post-
knowledge test scores with their specific regional evaluation liaison or sent quizzes 
(electronically or via paper mail) to their regional evaluation liaison for data entry. Each regional 

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   16 
 

evaluation team holds its own Institutional Review Board approval, and demographic data were 
not collected.  
 
Each regional evaluation team gathered their regional PSA knowledge assessment data 
collected from January 2019 through June 2022 for this study. The total number of cases 
collected across all four regions was 7,185. The aggregated regional data were reviewed, and 
cases with incomplete pre- or post-test scores were removed from the dataset, which yielded a 
final data set of 6,583 cases. For this study, the data were additionally coded for three 
independent variables: (a) year, (b) delivery modality, and (c) U.S. region. A paired samples t-
test was used to address objectives (a) and (b), a one-way ANCOVA was used to answer 
objectives (c) and (d), and a difficulty index based on the proportion of correct responses and 
discrimination index via a point-biserial correlation (Millman & Green, 1989) were calculated 
for objective (e). 
 

Findings 
 
Objective (a): Module-Level Knowledge Gain 
There are seven modules in the PSA GT course. Scores were standardized to the percent of 
correct responses in each module. Table 1 shows the average pre- and post-test scores across 
each module. There were statistically significant increases (p < 0.001) in participant scores 
between the pre- and post-test scores across all seven modules. Based on Cohen’s d, the effect 
size between pre-and-post-test scores was small for module 2, moderate for modules 1 and 3, 
and large for 4, 5,  6, and 7. 
 
Table 1  

Module-Level Knowledge Scores 

Module n 
% 

t (One-tailed) 
P Cohen’s d Pre-test Post-test 

1 5,147 72 82 34.55 <0.001 0.48 
2 5,147 94 97 10.20 <0.001 0.14 
3 5,150 74 86 29.00 <0.001 0.40 
4 5,161 54 77 48.66 <0.001 0.68 
5 5,165 58 83 57.13 <0.001 0.80 
6 5,138 52 77 53.85 <0.001 0.75 
7 5,153 44 69 59.38 <0.001 0.83 

Note. The Southern Region did not gather module-level data. Table 1 excludes data from that 
region. 

  

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   17 
 

Objective (b): Overall Knowledge Gain  
Participants demonstrated an increase in their knowledge of the PSR after completing the 
course. Based on a paired t-test, there was a statistically significant increase in participants’ (n = 
6,583*) pre-test and post-test scores (t = 108.39, p < 0.001). On average, participants scored 
15.94 (SD = 3.62) or 64% on the pre-test and 20.38 (SD = 3.93) or 82% on the post-test. Based 
on Cohen’s d, the PSA course had a large effect on participants’ knowledge of the PSR (d = 
1.34). 
 
Figure 1 

Pre- and post-test knowledge from the PSA Course. 

 
* The error bar represents the mean difference between groups. 

Objective (c): Net Knowledge Gain by Year 
A one-way ANCOVA was used to estimate the net change in knowledge by program year while 
controlling for the effects of pre-test scores. The pre-test was statistically correlated with the 
post-test (r = 0.617), but the correlation was below the recommended threshold for its effect 
on the dependent variable (r < 0.80) and, therefore, served as a valid covariate in the model. 
Results indicated a statistically significant difference in mean knowledge change by year (t = 
10.34, p < 0.001) when controlling for pre-test scores. However, the partial eta squared was 
very small (ηp

2 = 0.005), indicating that the differences in mean knowledge change across each 
program year was small and likely significant due to the large sample size. Therefore, while 
statistically significant, on a practical level, participants had similar knowledge gain between 
2019 to 2022 when controlling for pre-test scores.  
 
Mean post-test scores for each year were as follows; 2019 = 20.12 (SD = 3.53, n = 2,635); 2020 = 
20.54 (SD = 3.60, n = 1,720); 2021 = 20.64 (SD = 3.71, n = 1,263); 2022 = 20.45 (SD = 3.73, n = 
695). Figure 2 provides a comparison of pre-and-post-test scores between 2019 to 2022. 
Corroborating results of the ANCOVA model, Figure 2 shows that participants across each 
program year had similar steady levels of knowledge gain. 
  

15.94

4.43* 20.38

0

5

10

15

20

25

Pre-test                              Post-test

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   18 
 

Figure 2 

Change in knowledge scores between 2019 to 2022. 

 
*The error bars represent the mean difference between groups. 
 
Objective (d): Knowledge Gain by Delivery Modality 
Most participants attended the PSA GT course remotely in 2020 and 2021. A one-way ANCOVA 
was used to examine the differences in net knowledge gain by delivery modality while 
controlling for the effect of pre-test scores. There was a statistically significant difference in 
knowledge gain between remote and in-person delivery (t = 51.65, p < 0.001). Yet, the partial 
eta squared was small (ηp

2 = 0.008), indicating the difference in mean knowledge gain based on 
delivery modality was not practically significant and likely due to the large sample size. 
Therefore, while statistically significant, participants in remote and in-person classes had similar 
levels of knowledge gain.  
 
The mean post-test score was 20.09 (SD = 3.62) for in-person participants (n = 4,228) and 20.88 
for remote participants (n = 2,305) across all years. Figure 3 shows the pre- and post-test scores 
between in-person and remote participants. Consistent with findings from the ANCOVA model, 
remote and in-person participants experienced a similar level of knowledge gain from the PSA 
course as shown in Figure 3. While pre-test scores were controlled in the ANCOVA model, the 
figure shows remote participants entered the course with more content knowledge compared 
to in-person participants. 
 
  

15.87 15.93 15.97 16.13

4.25* 4.61 4.67 4.3220.12 20.54 20.64 20.45

0

5

10

15

20

25

2019 2020 2021 2022

M
ea

n 
te

st
 sc

or
e

Pre-test Post-test

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   19 
 

Figure 3 
 
Pre- and post-test differences between delivery modes. 

 
*The error bars represent the mean difference between groups. 
 
Prior to March 2020, the PSA GT course was only delivered in person. In 2020, 67% of 
participants (n = 1,302) completed the PSA course in-person, while 33% completed it remotely 
(n = 629). In 2021, 22% of participants (n = 294) completed the course in-person and 78% 
completed it remotely (n = 1,064). In 2022, 18% (n = 201) completed the course in-person and 
81% completed the course remotely (n = 876).  
 
A two-way ANCOVA was conducted to assess the difference in net knowledge gain by delivery 
year (2020 to 2022) and delivery modality (in-person vs. remote) while accounting for the 
effects of pre-test scores. Results showed a weak but statistically significant difference in 
knowledge based on the interaction between delivery modality and program year (t = 3.10, p < 
0.01, ηp

2 = 0.001). This suggests there were minor changes in knowledge gain for remote and in-
person participants each year during the pandemic. Given a minor interaction effect, a 
Bonferroni post hoc test was used to identify statistically significant differences in knowledge 
based on the interaction between year and delivery modality. Using adjusted family-wise p-
values, results showed remote participants in 2020 and 2021 scored significantly higher on the 
post-test (2020: M = 21.18, SD = 3.27; 2021: M = 21.02, SD = 3.57) compared to in-person 
participants (2020: M = 20.23, SD = 3.72; 2021: M = 19.04, SD = 3.92). However, remote and in-
person participants experienced similar levels of knowledge gain in 2022 when controlling for 
the effects of pre-test scores. Figure 4 provides a descriptive summary of net knowledge gain by 
delivery modality and program years. Scoring a net two-point difference, remote participants 
performed noticeably better than in-person participants in 2021. 
 
  

16.22 15.76

3.86* 5.1220.09 20.88

0

5

10

15

20

25

Remote In-person

M
ea

n 
te

st
 sc

or
e

Pre-test Post-test

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   20 
 

Figure 4 
 
Net knowledge gain between remote and in-person participants during the first years of the 
pandemic. 

 

*The error bars represent the mean difference between groups. 
 
Objective (e): Item Difficulty and Discrimination Index 
The PSA GT knowledge assessment contains 25 multiple-choice questions and was developed 
by a team of food safety specialists and Extension educators led by Catherine Shoulders of the 
University of Arkansas. The difficulty index and discrimination index were calculated using item-
level data from participants who completed the PSA training between 2019 to 2022 in three out 
of the four regional centers (n = 5,195, excluding the Southern region). Table 2 shows the 
corresponding indices for all items in the assessment.  
 
The Difficulty index represents the proportion of people who answered the item correctly. 
Ranging from 0 to 1, items with higher scores are easier, while lower scores are more difficult. 
From Table 2, 14 out of the 25 items (or 56% of items) were categorized as easy (> 0.80), while 
11 were moderately difficult (0.30 to 0.80). No item had a high level of difficulty. The 
discrimination index measured the effectiveness of an item to distinguish between high and low 
performers on a scale of -1 to 1. An effective test item should provide sufficient evidence to 
differentiate between students who attained topic mastery and those who did not. Most items 
had a good discrimination index (> 0.30), while three had relatively low discrimination power 
(Q5, Q6, and Q16). Overall, while the items had generally acceptable discriminant properties, 
the test was easy for respondents. 
 
  

20.23 19.04 20.270.95* 1.98 0.2221.18 21.02 20.49

0

5

10

15

20

25

2020 2021 2022

M
ea

n 
te

st
 sc

or
e

Post-test: In-person Post-test: Remote

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   21 
 

Table 2 
 
Item Analysis of the PSA GT Knowledge Assessment 

Item Difficulty index 
Discrimination index: Point-

biserial correlation 
Q1 .95 .35 
Q2 .81 .39 
Q3 .75 .45 
Q4 .94 .36 
Q5 .98 .19 
Q6 .96 .29 
Q7 .86 .43 
Q8 .86 .47 
Q9 .89 .41 

Q10 .83 .37 
Q11 .74 .51 
Q12 .80 .44 
Q13 .61 .37 
Q14 .95 .31 
Q15 .78 .43 
Q16 .95 .28 
Q17 .76 .43 
Q18 .69 .39 
Q19 .90 .39 
Q20 .86 .46 
Q21 .55 .44 
Q22 .59 .46 
Q23 .91 .32 
Q24 .58 .34 
Q25 .70 .51 

 
Conclusions, Discussion, and Recommendations 

 
The hierarchy that serves as the basis of the TOP model integrates evaluation into the overall 
program development process. The pre- and post-test tool was developed, in part, to assess the 
immediate knowledge gain of the PSA GT attendees. Looking at the results from pre- and post-
tests nationally from 2019 – 2022, participants experienced an increase in their knowledge of 
the FSMA PSR after completing the course.  
 
The key takeaway from these results is consistency in knowledge gain over the four-year 
period, regardless of delivery type or year of implementation. The differences in knowledge 
gain by implementation and year were statistically significant mainly because of the large 
sample size. While statistically significant, some differences (e.g., a 1.25-point difference in 

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   22 
 

knowledge change scores between remote and in-person participants) are not practically 
significant. For example, a 1- or 2-point difference suggests very little practical knowledge 
change, even though it is statistically different due to sample size and power, it is not practically 
different. Another important takeaway is the internal consistency within the dataset because 
the test was reliable in most cases, and there was an improvement in knowledge. Furthermore, 
the questions had sufficient discrimination power to differentiate between the low and top 
performers. Another key takeaway is that the questions were designed, in part, to measure 
immediate knowledge gain of produce safety concepts, and the practical significance of the 
data demonstrates that the objective has been met. Per the TOP model, there is a need to 
further the national evaluation model to assess medium- and long-term impacts of the PSA GT 
beyond knowledge gain and to measure synthesis and application of content. It is important to 
note the pre- and post-tests still have value in demonstrating program effectiveness. Trainers 
may continue to use the tool for other reporting purposes; regardless, the need for national 
aggregation of the data has been met. 
 
The FSMA PSR has evolved since its inception in 2015. Some portions of the rule have been 
finalized, while requirements for pre-harvest agricultural water and application intervals of 
untreated biological soil amendments of animal origin (BSAAO) have not. FSMA PSR inspections 
began in 2018, with growers moving beyond the need for produce safety knowledge to a need 
to implement that knowledge on their farms. Seven years later (2023), the focus has shifted to 
ensuring growers know how to implement the PSR on their farms.  
 
In conclusion, this research has shown that the initial objective of the assessment tool was met, 
and the PSA GT curriculum resulted in a significant knowledge gain in each module of the 
curriculum. With evolving educational needs based on PSR inspection findings and revised 
regulatory requirements, there is a need for a new assessment tool to measure knowledge 
change as well as growers’ capacity to incorporate food safety behaviors and processes. 
Specifically, the research team suggests replacing this assessment tool and developing a new 
tool capable of answering the following questions: 1) are growers learning what they need to 
know about the rule? 2) are growers learning how to produce safe food? and 3) is the 
assessment tool meaningful for regulatory application and education effectiveness measures?    

 
Acknowledgments 

 
Thank you to Dr. Amy Harder, former evaluator for the Southern Center for Food Safety 
Training, Outreach, and Technical Assistance, for conceptualizing and leading efforts to develop 
the PSA GT knowledge assessment tool. Thanks to Donna Pahl Clements and Ellen Johnsen for 
reviewing the manuscript and suggesting edits. This work was funded by the Food Safety 
Outreach Program from the USDA National Institute of Food and Agriculture, agreement 
numbers 2021-70020-35740, 2021-70020-35732, 2021-70020-35753, and 2021-70020-35497. 
 
L. Narine - data curation, formal analysis, funding acquisition, methodology, project 
administration, resources, visualization, writing-original draft, writing-review and editing; A. 

https://doi.org/10.37433/aad.v5i3.473


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   23 
 

Enderton - conceptualization, data curation, project administration, visualization, writing-
original draft, writing-review and editing; M. Benge – Data curation, writing-original draft; E. 
Bihn - writing-original draft, writing-review and editing; S. Brown - data curation, funding 
acquisition, methodology, project administration, resources, writing-original draft, writing-
review and editing; J. Kovacevic - conceptualization, data curation, funding acquisition, 
methodology, project administration, resources, writing-original draft, writing-review and 
editing; E. Newbold - conceptualization, data curation, funding acquisition, investigation, 
project administration, resources, writing-original draft, writing-review and editing; K. 
Schneider - funding acquisition, project administration, writing-review and editing; A. Shaw - 
data curation, formal analysis, funding acquisition, investigation, methodology, resources, 
validation, visualization, writing-review and editing. 

 
References 

 
Allen, M. (Ed.). (2017). The SAGE encyclopedia of communication research methods. Sage 

Publications, Inc. https://doi.org/10.4135/9781483381411.n293 
 
Ary, D., Jacobs, L. C., Razavieh, A., & Sorensen, C. (2006). Introduction to research in education 

(7th ed.). Thompson-Wadsworth. 
 
Bennett, C. (1975). Up the hierarchy. Journal of Extension, 13(2), 7-12. 

https://archives.joe.org/joe/1975march/1975-2-a1.pdf  
 
Bennett, C. (1976). Analyzing impacts of extension programs (ESC-575). United States 

Department of Agriculture, Science, and Education Administration. 
https://ia600708.us.archive.org/12/items/analyzingimpacts57benn/analyzingimpacts57
benn.pdf   

 
Bugingo, C., Stoeckel, D., Clements, D., George, L., Saunders, T., Blasini, D., Humiston, M., 

Melville, T., Way, R., Acuna-Maldonado, L., & Bihn, E. A. (2023). Remote PSA grower 
training delivery policy: Lessons learned and recommendations for future courses. 
Produce Safety Alliance. 
https://resources.producesafetyalliance.cornell.edu/documents/Remote-PSA-Grower-
Training-Delivery-Policy-Lessons-Learned-and-Recommendations-for-Future-
Courses.pdf  

 
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods 

approaches (4th ed.). Sage Publications, Inc.  
 
Harder, A. (2019). Using the TOP model to measure program performance: A pocket reference 

(WC092). University of Florida Department of Agricultural Education and 
Communication UF/IFAS Extension. https://edis.ifas.ufl.edu/publication/WC092  

  

https://doi.org/10.37433/aad.v5i3.473
https://doi.org/10.4135/9781483381411.n293
https://archives.joe.org/joe/1975march/1975-2-a1.pdf
https://ia600708.us.archive.org/12/items/analyzingimpacts57benn/analyzingimpacts57benn.pdf
https://ia600708.us.archive.org/12/items/analyzingimpacts57benn/analyzingimpacts57benn.pdf
https://resources.producesafetyalliance.cornell.edu/documents/Remote-PSA-Grower-Training-Delivery-Policy-Lessons-Learned-and-Recommendations-for-Future-Courses.pdf
https://resources.producesafetyalliance.cornell.edu/documents/Remote-PSA-Grower-Training-Delivery-Policy-Lessons-Learned-and-Recommendations-for-Future-Courses.pdf
https://resources.producesafetyalliance.cornell.edu/documents/Remote-PSA-Grower-Training-Delivery-Policy-Lessons-Learned-and-Recommendations-for-Future-Courses.pdf
https://edis.ifas.ufl.edu/publication/WC092


Narine et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i3.473   24 
 

Millman, J., & Green, J. (1989). The specification and development of tests of achievement and 
ability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 335-366). Macmillan 
Publishing Co, Inc; American Council on Education.  

 
Rockwell, K., & Bennett, C. (2004). Targeting outcomes of programs: A hierarchy for targeting 

outcomes and evaluating their achievement. Agricultural Leadership, Education, and 
Communication Department: Faculty Publications, 48. 
https://digitalcommons.unl.edu/aglecfacpub/48/?utm_source=digitalcommons.unl.edu
%2Faglecfacpub%2F48&utm_medium=PDF&utm_campaign=PDFCoverPages  

 
Stewart, D. W., & Kamins, M. A. (1993). Secondary research: Information sources and methods 

(2nd ed., Vol. 4). Sage Publications, Inc. 
 
Zimmerman, J. N., & Kahl, D. (2018). Finding publicly available data for extension planning and 

programming: Developing community portraits. The Journal of Extension, 56(3), Article 
1. https://doi.org/10.34068/joe.56.03.01 

 
 
© 2024 by authors. This article is an open access article distributed under the terms and conditions of 
the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/). 
 

https://doi.org/10.37433/aad.v5i3.473
https://digitalcommons.unl.edu/aglecfacpub/48/?utm_source=digitalcommons.unl.edu%2Faglecfacpub%2F48&utm_medium=PDF&utm_campaign=PDFCoverPages
https://digitalcommons.unl.edu/aglecfacpub/48/?utm_source=digitalcommons.unl.edu%2Faglecfacpub%2F48&utm_medium=PDF&utm_campaign=PDFCoverPages
https://doi.org/10.34068/joe.56.03.01

