Advancements in Agricultural Development Volume 5, Issue 4, 2024 agdevresearch.org 1. Amy Harder, Associate Dean for Extension, University of Connecticut, 1376 Storrs Rd., Unit 4134, W.B. Young Building, Room 233, Storrs, CT 06269, amy.harder@uconn.edu, https://orcid.org/0000-0002-7042-20283 2. Lendel Kade Narine, Extension Associate Professor, Utah State University, Old Main Hill Logan, Utah 84322, lendel.narine@usu.edu, https://orcid.org/0000-0001-6962-2770 53 Best Practices in the Application of the Ranked Discrepancy Model A. Harder1, L. K. Narine2 Article History Received: October 18, 2024 Accepted: November 4, 2024 Published: November 6, 2024 Keywords needs assessment; Borich model; ranked discrepancy scores; SDG 4: Quality Education Abstract In this brief article, we discuss the rationale for the Ranked Discrepancy Model (RDM) and best practices. An overview of the history of the RDM and its appropriate usages is offered to create clarity for researchers. We then provide an explanation of the role of tied ranks in determining ranked discrepancy scores (RDS), guidance on the interpretation of RDS for planning, and recommendations for comparing the RDM with the Borich model. A summary of prior research comparing the RDM to the Borich model is included. We conclude by encouraging researchers to use needs assessment models appropriate for the problem, population, and context. mailto:amy.harder@uconn.edu https://orcid.org/0000-0002-7042-20283 mailto:lendel.narine@usu.edu https://orcid.org/0000-0001-6962-2770 Harder and Narine Advancements in Agricultural Development https://doi.org/10.37433/aad.v5i4.559 54 Introduction In 2021, we introduced the Ranked Discrepancy Model (RDM) as an alternative to Borich’s (1980) model of needs assessment. We have been encouraged by the interest in the RDM and have welcomed feedback from researchers and practitioners around the world which has allowed us to revisit and strengthen the original concept to make it a more effective and efficient approach (Narine & Harder, 2024). A recent article from Johnson et al. (2024) provided another chance to review our work and assess a critique of it. We appreciate the opportunity provided by Advancements in Agricultural Development to respond and the willingness of Johnson et al. to allow such a response. We offer the following discussion on the rationale for the RDM and best practices in its usage in the spirit of collegiality and a desire to promote the advancement of agriculture and extension education research methods. Rationale for RDM Development and Appropriate Usages The impetus for developing the RDM arose from the longstanding debate regarding the use of means for ordinally scaled items, a central component in the use of the Borich model. Rather than enter the fray, we advocated for the pursuit of a solution which could avoid the argument entirely “while preserving the underlying rationale of the Borich model” (Narine & Harder, 2021, p. 97). The debate over using means for ordinal and interval variables will continue, so the RDM was presented as an alternative that moves away from the controversy. We initially presented the RDM as a novel approach to analyze professional development needs. Like the Borich model, the emphasis was on identifying which needs were most urgent by examining gaps between respondents’ perceptions of a competency’s importance and perceptions of their knowledge or ability. Unlike the Borich model, the RDM allows researchers to make these determinations by comparing positive ranks, tied ranks, and negative ranks within pairs of observations, a nonparametric approach to needs identification (for more details, see Narine & Harder, 2021). Later, we provided detailed steps as to how to use the same underlying logic of the RDM to analyze needs assessments with repeated measures data (Narine & Harder, 2024). The RDM has been applied, peer reviewed, and published in the professional development literature several times (e.g., Choi & Park, 2022; Flanagan et al., 2023; Seitz et al., 2022; Zickafoose et al., 2023). By 2024, we recognized and were forthcoming about a limitation of the RDM. The ranked discrepancy scores (RDS) produced by the RDM are sensitive to the size of the sample, as we illustrated in Narine and Harder (2024) using three sample sizes of 1,000, 500, and 100 across two populations. As a result, we recommended and continue to recommend that “researchers should be aware of this consideration if they intend to use the RDM for analysis” (Narine & Harder, 2024, p. 115). We note increasing the sample size is always going to affect data distribution and subsequent results from that data regardless of the parametric or nonparametric procedures adopted in the study (Serdar et al., 2021). To our knowledge, no research exists which has examined the validity or reliability of Mean Weighted Discrepancy https://doi.org/10.37433/aad.v5i4.559 Harder and Narine Advancements in Agricultural Development https://doi.org/10.37433/aad.v5i4.559 55 Score (MWDS) from the Borich model across sample sizes, so caution should also be applied before assuming the MWDS of a small sample are stable and representative of the target population. Best Practices in RDM Application Tied Ranks As mentioned, the RDM relies upon the frequency with which positive ranks, tied ranks, and negative ranks occur within paired data for a single item, such as a competency. The use of ranks is common in nonparametric procedures such Wilcoxon rank-sum and Mann-Whitney U (Krzywinski & Altman, 2014; Nahm, 2016; Schober & Vetter, 2020). Tied ranks are particularly important in the RDM because they represent equilibrium between two conditions, which is theoretically consistent with Lewin (1939). In the RDM, tied ranks substantially influence the RDS. When fewer negative or positive ranks are observed within pairs, and more tied ranks exist, the RDS moves closer to zero (or equilibrium) because of tied ranks. Note that assigning a score of zero does not cause tied ranks to be ignored. Their existence decreases the available percentages which can be assigned to positive or negative ranks when the rank counts are converted for the calculation of the RDS (Narine & Harder, 2021). Interpreting the RDS for Planning Johnson et al. (2024) rightly expressed concerns about investing organizational resources wisely and the need for clear data to drive those decisions. We agree. We offer these recommendations for interpreting RDS: 1. Any negative RDS indicates there is a negative discrepancy between the two variables (or conditions) being measured, meaning the situation is below a desired state of equilibrium. 2. RDS represent higher priorities for intervention as they trend towards -100. A negative score represents a deficit in an ideal condition. 3. The RDS should be used to determine what (if any) resources should be applied to addressing a gap. 4. Relative rankings (e.g., 1st, tied for 3rd, 16th) should not be exclusively relied upon when interpreting RDS. The use of relative rankings neglects the actual magnitude of the need based on contextual factors and only provides an ordinal interpretation as one source of evidence. Two items with remarkably similar scores may be many rankings apart when there is limited variation in the results across the group, which could lead to errors in interpretation. 5. Examining the distribution of positive ranks, tied ranks, and negative ranks can provide additional insight for decision-making when items share the same RDS. While this feature is not built into the formulas provided in the spreadsheet linked to in the appendix in Narine and Harder (2024), using the COUNTIF option in Excel allows such exploration. 6. Use good judgment. This is a practical solution which requires the individuals conducting needs assessments to themselves have a sufficient level of professional competency to interpret the results. In the absence of that, an alternative could be to triangulate the results with qualitative methods. https://doi.org/10.37433/aad.v5i4.559 Harder and Narine Advancements in Agricultural Development https://doi.org/10.37433/aad.v5i4.559 56 Comparing RDS to MWDS Researchers may wish to continue their exploration of RDM and how it performs when compared to the Borich model. We recommend a standard procedure for doing so to ensure transparency in such analyses. First, researchers should include frequency tables in their findings as a precursor to calculating MWDS or RDS, so that readers can better understand the data distribution and how that led to the reported MWDS or RDS. Second, we encourage researchers to use z-scores when examining relationships between MWDS and RDS as they do not share the same properties. Standardization is recommended to reduce error when comparing raw scores with different ranges (Hopkins & Rowlands, 2024). What Others Are Saying To our knowledge, the RDM has been examined twice in comparison to the Borich model in addition to Johnson et al. (2024). The first and most comprehensive examination was published in 2023 by Choi and Park. They assessed the needs of 75 school guidance teachers in South Korea using four analysis methods: (a) descriptive statistics, (b) the RDM, (c) the weighted total index (WTI), and (d) the Borich model. Following analysis, Choi and Park (2023) noted the priorities identified through the RDM and WTI approaches had few differences and concluded: “This indicates that the RDM has achieved concurrent validity, as the RDM results were almost the same as those from the WTI, which has already been tested and validated” (p. 11). Further, they found similarities between the RDM and Borich results, recommending: “it might be more appropriate to adopt the RDM as an alternative quantitative method for needs assessment relative to the Borich model in cases involving ordinal items, cross-sectional data, or non- normally distributed data” (Choi & Park, 2023, p. 12). Another comparison of RDM and the Borich model was presented by Eze and Siegmund (2024) in their study of the competency gaps of 141 UNESCO site actors. They did not explicitly seek to test the relationships between RDS and MWDS but presented both in their findings and noted the tendency for the two scores to provide similar results for the highest priority needs. Some variation existed when comparing relative rankings for the less pressing needs. Eze and Siegmund (2024) noted “expanding the sample sizes in future research facilitates more rigorous and robust statistical analyses, increased reliability of outcomes, higher accuracy in findings and generalizability” (p. 7). Conclusions Regardless of the model selected to analyze needs, researchers must be cognizant of its limitations and do their best to mitigate those limitations through informed methodological decisions. We encourage researchers to use needs assessment models appropriate for the problem, population, and context. We appreciate the contributions of Johnson et al. (2024) in advancing the discussion about how best to identify professional development needs given the importance of this task to advancing agricultural development and look forward to continuing to refine the RDM as an option to serve the needs of our discipline (and beyond). https://doi.org/10.37433/aad.v5i4.559 Harder and Narine Advancements in Agricultural Development https://doi.org/10.37433/aad.v5i4.559 57 Acknowledgments Author Contributions: A. Harder – conceptualization, writing – original draft, writing – review & editing; L. Narine – conceptualization, writing – review & editing References Choi, H., & Park, J. (2022). Exploring deficiencies in the professional capabilities of novice practitioners to reshape the undergraduate human resource curriculum in South Korea. Sustainability, 14(19), 12121. https://doi.org/10.3390/su141912121 Choi, H., & Park, J., (2023). Exploring the viability of the Ranked Discrepancy Model by comparing the Weighted Total Index approach and the Borich Model: A case of learning needs assessment of career guidance teachers. Education Sciences, 13(4), 414. https://doi.org/10.3390/edusci13040414 Eze, E., & Siegmund, A. (2024). Appraising competency gaps among UNESCO-designated heritage site actors in disaster risk reduction innovations. Progress in Disaster Science, 22, 100321. https://doi.org/10.1016/j.pdisas.2024.100321 Flanagan, B., Boren-Alpizar, A., Wingenbach, G., Lawver, D., & Strong, R. (2023). Evaluating agroforestry extension workers’ technical and human relation competencies: A Ranked Discrepancy Model needs assessment. Sustainability, 15(19), 14100. https://doi.org/10.3390/su151914100 Hopkins, W. G., & Rowlands, D. S. (2024). Standardization and other approaches to meta- analyze differences in means. Statistics in Medicine, 43(16), 3092-3108. https://doi.org/10.1002/sim.10114 Johnson, D. M., Doss, W., Estepp, C. M., Akwah, H. O., & Wardlow, G. W. (2024). A discrepancy in discrepancy scores? Comparing the Borich and Ranked Discrepancy Models for determining professional development needs. Advancements in Agricultural Development, 5(4), 42–52. https://doi.org/10.37433/aad.v5i4.515 Krzywinski, M., & Altman, N. (2014). Nonparametric tests. Nature Methods, 11(2014), 467–468. https://doi.org/10.1038/nmeth.2937 Lewin, K. (1939). Field theory and experiment in social psychology. American Journal of Sociology, 44(6), 868–896. https://doi.org/10.1086/218177 Nahm, F. S. (2016). Nonparametric statistical tests for the continuous data: The basic concept and the practical use. Korean Journal of Anesthesiology, 69(1), 8–14. https://doi.org/10.4097/kjae.2016.69.1.8 https://doi.org/10.37433/aad.v5i4.559 https://doi.org/10.3390/su141912121 https://doi.org/10.3390/edusci13040414 https://doi.org/10.1016/j.pdisas.2024.100321 https://doi.org/10.3390/su151914100 https://doi.org/10.1002/sim.10114 https://doi.org/10.37433/aad.v5i4.515 https://doi.org/10.1038/nmeth.2937 https://doi.org/10.1086/218177 https://doi.org/10.4097/kjae.2016.69.1.8 Harder and Narine Advancements in Agricultural Development https://doi.org/10.37433/aad.v5i4.559 58 Narine, L., & Harder, A. (2021). Comparing the Borich model with the Ranked Discrepancy Model for competency assessment: A novel approach. Advancements in Agricultural Development, 2(3), 96-111. https://doi.org/10.37433/aad.v2i3.169 Narine, L., & Harder, A. (2024). Analyzing and visualizing repeated-measures data using the Ranked Discrepancy Model. Advancements in Agricultural Development, 5(2), 105-118. https://doi.org/10.37433/aad.v5i2.321 Schober, P., & Vetter, T. R. (2020). Nonparametric statistical methods in medical research. Anesthesia and Analgesia, 131(6), 1862–1863. https://doi.org/10.1213/ANE.0000000000005101 Seitz, P., Strong, R., Hague, S., & Murphrey, T. P. (2022). Evaluating agricultural extension agent’s sustainable cotton land production competencies: Subject matter discrepancies restricting farmers’ information adoption. Land, 11(11), 2075. https://doi.org/10.3390/land11112075 Serdar, C. C., Cihan, M., Yücel, D., & Serdar, M. A. (2021). Sample size, power and effect size revisited: Simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochemia Medica, 31(1), 010502. https://doi.org/10.11613/BM.2021.010502 Zickafoose, A., Wingenbach, G., Baker, M., & Price, E., (2023). Developing youth agricultural entrepreneurship in Latin America and the Caribbean. Sustainability, 15(22), 15983. https://doi.org/10.3390/su1522115983 © 2024 by authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/). https://doi.org/10.37433/aad.v5i4.559 https://doi.org/10.37433/aad.v2i3.169 https://doi.org/10.37433/aad.v5i2.321 https://doi.org/10.1213/ANE.0000000000005101 https://doi.org/10.3390/land11112075 https://doi.org/10.11613/BM.2021.010502 https://doi.org/10.3390/su1522115983