J Forensic Sci Educ 2023, (5)1 2023 Journal Forensic Science Education Conte Forensic method validation: piloting a new course to increase student skills for the forensic science workforce Jillian Conte, Ph.D. 1,2 * 1 previously of Keystone College, Department of Biological and Physical Sciences, La Plume, PA 2 Conte Forensic Consulting, LLC, Peckville, PA *Corresponding author: jillianconte517@gmail.com Abstract: The process of method validation is critical prior to implementation of a new technology, product, or method into forensic casework. Despite its importance, gaps exist in educating forensic scientists and students on method validation. To combat this issue, a course in forensic method validation was piloted as an undergraduate, two-credit course at a small, liberal arts college within a forensic biology program. The course had both a lecture and a laboratory component which culminated in the joint submission of a formal validation report. Students increased their technical skills and knowledge of validation that they were then able to take with them into post-graduate employment. Based on a post-course survey, students would recommend this course to others, and they gained more laboratory and technical experience in this course compared to other undergraduate lecture and laboratory-based courses. The work herein serves as a model for offering educational experiences in forensic method validation that can be expanded and delivered in other learning platforms. Keywords: method validation, education, forensic science . Introduction Method validation is a process used to demonstrate that an analytical method or system performs satisfactorily to meet the requirements of its intended purpose (1). Data generated from a validated method or system is expected to produce the same or similar result when analyzed by a different laboratory or analyst under the same conditions within a measure of uncertainty (2). Once a method or system is validated, scientists can be confident that when the method is applied according to the developed procedure, it will perform as expected and produce reliable data for their customers. In the forensic sciences, method validation recommendations have been established by scientific working groups. Recently, standards have been created in multiple forensic science sub-disciplines to create consistency in how analytical methods and systems are validated (3). There are validation standards in the forensic toxicology, DNA analysis, bloodstain pattern, and wildlife sub-disciplines of forensic science, to date (3). The movement from ‘recommendations’ to ‘standards’ for method validation aligns with the 2009 National Research Council report, Strengthening Forensic Science in the United States: A Path Forward call for improving quality assurance in the work processes of forensic scientists (4). Validation is done in many ways and in many fields, from clinical testing to computer software (8,9). There are short courses, boot camps, textbooks and other resources where a person inexperienced with method validation can turn to learn how to conduct such a study. Two examples of resources are cited here (9,10). Validations may be performed by dedicated staff in crime laboratories, more tenured forensic scientists, a team of forensic scientists, and/or be outsourced to commercial entities. A search of 257 workshops from the past 10 years of conference proceedings for the American Academy of Forensic Science (AAFS) resulted in only 17 workshops that covered content on method validation (2013 – 2023). Understanding, planning, executing, and reporting a validation are therefore, knowledge, skills, and abilities (KSAs) that forensic scientists must acquire. Professional development and training opportunities exist in-house at laboratories, at regional and national conferences and workshops, virtual webinars, and through other innovative pathways in the forensic science community. However, minimal opportunities exist to gain KSAs in method validation before forensic scientists enter the workforce or during their professional development while employed. This article describes a pilot offering of an undergraduate course titled “Forensic Method Validation”. This two-credit, 400-level, lecture- and laboratory-based course was offered at a rural, primarily undergraduate college in the spring 2019 semester. The J Forensic Sci Educ 2023, (5)1 2023 Journal Forensic Science Education Conte course was offered as part of the undergraduate program in forensic biology. Students that completed this course have gained KSAs in forensic method validation and are better prepared for employment as forensic scientists. Methods New course proposal procedures were followed to create a new course at Keystone College in the Turock School of Arts and Sciences. The procedure included completing a “New Course Proposal Form” and submitting it to the Academic Committee for approval in the academic year prior to the first offering of the course. Students enrolled in the lecture- and laboratory-based course for two credit hours in the spring 2019 semester (N = 10 students). The class met together for one hour per week in a classroom and students worked asynchronously in the laboratory during the six available hours each week when the faculty was readily available to answer questions and provide support. Prerequisites included courses in statistics, forensic biology (lecture and lab course), and a survey of general forensic science (lecture and lab course). The Investigator Quantiplex Pro Kit (QIAGEN, Hilden, Germany) was the product the students were tasked with internally validating. The Quantiplex Pro Kit was chosen as it was readily available and did not require the higher cost of STR amplification and analysis. The validation was done on a QIAGEN RotorGene Q 6 Plex. All other supplies and consumables were purchased and made available on one laboratory bench for students to use, as needed. Lecture topics, lab tasks, and learning resources are outlined in TABLE 1. Topics the students had learned previously, such as DNA extraction and statistical calculations, were reviewed in the course prior to doing these tasks in the laboratory setting. The course objectives were: 1. Describe the documents and recommendations of a forensic validation. 2. Describe the studies conducted during a forensic validation. 3. Conduct laboratory experiments to complete one forensic validation study. 4. Critically analyze self-generated data from a forensic validation study and communicate the results in written form. In Week 3, students were asked to select which study they wanted to conduct in the validation (reproducibility, repeatability, sensitivity, specificity, stability, mixtures, stochastic effects, case-like samples) and studies were assigned based on a first-come, first-served basis. Two students were assigned to the reproducibility study due to the nature of having two students replicating the samples, and two students were assigned to the sensitivity study due to the larger sample sizes. Students were assessed based on their participation (80%) and the final joint validation report (20%). Students received letter grades for the course. Reviews were received by the author 1-2 months after the course ended. Additionally, two years after completion of the course, students completed course evaluations anonymously via an electronic portal. Hazards and Safety Precautions Students were required to wear goggles, lab coats, closed-toe shoes, and long pants with long hair pulled back during laboratory times. Universal precautions were followed when handling biological fluids (blood and saliva). Safety policies are reviewed during the first week of all laboratory-based courses and students sign that they have read, understood, and will follow all protocols and procedures. All students in the course signed the safety policy. TABLE 1 Outline of the forensic method validation course weekly topics, tasks, and resources. Week Lecture Topic Lab Tasks Resources Provided 1 Introduction to Validation (2,5) 2 Product overview Product handbook 3 Experimental Design 4 Review of DNA extraction process DNA Extraction practice (6) 5 Contamination in the DNA Laboratory Sample Collection 6 DNA Extraction 7 Review of qPCR (standard curves, data output, etc.) qPCR (6) 8 qPCR 9 Exporting data into Excel for calculations Data Analysis Bring laptops – working in Excel 10 Review of statistical calculations Statistics Statistics textbook 11 What goes into a validation report? Draft Validation Report (7) 12 Explanation of feedback from draft report Editing of Validation Report 13 Final Validation Report Due J Forensic Sci Educ 2023, (5)1 2023 Journal Forensic Science Education Conte Results Ten students enrolled in the pilot method validation course in the spring 2019 semester. Each student was assigned and held accountable for preparing the experimental design for their assigned study, executing the laboratory experiment, collecting and analyzing the data, and contributing their methods and results to the class-wide final validation report. The faculty member helped guide students throughout the course. The final experimental design from the students for the Investigator Quantiplex Pro Kit validation is found in FIGURE 1. In the experimental design phase of the course, students struggled with determining the sample size for their study. The appropriate sample size of a validation is still up for debate in the forensic community, thus, opening our classroom to a discussion topic of “how many samples are sufficient to validate a kit?”. An additional area the students struggled was how to design their study. Resources were provided that included published validations and students were encouraged to emulate these previous works. However, there was a lot of “trying to re- invent the wheel” observed when interacting with students during this class period, meaning, students were reluctant to design their experiment after previous studies. The student evaluation survey was completed by 5 students. There were 10 questions that students scored on a Likert Scale where 1 meant they “strongly agreed” and 5 meant they “strongly disagreed” (TABLE 2). There were three open-ended questions included in the survey and one lab competency question. The three open ended questions were: (1) Please identify area(s) where you think the course could be improved, (2) Please identify what you consider to be the strengths of the course, and (3) Please provide any additional information (such as if you’ve worked on a validation in a position, asked about this course in an interview, realized you never want to do validation, etc.). Of the respondents, 60% have completed or started graduate studies and 100% have gained employment in a scientific capacity. FIGURE 1 Student-designed experimental design of Investigator Quantiplex Pro Kit validation conducted in Forensic Method Validation course. TABLE 2 Results of Forensic Method Validation course evaluations. Survey Question Average Likert Scale Score (1 = Strongly Agree, 5 = Strongly Disagree) The instructional materials (i.e. readings, handouts, lab handbooks, articles) increased my knowledge and skills in forensic method validation. 1.6 The course was organized in a manner that helped me understand the underlying concepts. 1.6 The course gave me the confidence to do more advanced work with the laboratory skills. 1.4 I believe the concepts I was being asked to learn in this course are important. 1.2 I would recommend this course to other students. 1.2 The course provided an appropriate balance between instruction and practice. 1.6 The lab portions of the course complemented my understanding of the lectures. 1.4 The course developed my abilities and skills for forensic method validation. 1.6 I enjoyed this course. 1.8 I feel I gained more from this course compared to other lecture & lab-based courses. 1.8 J Forensic Sci Educ 2023, (5)1 2023 Journal Forensic Science Education Conte For the lab competency question, students were allowed to select as many skills as possible that they used after the course. Skills to choose from included: universal precautions, validation, experimental design, nucleic acid extraction, prevention of contamination in a lab, qPCR/PCR, data analysis, statistics for data generated, and report writing for science (FIGURE 2). Preventing contamination is a skill that all respondents have utilized after the course in their graduate studies and/or employment. During the offering of this course, a contaminating event occurred where a student contaminated samples with their own DNA. This contaminating event helped students experience the impact of contamination. Two students had to start DNA sample extraction over again before proceeding to obtain uncontaminated extracts. Eighty percent of respondents answered universal precautions, validation, experimental design, data analysis, and report writing as skills they have used in both this course and post-graduation. In open ended questions, feedback on areas to improve included: only having the course available to junior- and senior-level students that have coursework for concepts in the course (e.g., statistics and forensic biology), extending the length of the course, and having the ability to experience more than one study of a validation. In the open prompt question about strengths of the course, all responses included the teamwork of the course being similar to a workplace and felt like “real world” experience that they did not get in other lab courses. Additionally, respondents mentioned they have gained transferable skills for all areas of science in method validation. In other comments added, two respondents stated that they brought this course up in their job interview for their first job out of college and they got feedback from the interviewer that the course helped them land the job. A third response stated that she was able to get started more quickly during an internship and contribute more as she was involved in a validation in a forensic DNA crime lab. FIGURE 2 Assessment of skills students in the forensic method validation course had used in employment. Discussion and Conclusion Method validation is a critical process that takes place before implementing a new product, technology, or workflow into practice. Typically, entry level forensic scientists have their first exposure to method validation on the job, and after post-secondary education. This gap presents a learning opportunity to reinforce forensic scientific principles through exploring how a method is fit for purpose. The newly developed forensic method validation course presented in this paper is a step toward filling this gap and aligns with recent national strategies in forensic science that support fostering “the next generation of forensic Science researchers” (11). The course was successfully offered to 10 students in a forensic biology undergraduate program. The course had a lecture and a laboratory component, including aspects where students worked independently as well as on a team. In the open prompt feedback, it is evident that this course directly links to students entering the workforce – and at times, giving the student an edge over other job applicants. The method validation course also provided students with additional experience with laboratory techniques that they used on-the-job. One area in which students were apprehensive involved mimicking other validation studies in designing their experiments. It is hypothesized this apprehension stemmed from much emphasis in undergraduate education on experiments being novel and having unique hypotheses. This observation furthers the need for students and young scientists to understand method validation and how these experiments differ from foundational and applied research. Some ideas for how to improve this course include, but are not limited to, inviting a guest speaker that has recently done or is doing a method validation to offer firsthand experience, a class discussion at the conclusion of the class on lessons learned and pain points, having it worth more credits and thus more contact hours, and support from a statistics expert to reinforce statistical concepts. This forensic method validation course can be used for other forensic disciplines and methods or products, while keeping the same learning objectives – making it easier for faculty to get a new course approved and potentially offer different sections of the course with different validation focus areas (for example, one section for qPCR, a second section for validating a GC/MS method). The forensic method validation course can alternatively be offered in a “boot-camp-style” where students spend a few weeks, or weekends, learning the basics of validation while conserving their time and financial resources compared to a traditional college classroom setting (12). J Forensic Sci Educ 2023, (5)1 2023 Journal Forensic Science Education Conte Acknowledgements The author thanks Dr. Vicki Stanavitch for her support in the creation of a new course. The author thanks QIAGEN for donating the commercial kit that was validated in this course. The author thanks Jennifer Bonetti for providing feedback on a draft of the manuscript. References 1. Hibbert DB. Method validation of modern analytical techniques. Accred Qual Assur 1999;4(8):352–6. https://doi.org/10.1007/s007690050381. 2. Wille SMR, Peters FT, Di Fazio V, Samyn N. Practical aspects concerning validation and quality control for forensic and clinical bioanalytical quantitative methods. Accred Qual Assur 2011;16(6):279–92. https://doi.org/10.1007/s00769- 011-0775-0. 3. Academy Standards Board | American Academy of Forensic Sciences. https://www.aafs.org/academy- standards-board (accessed November 15, 2022). 4. Council NR, Sciences D on E and P, Statistics C on A and T, Affairs P and G, Law C on S Technology, and, Community C on I the N of the FS. Strengthening Forensic Science in the United States: A Path Forward. National Academies Press, 2009. 5. Scientific Working Group on DNA Analysis Methods. Validation Guidelines for DNA Analysis Methods, 2016. 6. Butler JM. Forensic DNA Typing: Biology, Technology, and Genetics of STR Markers. Elsevier, 2005. 7. (EN) - Investigator Quantiplex HYres Developmental Validation Report - QIAGEN. https://www.qiagen.com/us/resources/resourcedetail? id=6703bd2a-4f7c-4798-add9- 83381c3b0527&lang=en (accessed November 14, 2022). 8. Park SH, Choi J, Byeon J-S. Key Principles of Clinical Validation, Device Approval, and Insurance Coverage Decisions of Artificial Intelligence. Korean J Radiol 2021;22(3):442–53. https://doi.org/10.3348/kjr.2021.0048. 9. Adrion WR, Branstad MA, Cherniavsky JC. Validation, Verification, and Testing of Computer Software. ACM Comput Surv 1982;14(2):159–92. https://doi.org/10.1145/356876.356879. 10. Lab Training | CDC. 2023. https://www.cdc.gov/labtraining/index.html (accessed March 6, 2023). 11. National Institute of Justice, "Forensic Science Strategic Research Plan, 2022-2026," June 20, 2022, nij.ojp.gov: https://nij.ojp.gov/topics/articles/forensic-sciences- strategic-research-plan-2022-2026 12. Zumwalt K. Simple Answers: Alternative Teacher Certification. Educational Researcher 1996;25(8):40–2.