Swan_533 Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 189 Querying Rural Content Experts Using an Online Questionnaire Marilyn A Swan, PhD 1 Barbara B. Hobbs, PhD, RN 2 1 Assistant Professor, School of Nursing, Minnesota State University, Mankato, marilyn.swan@mnsu.edu 2 Associate Professor and Assistant Dean, College of Nursing, South Dakota State University, barbara.hobbs@sdstate.edu Abstract Purpose: Obtaining feedback from rural content experts is critical in developing valid and reliable instruments to advance the science of rural health. However, traditional methods, i.e., focus groups are impractical due to location and distance. Using an online questionnaire combined with telephone and email contacts to obtain content experts’ feedback is discussed. Item statement analysis and efficiency and effectiveness of the process are presented. Methods: The process included the development of an online questionnaire, asking experts to rate 51 item statement for their relevancy, sufficiency of description, and clarity and readability. To increase the response rate, a series of four contacts (one telephone and three email) were planned and implemented. An item content validity index (I-CVI) was calculated for all items. Results: Distribution of the online questionnaire to rural content experts separated by geographic distance was efficient and effective in gathering feedback on item statements for content Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 190 validity. Content experts completed the questionnaire in less than one hour suggesting the overall efficiency of the process; an 84% response rate supports process effectiveness. Following I-CVI evaluation, item statements were reduced from 51 to 24. The analysis resulted in retaining, with or without revision, 47% of the item statements. Conclusions: The online questionnaire and four-contact strategy were effective in gathering input from a representative sample of rural content experts separated by great distances; thereby, strengthening the content validity of the item statements. The process demonstrates new opportunities for using online technologies to reach rural content experts. Keywords: Content validity, rural, content experts, instrument development, lack of anonymity Querying Rural Content Experts Using an Online Questionnaire Feedback from rural experts who live in remote, sparsely populated areas is critical to developing valid and reliable instruments that advance the science of rural health. One challenge reported in the literature when conducting rural health studies is obtaining feedback from a representative sample (McCauley et al., 2006; Prinz, Kaiser, Kaiser, & Von Essen, 2009). Remoteness and rural isolation can be factors when seeking rural content expert’s participation and feedback on rural health issues and research (Schlairet, 2017; Williams, 2012). Identifying and recruiting content experts was necessary as a key component during new instrument development for the rural health concept lack of anonymity. A frequently used method to gather content experts’ input is a focus group; experts can interact and share their knowledge and perspectives on a topic (Carter, Bryant-Lukosius, DiCenso, Blythe, & Neville, 2014). However, the use of focus groups is not practical for gathering feedback from rural experts separated by vast geographic distance. Multiple disciplines report using online questionnaires with rural populations for collecting data (Dillman, Smyth, & Christian, 2014; Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 191 Smyth, Dillman, Christian, & O’Neill, 2010). Although there are no reports of online questionnaires being used to reach content experts, challenges and barriers to using online questionnaires with rural populations have been identified, e.g., lack of internet access, and sampling methodology. Additionally, there is an ethical concern related to the importance of establishing a relationship with an individual before sending an online questionnaire (Smyth et al., 2010). The lack of a previously established strategy for reaching rural experts resulted in exploring technological options for constructing an efficient and effective process for gathering rural content expert feedback. This paper discusses the development and implementation of a process, using an online questionnaire, to gather rural content expert feedback on item statements for use in a measure on lack of anonymity. The online questionnaire was also used to establish content validity. To strengthen the response rate, a series of four contacts (one by telephone and three by email) were planned (Dillman, Smyth, & Christian, 2009). The effectiveness and efficiency of both the online content expert questionnaire and the contact plan are discussed. The actual instrument items, scoring, pilot testing, and psychometrics are not presented in this article. Background Rural nursing theory recognizes lack of anonymity as a component of living in rural areas; within the theory, rural is defined as “living in sparsely populated areas” (Long & Weinert, 2018, p. 1). Therefore, for this project, rural was defined as U.S. counties with a population of less than 10,000 (Health Resources and Services Administration, 2014). A clear understanding of the concept and establishment of content validity are essential to instrument development (Grant & Davis, 1997; Norbeck, 1985). Concept analysis is frequently used to define concept attributes and to determine empirical referents, how the concept is Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 192 experienced in everyday life (Walker & Avant, 2011). Analysis of the concept informs the development of potential item statements that fully represent the concept the researcher is investigating (Grant & Davis, 1997; Lynn, 1986). Prior to this project, a concept analysis of lack of anonymity was completed and 51 item statements were generated (Lynn, 1986; Swan & Hobbs, 2017). The next step in the initial instrument development process was to establish content validity (Lynn, 1986). Establishing content validity ensures that the instrument includes items that accurately represent the intended concept and are relevant to the content domain (Houser, 2008; Waltz, Strickland, & Lenz, 2017). An approach to validate content is by asking others, who have experience or knowledge with the concept, for their feedback (Lynn, 1986; Streiner & Kottner, 2014). As such, content expert input and analysis of the individual item statements is essential to establish content validity during instrument development (Fehring, 1987; Lynn, 1986). The scoring by experts of each item statement relevance is needed to calculate a content validity index (I-CVI) for that item. Once the item statements have been analyzed and validated based on the expert feedback, the item statements can be used in the instrument for data collection. Method The first step in developing the online questionnaire was to gain a solid working knowledge of the software, QualtricsTM. Advanced tutorials within the questionnaire software were viewed to learn how to maximize the software capabilities. Knowledge acquisition on the questionnaire software was iterative; tutorials were repeatedly viewed to enhance learning that was then applied to designing the questionnaire. A major design consideration was to limit the burden on the rural expert, who would need to review 51 item statements along with related questions. The questionnaire needed to be easy to use and function efficiently to help compensate for the length. Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 193 However, the first attempt at designing the questionnaire did not meet these criteria; the layout and appearance of the questionnaire was difficult for users to navigate and it lacked question logic. In the second design, an item structure was developed that incorporated question logic as outlined in Figure 1. Figure 1. Item Statement Structure for the Online Questionnaire Question logic allows questions to be displayed based on the expert’s answer to the previous question. Using the question logic in Figure 1, all 51 item statements were listed as individual questions and rated using a four-point Likert-type scale, ranging from 4, highly relevant, to 1, not relevant, to evaluate relevancy and adequacy of the statement description (Waltz, Strickland, Lenz, & Soeken, 2010). Items answered as highly relevant and quite relevant presented the expert with the next question about the item. Items answered as somewhat relevant or not relevant advanced to the next item statement to be evaluated. The application of question logic had the potential to reduce the overall number of questions each expert would need to answer. For clarity and readability, experts were asked if the item statement was clear and readable; rating options of yes, no, or yes, but requires revision were used. Following the yes, but requires revision option, a text box allowed experts to enter suggested item statement revisions. Similar to the qualitative nature Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 194 of focus groups or meetings, the multiple opportunities for written feedback captured the expert’s thoughts on each item statement. At the end of the questionnaire, content experts were asked if the item statements they identified as highly and quite relevant, comprehensively describe the concept of lack of anonymity. This final question was followed by an opportunity for the content experts to share any additional feedback, thoughts, or revisions in a text field. Using the process described above, experts could answer as few as 51 or as many as 153 questions. To address the potential time burden, the software allowed experts to enter and exit the questionnaire as needed, saving their answers before exiting. This strategy ensured that data were not lost and reduced expert burden in completing a lengthy questionnaire. Questionnaire Development Following the development of the online questionnaire, a small feasibility test was conducted on the questionnaire and email instructions to ensure that the online questionnaire was efficient and ready for expert use. Two colleagues with a background in higher education were asked to complete the questionnaire. Each tester brought a different perspective--one as a nurse educator with rural health expertise and the other as a library and information technology expert to inform the questionnaire development process. The testers were asked to review and use the email instructions and provide feedback on the usability of the directions and, online questionnaire, issues they encountered, and the time it took to complete the questionnaire. The testers reported that the online questionnaire functioned as designed and could be completed in approximately one hour. Both testers reported that, due to the questionnaire length, they felt lost at times. They felt that having more information about the findings from the concept analysis would have helped them navigate the questionnaire. Based on these comments, two documents were sent as email attachments to the content experts: findings from the concept Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 195 analysis, and a list of the 51 item statements. The testers concluded that access to the supporting information, before entering the online questionnaire, would allow for a more comprehensive review of each item statement by the experts. Sample and Setting Rural health experts were identified as best equipped to provide feedback on the rural health concept of lack of anonymity. Seventeen rural health and nursing experts from the United States and Canada were selected based on recognition as leaders in rural health care, research, or theory. Following a review by the South Dakota State University Internal Review Board, the project was considered exempt from human subject review and the rural content experts were contacted. Recruitment A series of four contacts was used to approach the content experts. The first contact was made by telephone with the principle investigator using a written script to introduce herself and explain the purpose for the expert review. An introduction was necessary as many of the experts were not personally known to the investigators. Experts were provided an overview of the online questionnaire, data collection process and timeline, and time commitment. They were subsequently asked if they would be willing to participate. Experts who verbally indicated willingness to participate confirmed preferred email address and were told that future communication would occur by email. Contacting the experts by telephone provided time to establish a relationship with the experts, served as pre-notice to receiving the online questionnaire, and provided transparency about the time commitment required to participate in the content validity process (Dillman et al., 2009). The initial telephone contact was anticipated to last for 15 to 20 minutes. Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 196 Following the initial telephone call, the second contact (1st email) was an introductory email. A total of three email contacts were made, seven days apart, over the course of three weeks (Dillman et al., 2009). Consent was implied by questionnaire completion. Once an expert completed the online questionnaire, no further emails were sent, and participation was complete. Each email contained appreciation and recognition to the content experts about the value of their feedback, and a thank you was sent when the questionnaire was completed. Content experts received information regarding the project and questionnaire completion, the defining attributes and empirical referents from the concept analysis, and a link to the online questionnaire. Additionally, a document containing the 51 item statements was sent as an email attachment. To ensure consistency of the instructions and information, the content provided by email was the same information as given during the initial telephone contact. Emailing consistent information connected the initial telephone call to the online questionnaire, and in turn, promote response (Dillman et al., 2009). The third contact (2nd email) served as a reminder about completing the questionnaire and the importance of their feedback. Last, the fourth and final contact (3rd email) re-introduced and provided the same information that was sent in the second contact (1st email). The online questionnaire remained open for seven days after the final email. In total, the online questionnaire remained open for data collection for four weeks. Results Seventeen rural health and nursing experts were initially contacted by telephone and asked to participate in an online process for gathering expert feedback for instrument development. Thirteen rural experts agreed to participate as content experts and received the series of three email contacts; the rural experts lived as close as 25 miles to the principle investigator and as far away Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 197 as 1500 miles. At the completion of the series of email contacts, 11 of 13, or 84% of the content experts, completed the online questionnaire. Following the initial telephone contact, three email contacts were made over the course of four weeks. One rural expert responded after the 1st email (second contact); 8 responded following the 2nd email (third contact); and, 2 experts responded after the 3rd email (fourth and final contact). Effectiveness and Efficiency of the Process The questionnaire data were downloaded from the software server into a spreadsheet for initial review. All responses were de-identified. Analysis required that expert responses to individual item statements be collated for comprehensive review and refinement. Additionally, each item was reviewed independently for relevancy, adequacy of description, clarity and readability. The item content validity index was calculated to determine retention, revision, or removal of item statements. The response rate (84%) exceeded the average online response rate of 33% reported by Nulty (2008) in a review of face-to-face compared to online survey response rates. Furthermore, the 84% response rate is comparable, or exceeds, a face-to-face survey method. Response rate is an indirect indicator of the quality of the questionnaire (Dillman et al., 2014). Additionally, 100% of the participating content experts completed the total questionnaire. No financial incentive was provided to experts for participating. The possible lack of internet access was not a concern for this population, as the rural experts would have internet access through employers, such as a government or educational institution. The total number of questions that each content expert could have answered ranged from 51 to 153. The actual number of questions answered by the content experts ranged from 81 to 153 (M = 126), with only one content expert answering all the questions. Nine of the 11 content experts Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 198 (82%) suggested item statement revisions; ten of the 11 content experts (91%) provided written feedback at the end of the questionnaire. The time from when the online questionnaire was entered in the QualtricsTM software until the expert exited the questionnaire ranged from 12 minutes to two-hours and 22 minutes. The mean time to complete the online questionnaire was 47 minutes. The software allowed the experts to enter and exit the questionnaire; however, no data were obtained to determine if the experts entered the questionnaire more than once. One expert’s time was not considered in this calculation, as the time exceeded 31 hours, indicating that the link was left open, making it impossible to determine the actual questionnaire completion time. Content Validity Index Experts rated the relevancy of each item statement as highly relevant, quite relevant, somewhat relevant, or not relevant using a four-point Likert-type scale. An item content validity index (I-CVI) score was calculated based on the relevancy of the item statements. The I-CVI number represents the proportion of experts who agreed with the relevance of the item statement and is calculated by dividing the number of experts who found the item statement to be highly or quite relevant by the total number of content experts (Polit & Beck, 2012). A value of one indicates complete agreement among the experts (Waltz et al., 2010). For rigor in developing a new instrument, an item statement rating of 0.8 or higher is considered acceptable (Dillman et al., 2009; Polit & Beck, 2012; Waltz et al., 2010). To help track the 51 item statements, each item statement was assigned a number that was used throughout the analysis. A spreadsheet was developed that listed each item statement by defining attribute and empirical referent and included: the number of experts who rated the item statement as highly and quite relevant; the I-CVI score; a summary of the written comments from the content experts for each item; and, a section for investigator notes and rationale for the Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 199 disposition of an item statement. The spreadsheet document was used to track the overall analysis, however, each round of analysis was recorded and saved in a separate document. Saving a document that represented each stage of the analysis ensured that data were not lost and that item statement decisions throughout the analysis process were captured. The I-CVI number was calculated for each of the item statements. The I-CVI numbers ranged from 0.36 to 1; an I-CVI score of 0.8 or higher was considered acceptable. Seventeen item statements (33%) had ratings below 0.7 and were deleted. A number of item statements fell just below the acceptable score or 0.8, with an I-CVI score of 0.7 to 0.79 (n = 11; 22%). A total of 23 item statements had an I-CVI score of 0.8 or higher (45%). Item statements with an I-CVI score of 0.7 or higher were evaluated further to determine if they should be retained, revised, or deleted. Evaluation of Item Statements Following the calculation and interpretation of the I-CVI scores, individual and collective content expert written responses for each item statement were evaluated. Also included in the evaluation was a review of the sufficiency of the description, clarity, and readability scores for the item statements from the online questionnaire. The data were analyzed to refine each item statement and to ensure that each item supported the content domain (Grant & Davis, 1997). For example, expert feedback indicated that one item statement lacked a conceptual link to lack of anonymity, and the item statement was deleted. Similarly, expert feedback informed item statement revisions to ensure clarity and proper wording. Discrepancies or inconsistencies in content expert feedback were discussed between the investigators, and taken into consideration when making decisions to revise, delete, or retain an item statement. Through this interpretive process, the item statements were refined. Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 200 Item statements with an I-CVI of .8 or higher accounted for 45% of the 51 item statements; the majority (13) of which did not require revision. The evaluation of the remaining 10 item statements with an I-CVI score of .8 or higher resulted in deletion of three item statements with wording and meaning similar to other item statements; the remaining seven item statements were revised. The evaluation of the 11 item statements with an I-CVI score of 0.7 to 0.79 resulted in seven item statements being deleted; four item statements were revised and retained. See Table 1 for the complete listing and disposition of the items. Table 1. Breakdown of Items by I-CVI Score Disposition of Item Statements I-CVI Score # of Items % of Items No Revision Revised Deleted ≥ .8 23 45 13 7 3 .7-.79 11 22 0 4 7 < .7 17 33 0 0 17 Total 51 100 13 11 27 Note. n = 51 Of the original 51 item statements, 24 (47%) were retained. Of the remaining 24 item statements, 13 (55%) required no revision and, 11 (45%) were revised. At the completion of the analysis, the 24 item statements were used in the development of the lack of anonymity instrument (LOAN-24). At the end of the questionnaire, content experts were asked if the items they rated as highly and quite relevant adequately describe the concept of lack of anonymity. Ten of the 11 content experts (91%) agreed that the item statements comprehensively described the concept. For Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 201 example, one expert shared, “I think you have captured the true essence of the concept.” This feedback supported that the content domain was sufficiently covered in the items. Before exiting the questionnaire, experts were asked to share any additional feedback, thoughts, or revisions; ten experts (91%) provided written feedback. The written feedback was used to ensure full conceptualization of lack of anonymity in the item statements. Discussion The intent of this paper was to discuss a process used to contact rural content experts and the development and use of an online questionnaire to gather feedback on item statements for a new measure. The development of valid and reliable measures on rural health concepts requires feedback from rural experts who live in remote, sparsely populated areas. Item statements were evaluated using an I-CVI score. Polit and Beck (2006, p. 496) suggest that excellent content validity results from a solid understanding of the concept, good item statements, carefully selected content experts, and clear instructions that enable experts to engage in thoughtful rating. Effectiveness and Efficiency The series of four contacts was effective in achieving an 84% questionnaire response rate. Use of a pre-notice telephone call as a first contact to prospective content experts supports the importance of social interaction and personal connection in questionnaire response (Dillman et al., 2009). The personal connection made during the pre-notice telephone call between the principle investigator and each content expert may have provided incentive for experts to complete the questionnaire. The significance of the initial telephone call on prompting response increases when considering that most of the rural content experts contacted were not known to the investigators. Evidence of the personal connection was revealed in content expert comments, including “I appreciate being asked for my input.”, “Thank you for allowing me to comment on the item Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 202 statements.”, and “I will be happy to provide any explanations or discuss my comments for clarification. This will be an interesting and relevant study.” Another benefit of the pre-notice telephone call was that the email contacts sent to the content experts were expected. Each expert had the option to delete the email or to review and complete the questionnaire. Again, the high response rate indicates the importance of making a personal connection before sending out an online questionnaire. Questionnaire ethics requires that a relationship be established before sending out an online questionnaire; however, receiving email questionnaires without knowing the investigators is not an uncommon practice (Smyth et al., 2010). The process of placing ‘cold’ telephone calls to experts was daunting, but is an ethically sound process (Dillman et al., 2009; Smyth et al., 2010). The content validity process discussed supports the use of a pre-notice telephone contact and demonstrates the importance of making a personal connection with experts. The use of question logic, within the questionnaire, reduced the overall number of questions each expert needed to answer, suggesting experts moved efficiently through the questionnaire. Efficiency was further indicated by the average time experts were in the questionnaire, which was less than the anticipated 60 minutes determined from the feasibility testing (M = 47 minutes). A majority of the questionnaires were completed over a four-day period following the second email contact. It is not known if experts may have used the time after the first email contact to review the attached conceptual information and item statements. Thus, attaching informative documents to the email contact may support questionnaire completion and time efficiency. Based on this information, the online questionnaire appeared to be an effective tool in reducing content experts’ time. Efficiency of time was a key consideration during the development process. Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 203 The use of a well-planned online questionnaire was an effective and efficient process to obtaining rural expert feedback. Additionally, the online questionnaire was a practical strategy to overcome the issue of accessing content experts in remote areas. Content Validity The multiple opportunities to provide written feedback allowed the questionnaire to function similarly to a face-to-face focus group; rural experts could provide direct feedback about the concept and item statements to the investigators. The high response rate (91%) for written feedback suggests that online questionnaires should be designed to provide multiple opportunities for content experts to write feedback. Revision of the item statements was supported by the rich, insightful feedback from the rural experts. The amount, and quality, of the written feedback was extremely helpful in refining the item statements for the development of a new measure (Lynn, 1986). The strong agreement among the experts that the item statements fully covered the domain of content, supports that lack of anonymity was fully conceptualized (Grant & Davis, 1997; Waltz et al., 2017). In turn, this established the conceptual clarity of lack of anonymity for the newly developed instrument. Making revisions without the feedback may have limited full conceptualization of lack of anonymity for instrument development. Thus, it is possible to capture the knowledge and insight of content experts using an online questionnaire. This is an important finding for investigators who work with rural and remote populations, separated geographically by distance; a planned contact process and well-planned online questionnaire is an effective strategy to obtain detailed feedback from experts. Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 204 The I-CVI score is a widely used measure for establishing content validity for specific items (Polit & Beck, 2006). In this process, the I-CVI score provided information on the relevance of each item statement that served as a basis for evaluation. Deleting item statements with a low I- CVI score, rather than trying to revise and keep them in some way, ensured the relevancy and conceptualization of lack of anonymity was preserved. Similarly, grouping the item statements by I-CVI score and reviewing item statements that fell just below the established benchmark of 0.8 provided opportunity to incorporate meaningful expert feedback to refine the item statements. At the outset, the investigators anticipated the content validity process would reduce the number of item statements. In the end, the item statements were reduced by 47%; from 51 potential item statements to a manageable number of 24. The 24 item statements were incorporated into the lack of anonymity measure (LOAN-24) and prepared for further testing. Further testing will include calculating a scale-level content validity index (S-CVI) using two raters to establish content validity for the overall scale, or measure (Polit & Beck, 2006). Importance of Planning Developing a process to access rural experts, and creating an online questionnaire required a substantial time commitment for the investigators. Time spent at the beginning of the project, including planning, developing, and testing the online questionnaire, reduced the burden and time for the content experts. Essential to the success of this process was learning the capabilities of the questionnaire software to promote a high expert response rate. Learning the questionnaire software took time, as did creating a questionnaire with 153 questions, but the knowledge is transferrable for future use. Further, testing the questionnaire prior to sending it to the experts was key in understanding the information needed to complete the online questionnaire. Implications for Nursing Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 205 Given the speed and advances in technology communication, we assert that our process successfully moves existing content expert processes into technology modalities. The success of this project supports the use of online questionnaires as a viable strategy to reach content experts in rural and remote areas. The process could be used in nursing research to support instrument development as a replacement to presence at traditional focus groups. A benefit of the online questionnaire is the ability to gather diverse feedback from individuals separate by geographic distance and time zones. As such, the process has many potential applications to capture individual feedback on issues in rural practice, research and health policy. The use of a series of four contacts demonstrated effectiveness in making a personal connection with rural experts, making the process a viable alternative when face-to-face focus groups are not practical. More research is needed on how to elicit rural content experts’ feedback using online questionnaires. Additionally, prompting a response through personal connection to the investigator gathers input about issues affecting rural practice and health policy that may have been missed in more traditional methods. Conclusion A series of four contacts, including the use of an online questionnaire, was successful in accessing rural content experts in remote areas across the United States and Canada to establish content validity for a new instrument to measure lack of anonymity. Developing processes that effectively and efficiently reach rural experts is necessary to ensure that rural expertise is accurately represented in rural research. The findings from this project suggest that a pre-notice telephone call and a well-planned online questionnaire can obtain feedback essential for content validity. Finally, the findings suggest that an online questionnaire methodology may be a suitable replacement to focus groups that may be impractical in rural and remote locations. Further, testing Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 206 of this approach is needed to demonstrate its generalizability, effectiveness, and efficiency in accessing rural experts when developing measures to improve rural health. References Carter, N., Bryant-Lukosius, D., DiCenso, A., Blythe, J., & Neville, A. J. (2014). The use of triangulation in qualitative research. Oncology Nursing Forum, 41(5), 545-547. https://doi.org/10.1188/14.ONF.545-547 Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys: The tailored design method (3rd Ed.). Hoboken, NJ: John Wiley & Sons, Inc. Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th Ed.). Hoboken, NJ: John Wiley & Sons, Inc. Fehring, R. J. (1987). Methods to validate nursing diagnoses. Heart & Lung, 16(6), 625-629. Grant, J. S., & Davis, L. L. (1997). Focus on quantitative methods. Selection and use of content experts for instrument development. Research in Nursing & Health, 20(3), 269-274. http://dx.doi.org/10.1002/(SICI)1098-240X(199706)20:3<269::AID-NUR9>3.0.CO;2-G Health Resources and Services Administration (2014). Defining the rural population. Retrieved from http://www.hrsa.gov/ruralhealth/policy/definition_of_rural.html Houser, J. (2008). Scientific inquiry. Precision, reliability, and validity: Essential elements of measurement in nursing research. Journal for Specialists in Pediatric Nursing, 13(4), 297- 299. http://dx.doi.org/10.1111/j.1744-6155.2008.00171.x Lynn, M. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382-385. http://dx.doi.org/10.1097/00006199-198611000-00017 Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 207 Long, K. A., & Weinert, C. (2018). Rural nursing: Developing the theory base. In C. A. Winters & H. J. Lee (Eds.), Rural nursing: Concepts, theory, and practice (5th ed.). New York, NY: Springer Publishing Company, LLC. https://doi.org/10.1891/9780826161710.0002 McCauley, L. A., Anger, W. K., Keifer, M., Langley, R., Robson, M. G., & Rohlman, D. (2006). Studying health outcomes in farmworker populations exposed to pesticides. Environmental Health Perspectives, 114(6), 953-960. http://dx.doi.org/10.1289/ehp.8526 Norbeck, J. S. (1985). What constitutes a publishable report of instrument development? Nursing Research, 34(6), 380-382. https://doi.org/10.1097/00006199-198511000-00022 Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment & Evaluation in Higher Education, 33(3), 301-314. http://dx.doi.org/10.1080/02602930701293231 Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489-497. http://dx.doi.org/10.1002/nur.20147 Polit, D. F., & Beck, C. T. (2012). Nursing research: Generating and assessing evidence for nursing practice (9th Ed.). Philadelphia, PA: Lippincott Williams & Wilkins. Prinz, L., Kaiser, M., Kaiser, K. L., & Von Essen, S. G. (2009). Rural agricultural workers and factors affecting research recruitment. Online Journal of Rural Nursing and Health Care, 9(1), 69. Schlairet, M. (2017, July). Complexity compression in rural nursing. Online Journal of Rural Nursing and Health Care, 17(2), 2-33. http://dx.doi.org/10.14574/ojrnhc.v17i2.445 Smyth, J. D., Dillman, D. A., Christian, L. M., & O'Neil, A. C. (2010). Using the internet to survey small towns and communities: Limitations and possibilities in the early 21st century. Online Journal of Rural Nursing and Health Care, 18(2) http://dx.doi.org/10.14574/ojrnhc.v18i2.533 208 American Behavioral Scientist, 53(9), 1423-1448. http://dx.doi.org/10.1177/00027642103 61695 Streiner, D. L., & Kottner, L. (2014). Recommendations for reporting the results of studies of instrument and scale development and testing. Journal of Advanced Nursing, 70(9), 1970- 1979. http://dx.doi.org/10.1111/jan.12402 Swan, M. A., & Hobbs, B. B. (2017). Concept analysis: Lack of anonymity. Journal of Advanced Nursing, 73(5), 1075-1084. http://dx.doi.org/10.1111/jan.13236 Walker, L. O., & Avant, K. C. (2011). Strategies for theory construction in nursing (5th ed.). Columbus, OH: Prentice Hall. Waltz, C. F., Strickland, O. L., & Lenz, E. R. (Eds.). (2017). Measurement in nursing and health research (5th Ed.). New York, NY: Springer Publishing Company. Waltz, C. F., Strickland, O. L., Lenz, E. R., & Soeken, K. L. (2010). Validity of measures. In C. F. Waltz, O. L. Strickland, & E. R. Lenz (Eds.), Measurements in nursing and health research (4th ed., pp. 163-201). New York, NY: Springer Publishing Company. Williams, M. A. (2012). Rural professional isolation: An integrative review. Online Journal of Rural Nursing and Health Care, 12(2), 3.