Advances in Social Science and Culture ISSN 2640-9682 (Print) ISSN 2640-9674 (Online) Vol. 6, No. 5, 2024 www.scholink.org/ojs/index.php/assc 41 Original Paper Using the Web-first Tailored Design Method in a Rural Population: A Follow-Up Investigation M. E. Betsy Garrison1*, Timothy Killian1, Kelly A. Way1 & Zola Moon1 1School of Human Environmental Sciences, University of Arkansas, Fayetteville, AR, USA * Corresponding author Received: July 29, 2024 Accepted: September 02, 2024 Online Published: September 24, 2024 doi:10.22158/assc.v6n5p41 URL: http://dx.doi.org/10.22158/assc.v6n5p41 Abstract Surveys remain a mainstay of social sciences research. The study’s primary purpose was to replicate a web-first tailored design method (TDM) in a previously surveyed rural population from a state in the mid-south of the U.S. to investigate changes in response mode (web versus paper). Most respondents did not change their response mode between the two data collection periods. They chose to complete their surveys on paper rather than the internet, even though most had access to and used the internet. The use of TDM continues to be an effective data collection strategy for a rural population. Given the emphasis on increasing internet access, speed, and reliability in rural areas of the U.S., along with other web-based services, future data collection efforts will likely involve web-based surveys. Concerns about false identities, bots, and security will continue to need to be addressed, and personal characteristics related to survey completion considered. Keywords address-based sampling, push-to-web, rural populations, survey research, tailored designed method 1. Introduction Surveys remain a mainstay of social sciences research. Collecting data using web-based surveys creates attractive efficiencies for researchers by reducing time and cost compared to collecting data through a postal service. However, important questions about internet access and utilization persist, especially when collecting data from rural populations, where broadband can be unreliable or unavailable. Data collection strategies, including sampling and population characteristics, are paramount in survey research. The generalizability of research findings and inferences is only as sound as the data, including the response rates upon which they are drawn. For several decades, Dillman’s survey design methods (1978) have been considered a gold standard of survey research, initially with the Total Design Method, www.scholink.org/ojs/index.php/assc Advances in Social Science and Culture Vol. 6 No. 5, 2024 Published by SCHOLINK INC. 42 then broadened to the Tailored Design Method, and more recently push-to-web strategies (Dillman et al., 2014; Dillman, 2017; Dillman, 2022). Collecting data via a web survey has drawbacks relative to paper surveys. Among these drawbacks are pre-existing and emerging issues, including technical incompatibility, data security, over-surveying, and spam emailing (Daikeler et al., 2020). Chief among these drawbacks is that researchers may need to be made aware of whether a person or a bot is completing the survey, particularly in studies that rely upon Amazon’s Mechanical Turk (MTurk) for the recruitment of survey respondents. In a review of the use of MTurk in management research, Aguinis et al. (2021) identified 10 validity threats, including its vulnerability to bots. The remaining threats can be categorized into respondent challenges, such as inattention, self-misrepresentation, self-selection bias, high attrition rates, English fluency, and MTurk exposure and involvement in MTurk communities. Two meta-analyses, one more recent than the other, found that the response rates of web surveys were lower than for other modes in a sample of 114 studies (Daikeler et al., 2020; Manfreda et al., 2008). However, these analyses also suggested that response rates could be increased if researchers used prenotifications, enhanced recruitment and solicitation strategies, and increased the number of contact attempts. From a sample of 1071 studies in education-related fields, Wu (2022) found an average response rate of 44%, albeit with a wide range, and neither sending a web survey to more participants nor using incentives resulted in a higher response rate. Similar to Daikeler et al.’s (2020) findings, clearly defined populations, pre-notifications, multi-modal approaches, and the age and occupation of respondents were related to response rates. Admittedly, some of these factors are easier for researchers to control than others—mainly the first three compared to the last two. A limitation of Wu et al.’s (2022) study is the reliance on education research with student respondents. They often are given extra credit in courses or other incentives to participate (e.g., pizza parties and competition prizes), and whose ages are typically younger than the general population, particularly in rural areas of the U.S. Both meta-analyses (Daikeler et al., 2020; Wu et al., 2022) failed to consider location, particularly rural versus urban, a limitation addressed in the current study. In summary, data collection strategies in the social sciences vary, but there is concern that delivery methods may be problematic in locations where internet service is inadequate, if even available. Moreover, the credibility of potential respondents is questionable if researchers are using internet-based databases, such as MTurk, with the proliferation of artificial intelligence and bots. The primary purpose of the current study was to replicate the use of the web-first TDM (Dillman et al., 2014) in a previously surveyed rural population (citation omitted for peer review) to investigate changes in response mode (i.e., web versus paper) over time. A secondary purpose of the current study was to examine internet access, reliability, and use within a rural population. www.scholink.org/ojs/index.php/assc Advances in Social Science and Culture Vol. 6 No. 5, 2024 Published by SCHOLINK INC. 43 2. Method Data were initially collected following the recommended multi-step with repeated reminders TDM in 2017 from a rural population in a mid-south U.S. state (Martinez et al., 2020). In 2020 data were collected again from the same population using the same TDM process, with its repeated mailings and two-dollar incentives. Almost half (47%) of the original 2017 study participants (n=812) responded in 2020, an improvement from the 30% response rate in 2017. Most respondents were white, middle-aged, married females with at least some college with a middle-class income level. Most of the survey in 2020 was the same as the 2017 survey (Martinez et al., 2020), including questions about health and well-being. Unlike in 2017, questions about COVID-19 and the internet and cell phones were added to the 2020 survey. There were several questions in the survey about cell phones and the internet. Of the questions that did not require a yes or no response, the response set for cell phone coverage ranged from almost never to very often, and the response set for internet usage ranged from never to several times a day. 3. Results More than half of the respondents (57%) completed the survey by mail, and 43% finished the survey by web. Most respondents (69%) did not change their response mode between the two data collection periods. However, the chosen response mode changed significantly (χ2 = 53.66, p < .001), with more respondents opting to complete the paper version than initially. As previously mentioned, questions about cell phones and internet access, reliability, and use were added to the 2020 survey. Almost all (96%) of the respondents had a cellphone, and of those, 84% had a smartphone. They reported that their cell phone coverage was frequently problematic because of poor signals or dropped calls, indicating somewhat unreliable service. Almost all (90%) indicated they had some form of internet access, and of those that had internet access at home, 72% reported it was broadband or high-speed access. Most accessed the internet several times a day at home by either a cell phone (78%) or a computer (77%). They primarily used the internet to stay connected with family and friends (77%), shop (76%), pay bills (66%), and for entertainment (64%). The results indicated that many respondents utilized the internet daily for life activities regardless of living in a rural location, although less for survey completion. Technological advancements have assisted populations in daily tasks, employment, lifestyle, and communication. 4. Discussion In the present study, rural respondents chose to complete the paper version of the surveys despite having internet access. These findings are similar to those conducted during the advent of web-based surveys (Manfreda et al., 2008). In a more recent meta-analysis, however, Holcomb et al. (2022) did not find a significant difference in response rates between paper and web surveys. Because of TDM’s address-based sampling approach, concerns about bots are avoided when employed. www.scholink.org/ojs/index.php/assc Advances in Social Science and Culture Vol. 6 No. 5, 2024 Published by SCHOLINK INC. 44 The COVID-19 pandemic might have influenced the findings of the current study. Possibly, there was more comfort in completing the paper version questionnaire from this older, mostly retired sample, particularly during uncertain and socially isolated times. They might have used the internet for more necessary or pleasurable activities, especially if they had data or Wi-Fi plans with limited usage. Unsurprisingly, respondents reported using the internet primarily for staying connected with family and friends, closely followed by shopping. COVID-19 might have influenced both of these usages as the data were collected early in the pandemic and when a vaccine was not yet available and sheltering in place was highly encouraged. As television program streaming becomes ever more available and, in some cases, necessary to watch specific series, movies, and sporting events, it is not unrealistic to expect that using the internet for entertainment will be reported more frequently. Future research should address this trend, particularly in rural populations within and outside the U.S. More sources of errors in survey research exist besides sampling and nonresponses (Ponto, 2015). These errors include both coverage and measurement errors. Holtom et al. (2022) recommended that a response-rate validity framework be utilized during a research project’s planning and design phase(s) to ensure sample quality, appropriateness, and representativeness. Future survey research should also be planned and designed explicitly to detect bots and false identities (Bernerth et al., 2021). The use of web-first TDM continues to be an effective data collection strategy for a rural population. An overlooked advantage of paper surveys is that they are completed by a person and not a bot. Circumstances indeed exist in which paper surveys are warranted over web-based. Concerning sound survey research—one size does not fit all. It is also hard to overstate the importance of research design and planning phases, including establishing trust with intended respondents and carefully appraising personal characteristics, such as age and occupation, known to be related to survey completion (Daikeler et al., 2020). Given the emphasis on increasing internet access, speed, reliability, and unlimited plans in rural areas of the U.S., along with other web-based services (such as telemedicine), future data collection efforts will likely emphasize web-based data collection. These efforts will become even more true as additional protection from bots are made available and security upgraded. As the digital divide narrows in rural and urban areas, studies like the current one need to be replicated, adapted, and updated to comprise both rural and non-rural populations. Web-based data collection is not a panacea, however. The success of the drop-off/pick-up method with hard-to-reach populations was demonstrated more than 10 years ago (Allred & Ross-Davis, 2011; Jackson-Smith et al., 2016; Trentelman et al., 2016) and more recently by the vaccine hesitancy surrounding COVID-19 within rural and historically underserved communities (Willis et al., 2021). Response rates and nonresponse bias often improve with the involvement of trusted community leaders, including those from faith-based communities. It is also incumbent on social scientists to develop mutually respectful, culturally competent relationships with the populations they study before collecting data, regardless of quantitative or qualitative methods or web or paper surveys. www.scholink.org/ojs/index.php/assc Advances in Social Science and Culture Vol. 6 No. 5, 2024 Published by SCHOLINK INC. 45 Acknowledgments The University of Arkansas School of Human Environmental Sciences and the Arkansas Agricultural Experiment Station partially supported the project, which was approved as project number ARK02480. Thanks to our participants and the Center for Survey Statistics & Methodology, Survey Research Services at Iowa State University. References Aguinis, H., Villamor, I., & Ramani, R. S. (2021). MTurk research: Review and recommendations. Journal of Management, 47, 823-837. https://doi.org/10.1177/0149206320969787 Allred, S. B., & Ross-Davis, A. (2011). The drop-off and pick-up method: an approach to reduce nonresponse bias in natural resource surveys. Small-Scale Forestry, 10, 305-318. https://doi.org/10.1080/21645515.2018.1496767 Bernerth, J., Aguinis, H., & Taylor, E. C. (2021). Detecting false identities: A solution to improve web-based surveys and research on leadership and health/well-being. Journal of Occupational Health Psychology, 26, 564-581. https://doi.org/10.1037/ocp0000281 Daikeler, J., Bošnjak, M., & Manfreda, K. L. (2020). Web versus other survey modes: an updated and extended meta-analysis comparing response rates. Journal of Survey Statistics and Methodology, 8, 513-539. https://doi.org/10.DOI:10.1093/jssam/smz008 Dillman, D. A. (1978).Mail and telephone surveys: The total design method. New York: Wiley. Dillman, D. A. (2017). The promise and challenge of pushing respondents to the web in mixed-mode surveys. Survey Methodology, 43, 3-31. Retrieved from http://www.statcan.gc.ca/pub/12-001-x/2017001/article/14836-eng.htm Dillman, D. A. (2022). Fifty years of survey innovation. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 154, 9-38. Dillman, D. A., Smyth, J. A., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. New York: Wiley Holtom, B., Baruch, Y., Aguinis, H., & Ballinger, G. A. (2022). Survey response rates: Trends and a validity assessment framework. Human Relations, 75, 560-1584. https://doi.org/10.1177/00187267211070769 Jackson-Smith, D., Courtney, G., Flint, C. F., Dolan, M., Trentelman, C. K., Holyoak, G., Thomas, B., & Ma, G. (2016). Effectiveness of the drop-off/pick-up survey methodology in different neighborhood types. Journal of Rural Social Sciences, 31, 35-67. https://egrove.olemiss.edu/jrss/vol31/iss3/3 Manfreda, K. L., Bosnjak, M., Berzelak, J., Haas, I., & Vehovar, V. (2008). web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50, 79-104. https://doi.org/10.1177/147078530805000107 www.scholink.org/ojs/index.php/assc Advances in Social Science and Culture Vol. 6 No. 5, 2024 Published by SCHOLINK INC. 46 Martinez, D., Moon, Z., Killian, T., Way, K., & Garrison, M. E. B. (2020). Does a web-first tailored design method work with rural populations? Journal of Rural Social Sciences, 35(2). Ponto, J. (2015). Understanding and evaluating survey research. Journal of the Advanced Practitioner in Oncology, 6, 168. Trentelman, C. K., Irwin, J., Petersen, K. A., Ruiz, N., & Szalay, C. S. (2016). The case for personal interaction: drop-off/pick-up methodology for survey research. Journal of Rural Social Sciences, 31, 4. https://egrove.olemiss.edu/jrss/vol31/iss3/4 Willis, D. E., Andersen, J. A., Bryant‐Moore, K., Selig, J. P., Long, C. R., Felix, H. C., Curran, G. M., & McElfish, P. A. (2021). COVID‐19 vaccine hesitancy: Race/ethnicity, trust, and fear. Clinical and Translational Science, 14, 2200-2207. https://DOI.org/10.1111/cts.13077 Wu, M., Zhao, K., & Fils-Aime. F. (2022). Response rates of online surveys in published research: A meta-analysis. Computers in Human Behavior Reports, 7, 100206. https://doi.org/10.1016/j.chbr.2022.100206