































   Advancements in Agricultural Development 
  Volume 5, Issue 2, 2024 
  agdevresearch.org 

 

1. Lauri M. Baker, Associate Professor, Agricultural Education and Communication Department, University of Florida, 1408 
Sabal Palm Drive, Level 2, Gainesville, FL 32611, lauri.m.baker@ufl.edu,  

 https://orcid.org/0000-0002-4241-60773 
2. Ashley McLeod-Morin, Associate Director of Strategic Communication, Southeastern Coastal Center for Agricultural Health 

and Safety, University of Florida, 1408 Sabal Palm Drive, Level 2, P.O. Box 110126, Gainesville, FL 32611, 
ashleynmcleod@ufl.edu,  https://orcid.org/0000-0002-8649-9783 

3. Anissa M. Mattox, Research Coordinator, UF/IFAS Center for Public Issues Education in Agriculture and Natural Resources, 
University of Florida, 1408 Sabal Palm Drive, Level 2, Gainesville, FL 32611, azagonel@ufl.edu,  

 https://orcid.org/0000-0001-6359-58204  
135 

 

All in A Moment: Continuous Response Measurement Method 
for Analyzing Dynamic Science Communication Content 

 
Lauri M. Baker1, Ashley McLeod-Morin2, and Anissa Mattox3  

 
 

Article History 
Received: June 1, 2023 
Accepted: July 31, 2023 
Published: January 31, 2024 
 
 
Keywords 
perception analyzer dials; real time 
response measurement; dual 
processing theory 
  

Abstract 
Science communication strategy plays a crucial role in effectively 
conveying scientific information to target audiences. While various 
models exist for developing communication strategies, many focus on 
measuring the effectiveness of communication efforts and adjusting 
these based on audience feedback. However, traditional methods of 
evaluating communication effectiveness often measure one component 
at a time and do not consider how people make decisions in real-life 
situations. This paper proposes the use of continuous response 
measurement (CRM) as a method to evaluate science communication in 
agricultural social science research. CRM allows for real-time 
measurement of how individuals make decisions in response to dynamic 
communication content. This paper compares different types of CRM, 
including in-person and virtual dial CRM, and provides resources for 
researchers interested in implementing this methodology. The paper also 
discusses various research designs that can be used with CRM, such as 
experimental designs, survey designs, time-series designs, focus groups, 
and coding content and behavior. The benefits and limitations of CRM are 
outlined, highlighting the need for immediate feedback and real-time 
response in science communication campaigns. In-person CRM is 
discussed, including the selection of stimuli, response prompts, data 
collection procedures, and data analysis. Virtual CRM is also examined, 
highlighting its advantages in terms of flexibility and cost-effectiveness. 
The paper concludes by discussing data output and analysis methods for 
CRM data. Overall, this paper serves as a methodological proposal for the 
use of CRM in agricultural social science research, emphasizing the 
importance of real-time measurement and response in science 
communication. 

 

mailto:lauri.m.baker@ufl.edu
https://orcid.org/0000-0002-4241-60773
mailto:ashleynmcleod@ufl.edu
https://orcid.org/0000-0002-8649-9783
mailto:azagonel@ufl.edu
https://orcid.org/0000-0001-6359-58204


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   136 
 

Introduction and Problem Statement 
 
Science communication strategy involves developing content that resonates with target 
audiences while emphasizing major tenants of science communication: timeliness, accessibility, 
trustworthiness, credibility, and usefulness. While models for developing a communication 
strategy vary, many models are cyclical in nature with an emphasis on measuring the 
effectiveness of communication efforts followed by adjusting communication based on 
feedback from target audiences (Percy, 2023; Villarreal, 2010). Primary and secondary 
consumer research play a vital role in developing a communication strategy and evaluating the 
strategy. Traditional methods for evaluating communication efforts include both quantitative 
and qualitative methodologies including surveys, sales data, behavioral observations, focus 
groups, interviews, and social media listening (Juska, 2021). Each of these methods has benefits 
in understanding how effective science communication is, but these tend to measure 
effectiveness in an artificial way, by focusing on measuring one component of a strategy at a 
time. However, modern behavioral economics indicates that people make decisions based on 
more than one factor at a time (Grayot, 2020). When you ask a question on a survey about a 
respondent’s perceptions of a campaign, you are forcing them to think about one element of a 
message at a time, but in real life they are making decisions based on multiple components at a 
time. Dual processing theory proposes that people make decisions two different ways (a) 
intuitively and (b) deliberately (De Neys & Pennycook, 2019). When you ask a question in a 
survey or interview you force people to deliberately decide based on one element. In a natural 
setting, they would decide intuitively based on a blend of message, graphics, audio, logo, color 
scheme, speaker characteristics, etc. For these reasons, it is imperative that researchers 
establish and embrace new methods for gathering real-time data from participants. 

Currently, many methods do not allow for real-time response or immediate feedback to change 
course in science communication campaigns. Continuous response measurement (CRM) offers 
a way to measure how people make decisions on dynamic, communication content in real time 
from moment-to-moment. CRM is commonly used in the communication industry for real-time 
measurement, but it has not yet been widely adopted in social science research (Maier et al., 
2009). The purpose of this paper is to propose the use of CRM in agricultural social science 
research by describing the process and how to analyze data reaped from the method. Through 
these objectives, the authors will also compare types of CRM, including in-person dial testing 
and virtual dial testing, and provide resources for other researchers who want to apply this 
methodology in their own research.   

Continuous Response Measurement 
To effectively respond to public concerns and solve real problems, immediate feedback is 
needed. One of the public criticisms of scientists during the COVID-19 pandemic was a lack of 
transparent and immediate response (Hamel et al., 2020; Kenen & Roubein, 2020; Pollard & 
Davis, 2021). Particularly in the agriculture and natural resource sectors, immediate response is 
often necessary due to the essence and urgency of the industries, such as dealing with natural 
disasters (Mike et al., 2020), food safety issues (Opat et al., 2018), biosecurity risks (Sellnow et 
al., 2017), and animal welfare issues (Steede et al., 2018). CRM offers a way for researchers to 

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   137 
 

operate in a more immediate way than traditional research methods while being more 
authentic in the way messages are measured. Messages are a “continuous stream of sensory 
stimuli arranged in patterns,” (Biocca et al., 1994, p. 15) which convey meaning through and are 
interpreted based on individuals’ cognitive state.  

Purpose 
 
Science communication strategy is vital for delivering content to target audiences effectively. 
However, a gap exists in the ability of current methods to measure real-time response and 
feedback. This paper aims to address this gap by reviewing relevant literature that can serve as 
a guide to researchers, specifically those in agricultural social science research, considering 
Continuous Response Management (CRM) as a methodology. The following research objectives 
guided this study: 

RO1: Describe the planning process for conducting a CRM study. 

RO2: Describe how to analyze data gathered from a CRM study. 

Methods 
 
To address the research objectives of this study, research synthesis methodology was used to 
comprehensively review and synthesize existing literature on CRM methods to explain how to 
plan and conduct CRM studies, data analysis approaches for CRM, and best practices for CRM in 
academic settings. The authors’ search strategy included a systematic search of relevant 
scholarly databases (e.g., PubMed, Scopus, Web of Science) to retrieve articles published up 
until January 1, 2023. Additionally, grey literature, offline sources, and popular press references 
were explored to minimize publication bias and ensure all relevant resources were found. 

Search Terms and Inclusion Criteria 
The search strategy employed a combination of keywords and Boolean operators to identify 
relevant studies. These keywords included the exact terms and variations of the following 
terms: continuous response measurement, perception analyzer dials, online/virtual dial testing, 
dial testing. Articles were included if these met the predefined inclusion criteria based on if the 
articles used CRM methods, explained CRM planning, or analysis of CRM data.  

Study Selection 
Two independent reviewers screened the retrieved literature based on title and abstract, 
followed by a full-text review of potentially relevant studies. Any discrepancies between 
reviewers were resolved through discussion or consultation with a third reviewer. Articles were 
included if they met the predefined inclusion criteria and provided pertinent information 
related to the research purpose. Studies were excluded if they did not meet the inclusion 
criteria, were duplicates, or lacked relevant data. 

 

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   138 
 

Data Extraction 
A standardized data extraction form was utilized to extract key information from included 
studies, such as author information, publication year, study design, analysis methods, 
interventions/exposures, research stimuli, research setting. The methodological quality and risk 
of bias of included studies were assessed using the Critical Appraisal Skills Programme 
Systematic Review Checklist (CASP, 2018) by two independent reviewers. Discrepancies were 
resolved through discussion or involvement of a third reviewer. 

Synthesis of Results and Reporting 
An a priori qualitative narrative synthesis was conducted to deductively summarize the findings 
from the included studies within the framework of the two study objectives. This deductive 
coding approach matched the purpose of this research as recommended in qualitative coding 
procedures (Silverman, 2019). The synthesis methods in this paper adhere to the Preferred 
Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to ensure 
transparency and comprehensive reporting of the review process (Page et al., 2021). 

Findings 
 
RO1: Describe the planning process for conducting a CRM study. 
CRM technologies allow for measurement in self-reported shifts in message processes using 
electronic technologies (Biocca et al., 1994). CRM has been used in public speaking, advertising, 
film and television, political and health communication (Biocca et al., 1994), and agricultural 
communication (LaGrande et al., 2021; Opat et al., 2022; Tarpley et al., 2020). Research designs 
within this method can use experimental, survey and quasi-experimental, time-series, focus 
groups, and coding content and behavior designs (Biocca et al., 1994). Further descriptions of 
type of research design can be seen in Table 1. The majority of previously published work, 
regardless of the type of research design, has focused on in-person use of CRM, but CRM 
technology providers have recently started introducing online CRM options (Paull, 2023).  

 

 

 

 

 

 

 

 

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   139 
 

Table 1 
 
Research Designs using CRM with Design Considerations and Previous Studies 

Design Considerations  Previous Studies 
Experimental Experimental design is the most used design in CRM 

research and has been used with standard between-
subject and within subject factorial design. Can use CRM 
to substitute for self-report measures. 

Biocca & David, 1990a; Biocca et 
al., 1987; Caspi et al., 2019; 
Jennings et al., 2021; Jeon & Lang, 
2021; Keppel, 1982; Reeves et al., 
1983; Reeves, 1984; Rossiter & 
Thornton, 2004; Thorson & Reeves, 
1985; West & Biocca, 1992; Saks et 
al, 2016; Wöllner & Auhagen, 2008. 

Quasi 
experimental 
and survey 

Television research typically selects this design. Within 
this design, CRM is used as an extension of a survey with 
survey data collected within the CRM software or in a 
post/pre survey. In in-person settings it is difficult to 
obtain representative samples in this design, but the 
online CRM option is ideal for this design. 

Baggley, 1987; Baggaley, 1986a, 
1986b; Biocca & David, 1990a; 
Beville, 1985; D'Ambrosio, 2019; 
Millard, 1989; Papakonstantinou et 
al., 2002; Philport, 1980; West & 
Bocca, 1992; West et al., 1991. 
 

Time series  This design works well for public opinion on scientific 
issues that changes over time in response to current 
events or legislation. It can also be used when purposeful 
intervention is used by the researcher to measure the 
effectiveness of an intervention over time. It can be used 
in a single group with multiple interventions, multiple 
groups with a single intervention, or multiple groups with 
multiple interventions.  
 

Glass et al., 1975 (for more on 
time-series design generally); 
Palmgren, 1996; Wöllner & 
Auhagen, 2008;  

Focus groups Advertising and marketing studies have historically 
preferred this design, but in recent years science 
communication researchers have also used this design. It 
allows for mixed method data collection with 
quantitative data from CRM technology and qualitative 
data from focus group discussion about CRM results from 
the moment-to-moment data collection. This design 
allows for collection of individual thoughts on dynamic 
content to be collected before group discussion may 
alter individual responses. This design also allows for 
deeper understanding of the meaning of CRM data 
collection and offers participants a chance to interpret 
meaning rather than researchers relying solely on 
quantitative data analysis to decipher. 
 

Baggaley, 1987; Boussalis & Coan, 
2021; Chandler et al., 2022; 
Cummins et al., 2018; Hughes, 
2015; Schwarzkopf, 2021. 

Coding 
content and 
behavior 

CRM allows for quick coding of messages, context, and 
interaction from live or recorded content. Researchers 
can use the tool to record codes from multiple coders 
simultaneously to better understand participants 
behaviors and responses to content (when participants 
are the subject being viewed) or codes for variables like 
camera distance, presence of controversial images, 
changes in speech patterns, visual ques.  

Biocca & David, 1990b; Palmer & 
Cunningham, 1987; Waddell, & 
Williamson, 2017. 

 

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   140 
 

Choosing a research stimulus is one of the most important steps in collecting CRM data (Lawson 
et al., 2020). CRM can be used to evaluate any dynamic content from a persuasive campaign 
video, a political debate, a teaching presentation, or a podcast. Table 2 provides a list of 
recommended stimuli and unique considerations for each type that arose from the themes in 
the literature. Once a stimulus is selected, researchers must choose a response prompt. 
Response prompts may include the trustworthiness of the content, emotion, or the 
persuasiveness of the message. Participants can only respond to one option at a time, so it is 
suggested that researchers limit the options they ask participants to respond to in a study. 
Participants respond to the prompt on a continuous scale with responses ranging from 0 to 
100.  

Table 2 
 
Recommended Stimulus Types for CRM and Considerations for Each 

Stimulus Unique Considerations Prior Studies Using  
Radio or podcasts CRM use began with radio content, but modern-day 

researchers have used it with podcasts. While radio 
content and podcasts can be listened to in a group 
setting, it is typically content people listen to alone, so 
an in-person CRM setting may be unnatural for this 
content; virtual CRM may provide a more natural 
context for these stimuli. 
 

Bolce et al., 1996; Opat et 
al., 2022. 

Live events, public 
speeches, or lectures 

Any live event can be used with in-person CRM, 
although outside events can be challenging for 
hardware with difficulty viewing screens on dials and 
monitors and potential for adverse weather. With a live 
event as stimuli, it is important to record the event so 
you have the content for reference later and to make 
sure you have a precise starting point for when dial use 
began with participants. Alternatively live events can be 
recorded to use as video stimuli in in-person or virtual 
CRM.  
 

Biocca & David, 1990a; 
Cummins et al., 2018; 
Hughes, 2015; West & 
Biocca, 1992. 

Videos  Videos can range from lengthy heavily produced 
content like full length documentaries to social media 
videos. CRM software has a minimum time of 30 
seconds, so shorter social media videos need to be 
combined for use. The type of video content will 
depend on if it is most appropriate for use in a group in-
person CRM setting or an individual CRM setting, which 
is more typical and effective in a virtual CRM setting. 

Biocca et al., 1987; Philport, 
1980; LaGrande et al., 2021; 
Reeves et al., 1983; Reeves, 
1984; Tarpley et al., 2020; 
Thorson & Reeves, 1985. 

 
In-person CRM   
CRM is a measurement tool that allows researchers to pinpoint specific perceptions of 
communication messages and elements of communication in real-time (Izenson, 2016). 
Traditionally, CRM is conducted in-person using handheld dials (Figure 1) that participants use 
to respond to the research stimulus. 

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   141 
 

Figure 1 
 
Participant Using a Handheld Dial in An In-Person CRM Setting 

 
 
 
 
 
 
 
 
 
 
 
 
 
 

It is recommended to have participants practice using the dials before responding to the 
research prompt. For most participants, this is the first time they have used a CRM dial. 
Researchers should ensure the participants are comfortable using the dial and understand the 
prompt. For example, if the prompt is trustworthiness, the researchers should provide an 
example of how extremely untrustworthy and extremely trustworthy is rationalized in the 
study. As participants respond to the prompt, researchers should observe the participants and 
how they are using the dials. Researchers may need to remind participants to turn their dial 
continuously. If a participant is inactive, researchers may need to remove their data from the 
study. For a thorough step-by-step guide on how to use in-person CRM in an agricultural social 
science context, refer to Lawson et al. (2020). Quantitative data is automatically collected 
through CRM software, but to gain additional insight it is recommended that researchers pause 
at key moments in data collection and/or at the end of data collection to show results and 
discuss the reasons why participants felt the way they did about the critical moments in the 
data. Using a qualitative methodology for this questioning route is recommended.   

While in-person dial testing offers many unique benefits, limitations do exist. Two of the most 
burdensome limitations are the cost and participant recruitment. Software and equipment, 
including the console and 25 handheld dials, necessary to collect in-person CRM data will cost ~ 
$25,000. Each participant must use a handheld dial when participating in a CRM study so if 
researchers have 25 dials, only 25 people can be included in each study session. To collect data 
from 120 participants, the session would need to be held at least four times. Even if researchers 
have access to more handheld dials, it is recommended to keep the study groups a manageable 
size to avoid nonresponse and technical difficulties with the dials. Additional costs will be 
incurred to recruit participants and record and transcribe qualitative data.   

 

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   142 
 

Virtual CRM  
Online CRM has recently been introduced by CRM technology providers, such as DialSmith so, 
as such, fewer academic publications exist. However, online CRM provides researchers with 
more flexibility compared to in-person dial testing. Recorded video or audio is uploaded to the 
dial testing platform. Participants use a unique web link to view or listen to the content being 
tested, and participants use their own computer or tablet to indicate their moment-to-moment 
response. Respondents receive instructions and have an opportunity to practice rating content 
not related to the study. A slider appears on the screen and participants are asked to respond 
to the research prompt by moving the slider (Izenson, 2023). The feedback is automatically 
available to researchers through the reporting portal. Visuals of the data, including moment-to-
moment video overlays, are available in the reporting portal but researchers can also download 
raw data to conduct additional analyses.   

The online CRM platform supports recorded audio and video files, including .mp4 and .mov 
files, but cannot be used to test live streaming content (Paull, 2023). Live stimuli must use in-
person CRM. Since online CRM participants do not have to be physically present, a more 
geographically diverse sample can be obtained (Alzuhn, 2023). Researchers are not limited by 
the number of handheld devices when conducting online CRM so more responses can be 
collected over a shorter period (Alzuhn, 2023). Online CRM can also be more cost effective, 
particularly when researchers have a limited number of studies they wish to conduct. Costs vary 
depending on the type of content being tested, length of content, and number of desired 
participants. Costs range from $1,500 to $2,000 USD to upload the content and manage the 
project with an additional cost to recruit participants. One distinct disadvantage to online CRM 
is respondent engagement, particularly when testing lengthier content. When dial testing in-
person, moderators can observe when participants are no longer engaging with the handheld 
device and can remind them to turn the dial or indicate when a response should be removed 
from data analysis. With online CRM, researchers cannot observe participants and help with 
focus, but a reminder is displayed on the screen to keep rating the content if a respondent 
stops moving the onscreen slider. DialSmith reports that online and in-person CRM results are 
similar in response sentiments (Paull, 2023).   

Reliability and Validity 
As with any type of research study, establishing reliability is an important step to gathering 
valid and useful data. Reliability is concerned with determining if the same results can be 
expected if measured the same way (Maier et al., 2009). This can be tested in CRM through 
test-retest, split-half, and parallel-test designs. Reliability can be a challenge in test-retest 
because of the spontaneous nature of psychological measurement within CRM studies, but 
some studies have obtained reliability this way. Split-half is a more common method for 
determining reliability in CRM studies, but parallel-test designs can also be used (Maier et al., 
2009). External validity, or generalization, in CRM studies is a matter of properly designing the 
study regarding sampling procedures that are appropriate and representativeness of the target 
population. Additionally, it is important to match the measurement setting as close to a real life 
setting as possible and select the appropriate stimuli and data collection site. Like other 
experimental and quasi-experimental designs, pre-testing of CRM instruments can help 

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   143 
 

establish validity (Maier et al., 2009). For focus group designs, researchers should focus on 
appropriate sampling for their research questions and consider transferability to other settings. 
Verbatim transcripts, audit trails, data triangulation, member checks, and multiple coders can 
be used to ensure data accuracy (Lincoln & Guba, 1985).  

Limitations 
The cost of in-person and online CRM can be a significant limitation. In-person CRM requires 
more software and equipment, which can cost an estimated $25,000 USD. Online CRM can be a 
more cost efficient alternative but still comes with a cost of $2,000 USD or more depending on 
recruitment costs. Limited training is available for CRM data collection and analysis, which can 
be a limitation for researchers wanting to implement CRM research. Some CRM software 
companies, such as DialSmith, offer training on their specific software at an additional cost. 
These companies also offer consulting services to provide input on overall study design, data 
collection, and data analysis. CRM data is continuous, resulting in a data point per second per 
participant. As such, this methodology results in a large amount of data, even when working 
with a short stimulus and a small sample size, which can be a limitation. Researchers must be 
extremely organized and attentive to details to effectively manage the large amount of data 
and tie participant responses to stimuli.   

RO2: Describe how to analyze data gathered from a CRM study. 
Output of data from CRM technologies allows researchers to interpret in multiple ways. When 
collecting data in-person, you can export data as: PowerPoint, Excel, or SPSS files. When 
conducting virtual CRM, the same data output as in-person CRM is provided. Data visualizations 
are more readily available in the virtual CRM interface and, when using DialSmith, a project 
manager can support the data output and analysis phase.  

Analysis 
Data can be analyzed in two different ways. We have divided this into real time and reflective 
analysis. Real time analysis can provide immediate feedback, course correction, and/or 
discussion within a focus group setting. CRM allows researchers to view data in real time during 
collection in aggregate, by a single participant, or by a specific demographic. Summary statistics 
for each study question are projected on the researcher monitor throughout the data collection 
and immediately following. Data are saved on the designated CRM computer within the CRM 
software for future analysis. A researcher can choose to show the results to participants to 
collect their perceptions of the results and to discuss peaks and valleys in the data (Figure 2). 
This will offer additional insight beyond the quantitative data analysis and fits into the focus 
group design described in Table 1. Peaks and valleys in the data allow for visual understanding 
of the data through identification of high and low points. Data can be grouped by other study 
variables to visually determine differences (Figure 2). Depending on the research goal, some 
science communication projects may be able to use the real-time data analysis to adjust 
communication campaigns without deeper analysis. However, to publish data in a journal, 
reflective analysis is needed.  

 

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   144 
 

Figure 2 
 
Example of Real Time Data Analysis Demonstrating Peaks and Valleys 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Note. Data outputs are visualized as peaks and valleys, with 50 being the neutral point on a 
100-point scale (DialSmith, 2023).  
 
Reflective analysis offers additional understanding for research purposes. Data can be exported 
from the CRM software directly to SPSS statistical software or Excel in ASCII (American Standard 
Code for Information Interchange) file format for statistical analyses. The next steps in data 
analysis depend on the research questions and study design. If researchers collected survey 
data separately from the CRM data, they need to merge the data files and clean the data. 
Researchers may choose to run crosstabs or mean comparisons initially to understand the data. 
In the virtual CRM option, crosstabs will be run by the project manager at DialSmith. Because of 
the large quantity of data, researchers often chose to compress the data from 1 second 
increments to 5 or 10 second increments depending on the study size. Normalizing the data 
through Z scores is another step researchers may take before moving to deeper analysis, 
particularly when working to identify statistical significance of peaks and valleys in the data. 
Researchers may select to identify Z scores greater than 1.96 or less than -1.96 (Biocca et al., 
1994), or they may choose to run independent samples t-test with an a priori established level. 
Once peaks and valleys are established as significant, then it is important to review the stimuli 
to determine what was happening at these critical moments. This will help researchers 
determine what is happening at these moments that may be affecting participants’ responses 
and to understand patterns in responses. Researchers may decide to categorize what was 
happening at critical moments to interpret the data. For example, if a particular word is used in 
all peaks, or if a particular speaker is on screen during every valley.  

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   145 
 

From this point, the data are handled similarly to how you would handle any other quantitative 
data set based on the study design and research questions. Studies have used basic descriptive 
statistics and mean comparisons. However, with such large data sets, deeper analysis methods 
are available for use and can offer deeper understanding.  Beyond means and peaks and 
valleys, researchers can understand group differences with ANOVA. They can also create 
alternative series to look at standard deviation series and absolute, mean, and percentage 
activity series (Biocca et al., 1994). Time series analysis may include autoregressive integrated 
moving average (ATIMA) techniques (Box et al., 2015). Confirmatory factor analysis may also be 
an appropriate method as an extension of regression models; spectral or Fourier analysis may 
be appropriate to understand complex problems that were periodic in nature within the CRM 
study design (Biocca et al., 1994). For those interested in a deeper understanding of data 
analysis related to CRM, Biocca et al. (1994) provides formulas, examples, and strategies.  

Focus group designs typically combine the peak and valley analysis with qualitative data 
analysis methods (Williams & Moser, 2019) to compare quantitative with qualitative results in a 
mixed methods design (O’Cathain et al., 2010). Focus group results can offer insight into 
participants thought processes while viewing stimuli more than CRM results alone.  

Conclusions, Discussion, and Recommendations 
 
The benefits of CRM in agricultural social science research are abundant, including the depth of 
data collected, real-time response and visualization of the data, reduction of recall bias, and 
limiting group think. CRM allows for researchers to quickly collect a large amount of data that 
can be analyzed in a variety of ways, including audience segmentations, experimental designs, 
qualitative research, and self-report items (Lawson et al., 2020). Even though data can be 
analyzed in multiple ways, data can be viewed in real-time as participants are responding and 
immediately exported into visualizations that can be shared with participants or relevant 
stakeholder groups. When asking participants to respond to a survey question or a question 
posed during an interview or focus group, participants must often reflect on a past event or 
experience and may not remember accurately. Since CRM is collected in real-time of the 
participants interacting with the stimuli, the need for recall is removed. Additionally, CRM 
offers a way to measure messaging in a dual process way while all elements of a message are 
considered at the same time rather than isolating one specific component. Groupthink is often 
a limitation in qualitative research and occurs when one participant or a small group of 
participants influences others’ opinions (Janis, 1997). In CRM, you can collect individual 
opinions without the influence of others, but also have the option to collect qualitative data in 
a focus group using CRM to gain individual and group insight into specific points in the data. 
Participants respond to CRM individually without the influence of other participants, limiting 
groupthink.  

CRM has not been used to its full potential in agricultural social science research, but it could 
take science communication evaluation to the next level with quantification of perceptions and 
response to communication stimuli in a moment-to-moment format. Multiple opportunities 
exist to enhance strategic science communication in agriculture and natural resources. In-

https://doi.org/10.37433/aad.v5i2.357


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   146 
 

person CRM use can offer real time testing of science and outreach products and allow for 
discussion with stakeholders on the reasons behind their views on science communication 
products. With the instantaneous nature of social media, CRM can render a similar 
environment and gather feedback. Moreover, the immediate response and course correction 
offers the ability to respond rapidly during a crisis or as a crisis develops. Because in-person 
CRM requires more equipment and a larger research team, it may not be an option for all 
researchers. However, online CRM has a lower entry point with costs allocated by online 
response and support for developing integration into surveys. This option also allows for help in 
the recruitment of participants.   

In addition to being a unique and useful research tool in science communication, CRM also 
provides opportunities for Extension, teaching, and leadership. Notably, CRM can be used to 
determine preferred teaching styles, perceptions of presenters, and response to messages 
used. Utilizing CRM can provide real-time insight to effective messaging, while also providing 
insight on how to become better teachers, presenters, and leaders.  

Acknowledgements 
 
This publication was possible because of support from the UF/IFAS Center for Public Issues 
Education in Agriculture & Natural Resources (PIE Center), https://piecenter.com  
 

References 
 
Alzuhn, E. (2023). Moment-to-moment cheat sheet: Online research vs in-person research. 

DIALSMITH. http://www.dialsmith.com/blog/moment-research-best-online-research/  

Baggaley, J. (1987). Continual response measurement: Design and validation. Canadian Journal 
of Educational Communication, 16(3), 217–238. https://doi.org/10.21432/T2VG8M   

Baggaley, J. (1986a). Developing a televised health campaign: I. Smoking prevention. Media in 
Education & Development, 19, 29–43.  

Baggaley, J. (1986b). Developing a televised health campaign: II. Skin cancer prevention. Media 
in Education & Development, 19, 173–176. 

Beville, H. M. (1985). Audience ratings: Radio, television, and cable. Lawrence Erlbaum 
Associates. 

Biocca, F., Neuwirth, K., Oshagan, H., Zhongdang, P., & Richards, J. (1987, May). Prime-and-
probe methodology: An experimental technique for studying film and television [Paper 
presentation]. International Communication Association, Montreal, QC, Canada. 

Biocca, F., David, P., & West, M. (1994). Continuous response measurement (CRM): A 
computerized tool for research on the cognitive processing of communication messages. 

https://doi.org/10.37433/aad.v5i2.357
https://piecenter.com/
http://www.dialsmith.com/blog/moment-research-best-online-research/
https://doi.org/10.21432/T2VG8M


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   147 
 

In A. Lang (Ed.), Measuring psychological responses to media messages (pp. 15–65). 
Lawrence Erlbaum Associates. 

Biocca, F., & David, P. (1990a). How camera distance affects the perception of candidates during 
a presidential debate [Unpublished manuscript]. Center for Research in Journalism and 
Mass Communication, University of North Carolina at Chapel Hill. 

Biocca, F., & David, P. (1990b, May). Micro-shifts in audience opinions: A second-by-second 
analysis of the first 1988 presidential debate [Paper presentation]. International 
Communication Association, Dublin, Ireland. 

Bolce, L., De Maio, G., & Muzzio, D. (1996). Dial-in democracy: Talk radio and the 1994 
election. Political Science Quarterly, 111(3), 457–481. https://doi.org/10.2307/2151971 

Boussalis, C., & Coan, T. G. (2021). Facing the electorate: Computational approaches to the 
study of nonverbal communication and voter impression formation. Political 
Communication, 38(1-2), 75–97.  https://doi.org/10.1080/10584609.2020.1784327  

Box, G. E., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time series analysis: forecasting 
and control (5th ed.). John Wiley & Sons. 

Caspi, A., Bogler, R., & Tzuman, O. (2019). “Judging a book by its cover”: The dominance of 
delivery over content when perceiving charisma. Group & Organization Management, 
44(6), 1067–1098. https://doi.org/10.1177/1059601119835982 

Chandler, R., Ross, H., & Guillaume, D. (2022). Innovative perception analysis of HIV prevention 
messaging for black women in college: A proof of concept study. BMC Public 
Health, 22(1), 1255. https://doi.org/10.1186/s12889-022-13564-4  

Critical Appraisal Skills Programme. (2018). CASP Checklists: Critical Appraisal Checklists. CASP. 
https://casp-uk.net/casp-tools-checklists 

Cummins, R. G., Smith, D. W., Callison, C., & Mukhtar, S. (2018). Using continuous response 
assessment to evaluate the effectiveness of extension education products. The Journal 
of Extension, 56(3), 26. https://tigerprints.clemson.edu/joe/vol56/iss3/26   

D'Ambrosio, L. (2019, December 15). Understanding the adoption of and education about new 
auto technologies among older adults. ROSAP. https://rosap.ntl.bts.gov/view/dot/43817  

De Neys, W., & Pennycook, G. (2019). Logic, fast and slow: Advances in dual process theorizing. 
Current Directions in Psychological Science, 28(5), 503-509. 
https://doi.org/10.1177/0963721419855658 

DialSmith. (2023). Our technology. https://www.dialsmith.com/technology/  

https://doi.org/10.37433/aad.v5i2.357
https://doi.org/10.2307/2151971
https://doi.org/10.1080/10584609.2020.1784327
https://doi.org/10.1177/1059601119835982
https://doi.org/10.1186/s12889-022-13564-4
https://casp-uk.net/casp-tools-checklists
https://tigerprints.clemson.edu/joe/vol56/iss3/26
https://rosap.ntl.bts.gov/view/dot/43817
https://doi.org/10.1177/0963721419855658
https://www.dialsmith.com/technology/


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   148 
 

Glass, G. V., Willson, V. L., and Gottman, J. M. (1975). Design and Analysis of Time Series 
Experiments. Colorado Associated University Press. 

Grayot, J. D. (2020). Dual process theories in behavioral economics and neuroeconomics: A 
critical review. Review of Philosophy and Psychology, 11(1), 105–136.  
https://doi.org/10.1007/s13164-019-00446-9  

Hamel, L., Kearney, A., Kirzinger, A., Lopes, L., Muñana, C., & Brodie, M. (2020, September 10). 
KFF health tracking poll - September 2020: Top issues in 2020 election, the role of 
misinformation, and views on a potential coronavirus vaccine. KFF. 
https://www.kff.org/coronavirus-covid-19/report/kff-health-tracking-poll-september-
2020/  

Hughes, S. R. (2015). Moving the needle: A comparative analysis of message reception during 
televised presidential debates [Doctoral dissertation, Texas Tech University]. Texas Tech 
University Libraries. http://hdl.handle.net/2346/66113  

Izenson, B. (2016). Your memory stinks: How flawed recall & memory bias pollute market 
research and what can be done about it. DIALSMITH. 
https://www.dialsmith.com/blog/flawed-recall-memory-bias-pollute-market-research-
can-done/ 

Izenson, B. (2023). What is Online Dial Testing? DIALSMITH. 
https://www.dialsmith.com/blog/what-is-online-dial-testing/  

Janis, I. L. (1997). Groupthink. In R. P. Vecchio (Ed.), Leadership: Understanding the dynamics of 
power and influence in organizations (pp. 163–176). University of Notre Dame Press. 
https://psycnet.apa.org/record/1997-36918-010  

Jennings, F. J., Allen, M. W., & Le Vu Phuong, T. (2021). More plastic than fish: Partisan 
responses to an advocacy video opposing single-use plastics. Environmental 
Communication, 15(2), 218–234. https://doi.org/10.1080/17524032.2020.1819363  

Jeon, Y. A., & Lang, A. (2021). The vicious cycle of stressors, food advertising and overeating: 
How stigmatizing anti-obesity PSAs, which precede food commercials, influence 
subsequent implicit and explicit attitudes toward high-and low-calorie food. Media 
Psychology, 24(5), 606–636. https://www.doi.org/10.1080/15213269.2020.1773854  

Juska, J. M. (2021). Integrated marketing communication: Advertising and promotion in a digital 
world (2nd ed.). Routledge. https://doi.org/10.4324/9780367443382    

Kenen, J., & Roubein, R. (2020, March 31). Why America is scared and confused: Even the 
experts are getting it wrong. POLITICO. 
https://www.politico.com/news/2020/03/31/experts-coronavirus-cdc-158313 

Keppel, G. (1982). Design and analysis: A researcher's handbook. Prentice Hall. 

https://doi.org/10.37433/aad.v5i2.357
https://doi.org/10.1007/s13164-019-00446-9
https://www.kff.org/coronavirus-covid-19/report/kff-health-tracking-poll-september-2020/
https://www.kff.org/coronavirus-covid-19/report/kff-health-tracking-poll-september-2020/
http://hdl.handle.net/2346/66113
https://www.dialsmith.com/blog/flawed-recall-memory-bias-pollute-market-research-can-done/
https://www.dialsmith.com/blog/flawed-recall-memory-bias-pollute-market-research-can-done/
https://www.dialsmith.com/blog/what-is-online-dial-testing/
https://psycnet.apa.org/record/1997-36918-010
https://doi.org/10.1080/17524032.2020.1819363
https://www.doi.org/10.1080/15213269.2020.1773854
https://doi.org/10.4324/9780367443382
https://www.politico.com/news/2020/03/31/experts-coronavirus-cdc-158313


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   149 
 

LaGrande, L. E., Meyers, C., Cummins, G. R., & Baker, M. (2021). A moment-to-moment analysis 
of trust in agricultural messages. Journal of Applied Communications,105(2), 2. 
https://doi.org/10.4148/1051-0834.2375   

Lawson, C., Fischer, L., LaGrande, L., & Opat, K. (2020). Do touch that dial: A guide to 
continuous response measurement in agricultural communications. Journal of Applied 
Communications, 104(3), 1–18. https://doi.org/10.4148/1051-0834.2333   

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Sage Publications, Inc. 

Mike, M. R., Rampold, S. D., Telg, R. W., & Lindsey, A. B. (2020). Utilizing extension as a resource 
in disaster response: Florida extension’s communication efforts during the 2017 
hurricane season. Journal of Applied Communications, (104)1, 1–13.  
https://doi.org/10.4148/1051-0834.2308 

Maier, J., Maier, M, Maurer, M., Reinemann, C., & Meyer, V. (2009). Real time response 
measurement in the social sciences: Methodological perspectives and applications. Peter 
Lang.  

Millard, W. (1989). Research using the Millard System (Televac) [Research report]. W. J. Millard. 

O’Cathain, A., Murphy, E., & Nicholl, J. (2010). Three techniques for integrating data in mixed 
methods studies. bmj, 341. https://doi.org/10.1136/bmj.c4587  

Opat, K., Magness, H., & Irlbeck, E. (2018). Blue Bell's Facebook posts and responses during the 
2015 Listeria crisis: A Case Study. Journal of Applied Communications, (102)4, 1–16.  
https://doi.org/10.4148/1051-0834.2232 

Opat, K., Irlbeck, E., Cummins, R. G., Li, N., & Boren-Alpizar, A. E. (2022). Get the story straight: 
Comparing narrative and logical-scientific communication to capture Gen Z’s interest in 
science podcasts. Journal of Radio & Audio Media, 1–19. 
https://doi.org/10.1080/19376529.2022.2145481 

Page, M. J., McKenzie, J. E., Bossuyt, P.M., Boutron, I., Hoffmann, T.C., Mulrow, C. D., Shamseer, 
L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., 
Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., 
McGuiness, L. A., Stewart, L. A., Thomas, J., … Moher, D. (2021). The PRISMA 2020 
statement: An updated guideline for reporting systematic reviews. bmj 372(71). 
https://www.doi.org/10.1136/bmj.n71    

Palmgren, J. (1996). Analysis of longitudinal data. Statistics in Medicine, 15(11), 123–
1232. https://doi.org/10.1002/(SICI)1097-0258(19960615)15:11<1231::AID-
SIM282>3.0.CO;2-Z 

https://doi.org/10.37433/aad.v5i2.357
https://doi.org/10.4148/1051-0834.2375
https://doi.org/10.4148/1051-0834.2333
https://doi.org/10.4148/1051-0834.2308
https://doi.org/10.1136/bmj.c4587
https://doi.org/10.4148/1051-0834.2232
https://doi.org/10.1080/19376529.2022.2145481
https://www.doi.org/10.1136/bmj.n71
https://doi.org/10.1002/(SICI)1097-0258(19960615)15:11%3C1231::AID-SIM282%3E3.0.CO;2-Z
https://doi.org/10.1002/(SICI)1097-0258(19960615)15:11%3C1231::AID-SIM282%3E3.0.CO;2-Z


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   150 
 

Palmer, M., & Cunningham, R. (1987). Computerizing the collection and coding of 
communication interaction data: The AIDE system [Unpublished manuscript]. 
Communication Arts Dept., University of Wisconsin at Madison. 

Papakonstantinou, E., Hargrove, J. L., Huang, C., Crawley, C. C., Canolty, N. L. (2002). 
Assessment of perceptions of nutrition knowledge and disease using a group interactive 
system: The Perception Analyzer®. Journal of the American Dietetic Association, 102(11), 
1663–1668, https://doi.org/10.1016/S0002-8223(02)90354-8 

Paull, A. (2023). Online dial testing FAQ. DIALSMITH. https://www.dialsmith.com/blog/online-
dial-testing-faq-frequently-asked-questions/ 

Percy, L. (2023). Strategic integrated marketing communications (4th ed.). Routledge. 
https://doi.org/10.4324/9781003169635    

Philport, J. (1980). The psychology of viewer program evaluation. In J. Anderson (Ed.), 
Proceedings of the 1980 technical conference on qualitative television ratings (pp. B1–
B17). Corporation for Public Broadcasting. 

Pollard, M. S., & Davis, L. M. (2021). Decline in trust in the Centers for Disease Control and 
Prevention during the COVID-19 pandemic. RAND Corporation. 
https://www.rand.org/pubs/research_reports/RRA308-12.html 

Reeves, B. (1984, June). Attention to television: Psychological theories and chronometric 
measures (pp. 251–279). University of Wisconsin. 

Reeves, B., Rothschild, M., & Thorson, E. (1983). Evaluation of the Tell-Back audience response 
system [Research Report for ABC]. Mass Communication Research Center, University of 
Wisconsin-Madison.  

Rossiter, J. R., & Thornton, J. (2004). Fear-pattern analysis supports the fear-drive model for 
antispeeding road-safety TV ads. Psychology & Marketing, 21(11), 945–960. 
https://psycnet.apa.org/doi/10.1002/mar.20042  

Saks, J., Compton, J. L., Hopkins, A., & El Damanhoury, K. (2016). Dialed in: Continuous response 
measures in televised political debates and their effect on viewers. Journal of 
Broadcasting & Electronic Media, 60(2), 231–247. 
http://dx.doi.org/10.1080/08838151.2016.1164164  

Schwarzkopf, S. (2021). Ten little jurors in the training camp: a genealogy of audience 
simulation. Journal of Cultural Economy, 14(6), 732–749. 
https://doi.org/10.1080/17530350.2021.1882535  

 

https://doi.org/10.37433/aad.v5i2.357
https://doi.org/10.1016/S0002-8223(02)90354-8
https://www.dialsmith.com/blog/online-dial-testing-faq-frequently-asked-questions/
https://www.dialsmith.com/blog/online-dial-testing-faq-frequently-asked-questions/
https://doi.org/10.4324/9781003169635
https://www.rand.org/pubs/research_reports/RRA308-12.html
https://psycnet.apa.org/doi/10.1002/mar.20042
http://dx.doi.org/10.1080/08838151.2016.1164164
https://doi.org/10.1080/17530350.2021.1882535


Baker et al.  Advancements in Agricultural Development 
 

https://doi.org/10.37433/aad.v5i2.357   151 
 

Sellnow, T. L., Parker, J. S., Sellnow, D. D., Littlefield, R. S., Helsel, E. M., Getchell, M. C., Smith, J. 
M., & Merrill, S. C. (2017). Improving biosecurity through instructional crisis 
communication: Lessons learned from the PEDv outbreak. Journal of Applied 
Communications, 101(4), 1–15. https://doi.org/10.4148/1051-0834.1298 

Silverman, D. (2019). Interpreting qualitative data (6th ed.). SAGE Publications. 

Steede, G. M., Meyers, C., Li, N., Irlbeck, E., & Gearhart, S. (2018). A sentiment and content 
analysis of Twitter content regarding the use of antibiotics in livestock. Journal of 
Applied Communications, 102(4), 1–16. https://doi.org/10.4148/1051-0834.2225 

Tarpley, T. G., Fischer, L. M., Steede, G. M., Cummins, R. G., & McCord, A. (2020). How much 
transparency is too much? A moment-to-moment analysis of viewer comfort in 
response to animal slaughter videos. Journal of Applied Communications, 104(2), 6. 
https://doi.org/10.4148/1051-0834.2302 

Thorson, E., & Reeves, B. (1985). Effects of over-time measures of viewer liking and activity 
during programs and commercials on memory for commercials. In R. Lutz (Ed.), 
Advances in consumer research (pp. 549-553). Association of Consumer Research. 

Villarreal, R. (2010). Integrated marketing communication strategy. Wiley International 
Encyclopedia of Marketing. https://doi.org/10.1002/9781444316568.wiem01027 

Waddell, G., & Williamon, A.  (2017). Measuring the audience. In Lee, S. (Ed.), Scholarly research 
for musicians. (pp. 148-155). Routledge. 

West, M., & Biocca, F. (1992). "What if your wife were murdered": Audience responses to a 
verbal gaffe in the 1988 Los Angeles presidential debates [Paper presentation]. The 
Annual Conference of the American Association for Public Opinion Research, St. 
Petersburg, FL. 

West, M., Biocca, F., & David, P. (1991). "You're no Jack Kennedy": Audience responses to a 
verbal barb in the 1988 Omaha vice-presidential debates [Paper presentation]. The 
Annual Conference of the American Association for Public Opinion Research, Phoenix, 
AZ. 

Williams, M., & Moser, T. (2019). The art of coding and thematic exploration in qualitative 
research. International Management Review, 15(1), 45-55. 
http://www.imrjournal.org/uploads/1/4/2/8/14286482/imr-v15n1art4.pdf  

Wöllner, C., & Auhagen, W. (2008). Perceiving conductors' expressive gestures from different 
visual perspectives. An exploratory continuous response study. Music Perception, 26(2), 
129-143. https://doi.org/10.1525/mp.2008.26.2.129  

© 2024 by authors. This article is an open access article distributed under the terms and conditions of 
the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/). 

https://doi.org/10.37433/aad.v5i2.357
https://doi.org/10.4148/1051-0834.1298
https://doi.org/10.4148/1051-0834.2225
https://doi.org/10.4148/1051-0834.2302
https://doi.org/10.1002/9781444316568.wiem01027
http://www.imrjournal.org/uploads/1/4/2/8/14286482/imr-v15n1art4.pdf
https://doi.org/10.1525/mp.2008.26.2.129

