




































 
 
 

1 
 

Ann. psychophysiol. 
ISSN 2412-3188 (Online) | 2410-1354 (Print) 

APP| Published By AEIRC| https://doi.org/10.29052/2412-3188.v8.i1.2021.1-5 
 

 
 

Editorial  

Changes in the process used to critique 
articles based on Psychophysiologically 
Based Research Studies 
Richard A. Sherman 
Editor, Annals of Psychophysiology. 
     

Large numbers of audits have shown that we are inundated with faked 
studies, poorly done studies, improperly massaged data, sales pitches, etc. Few 
of the major studies can be replicated, and many journals still refuse to publish 
replications – especially if they don't support the original study's results. Thus, 
the way we need to critique studies has shifted from a relatively 
straightforward evaluation of the study to a detective process, including 
evaluating the author(s) and the journal in which the study appeared. 

 
This set of criteria is only applicable to research studies using human or non-
human subjects. Studies appropriate for applying the following criteria can be 
from any area within psychophysiology, including clinical, sports, education, 
military, etc. It is not for theoretical articles, thinly veiled sales pitches, etc. The 
critique process is active and generally involves more than reading an article 
then accepting its conclusions at face value: 
 
The person critiquing a research article needs to gain some perspective on the 
area the article discusses, the authors' qualifications and experience (are they 
sales folk selling something, etc.), the literature the authors included in their 
review as opposed to what is published, etc. It is also likely that the critiquer 
will be checking the statistics and other crucial portions of the article by using 
statistical software. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Citation: Sherman RA. Changes in the 
Process used to Critique an Article 
Based on a Research Study from the 
Field of Psychophysiology. APP.2021; 
8(1):1-5 
 
Corresponding Author Email: 
drrichsherman@gmail.com 
 
DOI: 10.29052/2412-3188.v8.i1.2021.1-5 
 
Received 22/03/2021 
 
Accepted 22/04/2021 
 
Published 01/06/2021 
 
Copyright © The Author(s). 2021 This  
 is an open access article distributed 
under the terms of the Creative 
Commons Attribution 4.0 International 
License, which permits unrestricted use, 
distribution, and reproduction in any 
medium, provided the original author 
and source are credited.  
 

 

Funding: The author(s) received no 
specific funding for this work. 

Conflicts of Interests: The authors 
have declared that no competing 
interests exist. 
 

https://doi.org/10.29052/2412-3188.v8.i1.2021.
http://creativecommons.org/licenses/by/4.0/)
http://creativecommons.org/licenses/by/4.0/)
http://creativecommons.org/licenses/by/4.0/)


 
 
 

2 
 

ISSN 2412-3188 (Online) | 2410-1354 (Print) 

 

APP| Published By AEIRC| Volume 8 Issue 1  

 

The Author(s) 
 Check their CVs online.  
 Do their degrees and experience match 

the expertise needed for the study? 
 Are they from institutions (academic, 

organizations/government labs, 
corporations, etc.) that seem 
legitimate? If you haven’t heard of the 
institution, take a moment to look it up. 
Many are fake. 

 Do one or more of the authors work for 
or seem to get funding from groups 
selling products associated with the 
study? 

 Is this one of a string of articles on the 
topic? 

 Does it look like they are selling 
something?  

 Is there a conflict of interest? 
 

The Journal 
 Remember that there are now so many 

thousands of journals that anybody 
can get any “study” published. Many 
of the journals are predatory (charge to 
publish) and have fake peer-reviews. 
These journals will publish anything 
submitted to them. 

 Is the journal in which the article was 
published appropriate for the audience 
or a very odd journal choice? 

 Does the journal seem to be peer-
reviewed with a reasonable impact 
factor or a predatory journal that will 
publish anything? If the journal does 
not have an impact factor, it means that 
virtually nobody is citing articles from 
the journal. Very legitimate specialty 
journals such as Applied 
Psychophysiology and Biofeedback are 
read by far fewer people and cited by 
fewer authors than the top general 
scientific journals. So, while the New 
England Journal of Medicine has an 
impact factor of about 75, Applied 
Psychophysiology has a relatively 

respectable impact factor of about 2. If 
the impact factor is below 0.5, 
something is probably wrong with the 
journal. 

 Does the journal charge the author to 
publish in the journal? If so, it is not 
likely to be a legitimate journal. The 
exception is when a granting agency 
demands that articles based on work 
they support be published as “open 
access” so anybody can read the entire 
results of the work they supported. 
This is a requirement for all Federal 
grants. Journals are permitted to 
change open access fees. 

 Is the journal dedicated to selling a 
product? 

 Does the journal's website look 
legitimate, or is it poorly set out with 
mistakes? 

 

The Abstract 
Does the abstract match the findings in the 
article, or did the authors add extra, 
exaggerate significance, etc.? 
 

Introduction & literature review 
 Does the introduction and literature 

review differentiate between actual 
studies and sales pitches? 

 Is the basic idea of the study plausible 
or so far from anything that makes sense 
that you would have a difficult time 
believing the results? If this is the case, 
do the authors present a reasonable case 
for presenting proof for an 
extraordinary idea? In other words, 
they need to convince you that they did 
the study and had sufficient safeguards 
against data manipulation and cheating 
to have gotten the results they claim. 
Remember that nearly all (but not all) 
such studies fail replication and are 
found to be either fraudulent in some 
way or simply poorly performed. Think 
of the infamous “prison” experiment 



 
 
 

3 
 

ISSN 2412-3188 (Online) | 2410-1354 (Print) 

 

APP| Published By AEIRC| Volume 8 Issue 1  

 

where it turns out that the college 
student “guards” were told just what to 
do to influence the outcome.  

 Is the problem being studied identified 
clearly and precisely? 

 Is it sufficiently limited in scope so the 
study could be done? 

 Is the problem justified in light of 
theoretical and empirical work relevant 
to the topic? 

 What does the literature say?  How does 
the study fit with what is known?  How 
does it contribute to gaps in knowledge?  

 Is the theoretical and practical 
significance of the problem discussed? 

 Do the authors discuss the importance 
of studying the problem relative to the 
risks to the subjects? 

 Are the hypotheses tied to the problem 
being investigated and then clearly 
stated in a testable way using the 
proposed outcome measures? 

 Given the material in the introduction 
and the literature review, is the 
experimental design appropriate to the 
stage of establishment of efficacy, etc. 
(are the authors performing a placebo-
controlled study when it is only time for 
an open study)? 

 Is the study very oddly designed or 
typical for what the investigators are 
working on? 

 Is the literature analyzed or just listed 
without comment? Did they mix sales 
pitches and testimonials with actual 
studies? 

 In the literature review, can you tell if 
the authors did a solid job or just listed 
articles they found which seem to bear 
on the study? 

 For example, if they cited Bem’s 2011 (J 
of Personality and Social Psychology 
100) incredible claim that students who 
take a test then (after taking the test) 
practice for it, will do better on the 
initial test if they are given information 
in the post-test practice relevant to the 

test they just took, did the authors cite 
the three failures to replicate and point 
out the fatal flaws in the design? 

 Are the studies referenced by the 
authors unpublished and only referred 
to in books without substantial data & 
detail + lacking peer review? 

 Were the outcome measures justified in 
the introduction/literature review? 

 

Methods 
 Is the study well designed and using an 

appropriate design for what the authors 
want to find out? If the design is not 
typical for developing the idea (pilot, 
single group, etc.), be very suspicious. 

 Is the study design plausible? (e.g. 
could it be performed as stated, or is 
there too much for a subject to do or 
remember?) 

 Are the outcome measures related to the 
problem? 

 Are they sensitive enough to pick up 
likely changes? 

 Are they reliable and valid for the 
population being studied? 

 Are they likely to be the best, or is there 
something fishy about them as common 
outcome measures for the problem 
aren't listed? 

 Is there sufficient detail for you to 
perform the study without having to 
contact the authors? If not, something is 
wrong!  The intervention must be 
clearly described in great detail, as must 
how the outcome measures were 
utilized. 

 Did a separate team apply the outcome 
measures than the one performing the 
treatment? (Subjects regularly lie to the 
treatment team about how well the 
treatment worked.) 

 Can you tell how the subjects were 
recruited? Would the method lead to 
bias in results (e.g. subjects were from 
only specialty clinics when the subjects 



 
 
 

4 
 

ISSN 2412-3188 (Online) | 2410-1354 (Print) 

 

APP| Published By AEIRC| Volume 8 Issue 1  

 

are supposed to represent a general 
population, were the subjects found on 
a web site populated mainly by people 
who have more extensive problems 
with the disorder than might be found 
in a general population?) 

 Are the variables being 
measured/recorded relevant to the 
hypotheses and basic question the 
study is trying to answer (e.g. for a pain 
study, do they have several ways to 
assess pain or just ask how happy the 
subjects are with the outcome?) 

 Are the outcome measures as objective 
as possible? For outcome measures that 
count on scoring intensity of a problem 
by various investigators, was a way to 
assure that all investigators score the 
same way? 

 Is there evidence that the person 
applying the intervention already had 
sufficient expertise in using the 
intervention, so there is no significant 
learning curve during the study? 

 If this is a double-blind, placebo-
controlled study, was it registered in 
advance? Remember that nearly all 
published studies show significant 
findings in the direction of the 
hypothesis without preregistration, but 
with registration, only a few do. This is 
because investigators conducting 
preregistered studies have great 
difficulty hiding studies that didn’t 
work out or changing them to make it 
seem they produced significant results.  

 Can you tell the method used for 
randomizing? 

 Were the subjects adequately diagnosed 
(even for sports studies, this is crucial)? 
Were the diagnostic/sorting criteria 
listed? 

 Was the intensity of the intervention 
sufficient to produce an effect? (e.g. two 
muscle tension biofeedback treatments 
for jaw pain probably won’t help  
much.) 

 Was there an adequate pre-treatment 
baseline so normal symptom variability 
was established? 

 How were the number of subjects likely 
to be needed for the study determined 
(e.g. power analysis based on pilot 
work, etc.)? 

 

Results / Statistics 
 Did the data analysis use statistical 

techniques appropriate to the design, or 
are they reporting odd techniques? If 
the expected techniques are not 
reported, then the odds are they did not 
show the results the investigators 
wanted, so they looked for obscure tests 
that would give the results they wanted.  

 When you look at the demographic 
breakout of the groups, are they too 
similar to be believable? 

 Were any subjects excluded from the 
final analysis? If so, were details of their 
results presented? 

 Were the pre and post-treatment (or 
between groups) symptom intensities 
typical of the general population (e.g. 
number of headaches per month?). 
Remember that small groups can have 
very different levels than the general 
population by random choice of 
subjects. A small but consistent change 
that seems statistically significant can 
still fall within the general population's 
normal variability. 

 Were there sufficient subjects to detect a 
difference if there is one given inter 
subject or intra subject variability? 

 Were there sufficient subjects for the 
results to be believable regardless of 
finding a “significant” difference? 

 Was the follow-up long enough so the 
duration of any changes could be 
estimated? 

 Can you tell which of the subjects 
learned the technique, used the 
medication, etc.? The results need to 



 
 
 

5 
 

ISSN 2412-3188 (Online) | 2410-1354 (Print) 

 

APP| Published By AEIRC| Volume 8 Issue 1  

 

divide subjects who didn’t learn/use 
the medication from those who did. 
(The subjects who didn’t learn the 
technique shouldn’t do well.) 

 For parametric techniques, can you tell 
if the data are normally distributed and 
about the same amount of spread for 
each group? 

 Can you tell if there were sufficient 
subjects to find a difference between 
groups or before/after if there is one? 

 Were sufficient descriptive statistics 
presented, so the results are clear? 

 If you are suspicious of the statistics, 
there should be sufficient descriptive 
statistics to run your test. 

 Was any statistical difference related to 
clinical significance (A small but 
consistent difference showing as 
statistically significant but meaningless 
in the real words)? Did the authors 
discuss effect size as well as p values? 
The major concern here is the “effect” 
size of the result. 

 

The conclusion/discussion 
 In the conclusion and the abstract, is this 

a sales pitch, or do the results support 
the conclusion? 

 Does political, economic, or other bias 
show in the discussion? 

 Are the findings related to research 
noted in the introduction/literature 
review? 

 Is it obvious that some articles were left 
out of the literature review and then 
skipped in conclusion? 

 Are the conclusions valid and justified 
given the actual results of the analysis 
and the study’s limitations? 

 Are the generalizations based on the 
study appropriate or grandiose? 

 The bottom line: Do you trust what the 
authors did and said? Would you 
change your practice based on the 
study? 

 
 
 

 
 

 

 

 

 

 

 

 

 

 

 

https://crossmark.crossref.org/dialog/?doi=10.29052/2412-3188.v8.i1.2021.1-5

