1 Ann. psychophysiol. ISSN 2412-3188 (Online) | 2410-1354 (Print) APP| Published By AEIRC| https://doi.org/10.29052/2412-3188.v8.i1.2021.1-5 Editorial Changes in the process used to critique articles based on Psychophysiologically Based Research Studies Richard A. Sherman Editor, Annals of Psychophysiology. Large numbers of audits have shown that we are inundated with faked studies, poorly done studies, improperly massaged data, sales pitches, etc. Few of the major studies can be replicated, and many journals still refuse to publish replications – especially if they don't support the original study's results. Thus, the way we need to critique studies has shifted from a relatively straightforward evaluation of the study to a detective process, including evaluating the author(s) and the journal in which the study appeared. This set of criteria is only applicable to research studies using human or non- human subjects. Studies appropriate for applying the following criteria can be from any area within psychophysiology, including clinical, sports, education, military, etc. It is not for theoretical articles, thinly veiled sales pitches, etc. The critique process is active and generally involves more than reading an article then accepting its conclusions at face value: The person critiquing a research article needs to gain some perspective on the area the article discusses, the authors' qualifications and experience (are they sales folk selling something, etc.), the literature the authors included in their review as opposed to what is published, etc. It is also likely that the critiquer will be checking the statistics and other crucial portions of the article by using statistical software. Citation: Sherman RA. Changes in the Process used to Critique an Article Based on a Research Study from the Field of Psychophysiology. APP.2021; 8(1):1-5 Corresponding Author Email: drrichsherman@gmail.com DOI: 10.29052/2412-3188.v8.i1.2021.1-5 Received 22/03/2021 Accepted 22/04/2021 Published 01/06/2021 Copyright © The Author(s). 2021 This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: The author(s) received no specific funding for this work. Conflicts of Interests: The authors have declared that no competing interests exist. https://doi.org/10.29052/2412-3188.v8.i1.2021. http://creativecommons.org/licenses/by/4.0/) http://creativecommons.org/licenses/by/4.0/) http://creativecommons.org/licenses/by/4.0/) 2 ISSN 2412-3188 (Online) | 2410-1354 (Print) APP| Published By AEIRC| Volume 8 Issue 1 The Author(s)  Check their CVs online.  Do their degrees and experience match the expertise needed for the study?  Are they from institutions (academic, organizations/government labs, corporations, etc.) that seem legitimate? If you haven’t heard of the institution, take a moment to look it up. Many are fake.  Do one or more of the authors work for or seem to get funding from groups selling products associated with the study?  Is this one of a string of articles on the topic?  Does it look like they are selling something?  Is there a conflict of interest? The Journal  Remember that there are now so many thousands of journals that anybody can get any “study” published. Many of the journals are predatory (charge to publish) and have fake peer-reviews. These journals will publish anything submitted to them.  Is the journal in which the article was published appropriate for the audience or a very odd journal choice?  Does the journal seem to be peer- reviewed with a reasonable impact factor or a predatory journal that will publish anything? If the journal does not have an impact factor, it means that virtually nobody is citing articles from the journal. Very legitimate specialty journals such as Applied Psychophysiology and Biofeedback are read by far fewer people and cited by fewer authors than the top general scientific journals. So, while the New England Journal of Medicine has an impact factor of about 75, Applied Psychophysiology has a relatively respectable impact factor of about 2. If the impact factor is below 0.5, something is probably wrong with the journal.  Does the journal charge the author to publish in the journal? If so, it is not likely to be a legitimate journal. The exception is when a granting agency demands that articles based on work they support be published as “open access” so anybody can read the entire results of the work they supported. This is a requirement for all Federal grants. Journals are permitted to change open access fees.  Is the journal dedicated to selling a product?  Does the journal's website look legitimate, or is it poorly set out with mistakes? The Abstract Does the abstract match the findings in the article, or did the authors add extra, exaggerate significance, etc.? Introduction & literature review  Does the introduction and literature review differentiate between actual studies and sales pitches?  Is the basic idea of the study plausible or so far from anything that makes sense that you would have a difficult time believing the results? If this is the case, do the authors present a reasonable case for presenting proof for an extraordinary idea? In other words, they need to convince you that they did the study and had sufficient safeguards against data manipulation and cheating to have gotten the results they claim. Remember that nearly all (but not all) such studies fail replication and are found to be either fraudulent in some way or simply poorly performed. Think of the infamous “prison” experiment 3 ISSN 2412-3188 (Online) | 2410-1354 (Print) APP| Published By AEIRC| Volume 8 Issue 1 where it turns out that the college student “guards” were told just what to do to influence the outcome.  Is the problem being studied identified clearly and precisely?  Is it sufficiently limited in scope so the study could be done?  Is the problem justified in light of theoretical and empirical work relevant to the topic?  What does the literature say? How does the study fit with what is known? How does it contribute to gaps in knowledge?  Is the theoretical and practical significance of the problem discussed?  Do the authors discuss the importance of studying the problem relative to the risks to the subjects?  Are the hypotheses tied to the problem being investigated and then clearly stated in a testable way using the proposed outcome measures?  Given the material in the introduction and the literature review, is the experimental design appropriate to the stage of establishment of efficacy, etc. (are the authors performing a placebo- controlled study when it is only time for an open study)?  Is the study very oddly designed or typical for what the investigators are working on?  Is the literature analyzed or just listed without comment? Did they mix sales pitches and testimonials with actual studies?  In the literature review, can you tell if the authors did a solid job or just listed articles they found which seem to bear on the study?  For example, if they cited Bem’s 2011 (J of Personality and Social Psychology 100) incredible claim that students who take a test then (after taking the test) practice for it, will do better on the initial test if they are given information in the post-test practice relevant to the test they just took, did the authors cite the three failures to replicate and point out the fatal flaws in the design?  Are the studies referenced by the authors unpublished and only referred to in books without substantial data & detail + lacking peer review?  Were the outcome measures justified in the introduction/literature review? Methods  Is the study well designed and using an appropriate design for what the authors want to find out? If the design is not typical for developing the idea (pilot, single group, etc.), be very suspicious.  Is the study design plausible? (e.g. could it be performed as stated, or is there too much for a subject to do or remember?)  Are the outcome measures related to the problem?  Are they sensitive enough to pick up likely changes?  Are they reliable and valid for the population being studied?  Are they likely to be the best, or is there something fishy about them as common outcome measures for the problem aren't listed?  Is there sufficient detail for you to perform the study without having to contact the authors? If not, something is wrong! The intervention must be clearly described in great detail, as must how the outcome measures were utilized.  Did a separate team apply the outcome measures than the one performing the treatment? (Subjects regularly lie to the treatment team about how well the treatment worked.)  Can you tell how the subjects were recruited? Would the method lead to bias in results (e.g. subjects were from only specialty clinics when the subjects 4 ISSN 2412-3188 (Online) | 2410-1354 (Print) APP| Published By AEIRC| Volume 8 Issue 1 are supposed to represent a general population, were the subjects found on a web site populated mainly by people who have more extensive problems with the disorder than might be found in a general population?)  Are the variables being measured/recorded relevant to the hypotheses and basic question the study is trying to answer (e.g. for a pain study, do they have several ways to assess pain or just ask how happy the subjects are with the outcome?)  Are the outcome measures as objective as possible? For outcome measures that count on scoring intensity of a problem by various investigators, was a way to assure that all investigators score the same way?  Is there evidence that the person applying the intervention already had sufficient expertise in using the intervention, so there is no significant learning curve during the study?  If this is a double-blind, placebo- controlled study, was it registered in advance? Remember that nearly all published studies show significant findings in the direction of the hypothesis without preregistration, but with registration, only a few do. This is because investigators conducting preregistered studies have great difficulty hiding studies that didn’t work out or changing them to make it seem they produced significant results.  Can you tell the method used for randomizing?  Were the subjects adequately diagnosed (even for sports studies, this is crucial)? Were the diagnostic/sorting criteria listed?  Was the intensity of the intervention sufficient to produce an effect? (e.g. two muscle tension biofeedback treatments for jaw pain probably won’t help much.)  Was there an adequate pre-treatment baseline so normal symptom variability was established?  How were the number of subjects likely to be needed for the study determined (e.g. power analysis based on pilot work, etc.)? Results / Statistics  Did the data analysis use statistical techniques appropriate to the design, or are they reporting odd techniques? If the expected techniques are not reported, then the odds are they did not show the results the investigators wanted, so they looked for obscure tests that would give the results they wanted.  When you look at the demographic breakout of the groups, are they too similar to be believable?  Were any subjects excluded from the final analysis? If so, were details of their results presented?  Were the pre and post-treatment (or between groups) symptom intensities typical of the general population (e.g. number of headaches per month?). Remember that small groups can have very different levels than the general population by random choice of subjects. A small but consistent change that seems statistically significant can still fall within the general population's normal variability.  Were there sufficient subjects to detect a difference if there is one given inter subject or intra subject variability?  Were there sufficient subjects for the results to be believable regardless of finding a “significant” difference?  Was the follow-up long enough so the duration of any changes could be estimated?  Can you tell which of the subjects learned the technique, used the medication, etc.? The results need to 5 ISSN 2412-3188 (Online) | 2410-1354 (Print) APP| Published By AEIRC| Volume 8 Issue 1 divide subjects who didn’t learn/use the medication from those who did. (The subjects who didn’t learn the technique shouldn’t do well.)  For parametric techniques, can you tell if the data are normally distributed and about the same amount of spread for each group?  Can you tell if there were sufficient subjects to find a difference between groups or before/after if there is one?  Were sufficient descriptive statistics presented, so the results are clear?  If you are suspicious of the statistics, there should be sufficient descriptive statistics to run your test.  Was any statistical difference related to clinical significance (A small but consistent difference showing as statistically significant but meaningless in the real words)? Did the authors discuss effect size as well as p values? The major concern here is the “effect” size of the result. The conclusion/discussion  In the conclusion and the abstract, is this a sales pitch, or do the results support the conclusion?  Does political, economic, or other bias show in the discussion?  Are the findings related to research noted in the introduction/literature review?  Is it obvious that some articles were left out of the literature review and then skipped in conclusion?  Are the conclusions valid and justified given the actual results of the analysis and the study’s limitations?  Are the generalizations based on the study appropriate or grandiose?  The bottom line: Do you trust what the authors did and said? Would you change your practice based on the study? https://crossmark.crossref.org/dialog/?doi=10.29052/2412-3188.v8.i1.2021.1-5