Meta-Psychology, 2022, vol 6, MP.2020.2637 https://doi.org/10.15626/MP.2020.2637 Article type: Commentary Published under the CC-BY4.0 license Open data: Not Applicable Open materials: Not Applicable Open and reproducible analysis: Not Applicable Open reviews and editorial process: Yes Preregistration: No Edited by: Danielsson H., Carlsson, R. Reviewed by: Shirlene Wang, Oscar Olvera Astivia Analysis reproduced by: Not Applicable All supplementary files can be accessed at OSF: https://doi.org/10.17605/OSF.IO/X5QHF Careless Response Processes are Heterogeneous: Comment on Goldammer et al. (2020) Alexander J. Denison Department of Psychology, University of South Florida; Department of Education and Human Development, Clemson University Brenton M. Wiernik Department of Psychology, University of South Florida Abstract Goldammer et al. (2020) examined the performance of careless response detection indices by experimentally ma- nipulating survey instructions to induce careless responding, then compared the ability of various indices to detect these induced careless responses. Based on these analyses, Goldammer et al. concluded that metrics designed to detect overly consistent response patters (i.e. longstring and IRV) were ineffective. In this comment, we critique this conclusion by highlighting critical problems with the experimental manipulation used. Specifically, Goldammer et al.’s manipulations only encouraged overly inconsistent, or random, responding and thus did not induce the full range of behavior that is present in natural careless responding. As such, it is unsurprising that metrics designed to detect overly consistent responding appeared to be ineffective. Because the full range of careless behavior was not induced, Goldammer et al.’s study cannot address the utility of longstring or other consistency metrics. We offer recommendations for alternative experimental manipulations that may produce more naturalistic and diverse careless responding. Keywords: careless responding, insufficient effort responding, data cleaning, methodology Comment on Goldammer et al. (2020) Goldammer et al. (2020) presented two studies aimed at investigating the performance of various statis- tical methods for detecting careless respondents. Their first study experimentally manipulated response pat- terns using various instruction sets and examined the performance of commonly used careless response de- tection methods. Based on these results, Goldammer et al. provided recommendations about which detection methods are effective and ineffective. In this comment, we discuss several important limitations of the care- less response manipulations used in this study, whether these manipulations produced behavior consistent with natural careless responding, and the appropriateness of Goldammer et al.’s recommendations. https://doi.org/10.15626/MP.2020.2637 https://doi.org/10.17605/OSF.IO/X5QHF 2 Defining Careless Responding Careless or insufficient effort responding1 is part of a larger construct of carelessness or inattentiveness, which has been used to describe response behaviors that arises when individuals are not motivated to give hon- est, thoughtful responses to questions (Curran, 2016; Johnson, 2005). Careless responding occurs when this inattentiveness results in individuals answering items in a content non-responsive manner – i.e. without pay- ing attention to the content or instructions of the items (Curran, 2016; Meade & Craig, 2012; Nichols et al., 1989). This is distinct from other aberrant response pat- terns such as faking, which is content-responsive. When faking, individuals provide invalid data, but their re- sponses are contingent on the content of the items they are responding to (Nichols et al., 1989). Content non-responsive behavior has a long history of study in psychology, dating back over 30 years to examinations of detecting such behavior in the MMPI (Baer et al., 1997; Berry et al., 1992; Nichols et al., 1989) and examining factors arising from negatively keyed items (Schmitt & Stuits, 1985). The construct of careless responding grew out of these early investi- gations as a way to describe content non-responsive be- havior that was due to a lack of motivation. While early studies focused on overly inconsistent or “random” re- sponding, it is now well accepted that this behavior can take on two different forms—overly consistent or overly inconsistent responding. Overly inconsistent responding is often conceptual- ized as randomness or random responding, as it is as- sumed that individuals select their response to each item completely at random. While this behavior is dis- cussed as being completely random, researchers gen- erally seem to use the word random in the colloquial and not the statistical sense of the word (Curran, 2016). Instead, this behavior may be better conceptualized as highly inconsistent responses and is generally charac- terized by a high degree of variance within response strings, such as using every response option at least once on a question block (Curran, 2016; Marjanovic et al., 2015). Conversely, overly consistent responses follow some pattern, such as responding with the same anchor point to every item or varying responses in some pattern, such as “1, 2, 1, 2, 1, 2. . . ” or “1, 2, 3, 4, 5, 4, 3, 2, 1. . . ” (Curran, 2016; Meade & Craig, 2012). While careless responding is the most com- monly investigated construct when examining content- nonresponsive behavior, it is important to note that not all studies in the literature investigate this con- struct in its entirety. Specifically, there are some stud- ies that explicitly investigate random responding (e.g., Berry et al., 1992; Credé, 2010; Osborne & Blanchard, 2011) with no investigation of consistent responding. Although random, or more aptly termed inconsistent, responding is one piece of the careless responding con- struct, it is not equivalent to the construct of careless responding as a whole. Any investigation of purely ran- dom responding will overlook a large chunk of the care- less responding construct that involves response consis- tency. The conflation of random responding with all careless responding is something most authors are care- ful to avoid. Indeed, in all of the aforementioned ex- aminations of random responding the authors are care- ful to separate random and careless responding, and at most talk about carelessness as the underlying cause of random responding. These two conceptualizations of careless response be- havior are reflected in the metrics that have been devel- oped to catch these respondents. As Goldammer et al. note there are two general categories of indirect care- less response metrics. The first category is invariabil- ity metrics, which detect overly consistent response be- havior. This category includes longstring analysis and intraindividual variability. The second category is con- sistency metrics, which detect overly inconsistent behav- ior. This category includes a person’s response reliabil- ity and semantic or psychometric antonyms/synonyms. See Goldammer et al. (2020) or Curran (2016) for a description of these metrics. Inducing Careless Responding Given the above definition of careless responding as content non-responsive, it is critical to ensure that any manipulation intended to induce careless behavior is consistent with this definition. It is also important to consider whether experimentally induced behavior pro- vides a good proxy for what that behavior is like in the real world. For comparison, research on the aberrant response pattern of faking has consistently found that faking induced through researcher instructions does not approximate non-directed or “real-world” faking (Kun- cel et al., 2011; Viswesvaran & Ones, 1999). It is ques- tionable whether either of Goldammer et al.’s manipula- tions induce behavior that is a good proxy for real-world careless responding or that is even consistent with the definition of careless responding. Goldammer et al.’s (2020) first manipulation is “op- posite responding,” wherein participants are instructed to respond to items using the opposite scale point they 1The terms careless responding and insufficient effort re- sponding are used to refer to the same underlying construct in almost all cases. This behavior has also been called inatten- tive responding, random responding, and several other terms. In this paper we opt to use the term careless responding when discussing the construct. 3 would normally choose. This response pattern clearly violates the definition of careless responding as it is not content non-responsive behavior; participants are instructed to attend to the item content and then to re- spond in a specific aberrant way based on what they would normally answer. That is, respondents must at- tend to the content of a given item to generate their initial response, then reverse that response. Although these data may be considered invalid, as half of their responses will not align with the other half, they are produced from content-responsive, not careless, behav- ior. This is not to say that careless response indices will be unable to detect such behavior; in fact, we can see that the metrics designed to detect inconsistent response patterns do a good job of capturing this “opposite re- sponding” pattern. However, this result is unsurprising because this manipulation, by design, makes a partici- pant’s responses inconsistent and introduces a high de- gree of variance. Conversely, it would be nearly impossi- ble for the “opposite responding” manipulation to pro- duce an overly consistent response pattern that would be detected by longstring analysis, for example. In fact, longstring would only detect the opposite responding pattern if a participant happened to respond to all items at the midpoint. Consistency indices were not designed to detect an “opposite response” pattern, and it is nei- ther unexpected nor informative that they, in fact, do not detect such behavior. Thus, the results from this manipulation do not have any bearing on the utility of inconsistency or consistency indices to detect careless responding because the responses are not careless.2 Goldammer et al.’s second manipulation is “random responding.” Participants in this condition were in- structed on alternating pages to either “complete the questions below exactly as they apply to you” or to “choose any response option, no matter whether it ap- plies to you or not” (p. 4). It is again unclear what real-world response pattern this represents. Under what conditions would participants repeatedly start and stop attending to item content? Even if participants were partially careless, it is unlikely they would exhibit this in alternating blocks of responses. By repeatedly chang- ing the instructions from careful to random and vice versa, this manipulation will increase the within-person variability of responses, but it is not clear that these re- sponses will resemble real-world random responding, much less careless responding as a whole. Even if these instructions are assumed to reason- ably approximate random responding, these responses would nevertheless not reflect the full scope of response behavior exhibited in real-world careless responding. Previous work has extensively documented that careless responding is not simply responding randomly to items, but can also manifest as certain forms of patterned or consistent responding (Curran, 2016; Jaso et al., 2021; Johnson, 2005; Meade & Craig, 2012). For example, a respondent might try to complete a survey quickly by clicking the same response option repeatedly. Invari- ability metrics are designed specifically to capture this type of behavior because we know overly consistent re- sponding is a type of behavior careless respondents en- gage in. Because the manipulation used by the authors is unlikely to produce overly consistent behavior, it is again unsurprising that the invariability metrics failed to detect these respondents.3 As such, the data have no bearing on the utility of invariability indices because in- variability indices were not designed to detect the ran- dom or inconsistent behavior induced by the authors manipulation. Are Invariability Metrics Useless? A key takeaway from our examination of both of Goldammer’s manipulations is that the response pat- terns they produce will be highly inconsistent. Thus, it is unsurprising that Goldammer et al. (2020) conclude that metrics designed to detect overly inconsistent be- havior (consistency metrics) are effective at detecting careless responding, whereas the invariability metrics designed to detect overly consistent responding are not. This finding does not mean that invariability metrics are useless, but rather that the manipulations were not de- signed to produce consistent responding. Goldammer et al.’s (2020) conclusions in their Study 1 and subsequent recommendations against the use of invariability indices are especially problematic because we know that actual careless respondents do sometimes produce overly consistent response patterns. For ex- ample, Johnson (2005) identified that 3.5% of partic- 2It is somewhat difficult to determine what sort of real- world response pattern the “opposite responding” is intended to reflect. Rarely would a person be expected to actively switch their scale point use partway through a survey. One possibility would be if the anchor points for scales were re- versed midway through a multi-part survey and the partic- ipant did not notice this reversal. However, such a change would go against general best practice for survey design (Stern et al., 2007) and would likely be better simulated by instructing respondents to respond normally and randomly reverse-coding items for a subset of respondents. 3The actual participation instructions used did not explic- itly state that participants should respond at random, but it seems highly likely that participants would interpret these in- structions this way. Indeed, Golammer et al. appeared to assume as much, as their descriptions of the manipulation clearly communicate that they thought participants were en- gaging in random responding. 4 ipants in their dataset responded by selecting the same response option repeatedly throughout the survey; sim- ilar patterns have been noted in other studies of careless responding (Curran, 2016; Jaso et al., 2021; Meade & Craig, 2012). Therefore, applying only consistency met- rics in a real dataset (as done in Goldammer et al.’s Study 2) will overlook a potentially large portion of careless respondents who are engaging in overly con- sistent response behavior. Even if few or no overly con- sistent respondents are found, there is little downside to computing these metrics. How Can We Experimentally Study Careless Responding? Above, we critique Goldammer et al.’s manipulations as unlikely to produce response behavior that resem- bles real careless responding. This raises the question of how careless responding could be experimentally stud- ied. We offer several possibilities. First, rather than instructing participants to respond in a specific way, researchers can instead instruct par- ticipants to optimize their responses based on a spe- cific goal that is relevant for the population being stud- ied. For example, study participants recruited for ex- tra credit in university classes or platforms like MTurk might aim to complete surveys as quickly as possible to receive their compensation. To simulate such a strat- egy, participants could be instructed to “answer items as quickly as possible” or to “answer items as quickly as possible, while still appearing to respond carefully.” (cf. Huang et al., 2012). This type of instruction may more accurately reflect the types of thought processes that lead to real-world careless responding, and partici- pants might engage in a variety of behaviors to achieve this goal (e.g., random responding, patterned respond- ing). This approach is not foolproof. It may be difficult to write instructions that capture the myriad of factors that research participants simultaneously weigh when choosing a response strategy (cf. directed “fake good” instructions do not produce the same response strate- gies test takers use in real high-stakes settings; Kuncel et al., 2011; Viswesvaran & Ones, 1999). Response speed is also not the only goal participants optimize toward when responding carelessly, otherwise response time would be the only metric needed to detect care- lessness. A second approach might be manipulations designed to decrease the probability that individuals respond carelessly. For example, participants in one condition could be warned that “The researchers will be able to detect if you have responded carelessly. You will not re- ceive compensation if you respond carelessly.” (cf. Gib- son, 2019; Huang et al., 2012). Researchers could also employ a virtual presence (such as a human, or more abstract entity) to make participants feel moni- tored, in addition to warning them (cf. Ward & Pond, 2015). When participants are made to feel monitored or warned, carelessness rates should be lower. Accord- ingly, careless response indices would be expected to detect lower rates of carelessness in these conditions, compared to control conditions without a warning. Sim- ilarly, performance of careless responding indices could be compared across data collected in comparatively high-stakes (e.g., a job application) versus low-stakes (e.g., an extra credit or MTurk study) settings; careless responding rates would be expected to be lower in high- stakes contexts. However, the exact prevalence of care- less responding in each sample would still be unknown, which could make comparisons between conditions dif- ficult. A third approach might be to directly ask participants whether they responded to a survey carelessly and then examine whether careless responding indices can detect participants who responded affirmatively to this item (cf. a single item asking participants about their data quality can effectively detect a high percentage of care- less respondents; Meade & Craig, 2012). While this solves the problem of determining who is careless, it’s possible that the worst cases of carelessness will still be missed. That is, if someone is truly paying no attention to any items, they may answer no to this question by chance. Infrequency, trap, or instructed response items could also be used. These are items that have a correct an- swer that any conscientious respondent should be an- swer correctly (e.g., “select option 5”) but are answered incorrectly by careless participants because they are not paying attention (Curran & Hauser, 2019; Huang et al., 2015). While these items are effective at detecting care- less participants, a potential disadvantage is that care- less participants could circumvent these items if they are skimming questions in an attempt to not be caught. This method also does not solve the problem of induc- ing careless responding in the first place, so relies on this behavior being naturally present in the data. Last, a fourth approach might be to induce content non-responsive responses by using items with nonsensi- cal or blank content (cf., Maul, 2017, but see also Cur- ran and Hauser, 2019; Rhemtulla et al., 2017). While this will produce behavior that is non-responsive to the item content, since that content does not exist, this may produce content-nonresponsive behavior that dif- fers from careless responding to substantive scales. As outlined above, each of these approaches has po- tential advantages and disadvantages, and experimen- tal studies of careless responding should triangulate re- 5 sults across multiple approaches. The largest problem with inducing careless behavior is the variety of poten- tial forms and motivations behind this behavior (e.g., some participants may want to finish as fast as possible, while others may want to exert as little effort as possi- ble), which makes a precise operationalization difficult. The fact that this behavior is caused by a lack of moti- vation or effort also means that any instructed manipu- lations may not induce naturalistic careless responding because participants are now exerting conscious effort to produce this behavior. This is again why using a va- riety of approaches to triangulate this behavior seems most useful and why the process that elicits this behav- ior should capture how a lack of motivation will influ- ence response patterns. While addressing the above problem is beyond the scope of this paper, applied researchers are advised to ask a different question in the interim: what consti- tutes a conscientious response pattern? By doing so researchers can uncover response patterns that are the- oretically impossible for a conscientious respondent to produce and screen for them with careless response metrics. For example, identical responses to every item on a positive affect scale might be theoretically possi- ble, if unlikely, but identical responses to every item, or even to half the scale, on the BFI-10 does not make the- oretical sense (Rammstedt & John, 2007). Thus, differ- ent theoretically impossible longstring cut scores could be produced for these different scales. In doing so, re- searchers should consider a variety of careless respond- ing metrics, including both consistency metrics and in- variability metrics, to identify the full range of poten- tially invalid responses. Author Contact Alexander J. Denison https://orcid.org/0000-0002-4291-8325 Brenton M. Wiernik https://orcid.org/0000-0001-9560-6336 Correspondence concerning this article should be ad- dressed to Alexander J. Denison or Brenton M. Wiernik. Email: adeniso@clemson.edu or brenton@wiernik.org Conflict of Interest and Funding The authors have no conflicts of interest or funding to disclose. Author Contributions Conceptualization: Alexander J. Denison and Bren- ton M. Wiernik. Investigation: Alexander J. Denison. Project Administration: Alexander J. Denison. Supervision: Brenton M. Wiernik. Writing - Original Draft Preparation: Alexander J. Denison. Writing - Review Editing: Alexander J. Denison and Brenton M. Wiernik. Open Science Practices This article is a commentary and as such had no data or materials to share, and it was not pre-registered. The entire editorial process, including the open reviews, is published in the online supplement. References Baer, R. A., Ballenger, J., Berry, D. T., & Wetter, M. W. (1997). Detection of random respond- ing on the mmpi–a [[Online; accessed 2020-08- 15]]. Journal of Personality Assessment, 68(1), 139–151. https : / / doi . org / 10 . 1207 / s15327752jpa6801_11 Berry, D. T. R., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). Mmpi-2 ran- dom responding indices: Validation using a self- report methodology. [[Online; accessed 2020- 09-15]]. Psychological Assessment, 4(3), 340– 345. https : / / doi . org / 10 . 1037 / 1040 - 3590 . 4.3.340 Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research [[Online; accessed 2020-02-04]]. Edu- cational and Psychological Measurement, 70(4), 596–612. https://doi.org/https://doi.org/10. 1177/0013164410366686 Curran, P. G. (2016). Methods for the detection of care- lessly invalid responses in survey data. Jour- nal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006 Curran, P. G., & Hauser, K. A. (2019). I’m paid bi- weekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items [[Online; accessed 2019-09-24]]. Journal of Research in Personality, 82, 103849. https://doi.org/10.1016/j.jrp.2019.103849 Gibson, A. (2019). Stop what you’re doing, right now! ef- fects of interactive messages on careless respond- ing (Doctoral dissertation). Ohio, Wright State University. https://corescholar.libraries.wright. edu / cgi / viewcontent . cgi ? article = 3260 & context=etd_all https://orcid.org/0000-0002-4291-8325 https://orcid.org/0000-0001-9560-6336 https://doi.org/10.1207/s15327752jpa6801_11 https://doi.org/10.1207/s15327752jpa6801_11 https://doi.org/10.1037/1040-3590.4.3.340 https://doi.org/10.1037/1040-3590.4.3.340 https://doi.org/https://doi.org/10.1177/0013164410366686 https://doi.org/https://doi.org/10.1177/0013164410366686 https://doi.org/10.1016/j.jesp.2015.07.006 https://doi.org/10.1016/j.jrp.2019.103849 https://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=3260&context=etd_all https://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=3260&context=etd_all https://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=3260&context=etd_all 6 Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020). Careless responding in questionnaire measures: Detection, impact, and remedies [[Online; accessed 2020-03-05]]. The Leader- ship Quarterly, 101384. https://doi.org/https: //doi.org/10.1016/j.leaqua.2020.101384 Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and par- ticipant reactions [[Online; accessed 2021-05- 11]]. Journal of Business and Psychology, 30(2), 299–311. https://doi.org/10/gd86zr Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys [[On- line; accessed 2018-08-21]]. Journal of Business and Psychology, 27(1), 99–114. https : / / doi . org / https : / / doi . org / 10 . 1007 / s10869 - 011 - 9231-8 Jaso, B. A., Kraus, N. I., & Heller, A. S. (2021). Identification of careless responding in eco- logical momentary assessment research: From posthoc analyses to real-time data monitor- ing. [[Online; accessed 2021-10-17]]. Psycho- logical Methods. https : / / doi . org / 10 . 1037 / met0000312 Johnson, J. A. (2005). Ascertaining the validity of in- dividual protocols from web-based personality inventories [[Online; accessed 2018-08-21]]. Journal of Research in Personality, 39(1), 103– 129. https://doi.org/https://doi.org/10.1016/ j.jrp.2004.09.009 Kuncel, N. R., Goldberg, L. R., & Kiger, T. (2011). A plea for process in personality prevarication [tex.ids: Kuncel2011PleaProcess, Kuncelpleaprocessper- sonality2011a]. Human Performance, 24(4), 373–378. https://doi.org/10.1080/08959285. 2011.597476 Marjanovic, Z., Holden, R., Struthers, W., Cribbie, R., & Greenglass, E. (2015). The inter-item stan- dard deviation (isd): An index that discrimi- nates between conscientious and random re- sponders [[Online; accessed 2020-03-12]]. Per- sonality and Individual Differences, 84, 79–83. https://doi.org/10/gddnqh Maul, A. (2017). Rethinking traditional methods of survey validation [pub- lisher: Routledge e print : htt ps : //doi.org/10.1080/15366367.2017.1348108]. Measurement: Interdisciplinary Research and Perspectives, 15(2), 51–69. https://doi.org/10. 1080/15366367.2017.1348108 Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data [[Online; accessed 2018-08-21]]. Psychological Methods, 17(3), 437–455. https://doi.org/https://doi. org/10.1037/a0028085 Nichols, D., Greene, R., & Schmolck, P. (1989). Crite- ria for assessing inconsistent patterns of item endorsement on the mmpi: Rationale, devel- opment, and empirical trials. Journal of Clin- ical Psychology, 45(2), 12. https : / / doi . org / https : / / doi . org / 10 . 1002 / 1097 - 4679(198903 ) 45 : 2{\textless } 239 :: AID - JCLP2270450210{\textgreater}3.0.CO;2-1 Osborne, J. W., & Blanchard, M. R. (2011). Random responding from participants is a threat to the validity of social science research results [pub- lisher: Frontiers]. Frontiers in Psychology, 1. https://doi.org/10.3389/fpsyg.2010.00220 Rammstedt, B., & John, O. P. (2007). Measuring per- sonality in one minute or less: A 10-item short version of the big five inventory in english and german [[Online; accessed 2021-10-22]]. Jour- nal of Research in Personality, 41(1), 203–212. https://doi.org/10.1016/j.jrp.2006.02.001 Rhemtulla, M., Borsboom, D., & Bork, v. R. (2017). How to measure nothing [pub- lisher: Routledge e print : htt ps : //doi.org/10.1080/15366367.2017.1369785]. Measurement: Interdisciplinary Research and Perspectives, 15(2), 95–97. https://doi.org/10. 1080/15366367.2017.1369785 Schmitt, N., & Stuits, D. M. (1985). Factors defined by negatively keyed items: The result of careless respondents? [[Online; accessed 2020-03-12]]. Applied Psychological Measurement, 9(4), 367– 373. https://doi.org/https://doi.org/10.1177/ 014662168500900405 Stern, M. J., Dillman, D. A., & Smyth, J. D. (2007). Vi- sual design, order effects, and respondent char- acteristics in a self-administered survey [[On- line; accessed 2013-08-10]]. Survey Research Methods, 1(3), 121–138. https : / / doi . org / 10 . 18148/srm/2007.v1i3.600 Viswesvaran, C., & Ones, D. S. (1999). Meta-analyses of fakability estimates: Implications for personal- ity measurement [[Online; accessed 2020-04- 08]]. Educational and Psychological Measure- ment, 59(2), 197–210. https://doi.org/https: //doi.org/10.1177/00131649921969802 Ward, M., & Pond, S. B. (2015). Using virtual presence and survey instructions to minimize careless re- sponding on internet-based surveys [[Online; accessed 2020-02-03]]. Computers in Human https://doi.org/https://doi.org/10.1016/j.leaqua.2020.101384 https://doi.org/https://doi.org/10.1016/j.leaqua.2020.101384 https://doi.org/10/gd86zr https://doi.org/https://doi.org/10.1007/s10869-011-9231-8 https://doi.org/https://doi.org/10.1007/s10869-011-9231-8 https://doi.org/https://doi.org/10.1007/s10869-011-9231-8 https://doi.org/10.1037/met0000312 https://doi.org/10.1037/met0000312 https://doi.org/https://doi.org/10.1016/j.jrp.2004.09.009 https://doi.org/https://doi.org/10.1016/j.jrp.2004.09.009 https://doi.org/10.1080/08959285.2011.597476 https://doi.org/10.1080/08959285.2011.597476 https://doi.org/10/gddnqh https://doi.org/10.1080/15366367.2017.1348108 https://doi.org/10.1080/15366367.2017.1348108 https://doi.org/https://doi.org/10.1037/a0028085 https://doi.org/https://doi.org/10.1037/a0028085 https://doi.org/https://doi.org/10.1002/1097-4679(198903)45:2{\textless}239::AID-JCLP2270450210{\textgreater}3.0.CO;2-1 https://doi.org/https://doi.org/10.1002/1097-4679(198903)45:2{\textless}239::AID-JCLP2270450210{\textgreater}3.0.CO;2-1 https://doi.org/https://doi.org/10.1002/1097-4679(198903)45:2{\textless}239::AID-JCLP2270450210{\textgreater}3.0.CO;2-1 https://doi.org/https://doi.org/10.1002/1097-4679(198903)45:2{\textless}239::AID-JCLP2270450210{\textgreater}3.0.CO;2-1 https://doi.org/10.3389/fpsyg.2010.00220 https://doi.org/10.1016/j.jrp.2006.02.001 https://doi.org/10.1080/15366367.2017.1369785 https://doi.org/10.1080/15366367.2017.1369785 https://doi.org/https://doi.org/10.1177/014662168500900405 https://doi.org/https://doi.org/10.1177/014662168500900405 https://doi.org/10.18148/srm/2007.v1i3.600 https://doi.org/10.18148/srm/2007.v1i3.600 https://doi.org/https://doi.org/10.1177/00131649921969802 https://doi.org/https://doi.org/10.1177/00131649921969802 7 Behavior, 48, 554–568. https://doi.org/https: //doi.org/10.1016/j.chb.2015.01.070 https://doi.org/https://doi.org/10.1016/j.chb.2015.01.070 https://doi.org/https://doi.org/10.1016/j.chb.2015.01.070