2037-8264-5-CE Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013, pp. 134 – 155. Who says: “No Fair!”? What personality and an experiment in educational value tell us about perceptions of costs and benefits of research pool requirements Lisa DeMarni Cromer1, Shannon M. Reynolds, and Mitchell D. Johnson Abstract: Human subject pools (HSPs) are the basis for much psychological research. There is an explicit assumption that participants receive benefits from their participation, however there is little empirical research about the costs/benefits of participation. We conducted two studies with undergraduate psychology students to evaluate factors that can affect the cost/benefit ratio. Study 1 (N=46) examined Big Five personality characteristics and number of psychology courses taken, in relation to perceived benefits. There were depreciating returns for on-going participation but no personality differences in ratings. Study 2 (N=50) used a quasi-experimental design to manipulate educational value. Half of the participants completed an educational assignment that integrated their HSP research experience into course material. Students who completed the educational assignment had a strong sense of contributing to scientific knowledge whereas students who had no such assignment did not. Implications for increasing educational value in HSPs are discussed. Keywords: Human subject pool, educational value, integrative assignment, student perceptions of research I. Introduction. Human Subject Pools (HSPs) are a valued resource for psychological research. They are used by three quarters of universities (Miller, 1981; Sieber & Saks, 1989) and one third of four-year institutions (Landrum & Chastain, 1999). Approximately 70% of personality and social psychology studies and 90% of perception studies are conducted with college students in HSPs (Kulich, Seldon, Richardson, & Servies, 1978). Given the widespread use of HSPs, studies about the costs and benefits of HSP research participants’ experiences have widespread applicability to psychological research and ethics in the United States. The relatively small body of research about HSPs has focused on researchers’ concerns. HSP meta-research has sought to identify idiographic differences among participation characteristics. For example, meta-research has focused on individual differences that relate to predictors of early or late participation in the semester, finding that women tend to participate earlier in the semester than do men (Aviv et al., 2002; Cooper, Baumgardner, & Strathman, 1991; Roman et al., 1995; Witt, Donnellen, & Orlando, 2011). Research into personality aspects of participation has found that participants higher in conscientiousness, agreeableness (Witt, Donnellen, & Orlando, 2011) and introversion (Aviv et al., 2002) participate earlier in the semester. Some studies have examined pragmatic obstacles to research participation from participants’ perspectives. Other research has focused on factors related to volunteerism, suggesting that there are individual differences in who may be more likely to see the benefit of 1 Department of Psychology, The University of Psychology, 800 S. Tucker Drive, Tulsa, OK 74104, lisa-cromer@utulsa.edu Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 135 research participation. We summarize the literature relevant to United States research pools and then focus on the issue of costs and benefits for research participants. A. Individual Differences in Willingness to Volunteer for Research. Presumably, volunteer research participants experience some value or benefit from their volunteerism. When we examine research on volunteerism in HSPs, there are some individual differences between volunteers and nonvolunteers. This may illuminate who is more likely to perceive benefits from HSP participation. Martin and Marcuse (1958) prescreened 400 psychology undergraduates. These students were categorized into “volunteers”, i.e., they volunteered to do additional studies and “nonvolunteers,” i.e., they did not respond to subsequent research invitations. Volunteers were higher on intelligence (as measured by self-report ACT scores) than nonvolunteers, and female volunteers were more sociable than female nonvolunteers (Martin & Marcuse, 1958). Numerous other traits have been observed in individuals who elect to participate in research. Jews are more likely to participate in interviews than are Protestants and Catholics (Fischer & Winer, 1969). Willingness to volunteer in an HSP has been related to having more left-wing political views and being less conventional (Rosen, 1951; Rosenthal & Rosnow, 1975) and to being lower on authoritarianism (Rosnow & Rosenthal, 1976). Volunteers are more aggressive and higher in need for achievement (Rosenthal & Rosnow, 1975), and they are more agreeable and open to experience than are nonvolunteers (Dollinger & Leong, 1993). Amongst volunteers, extraverts are more likely to participate in face-to face research than internet-based research (Aviv et al., 2002). B. Beneficence in Research. The Belmont Report (U.S. Department of Health, Education, and Welfare: The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979) articulated an ethical obligation of beneficence in research. In other words, researchers should work to maximize the possible benefit to research participants. This standard is also found in the American Psychological Association’s ethics code (American Psychological Association, 2002). Benefits, however, may be in the eye of the beholder, and as such, perception of benefits is an important consideration for Institutional Review Boards (IRB) and others (e.g., professors) who are responsible for assuring participants’ welfare. All studies must weigh the participants’ costs and benefits; this balance is explicitly stated in IRB applications. In the present research, we consider costs and benefits as they relate to participation in HSPs, over and above the usual consideration of cost and benefits for any particular study. At the level of HSP participation, we consider cost to relate to factors such as possibly feeling coerced to participate, as well as negative feelings and experiences such as boredom, stress, perceptions of fairness (or unfairness) with regard to a HSP requirement or offer for extra credit. Meanwhile, benefit relates to learning from the experience, satisfying curiosity, intellectual stimulation and interest, altruism, and the opportunity to earn extra credit. Importantly, these are not objective measures of cost or benefit. Rather, these are individuals’ subjective perceptions of their experiences. We review this scant literature on costs and benefits below. Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 136 C. Perceptions of Cost and Coercion in HSP Participation. Some authors have suggested that coercion may be a concern about research conducted with HSPs (Miller & Kreiner, 2008; Miller, Kreiner, Ryan, & Smith, 2010). In a 2005 survey of IRBs, Wille (2005 as cited in Miller et al., 2010), found that a third of IRBs had explicit procedural concerns in order to prevent coercion. The most recent edition of the American Psychological Association ethics code, Standard 8.04b, addresses coerciveness in requirements for course credit, noting that it is essential that participants be given choices that include equitable alternatives to research participation (APA, 2002). In short, instructors can require students to be directly involved in research and can require their participation in person, as long as there is an equitable alternative such as reading and summarizing empirical papers (Smith, 2003). Implicit in this assumption is that reading about research and writing a summary is equitable in time and effort to the research requirement. Some investigators have attempted to document whether the assumption of equitability is consistent with students’ experiences. Foot and Stanford (2004) found that students viewed these alternatives as either boring or too time-intensive compared to the research sessions. Another study found that students thought research participation and summarizing articles were equally coercive (Trafimow, Madson, & Gwizdowski, 2006). Miller and Kreiner (2008) found that students reported feeling coerced to participate and viewed offers of extra credit from instructors to also be coercive, but then reported the overall participation was worthwhile. Additionally, the participants reported that they did not object to receiving extra credit or being paid for their research participation (Miller & Kreiner, 2008). The authors noted the apparent contradiction and speculated that participants may not have fully understood the word “coercion” used in the query. Previous research that examined participants’ personality primarily focused on whether personality predicted differences in participants’ behavior. Researchers found that personality type was related to when participants signed up for studies (early or late in the semester). Only two studies examined personality as it related to perceptions of one’s HSP experience. Carver (1980) found that Type A (high strung) personality types perceived higher levels of coercion than did Type B (easy going) personality types. The idea that perceived costs of research participation could vary by personality type was later pursued by Miller and colleagues. Miller et al., (2010) used a common Five Factor Model (FFM) of personality called the Revised NEO Personality Inventory (NEO PI-R). The NEO PI-R measures five broad domains of personality: Neuroticism, the tendency to experience negative emotions, Extraversion, the tendency to be social and assertive, Openness, creativeness and be willing to try new experiences, Agreeableness, the ability to get along with others and be friendly, and Conscientiousness, being thorough and careful (Costa & McCrae, 1985). Miller and colleagues (2010) used different vignettes that depicted five different recruiting strategies to research studies and had participants rate perceived level of coercion. The authors found that personality did not impact views of coerciveness. Nonetheless, Miller and colleagues did find a within subjects effect for type of recruitment depicted in the vignettes. The highest level of coercion was perceived in a vignette in which a professor simply asked students to participate in research. There was a lower level of perceived coercion when a professor was said to offer monetary incentives, and even less perceived coercion when a professor was said to ask students to stay after class to participate. The vignette rated least coercive was a professor offering extra credit to those who participated. From this research, it appears participants view coercion as being related to how direct and personal the requests for participation were. Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 137 Taxing students’ time is another potential cost to an HSP requirement. When queried for criticisms of HSPs, participants complained about studies being difficult to schedule (Elicker, McConnell, & Hall, 2010). Elicker and colleagues pointed out that the most common reason students do not participate in research was having limited time to participate followed by general scheduling conflicts. This research pointed to important considerations for reducing costs to students, including having a reasonable number of study selection options and a variety of study session days and times. D. Perceptions of Benefits to HSP Participation. Objective measures of educational benefits have shown that research participants learn from their research experiences. Students’ knowledge about research methods was greater when they participated in experiments than when they studied from a textbook (Darling et al., 2007; Thieman, Clary, Olson, Dauner, & Ring, 2009). Objective tests of knowledge-based questions also showed a better understanding of ethics after participating in research (Rosell, Beck, Luther, Goedert, Shore, & Anderson, 2005). Nonetheless, there is no work that queries parameters for learning. In particular, it is unknown whether participants gain more knowledge with more hours of research participation (Miller, 1981). Perception of learning is also an important parameter to measure. The educational value of participating in research is often used to justify an HSP requirement and is the purported benefit that balances the cost of participation. Furthermore, an educational component to research participation is implicit in federal regulations about human subjects’ research, and is monitored by IRBs. If participants perceive educational value, it would provide support for the view that research offers benefits to participants. Similar arguments have been made regarding perceptions of coercion. Leak (1981) and Scott-Jones (2000) noted that it is not whether researchers state there is coercion but rather participants’ perceptions of coercion that should determine whether or not coercion is present. Regarding perceptions of educational benefits of research, participants’ perceptions present a generally positive picture (Darling, Goedert, Ceynar, Shore, & Anderson, 2007). Consistent with objective studies of knowledge, subjective studies evidenced that students reported having a better understanding of research methodology and of research ethics after participating in experiments. Furthermore, students reported that research participation augmented their knowledge more than equivalent hours of classroom instruction (Darling et al., 2007; Elliott, Rice, Trafimow, Madson, & Hipshur, 2010). In contrast to these positive impressions about research participation, Brody, Gluck and Aragon (2010) found an absence of positive views when they interviewed 65 students about their experiences. Although Brody and colleagues did not probe for positives, the authors also noted that positive responses were not spontaneously offered; this suggests that students’ first impression of research may not be that it is educational for them. These studies also did not evaluate whether or not there is an optimal amount of research participation that can provide benefits. If indeed research participation is beneficial and positive for students, it would be helpful to know how much participation provides the most benefit (e.g., number of hours or studies) and whether or not there are diminishing returns after some amount of research participation. Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 138 E. The Current Study. In the present research we sought to examine the individual benefits that may be gained from HSP participation, and if the perception of benefits is correlated to the amount of participation in research. We wanted to compare responses of students who were new to the HSP compared to those who had participated over multiple semesters. This would allow us to detect whether there is a limited value in HSP participation or whether there are cumulative benefits over repeated participation. We also were interested in bringing together the individual difference literature (i.e., conservatism, and personality) with perceived level of costs (e.g., coerciveness) and benefits (e.g., educational value) of research. HSP studies (reviewed above) have shown that individual differences do relate to many HSP variables, such as when during the semester a student will participate (Witt, Donnellen, & Orlando, 2011), which personality types are more likely to participate in any kind of research (Dollinger & Leong, 1993; Martin & Marcuse, 1958; Rosen, 1951; Rosenthal & Rosnow, 1975), and which personalities are more likely to participate in face- to-face studies (Aviv et al., 2002). However only two studies (Carver, 1980; Miller, 2010) have examined personality differences as they relate to perceptions of research participation. Unfortunately, Carver (1980) used broad Type “A” and “B” personality traits, which are not well-defined personality constructs (Friedman & Booth-Kewley, 1988; Ganster, Schaubroeck, Sime, & Mayes, 1991). Although Miller found no differences for personality type, the study did not unpack these analyses into specific hypotheses. Because personality has been examined and found to be related to other HSP variables, there may be a possible “file drawer” problem. The “file drawer” problem was first described by Rosenthal (1979). Rosenthal asserted that the literature was biased because null results generally did not get published; hence information about the absence of group differences or the absence of associations between variables was not disseminated. This issue remains prevalent today, yet null findings do need to be communicated to the scientific community (Howard, Lau, Maxwell, Venter, Lundy, & Sweeny, 2009). Given that personality differences that may relate to HSP participation is a relatively new area of inquiry, and a previous “file drawer” problem may have existed, we sought to examine personality variables as they related to different perceptions of research experiences. In particular, we were interested in whether Conscientiousness and Openness to experience would relate to positive or negative views of research participation, and if agreeableness would relate to the perceived educational value and fairness of required participation. To our knowledge, this is the first study that examines differences in personality as it relates to perceived educational value and fairness. Personality, by definition, characterizes the way one thinks, behaves, and feels (Allport, 1961), so it is extremely relevant to perceptions of costs and benefits. We were interested in extending earlier research by Rosen (1951) and Rosenthal and Rosnow (1975) that found research volunteers held more liberal political views than non- volunteers. Both of these former studies were conducted in traditionally liberal states (authors were in California, Massachusetts, and Pennsylvania). Given that the current research is conducted in a conservative state, we wanted to investigate the generalizability of these earlier findings; in short, we wondered if liberalism would also equate to attitudes about research when in a predominantly conservative state. Hypotheses 1 and 2. As an extension of our discussion about Miller’s (1981) observation that little is known about whether participants believe that they gain more knowledge with more hours of research participation, we sought to examine this relationship. Our prediction was that Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 139 (1) perceptions of benefits would decline as research hour requirements increase. To test this, we correlated the number of psychology courses taken prior to the most recent participation, and the number of positive and negative comments that each student generated in response to open- ended questions about the value and experience of HSP participation. (2) We predicted that students who had more negative views of the research and HSP would have been required to complete more total research hours in the current semester. Hypotheses 3 and 4. We sought to examine whether personality variables would account for individual differences in perceptions of research experiences. Because only one published study (Miller, 2010) sought to examine HSP experiences and personality (and failed to find differences), we thought that it would be useful to determine whether in fact individual differences could explain positive or negative experiences with research. Specifically, we predicted that: (3) Individuals who were higher on personality variables of Openness to experience, Conscientiousness, and political conservatism would be more likely to be positive (versus negative) in their views about their HSP experiences. (4) We also predicted that individuals who were higher on Big Five Agreeableness would be more likely to rate the research requirement as being fair (versus unfair). II. Methods - Study 1. Data were collected during one semester. Students participated either for a research requirement or extra credit, as determined by their course instructor. Participation was optional; this study was offered amongst several other studies. Data collection was anonymous. A. Procedure. Participants who had completed at least two hours of research in the semester were eligible to participate. This was tracked by an online system (SonaTM). SonaTM is an online software product that is licensed by the Psychology department. All research (online and on site) is administered through this system. Two hours of research was chosen as a baseline for all participants to ensure that they had at least minimal exposure to participation before answering questions about how they viewed those experiences. They were contacted by mass email notification through the SonaTM system. They were given a research assistant’s email to contact if they wished to receive a password for the online study. Responses were completed online on the SonaTM system affording anonymity and automatic credit granting. The survey took fewer than 30 minutes to complete. Debriefing was provided online, and participants were encouraged to print this out, or contact the researcher for a paper copy. B. Participants. The study was conducted at a private Midwestern university. At this university, 128 students had registered in the HSP (as determined by number of accounts in the online administration system, SonaTM). Seventy-four participants responded to the invitation to the complete the survey and received a password. Of these, 62% subsequently took the online survey (and all completed once started). One participant was excluded from analyses due to selecting an option to not be included in any publications. The final sample (N = 46) consisted of 40 women and 6 men. The mean age was 21.43 (SD = 5.60). Table 1 details additional demographic information. Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 140 *GPA compares to a mean of 3.22 for the College of Arts and Science at this university, per Dean’s office report. C. Measures. Demographic questions included gender, GPA, academic classification, academic major, and employment status. Participants also were asked about the number of psychology courses they had taken, number of required research hours that semester, intention to complete extra credit research hours, and how many research hours they planned to complete as compared to number of hours reading and summarizing articles. Political Conservatism. Political affiliation was measured with a single subjective question: “Please indicate which option best describes your political affiliation” and responses were provided on an 8 point Likert scale where Liberal = 1, Moderate = 4, and Conservative = 8. Big Five Inventory. Personality was measured with the 44-item Big Five Inventory (BFI; John, Donahue, & Kentle, 1991) self-report questionnaire. The BFI has five scales consistent with the five-factor model: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness. Internal consistencies for the five scales range from .79 to .87, with a mean .83 (John, Donahue, & Kentle, 1991; John, Naumann, & Soto, 2008). The internal consistencies in the current data set ranged from .69 (Openness and Neuroticism) to .85 (Extraversion). The BFI also has a mean convergent validity correlation of .95 with the NEO PI-R scales (John, Naumann & Soto, 2008). The BFI means for a sample (N = 6076) of 21 year-olds are: Extraversion = 3.25 (SD =.90), Agreeableness = 3.64 (SD = .72), Conscientiousness = 3.45 (SD = .73), Neuroticism = 3.32 (SD = .82), and Openness = 3.92 (SD = .66; John & Srivastava, 1999). Five one-sample t- Table 1. Study 1 Demographic information for the HSP (N = 46). Minimum Maximum Mean SD Age 18 53 21.43 5.60 Political affiliation 1 (liberal) 7 (conservative) 3.98 1.57 Year in school 1 5 2.61 1.26 GPA* 2.3 4.0 3.41 0.47 Total psychology classes taken 1 36 6.43 6.29 Extra credit hours 0 10 1.87 2.71 Required research hours 2 8 5.87 1.98 BFI Extraversion 2 5 3.32 0.78 BFI Agreeableness 3 5 4.09 0.60 BFI Conscientiousness 3 5 3.79 0.51 BFI Neuroticism 2 4 2.91 0.63 BFI Openness 2 5 3.47 0.59 Female (%) 87 View HSP experience positively (%) 72 View HSP experience as fair (%) 89 View SonaTM positively (%) 100 Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 141 tests showed our sample means statistically differed from the normative sample because our sample was higher on Agreeableness (t(45) = 5.09, p = .0001, d = -.68), and Conscientiousness (t(45) = 4.52, p = .0001, d = -.54), but lower on Neuroticism (t(45) = 4.41, p = .0001, d = .56), and Openness (t(45) = 5.17, p = .0001, d = .72). See Table 1. HSP Experiences Questionnaire (HSP-EQ). The HSP Experiences Questionnaire was developed by the first author in order to obtain feedback regarding students’ experiences in the HSP. The original intention for this measure was to provide evaluative information in order to learn about students’ experiences as participants and to potentially guide changes to the HSP. All questions were open-ended to allow students to express as much as they wanted about their HSP experiences. Coding categories were developed by evaluating participants’ responses. Coding of HSP-EQ. Question 1, “Why did you choose to do the experiments instead of the alternative assignment? (OR why did you do the alternative assignment instead of the experiments, if that applies to you)?” Open-ended responses were coded into five categories: (1) it was easier, (e.g., “For me, spending hours participating in a study was easier than doing the alternative assignment” (2) contribute to research, (e.g., “I think it is important to support the research that graduate students are doing.”) (3) for educational or learning purposes (e.g., “Better experience—allows you to get a different perspective,” “It's also a nice deviation from "normal," written schoolwork,” “I wanted first hand experience at what participating in psychology research was about.”), (4) it was more interesting, fun, or they were curious (e.g., “I thought participating would be more interesting,” “It’s fun to get the experience,” “I chose to participate in experience due to the curiosity.”), or (5) felt coerced (e.g., “I thought that the alternate assignment would result in a lower grade.”) A distinction we made between categories 3 and 4, was that category 3 was coded if the implied goal was to learn something and to contribute to an academic knowledge base, as compared to category 4 that suggested more of a novel component with a shorter term goal. Each participant’s response could have multiple components, each of which would be coded although no single component was ever double coded. Participants’ opinions of the research requirement were assessed in question 2, “What do you think of the experiment requirement?” Negative responses included statements suggesting (1) fewer required hours or (2) unfair, “It’s difficult for athletes to find spare time to do it.” Positive views were coded as (1) motivating the participant to become involved in research (e.g., “It’s helpful, otherwise I wouldn’t be motivated to get involved at all”), (2) educational (e.g., “It's nice to learn the research side of psychology, and to get to see the tests that we talk about in class.”), or (3) generally positive, such as “it was okay” and “I liked it.” Coders only provided one code for each participant because nearly all of the participants only provided a single response. For the few cases in which more than one response was given, we coded the first response on this question. Thus, for this question, response category frequencies are independent and dichotomous. These scores were included in the overall summed score of responses, and question 2 was also used as an stand alone dependent variable in the logistic regression used to test hypothesis 2. Question 3, “What do you think is valuable about participating in research studies?” was coded as (1) no value, (2) educational (e.g., “Learning how the process works, e.g. informed consent, debriefing, etc.” or (3) contribute to research (e.g., “It provides data for professor's research.”). Frequencies for these items do not sum to 100% because of the double coding and therefore are not independent, thus only descriptive statistics (frequencies) are provided. No Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 142 single portion of a statement was ever double coded, however a compound sentence that touched on two categories, would have each part of the sentence coded separately. Question 4 queried dislikes of the research requirement: “Is there anything about the research/experiment requirement that you do not [sic] like?” Responses included: (1) too few experiment choices, (2) too many hour requirements, (3) experiments took too long or were difficult to schedule, or (4) general complaints (e.g., “Some of the studies are lengthy and monotonous”).The modal response was (5) no dislikes; when this question was left blank it was also coded as “no dislikes.” Question 5a, “Is the experiment requirement fair?” was coded yes or no. Question 5b “Why or why not?” was coded (1) educational, (2) contributes to research, or (3) that it was a reasonable requirement, (e.g., “Because it's just like a participation credit in any other class, and students are given ample time to complete it”). Responses explaining why the experiment requirement was unfair included statements that (1) it was educational for others but not me, (2) the participation should be extra credit, (3) unfair study exclusion factors or too few options, and (4) exploitative or too demanding (e.g., “Other departments don’t require it and as a undergrad student we don’t get any results therefore for us it seems pointless”). Question 6 assessed the participant’s opinion of his/her debriefing experience. Responses towards the debriefing were coded as (1) positive/helpful/necessary, (2) negative/not helpful/unnecessary, (3) no debriefing experience, or (4) a neutral statement (e.g., “It was brief and to the point.”) Question 7 assessed the participants’ experience with the electronic sign-up system. This was coded as either positive (e.g., “I liked it a lot more than signing up on paper, it is fast, easy, and instant”) or negative. However, no negative responses were given. The faculty investigator, graduate student, and undergraduate student were all trained on the coding method. The graduate and undergraduate student independently coded all the participants’ responses. Coding meetings were held weekly to discuss and resolve discrepancies. The primary investigator served as judge for discrepancies, so the final code reflected her agreement with one of the two coders. The two independent raters agreed 96% of the time. Across all of the open ended questions, we summed the number of codeable positive responses in order to obtain a measure of how positive respondents were. The range was 2 to 9 for each participant. We also summed the number of codeable negative responses in order to obtain a measure of how negative respondents were. The range of values was 0 to 5 for each participant. All respondents had a summed positive and a summed negative score. These scores were negatively correlated (Pearson’s r = -.684, p = .0001), indicating a strong relationship with the variables, yet not so correlated as to be redundant. The more positive respondents were about their HSP experience, the fewer negative things they had to say. III. Results from Study 1. A. Descriptive Statistics. Seventy-two percent of participants thought the research experience was educational or positive. Most respondents (89%) thought the research requirement was fair. Although 41% reported liking everything about the research requirements, the most common complaint was that the sessions were hard to fit into their schedules (26%), followed by 11% of the sample not liking an aspect of one of the experiments or feeling coerced into participating. While 52.2% of the sample Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 143 thought the debriefing was helpful, 17.4% said it was unhelpful, and 30.4% did not report on debriefing or said they had never experienced one. To further examine how previous HSP experience and personality may influence these views of the HSP experience, we tested hypotheses 1-4 below. Because the sample was 87% female, we did not include gender in any of the analyses for lack of power which could have been misleading in the results. Hypothesis 1. We predicted that perceptions of benefits would decline as research hour requirements increase. We examined the number of psychology classes participants had taken at our institution and proportion of individuals who rated the experiment as educational and/or positive (see Figure 1). Negative comments per number of psychology classes are also shown in Figure 1. Figure 1. Study 1: Percentage of positive and negative responses across number of psychology classes taken by students. All the students who had taken only one psychology class provided a positive response to question 2, “What do you think of the experiment requirement?” In comparison, there were fewer positive responses from participants who had taken more than one class that required research participation. There were also more negative views about experiences for individuals who had taken two or more classes. As stated above, we computed a summed score of the number of positive responses for all of the questions and a summed score of the total number of negative responses for each respondent. These summed positive and negative values were correlated with the total number of reported psychology classes taken. Number of psychology classes taken was used because at the institution in which this study was conducted, all psychology classes require students to complete between 1-4 hours of research experience as part of the course requirement (either reading and summarizing an article or participating in a research study). Most students choose to participant in research over reading and summarizing an article. The more psychology classes a student takes, the more research experience s/he has. Number of psychology classes was a better estimate than year in school because some senior students were not psychology majors, had only taken one psychology class, and only participated in one hour of research. Both correlations were significant (one-tailed because we hypothesized more classes would result in more negativity and less positivity). More classes related to fewer 0   20   40   60   80   100   1  class     2-­‐4  classes     5-­‐  7  classes     8  or  more  classes     Study  1:  Percentages  of  pos4ve  and   nega4ve  response  across  psychology   classes  taken   Posi5ve  response  (n  =  33)   Nega5ve  response  (n  =  12)   Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 144 positive responses, Pearson’s r = -.248, p = .048, and more classes related to more negative responses about the HSP, Pearson’s r = .411, p = .005. Hypothesis 2: To determine whether negative views of research were related to the number of required research hours across all psychology courses in which participants were currently enrolled, we performed a logistic regression, regressing participants’ views of the research experience as positive = 1 or not = 0 (based on each participant’s dichotomously coded answer to question 2), on a categorical predictor of research hours required. We coded research hours required into a categorical variable because examination of a histogram revealed that number of hours required was essentially a bimodal distribution. Five or fewer hours were dummy coded 0, and 6 or more were dummy coded 1. The full model was statistically significant, χ2 (1, N = 46) = 4.70, p = .03, indicating that the model was able to distinguish between those who thought the research experience was positive and those who did not. The model as a whole explained between 9.7% (Cox and Snell R square) and 14% (Nagelkerke R squared) of the variance in views of the research experience, and correctly classified 71.7% of the cases. The odds ratio of .193 indicated that respondents were about 20% more likely to report the research pool experience as negative if they were required to complete 6 or more hours of studies in a semester. Hypothesis 3. To examine if individuals high on Openness to Experience, Conscientiousness, and political conservatism were more likely to rate their research experience as positive (versus negative), we conducted a direct logistic regression. The model contained three independent continuous variables (Openness, Conscientiousness, and political conservatism). The dependent variable was a dichotomous 1 = positive, 0 = negative experience score (based on question 2). The full model was not statistically significant, χ2 (3, N = 46) = 5.66, p = .129, indicating that the model did not distinguish between individuals who found the research experience to be positive or not. See Table 2 for means and standard deviations. These results may reflect the fact that the majority of the sample (72%) considered the research requirement a positive experience. Hypothesis 4. To examine whether individuals who were higher on Big Five Agreeableness were more likely to view the research requirement was fair, responses to question 5a were coded into 1 = fair, 0 = unfair. A logistic regression with Agreeableness as the predictor was not statistically significant, χ2 (1, N = 46) = 1.024, p = .312. Failure to find a significant difference may be due to the majority (89%) of the sample viewing the requirement as fair. Table 2. Study 1: Means and standard deviations for positive and negative groups. Group Openness scale score Conscientiousness scale score Political affiliation Mean (SD) Mean (SD) Mean (SD) Positive group (n = 33) 3.42 (0.63) 3.72 (0.45) 3.79 (1.60) Negative group (n = 12) 3.58 (0.45) 4.02 (0.60) 4.42 (1.51) Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 145 IV. Discussion for Study 1. The results indicate that participants of this HSP generally viewed the research experience as positive and fair. Nonetheless, we observed depreciating returns to students as they participated in the HSP over multiple classes. It is important to note that 100% of participants who were partaking in the HSP for the first time had a positive experience in the HSP. The proportion of positive views of the HSP diminished for the other groups, with 69% in 2-4 classes, 75% in 5-7 classes, and 63% in 8 or more classes having positive ratings. There were increased negative views across more HSP participation, with 31% in 2-4 classes, 25% in 5-7 classes and 38% in 8 or more classes reporting negative views of HSP experiences. Both trends (increasing negativity and decreasing positivity about HSP experiences) were significant when correlated to number of psychology classes taken. This provides evidence that there are educational benefits to students who participate for the first time. This statistically significant finding speaks to the strength of the effect because our sample was small, yet differences still emerged. Regarding the question about whether students continue to receive benefits from higher levels of HSP participation in a given semester, some but not all students reported benefits and these benefits were reduced with increase exposure to research studies. When we dichotomously coded participants into requirements for 6 or more hours compared to 5 or fewer hours, we found that those with higher demands were less likely to view their experience as positive (hypothesis 2). From a cost/benefit perspective, this suggests that upper division classes that require HSP participation may need to augment the HSP experience in order to maintain a high level of educational benefit for students. Alternatively, departments may consider setting a ceiling on the number of research hours that are required of students in a given semester. Some of the variability in responses may have to do with the different types of research in which students were able to participate. At the university where this research was conducted, there are several different types of research, and so it is possible that some students’ benefits were augmented if they participated in different kinds of studies over time (e.g., personality, psycho-physiological, trauma), exposing participants to different methodologies and different content areas of psychology. Despite the increase in negative views, it is important to note that overall participants continued to report positive views of their participation (that outweighed the negative views). As an anonymous reviewer pointed out, it may be that participants’ increasing experience in human subjects research and their additional courses in psychology may have taught them critical thinking skills that enabled them to see these experiences in a more balanced light. Personality did not relate to attitudes about HSP participation. This finding was consistent with Miller (2010) who also noted that personality did not relate to perceptions of research experience. This suggests that student reports about research experiences being positive, fair, or valuable, is not driven by a personality difference of being more agreeable or conscientious; rather, it suggests that there is a general positive experience for most students who participate in research. In contrast to previous work (Rosen, 1951; Rosenthal & Rosnow, 1976), we did not find that political affiliation related to perceptions of HSP experiences. This may be related to the current sample being in the moderate range on a self-report measure of conservatism. Rosen (1951) used a fascism scale, and in Rosenthal and Rosnow (1976) authoritarianism was the construct discussed. The measure in the current study may not have elicited as strong responses as a fascism or authoritarianism scale, which may have limited sample variance. The absence of significant differences is often regarded as a non-finding in research and can result in a “file drawer” problem. We include it here because it is noteworthy Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 146 that personality and individual differences may not have a large impact on attitudes toward HSPs, suggesting that variance in attitudes has more to do with different experiences (under the control of HSP administrators and researchers) rather than individual differences. In effect, this can put the onus on the HSP administrators and experimenters to ensure that all participants receive benefits. For example, a department’s HSP may incorporate an educational assignment that encourages integration of the research experience into course material. The value of an integrative assignment that relates the HSP participant experience to course material is the focus of Study 2 below. V. Study 2. Broadly speaking, when surveyed, HSP participants appear to feel satisfied with their experiences and they find participation to be educational. Nonetheless, participants who are new to HSP experiences rate their experiences more positively than do participants who have repeatedly participated. Possibly, the decrease in favorable perceptions could be explained by educational value, in that those with previous experience have less to learn from additional participation. It follows then, that if we could increase educational value of HSP participation, positive ratings would also increase. In the present study, we used a quasi-experimental design in order to test the relationship of attitudes toward HSP participation and educational value, by enhancing educational value of research participation for one group. A. Intentionally Educating about Research. The question of whether HSP participation is of implicit educational value, or whether educational value can be explicitly enhanced, is empirical. One suggestion for increasing the educational value of HSP participation was to incorporate an integrative assignment into the research experience (Richardson, Pegalis, & Britton, 1992). Richardson and colleagues (1992) administered an integrative assignment after each research session. The assignment was five brief questions about the research that asked, “(a) the area of psychology relevant to the study, (b) data collection techniques used (e.g. survey, interview, observation), (c) potential practical applications of findings, and (e) relation of the study to course material” (p. 12). They then gave a questionnaire assessing all students’ perceptions of the overall education benefit of the research sessions, the benefit of the debriefing experience, how related the research was to the course, and an overall evaluation of the research experience. In this naturalistic research, they found that students who completed the integrative assignment rated the research sessions as more educational and relevant to course content. Specifically, students who completed the integrative assignments thought the research sessions allowed them to better understand psychology experiments, generated more interest in the debriefing explanation, and had larger educational emphasis. This study was naturalistic and there was no experimental control for who took the integrative assignment. In the present study, we sought to replicate the Richardson et al. (1992) findings using a quasi-experimental design. Hypothesis 1. We predicted a significant effect for the integrative assignments, in that the group that completed the integrative assignments (Group 1) would have a greater proportion of individuals who rated their experiences as positive/ educational and fair than Group 2, which did not complete the integration assignments. Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 147 VI. Methods – Study 2. Data were collected at the end of two semesters as part of an assessment of students’ research pool experience. Because this was part of a regular educational assessment, IRB approval was not sought prior to data collection (American Psychological Association, 2002). The University IRB gave the authors permission to use these existing data after they were collected. A. Procedure. Data were collected in paper and pencil form, in class, for both groups. The professor gave the students the HSP-EQ described in Study 1, and asked them to compete it in order for her to understand their research experiences and to possibly help improve those experiences in the future. The professor then left the room while the students responded to the questions. This took about 10 minutes, and students did not receive any compensation or incentive for completing the questions. Participation was voluntary as well as anonymous. B. Participants. Participants were enrolled in an introductory psychology class at the same private Midwestern university in Study 1. Each student either participated in four hours of psychology research as part of their course requirement, or read and summarized four journal articles (of their choice). Demographic information was not collected because it was not pertinent to the standard course evaluation. Most students were White. It is a requirement that they be at least 18 years of age in order to participate in research studies (age of consent). Students who were not of the age to consent read and summarized the four journal articles for their course requirement. For Group 1 the class composition was: 35.4% Freshmen, 38.2% Sophomores, 17.6 % Juniors, and 8.8% Seniors. For Group 2 the class composition was: 36.7% Freshmen, 30% Sophomores, 20 % Juniors, and 13.3% Seniors. C. Research Design. Participants signed up for studies on the SonaTM electronic system, which could be accessed on and off campus at any time of the participants’ choosing. This is a quasi-experimental research design with the assignment to Group 1 or 2 being naturalistic because we collected evaluation data as part of regular course evaluations. The integration activity was initially incorporated into the course to enhance learning, based on previous research (Richardson, Pegalis, & Britton, 1992). The choice to not give the integration activity to the second group was an intentional manipulation for this study. In addition to their research experience, students in Group 1 completed two assignments that were designed to help them integrate their research participation into their psychology course. The assignment was to answer four questions about the research participation. These questions were adapted from Richardson et al. (1992) and were designed to help students see a didactic value to research participation. Specifically, students needed to identify the name of a study they had completed, a book chapter and section that would describe the study (e.g. personality), and to think critically about whether the validity of the study was compromised due to the college student sample. Responses were graded on a 10-point scale by a graduate teaching Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 148 assistant. The points for these two assignments accounted for 5% of students’ grades. Students who opted to do alternative assignments of reading empirical studies completed these assignments based on research articles they read and summarized. For the purposes of the current investigation, the integration assignment was eliminated from the course for Group 2 participants. So, although Group 2 completed the same number of study hours as Group 1 (or read and summarized four articles), they did not have an integrative assignment, in which they would have reflected on their research experience and incorporated it into the course material. D. Measures. HSP-EQ. The HSP-EQ that was used in Study 1 was also used in Study 2. We utilized the coding scheme described in Study 1 and the same two coders. Coding agreement was 92%. VII. Results from Study 2. There were a total of 50 participants in the two groups. All but one participant chose to participate in research rather than complete the alternative assignment. The most frequently reported reason for choosing the research option was curiosity (56.6%) about what the research session would be like, followed by the research session was easier (41.5%) than the alternative assignment. The majority (77.4 %) thought the HSP experience was educational or positive, and 71.7% of the sample reported viewing the research requirement as fair. Reasons why students viewed the research requirement as fair included it was a reasonable class requirement (37.7%), it was educational (15.1%), or the participation contributed to research (18.9 %). For those who said the research requirement was not fair (22.6%) the reasons included it was too demanding or exploitative (18.9%), the studies were too few in options or had unfair exclusion factors (1.9%), and finally 3.8% thought the research should be extra credit rather than a class requirement. The most frequent response to the question asking what the participant did not like about the research requirement was no complaints (37.7%), followed by having scheduling difficulties (26.4%), and then 20.8% of the sample not liking an aspect of the experiment or viewing the research requirement as exploitative. About half (49.1%) of the participants thought the debriefing was helpful, 9.4% thought it was unhelpful, 13.2% did not receive a debriefing, and another 5.7% gave a neutral comment about the debriefing (e.g., “It was short”). All participants reported positive views about SonaTM. Hypothesis. We predicted that Group 1, which completed the integrative assignments, would be more positive in their answers about whether or not the research experiences were educational or positive compared to Group 2, which did not complete the integration assignment. Figure 2 shows the proportions of students who viewed the experience as positive. Overall, students thought that the requirement was educational, regardless of condition. In addition, for the entire sample, 71.7% thought the research requirement was “fair” with proportions as follows: 72% of Group 1 and 71.4% of Group 2. Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 149 Figure 2. Study 2: Proportion of students who viewed HSP as educational/positive and negative. Figure 3 shows the reasons why students thought the research requirement was fair or unfair. The effect of not doing the integrative assignment (Group 2) appears to be that a greater proportion of students thought the research requirement was exploitative and the belief that they contributed to science was eliminated (i.e., a loss in this as an altruistic benefit of research participation). While Group 2 (no assignment) did contain the most participants who endorsed negative views (26%) of the research requirement as compared to Group 1 (17% participants reported negative views), this result was not statistically significant χ2(1, N = 50) = 0.527, p = .47. We were unable to conduct chi square analyses on between group differences for reasons why the research requirement was perceived to be fair or not fair, because assumptions of the analyses were not met (minimum n=5 in each cell). VIII. Discussion for Study 2. The second study was designed to evaluate whether educational value and overall ratings of the HSP experience could be increased by changing aspects of how HSP requirements are incorporated into the departmental curriculum. By experimentally manipulating the educational component of the research requirement, we were able to test its relationship to students’ perceptions of educational value and fairness. Based on these qualitative data, there was a consistent trend that overall participants viewed their HSP experiences as positive. Nonetheless, we saw a change in responses to open- ended questions when there is no integrative assignment. That is, more students appeared to feel exploited and none of them thought they were contributing to science. The integrative assignment asked students to determine, “If this study was published in your introductory psychology textbook, what chapter would be the best choice?” It appears that the effect of this assignment, possibly attributed to this question, was that students not only integrated their experience into what they were learning, but they gained an additional benefit of feeling like they were contributing to science. This is an important aspect of the research experience, when 0.00   10.00   20.00   30.00   40.00   50.00   60.00   70.00   80.00   90.00   100.00   Posi5ve  response   Nega5ve  response   Study  2:  Percentages  of  posi4ve  and  nega4ve   responses  for  opinions  on  HSP  experience   Group  1:  Assignment   Group  2:  No  assignment   Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 150 weighing the costs and benefits of research – that students may gain a sense of pride, altruism, or importance for their contribution. What specifically was driving this finding will be a question for future study. Figure 3. Study 2: Reasons why student thought research requirement was fair. VIII. Summary. Researchers and educators are ethically bound to ensure research requirements provide an educational experience for undergraduate student participants (APA, 2002). Our findings support the notion that this is occurring, and these findings are consistent with the extant literature. Students generally hold positive perceptions of their research participation and consider it an educational experience (Darling et al., 2007; Elliott et al., 2010). Study 1 contributes to the current literature by assessing the relationship between specific individual differences and demographic characteristics and students’ perceptions of fairness and educational value. We found that as students take more psychology classes, and get more experience participating in research, they retain a stable level of positive views of the HSP. However, they also gain more negative views of the HSP, which on the whole could mean that their overall views are more balanced. On the face of it, it may be that the greatest educational benefit is for students in introductory psychology courses who are getting exposed to research for the first time. On the other hand, there could be continued benefits while more balanced (i.e., also seeing the draw backs of participation) as students develop more nuanced thinking and critical thinking skills. Study 2 extends the literature by assessing methods designed to increase students’ positive perceptions of HSP experience. Since participants report more negative views of HSP as they participate in more research session, we attempted to enhance the educational experience of research participation for students by giving an integrative assignment that allows them to incorporate their research experience with textbook knowledge. We found that overall positive 0.00   10.00   20.00   30.00   40.00   50.00   60.00   Study  2:  Reasons  why  par4cipants  thought   research  requirement  was  fair  or  not  (%)   Group  1:  Assignment   Group  2:  No  assignment   Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 151 and educational views were not substantially different with the addition of the integrative assignment. Thus, general ‘happiness’ with participation may not change when an assignment is added. Nonetheless, it is noteworthy that the group which did an integrative assignment believed that they were contributing to science whereas the other group did not. Also, when participants completed the integrative assignment, they less frequently expressed views of having been exploited in the research. IX. Limitations and Future Directions. There are several strengths to the current set of studies. We were able to extend the literature by examining questions related to HSP experiences that have not yet been conducted. It is a strength that we were able to use a quasi-experimental design to examine the impact of an educational assignment. There are several limitations to the current work. Because this research was conducted at a small, private college, we had limited power; interesting trends that were not statistically significant in this Study may have been significant with a larger sample. Furthermore, the sample in study 1 was predominantly female, and subsequent research could benefit for examining gender differences in research experiences. Also, because this is a private college in a conservative state with incoming freshman ACT scores averaging 28 points and higher, we potentially have a limited sample in terms of variability of responses. Additionally, our sample statistically differed from the normative sample on four of the five personality variables. Our sample was more agreeable and conscientious and less neurotic and open to experience, which could contribute to the positive attitudes and perceptions of the HSP. It is possible that larger schools with a more diverse student body would have more variance in student responses. Because evaluating students’ experiences of HSP is a relatively new area of research, we used free-response questions. The limitation here is that students may well have had additional views –both positive and negative – that would be expressed if prompted. We were limited in coding responses to non-leading questions that were designed to provide students an unrestricted opportunity to express their opinions of the research requirement. The results of our study illustrate the need for a measure that can accurately capture the fine nuances and changes in students’ opinions. Given that the use of HSPs is widespread in the United States and given the ethical obligation psychologists have to positive benefit/cost ratios for participants, studies of this nature are important. Moyer and Franklin (2011) noted that psychology departments typically do not evaluate the educational benefit to participants. The potential for educational benefits and good attitudes from participants is an area that could have pervasive benefits for psychological research in general. It is possible (although as yet, not empirically documented) that increasing the positive attitudes of participants could decrease the costs of administering HSPs. There could be fewer email complaints, few no-shows, and generally easier administration if participants’ experiences are positive. Furthermore, we hope that studies in the future will attempt to relate the feeling of contributing to science to the overall quality of data that is collected by researchers using the HSP. For example, we would predict that there would be fewer invalidated subjects’ data when participants feel responsibility of contributing to science and this perception could increase motivation to follow instructions and to do one’s best to meet study demands. Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 152 Acknowledgements Thank you to the Dynamics Lab at the University of Oregon for feedback on a presentation of this data which resulted in helpful suggestions for data coding and analysis. We are grateful to Dr. Amy Criss at Syracuse University for helpful comments on an earlier version of this manuscript. References Allport, G. W. (1961). Patterns and growth in personality. New York, NY: Hold, Rinehart & Winston. American Psychological Association. (2002). American Psychological Association ethical principles of psychologists and code of conduct. Retrieved from http://www.apa.org/ethics/code2002.html Aviv, A. L., Zelenski, J. M., Rallo, L., & Larsen, R. J. (2002). Whocomeswhen: Personality differences in early and later participation in a university subject pool. Personality & Individual Differences, 33, 487-496. Brody, J. L., Gluck, J. P., & Aragon, A. S. (2000). Participants' understanding of the process of psychological research: Debriefing. Ethics & Behavior, 10(1), 13-25. doi: 10.1207/s15327019eb1001_2. Cooper, H., Baumgardner, A. H., & Strathman, A. (1991). Do students with different characteristics take part in psychology experiments at different times of the semester? Journal of Personality, 59, 109–127. Costa, P.T., & McCrae, R. R. (1985). The NEO personality inventory manual. Odessa, FL: Psychological Assessment Resources. Darling, J., Goedert, K., Ceynar, M., Shore, W., & Anderson, D. (2007). Learning about the means to the end: What US introductory psychology students report about experimental participation. Psychology Learning & Teaching, 6(2), 91-97. doi: 10.2304/plat.2007.6.2.91 Dollinger, S. J., & Leong, F. T. L. (1993). Volunteer bias and the five-factor model. Journal of Psychology, 127, 29-36. Elicker, J. D., McConnell, N. L., & Hall, R. J. (2010). Research participation for course credit in introduction to psychology: Why don't people participate? Teaching of Psychology, 37(3), 183- 185. doi: 10.1080/00986283.2010.488521 Elliott, L. J., Rice, S., Trafimow, D., Madson, L., & Hipshur, M. F. (2010). Research participation versus classroom lecture: A comparison of student learning. Teaching of Psychology, 37(2), 129-131. doi: 10.1080/00986281003626862 Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 153 Fischer, E. H., & Winer, D. (1969). Participation in psychological research: Relation to birth order and demographic factors. Journal of Consulting and Clinical Psychology, 33(5), 610-613. doi: 10.1037/h0028112 Foot, H., & Sanford, A. (2004). The use and abuse of student participants. The Psychologist, 17(5), 256-259. Friedman, H. S., & Booth-Kewley, S. (1988). Validity of the Type A construct: A reprise. Psychological Bulletin, 104(3), 381-384. doi: 10.1037/0033-2909.104.3.381 Ganster, D. C., Schaubroeck, J., Sime, W. E., & Mayes, B. T. (1991). The nomological validity of the Type A personality among employed adults. Journal of Applied Psychology, 76(1), 143- 168. doi: 10.1037/0021-9010.76.1.143 Howard, G. S., Lau, M. Y., Maxwell, S. E., Venter, A., Lundy, R., & Sweeny, R. M. (2009). Do research literatures give correct answer? Review of General Psychology, 13, 116-121. John, O. P., Naumann, L. P., & Soto, C. J. (2008). Paradigm Shift to the Integrative Big-Five Trait Taxonomy: History, Measurement, and Conceptual Issues. In O. P. John, R. W. Robins, & L. A. Pervin (Eds.), Handbook of personality: Theory and research (pp. 114-158). New York, NY: Guilford Press. John, O. P., & Srivastava, S. (1999). The Big Five Trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin, O. P. John, L. A. Pervin, O. P. John (Eds.), Handbook of personality: Theory and research (2nd ed.) (pp. 102-138). New York, NY US: Guilford Press. Kulich, R. J., Seldon, J. W., Richardson, K., & Servies, S. (1978). Frequency of employing undergraduate samples in psychological researchand subject reaction to forced participation. Paper presented at the meeting of the Midwest Psychological Association, Chicago. Landrum, R. E., & Chastain, G. D. (1999). Subject pool policies in undergraduate-only departments: Results from a nationwide survey. In G. D. Chastain & R. E. Landrum (Eds.), Protecting human subjects: Departmental subject pools and institutional review boards. (pp. 25- 42). Washington, DC US: American Psychological Association. Leak, G. K. (1981). Student perception of coercion and value from participation in psychological research. Teaching of Psychology, 8(3), 147-149. doi: 10.1207/s15328023top0803_4 Martin, R. M., & Marcuse, F. L. (1958). Characteristics of volunteers and nonvolunteers in psychological experimentation. Journal of Consulting Psychology, 22(6), 475-479. doi: 10.1037/h0041496 Miller, A. (1981). A survey of introductory psychology subject pool practices among leading universities. Teaching of Psychology, 8(4), 211-213. doi: 10.1207/s15328023top0804_4 Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 154 Miller, W. E., & Kreiner, D. S. (2008). Student perception of coercion to participate in psychological research. North American Journal of Psychology, 10(1), 53-64. Miller, W. E., Kreiner, D. S., Ryan, J. J., & Smith, J. W. (2010). Personality types and perceived coercion in psychological research participation. Individual Differences Research, 8(3), 140-150. Richardson, D. R., Pegalis, L., & Britton, B. (1992). A technique for enhancing the value of research participation. Contemporary Social Psychology, 16 (1/2), 11-13. Roman, R. J., Moskowitz, G. B., Stein, M. I., & Eisenberg, R. F. (1995). Individual differences in experiment participation: Structure, autonomy, and the time of the semester. Journal of Personality, 63, 113–138. Rosell, M. C., Beck, D. M., Luther, K. E., Goedert, K. M., Shore, W. J., & Anderson, D. D. (2005). The pedagogical value of experimental participation paired with course content. Teaching of Psychology, 32(2), 95-99. doi: 10.1207/s15328023top3202_3 Rosen, E. (1951). Differences between volunteers and non-volunteers for psychological studies. Journal of Applied Psychology, 35(3), 185-193. doi: 10.1037/h0061411 Rosenthal, R., & Rosnow, R. (1975). The volunteer subject. New York: Wiley. Rosnow, R. L., & Rosenthal, R. (1976). The volunteer subject revisited. Australian Journal of Psychology, 28(2), 97-108. doi: 10.1080/00049537608255268 Scott-Jones, D. (2000). Recruitment of research participants. In B. D. Sales & S. Folkman (Eds.), Ethics in research with human participants. (pp. 27-34). Washington, DC US: American Psychological Association. Sieber, J. E., & Saks, M. J. (1989). A census of subject pool characteristics and policies. American Psychologist, 44(7), 1053-1061. doi: 10.1037/0003-066x.44.7.1053 Smith, D. ( 2003). Five principles for research ethics. Monitor on Psychology, 34. Retrieved from http://apa.org/monitor/jan03/principles.aspx Thieman, T. J., Clary, E. G., Olson, A. M., Dauner, R. C., & Ring, E. E. (2009). Introducing students to psychological research: General psychology as a laboratory course. Teaching of Psychology, 36(3), 160-168. doi: 10.1080/00986280902959994 Trafimow, D., Madson, L., & Gwizdowski, l. (2006). Introductory Psychology Students' Perceptions of Alternatives to Research Participation. Teaching of Psychology, 33(4), 247-249. doi: 10.1207/s15328023top3304_7 Cromer, L.D., Reynolds, S.M., and Johnson, M.D. Journal of the Scholarship of Teaching and Learning, Vol. 13, No. 3, August 2013. josotl.indiana.edu 155 U.S. Department of Health, Education, and Welfare: The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (1979, April). The Belmont report: Ethical principles and guidelines for the pro- tection of human subjects of research. Retrieved January 3, 2008, from http://www.hhs.gov/ohrp/humansubjects/guidance /belmont.htm Witt, E. A., Donnellan, M. B., & Orlando, M. J. (2011). Timing and selection effects within a psychology subject pool: Personality and sex matter. Personality and Individual Differences, 50(3), 355-359. doi: 10.1016/j.paid.2010.10.019