Microsoft Word - singleton-jackson, final proof copy Johnny Still Can’t Write, Even if He Goes to College: A Study of Writing Proficiency in Higher Education Graduate Students Jill A. Singleton- Jackson Department of Psychology University of Windsor D. Barry Lumsden Ron Newsom Department of Counseling and Higher Education University of North Texas Biographical Information: Jill Singleton-Jackson is an Assistant Professor in the Department of Psychology at the University of Windsor in Windsor, Ontario, Canada. She teaches psychology courses and is also the coordinator of the Foundations of Academic Writing courses. Her research interests include student writing and student entitlement. Barry Lumsden is retired from the Department of Counseling and Higher Education at the University of North Texas. He is currently a Senior Research Fellow at Texas A&M University and Editor-in-Chief of the Community College Journal of Research and Practice, Educational Gerontology, and Christian Higher Educator. He is a National Faculty Development Leader on non-fiction scholarly writing and publishing. Ron Newsom serves as Associate Professor and Assistant Chair of the Department of Counseling and Higher Education at the University of North Texas. Correspondence should be addressed to: Jill Singleton-Jackson University of Windsor Department of Psychology 401 Sunset Avenue Windsor, Ontario Canada N9B 3P4 Phone: 519-253-3000, ext. 4706 Email: jjackson@uwindsor.ca Abstract This study explored the extent to which graduate students enrolled in Higher Education courses were proficient at writing. A total sample size of 97 graduate students from programs of Higher Education served as the sample. To assess writing proficiency the SAT II: Writing Test, Part B was used. The graduate students in this sample did not score significantly higher on the SAT II: Writing Test, Part B than the typical high school senior whose scores enter into the norm group. Key words: higher education, writing, writing proficiency, graduate students, SAT II Writing Test JOHNNY STILL CAN’T WRITE, EVEN IF HE GOES TO COLLEGE: A STUDY OF WRITING PROFICIENCY IN HIGHER EDUCATION GRADUATE STUDENTS Language, both written and spoken, serves as a critical cornerstone for our culture. “The faculty of language stands at the center of our conception of mankind; speech makes us human and literacy makes us civilized” (Olson, 1977, p. 257). However, writing can be a painful and anxiety producing task for many individuals. And while the final goal of any writing is the same - communication - this goal is not always reached. In an effort to explore the current nature of this phenomenon while adding a unique twist, this study focused specifically on writing proficiency among higher education graduate students. Interestingly, a review of the literature, while rich with studies relative to undergraduate writing assessment, revealed a marked paucity of studies involving graduate student writing assessment. Likewise, aside from the GRE-Written (GRE-W), launched in 1999, instruments for assessing graduate student writing specifically are not national norm-based instruments, but are instead idiosyncratic to institutions and departments. Graduate school is undeniably a writing intensive experience (Hadjioannou, Shelton, Fu, & Dhanarattigannon, 2007; Wasley, 2008). So it is interesting that while not a population that is often studied with regard to writing proficiency, graduate students are required to write as experts. This paradox is further complicated by the fact that faculty often complain about the amount of time they spend editing and discussing graduate student writing. Clearly there is a disconnect between the expectations and reality. Part of the problem may be the result of an assumption. Higher education operates based on the assumption that students entering graduate school should be more proficient writers than they were upon entering college for the first time. As summarized by Mullen (2006), “like their younger counterparts, graduate students need to demonstrate high-level skills in reading comprehension, thinking and reading critically (as in knowing how to identify various rhetorical structures and to distinguish between what should be said explicitly and implicitly), and communicating with particular audiences for specific purposes. They also should know how to collaborate on writing; how to use technology; and how to write for specific genres, both professional and academic” (p. 30). Meanwhile, it has been established historically and repeatedly that there are many issues surrounding undergraduate writing proficiency (Flateby, 2005). For example, Knudson, Zitzer-Comfort, Quirk, and Alexander (2008) have published an article indicating that when measured for proficiency in writing and reading in English, 46 percent of the first-year students at The California State University required remediation. Just as The California State University has put in a place a program to increase reading and writing proficiency for their freshmen, many undergraduate institutions provide abundant opportunities for their undergraduates to acquire writing skills and then attempt to assess those skills as students exit the institution (Devarics, 2006). It is logical to assume that with the recognition of undergraduate writing problems and the subsequent prevalence of opportunities for skill acquisition and repeated assessments of writing in the undergraduate curriculum (e.g., essay exams, research papers); students should be skilled in writing upon completion of an undergraduate degree. And while not everyone who obtains a bachelor’s degree is an appropriate candidate for graduate study, a bachelor’s degree is a minimum requirement for attending graduate school. As stated by Mullen (2006), “[i]n graduate circles, academic writing is presumed to be a solitary activity for which students already are prepared” (p.30). To summarize, there seems to exist an assumption that graduate students are competent writers; they have completed a bachelor’s degree, and it is assumed they have learned to write competently during that undertaking. This assumption is exhibited by the fact that graduate study is writing-intensive, yet graduate programs do not offer or require specific courses in writing. Instead, graduate students study their chosen field in-depth and are expected to write as scholars. But to what degree is this assumption correct? The purposes of this study were to explore this assumption by looking at graduate students’ writing proficiency as defined by their ability to 1) control the basic elements of written English (e.g., grammar, word choice, syntax, 2) recognize writing errors in usage and structure, and 3) use language with sensitivity to meaning. A fourth purpose was to determine if writing proficiency is correlated with selected demographic variables. Graduate Student Writing Proficiency and Assessment In addition to understanding the cognitive processes and skill levels involved in producing written text, measuring proficiency is an important element of written communication that is of practical interest to faculty and administrators at institutions of higher education. Writing proficiency is a term that can be hard to conceptualize and even harder to define because it is a "slippery term" that hides "an even more slippery concept" (White, 1994, p.150). Proficiency may be thought of as skill, adequacy, sufficiency for a defined purpose, or capability. Regardless of how proficiency is specifically defined, it is something that those who evaluate writing as part of their profession can readily identify and describe when asked to discuss it. And while it may not be easy to define, writing proficiency has long been on the minds of those who teach and study graduate students. In 1986, Scardamalia and Bereiter asserted that as expectations for scholars, including graduate students, rose so would expectations regarding competence in writing. This expectation of writing competence from over two decades ago still exists for graduate students today, and while we can assume that they certainly possess some degree of writing competence, what is that degree? The two most commonly used means of evaluating the writing ability of graduate school applicants are 1) the Graduate Record Exam (GRE) verbal score, and 2) committee evaluations of the personal statement. The GRE verbal score is a marker of general language ability, but it is not a measure of performance or of applied writing skill for the applicant. While a language skill score might be expected to correlate to some degree with writing ability, it can be argued that the GRE verbal score alone is not enough to ascertain the baseline writing proficiency levels of entering graduate students. Sternberg and Williams (1997), while not specifically studying graduate student writing, looked at the GRE as a meaningful predictor of success in graduate training. They based their research on the fact that the use of GRE scores as a criterion for admission is prevalent in graduate programs across numerous institutions. Their research was grounded in Sternberg's Triarchic Theory of Human Intelligence. The Triarchic Theory of Human Intelligence distinguishes among academic, creative, and practical abilities. It should, however, be noted that "these kinds of abilities are not wholly independent...but research indicates only a weak relationship among the three abilities" (Sternberg & Williams, 1997, p. 633). According to these researchers, performance on the GRE will be most affected by analytical abilities while creative and practical abilities will not be as readily apparent from GRE test scores. Sternberg and Williams (1997) hypothesized that the GRE, while a good predictor of initial graduate grades, would not be a strong predictor of success among advanced graduate students. Specifically, they predicted "that the GRE would be, at best, very weakly predictive of more meaningful criteria of graduate program success: in particular...of dissertations" (Sternberg & Williams, 1997, p. 634). Sternberg and Williams did find that GRE scores, while predictive of first-year graduate school grades, did not prove useful as predictors of a number of graduate school performance measures. Notably they were not correlated with faculty ratings of dissertation quality. This is of particular relevance to the current research study due to the intensive writing component involved in producing a quality dissertation. While analytical abilities are important in writing, creative and practical abilities are also of relevance. The GRE verbal scores, while relative to language, still rely heavily on analytical skills, not creative or practical skills (Sternberg, 1996; Sternberg, Ferrari, Clinkenbeard, & Grigorenko, 1996). Using this score as an indicator of graduate students' writing ability, especially the ability to produce writing as complex as the dissertation, is a misuse of the score and can lead to frustrated faculty and defeated students. And while studies that are more recent have furthered ideas about best practices for increasing chances of graduate student success, the dissertation remains a major obstacle (DiPierro, 2007). The second form of attempting to evaluate graduate student writing proficiency, the personal statement (sometimes called statements of purpose or goals, or letters of intent), is a popular method of direct and indirect assessment of writing proficiency. Many graduate programs require applicants to submit personal statements as part of their application materials. These statements, written as part of the applicants’ attempt to gain admission to a program of study, usually reflect the applicants' motivation for attending graduate school and outline the intended career goals of the applicants. While providing information about aspiring graduate students' personal qualities and disciplinary socialization, the personal statement is also often viewed as a valid indicator of writing ability (Powers & Fowles, 1997; Powers, Fowles, & Willard, 1994; Brown, 2004; Samraj & Monk, 2008). Although personal statements are typically used as direct assessments or measurements of writing ability, defining personal statements as direct measures of writing ability is somewhat misleading. Messick contends that measurement is direct only in a very limited sense: skills and knowledge are more correctly inferred from a product as opposed to being measured directly. Writing ability is, therefore, inferred from applicants' personal statements more than it is measured by such means (as cited in Powers, et. al., 1994). While it is "widely acknowledged that an instrument [such as a personal statement] is not valid in and of itself....it is the inferences about the meaning and use of test scores that are to be validated" (Powers & Fowles, 1997, p. 76). Research Studies Two studies, one by Powers et al. (1994) and the other by Powers and Fowles (1997) explored the use of the personal statement as both a criterion for admission to graduate school and as a valid indicator of graduate student writing ability. In 1992, the staff of the GRE program conducted a nationwide study. Graduate deans and faculty were surveyed in an effort to determine if there was interest in a GRE writing test. While a number of the respondents were receptive to the idea of a test designed specifically to evaluate writing ability, a number of the respondents indicated that the information obtained from a writing test would be redundant with the information contained in the personal statement (i.e., personal statements serve as sufficient indicators of writing proficiency). Many of the respondents to the GRE survey indicated a belief that "personal statements were better indicators of writing skill than were standardized writing measures" (Powers & Fowles, 1997, p. 77). This echoed the earlier findings of Anderson and Ekstrom (1994). In a similar investigation, they found that writing samples from graduate departments in arts and humanities, on average, carried as much weight as GRE verbal scores and undergraduate course of study. Additionally, the writing samples were more heavily weighted than GRE quantitative, analytical, and Subject Test scores, letters of recommendation, undergraduate institution quality, and personal interviews. One faculty respondent from an economics department made the following comment: "I cannot see what we would gain from another test score. The applications already contain a statement of purpose, which gives a direct writing sample" (Powers & Fowles, 1997, p. 77). To summarize the findings of the GRE survey, "those who viewed a writing test as largely unnecessary valued the personal statement as a sufficient indicator of writing ability" (Powers & Fowles, 1997, p. 77). This could have been the end of this line of inquiry had the responses of some of those surveyed been considered definitive. However, there are a number of problems associated with equating the personal statement with a measure of writing ability. Powers and Fowles (1997) attempted to answer the following question: Is a standardized measure of writing skill largely redundant with the personal statement (statement of purpose) for determining writing skills?" (p.79). They were interested in determining the relative merit of the two means of assessing writing skill: the personal statement and a standardized test. They hypothesized that, for a number of reasons, writing skill would be best reflected by a standardized writing measure as opposed to a personal statement. Their hypothesis was based on the following: 1) a standardized writing measure would "better reflect writing skill than the personal statement"; 2) a standardized test "is based on topics that have survived careful pre-testing to ensure that they are accessible to most examinees and they elicit a sufficient range of response"; and 3) because it is administered under standardized conditions, a formal test "should reflect fewer sources of irrelevant variation than the personal statement" (Powers & Fowles, 1997, p. 79). Powers and Fowles (1997) recruited 475 GRE General Test examinees from among those who took the GRE between January and May of 1994. The participants went to testing centers where, under standardized conditions, two-thirds of them wrote two expository essays. The study participants also provided several non-test indicators of writing proficiency, a recent sample of undergraduate writing, and a copy of their personal statement if they had submitted one as part of their application to graduate school. Of special interest was a survey question posed to the participants asking them "to indicate how much help they had received in drafting and in editing and revising their statements" (Powers & Fowles, 1997, p. 80). College faculty who qualified as trained essay readers then evaluated the personal statements and the expository essays. The essays were independently scored by two readers, and a 6-point holistic score scale was used. The results of this study proved quite interesting. In response to the question regarding the amount of help they had received when constructing their personal statements, 59% of the participants revealed that they had received at least some help in editing and revising their statements. Thirty-four percent indicated that they had received moderate or substantial help. Thirty-six percent acknowledged receiving assistance when drafting their statements and 19% indicated that they had received moderate or substantial help. Fifteen percent indicated that when drafting, editing, and revising their statements, they had received moderate or substantial help. Thirty-eight percent of the participants stated that they had not received any help in writing their personal statements. The inter-reader correlation for the personal statements was .78, and the inter-reader correlations for the test essays ranged from .77 to .80 across the four different essay prompts. Powers and Fowles (1997) report that "the correlation between scores assigned to the test essays and those assigned to the personal statements was low, only .15. Therefore, on the basis of our sample, the timed essay and the personal statement cannot be considered to provide interchangeable information" (p. 83). These findings underscore the idea that personal statements are not necessarily adequate in providing graduate school faculty with valid information about an applicant's writing ability, and it might be unsafe to assume that an effective personal statement is indicative of writing ability on par with faculty expectations regarding student writing. The researchers ultimately concluded that "although the personal statement may provide certain unique and important information about applicants...its validity as an indicator of writing skill...needs to be better established” (p.75). In a similar study conducted by Powers et al. (1994), writing assessment was examined with the focus on the relationship between direct assessment and direct validation. In this study, sets of six student-produced essays (ranging in quality from poor to excellent) were sent out to be scored on a 6-point scale by graduate school personnel (department chairs, faculty, and deans) at 115 graduate institutions. Powers, Fowles and Willard contacted 1,100 individuals; a total of 347 people responded to the survey with 231 of the respondents submitting satisfaction ratings for the sample essays. Interestingly, when a sample of non-respondents was contacted for follow-up, the main difference between respondents and non-respondents was that non-respondents indicated a lower interest in a GRE writing measure. The researchers concluded from this study that "some performance assessments, such as those involving direct measurement of writing skills, may enable validation that requires somewhat smaller inferential leaps than those needed for less direct measurements" (Powers et al., 1994, p. 97). However, the authors were careful to explain that unless direct assessments are constructed and administered carefully, the inferential leap is actually more hazardous. Namely, "limited content coverage, poor generalizability, difficulty in maintaining test security, and increased opportunity for bias due to subjective scoring all pose plausible threats to the validity of these assessments under some conditions (Powers et al., 1994, p. 97; Dunbar, Koretz, & Hoover, 1991). Powers et al. also added an interesting aside in their discussion; they related that the faculty respondents expressed a distrust of the writing samples currently required by several of the graduate schools surveyed. The faculty members commented that it is impossible to know how much help an applicant has received in writing, revising, and editing personal statements or other writing samples not obtained in a secured testing environment. This concern would seem to be consistent with Powers and Fowles' subsequent 1997 study findings. Other more recent studies explore aspects of graduate student writing outside the realm of the GRE or the personal statement. Kamler (2008) and Cuthbert and Spark (2008) both engaged in two studies that centered on the idea of graduate students needing to learn how to write for publication and the problems contained therein. Kamler’s study starts with the premise that “writing for publication is an important activity for established academics and doctoral students alike” (2008, p. 283). Green, Hutchison, and Sra (as cited in Kamler, 2008) have concluded that publishing from the dissertation is a critical factor when predicting scholarly productivity. Kamler’s paper concludes with her assertion that through publication graduate students become productive members of a discipline through, among other things, publication. Similarly, Cuthbert and Spark (2008) look at graduate students writing for publication by exploring the results of a pilot program in Australia that used writing groups in an attempt to help graduate students gain skills and find support needed for writing for academic publication. In another study of graduate student writing, Lavelle and Bushrow (2007) explore the writing processes and beliefs pertaining to graduate students as they attempt to learn and master the highly specialized task of high-level academic writing. These researchers make the point that graduate school demands that students produce higher quantities and quality of writing; however, “[a]cademic writing at the graduate level is a complex and often novel undertaking for the student. Indeed, expectations as regards breadth and depth, and the diverse range of writing demands (article critiques, academic papers, grant writing), call for new insights and increased levels of skill” (Lavelle & Bushrow, p. 807). They then go on to describe an in-depth study of writing processes of graduate students and their attempt to develop psychometric model and measurement standards that would help explain and support graduate student writing. Interestingly, these authors cite as one of the rationales for their study the fact that while it is a commonly known that graduate students need to write a lot and write well, as early as the 1970s discussions have ensued regarding the need to address graduate student writing by providing support and instruction, while still little has been done to actually assess what graduate students go through (process) when they write. To underscore both the longstanding nature or this issue and the concept of a need for increased understanding of graduate student writing, in her 1998 presentation to the annual meeting of the Mid-South Educational Research Association, Debbie Hahs presented findings from a survey of graduate students that asked them what they thought they needed to succeed and persist in graduate study. Her survey results discovered many interesting things. Most relevant to this study was the respondents indication that they would like to attend workshops to gain knowledge about various writing tasks common to graduate school (e.g. dissertation, CV, grant) (Hahs, 1998). When given 11 sample workshops and asked which they would be most likely to attend, of the top five, three of them pertained to writing. The respondents in her study seemed to understand the need to master their field and then be able to communicate their expertise in writing to others in their field as well as to those not in their field. We can infer, as Hahs has, that “students who are requesting information or services or resources have not satisfactorily received this information through current channels” (1998, p. 2). These studies by Lavell and Bushrow and Hahs, as do the others previously discussed, speak to the assumption that graduate students by virtue of being in graduate school, are prepared to write at an acceptable level regardless of actual scrutiny of graduate student writing or of that assumption. There seems to exist, as summarized by Newman (2004) a “vexed relationship” between undergraduate writing and how this first level of education relates to writing at levels of higher study (p. 31). These studies, while focusing on different aspects of graduate student writing, reflect the same basic premise that there are issues with graduate student writing that require exploration of process and proficiency that ranges from studying the personal statement to creating writing groups to supporting graduate student publication efforts. So while all the previously discussed studies approached the topic of graduate student writing from different angles and with different agendas, these studies do allow for a general conclusion. Namely, graduate student writing assessment is a multi-faceted area of study; using the personal statement as an indicator of writing proficiency is erroneous; graduate students need to be competent writer; and graduate student writing merits further investigation. Perhaps the most obvious message to be concluded for this brief review of research in the area is that it should not be taken for granted that being a graduate student equates with being a proficient writer. Method Participants An availability sample of graduate students enrolled in at least one course in Higher Education at public institutions of higher education in the United States participated in the study. A total sample size of 97 students was obtained. Participants came from seven universities in the United States. Demographic data were collected from the study participants. The majority of respondents were female (65.6%). Participants were also asked to report the grade they earned on their most recent writing assignment. A few of the respondents (6.2%) appeared to be fairly new in their program (having fewer than 12 hours); the majority (68.0%) had accumulated 25 or more hours of graduate credit. Table 1 contains descriptive statistics for the sample demographic variables. Each participant received a cover letter to keep explaining the requirements of participation in the study. In this letter, the participants were informed of the right to withdraw from the study at any time. Participants were also provided with a detailed set of directions for completing the instruments. In the case of the data being collected in a group setting, the administering professor was given data collection instructions to read to the group of participants. In lieu of signatures, participants’ returning the completed instruments served as an indication of their consent to participate. This recognition of completion and submission of the instruments as consent was used for both individual participants and those who participated in classroom settings. Participants who participated in classroom settings did so on a voluntary basis and in no way was their participation linked to a grade in the course. The administering professor did not score or review the instruments; he or she mailed the completed instruments back to the primary researcher for analysis. Data collection materials were coded with a three- digit number for the purpose of being able to collate the data, but participants could not be linked to the submitted data.1 1 This study was approved by the University of North Texas Institutional Review Board Committee for the Protection of Human Subjects. Table 1 Participant Demographic Variables First Language Percent English 91.75 Other 8.25 Age Group Less than 25 Years 18.56 25 to 28 Years 9.28 29 to 35 Years 20.62 36 to 50 Years 34.02 Over 50 Years 17.53 Gender Male 34.37 Female 65.63 Graduate Hours Completed Less than 12 6.19 12 to 24 25.77 25 to 40 19.59 41 to 60 23.71 More than 60 11.34 All But Dissertation (ABD) 13.4 Program Masters in Higher Education 26.8 Doctorate in Higher Education 54.64 Masters in other field 3.09 Doctorate in other field 15.46 Doctoral Degree Sought Ed. D. 50.7 Ph. D. 49.3 Note: For table 1, all percentages are based on n=97 except a) 1 person did not answer the gender question, b) 3 people did not identify their current program. Furthermore, for type of doctorate in higher education, the percentages are based on n=71 (n=79 respondents indicated they were pursuing a doctorate in higher education, and of those, eight respondents did not indicate the type of doctorate). Instruments To assess writing proficiency for the participants, the SAT II: Writing Test, Part B was used. The SAT II: Writing Test, Part B is a timed, 60-item multiple-choice test developed by Educational Testing Service (ETS). The purpose of the test is to “measure [test takers’] ability to…recognize faults in usage and structure, and to use language with sensitivity to meaning” (Educational Testing Service, 1999-2000, p.7). The multiple-choice questions deal with such common writing problems as “being consistent, expressing ideas logically, being clear and precise, and following conventions” (Educational Testing Service, 1999-2000, p.7). This instrument has demonstrated reliability (R. Goodman, personal communication, August 22, 2000). Although it might seem unconventional to use the SAT II, an instrument typically associated with high school level testing, the use of this instrument was warranted based on a number of reasons: • ETS professionals who developed the test assert that “most students take the…SAT II tests…during their junior or senior year in high school, but there are no age or grade restrictions [italics added] (Handbook for the SAT Program, 1999-2000, p.5). • The SAT II is typically taken by college-bound seniors and is used by many colleges for “admission, placement, and advising” purposes (Handbook for the SAT Program, 1999- 2000, p.7). • Given the assumption being tested (i.e., after earning a bachelor’s degree, students who wish to pursue a graduate degree and are accepted to a graduate program of study are prepared to write at a level adequate for graduate level work), it is sensible to use a test typically given to college-bound seniors. In other words, graduate students should score significantly higher than the SAT II average if they did indeed obtain the writing skills assumed of a bachelor’s degree holder. • The SAT II is a normed instrument. A normed instrument was ideal for this study based on a number of reasons. Namely, by using a normed instrument, the scores obtained were scaled relative to people who have previously taken the test. If a raw score is obtained (e.g., 40 correct), it is meaningless unless information is available about how other people scored on the same test; norms anchor a test. The SAT II served as an anchor, or point of comparison, for the scores obtained from the participants. This instrument has “been established as a common standard against which students performance can be compared. Finally, to investigate students’ performance on the instrument, cursory observations about the instrument were made through the use of a pilot study of the instrument. To determine variation among responses to the 60 items on the test, the SAT II: Writing Test, Part B was piloted with six graduate students in Higher Education at the University of North Texas. The data were collected, and two of the students’ response sets were removed because they indicated that English was not their first language. Only one of the four remaining students was able to complete the 60 items in the allotted time. Interestingly, it was found that there was variance among two-thirds of the items on the instrument. This suggested that the instrument would yield useful information since there were individual differences in the responses of the pilot sample. If everyone had selected the same responses to the test items, this instrument would not yield useful information for the study. However, because variance was discovered and the instrument has been determined to be reliable by ETS, it is safe to assume that this instrument provided useful and informative results. Further, this should dispense with any concern about using an instrument typically associated with high school testing, since it is clear that even with a small pilot sample (n = 4) variation was present. Procedure In the original attempt at data collection, a list of all public institutions of higher education in the United States that offer graduate degrees in Higher Education was obtained from the ERIC Clearinghouse on Higher Education (www.eriche.org/resource/public.html). The institutions were sampled to ensure that both master’s and doctoral programs were represented. This resulted in a non-proportional stratified sample of institutions that offer graduate degrees in Higher Education (Gall, Borg, & Gall, 1996). Following the selection of the program pool, the researcher accessed program websites to obtain the names of contact persons. The identified contact person for each program in the sample was then contacted by phone or e-mail. The study was described, and a request was made that the contact person help recruit graduate students to participate in the study. After considerable effort, it was discovered that this was an ineffective means of recruiting study participants. Twenty-five institutions were contacted and only two professors had agreed to help collect data from graduate students. It was then determined that multiple means of collecting data would be necessary. Study participants subsequently were recruited in three ways: 1) by contacting Higher Education professors at various programs, 2) through Higher Education graduate student organizations, and 3) through a request for study participants posted on the Association for the Study of Higher Education (ASHE) list serve. This resulted in a convenience sample of graduate students in Higher Education. Professors of Higher Education courses were contacted and asked if they would be willing to recruit volunteers for the study from their classes. If they agreed, data collection materials were sent to the professor, the instrument was administered in a group setting by the professor, and the completed materials were returned to the researcher by the professor. Seventy-three participants were obtained using this method of recruitment. Individuals were also recruited to participate in the study. This was done in two ways: by obtaining permission to post a request for subjects on the ASHE list serve, and by asking graduate student coordinators or graduate student organization officers to forward an e-mail request for participants to students enrolled in courses in Higher Education. Students who received the posted messages and were interested in participating in the study contacted the researcher by e-mail, and arrangements were made to send a packet of data collection materials to the student. Twenty-four participants were obtained using these methods of recruitment. In the case of an individual providing data, as opposed to the data being collected in a group setting, a Statement of Honesty was required in an attempt to ensure that all data collected were legitimate and provided under similar circumstances. Each program or individual that participated in the study was offered a copy of the findings for the study. Results The SAT II: Writing Test, Part B Participants in this study completed the SAT II: Writing Test, Part B. The test was scored using the answer key provided by SAT, and raw scores were converted to scaled scores according to the conversion table provided by SAT, (SAT II: Subject Tests, 2000). The average multiple-choice scaled score from the SAT Writing Test, Part B was 59.30 (the range was 20 to 80), and the sample standard deviation was 10.27. The distribution had a very slight negative skew (-0.129), implying that the mean score was slightly lower than the median. Based on published norms, the population mean is 59 and the population standard deviation is 10 (Inside SAT II, 1999). This norm is based on “scores earned by 1998 college-bound seniors who took the SAT II: Writing Test at any time during high school” (Inside SAT II, 1999, p.12). When a sample of data is obtained and the population mean and standard deviation are known, an appropriate test for a significant difference between the sample and population mean is the z-Statistic for a Single Sample Mean (Howell, 2002). Hence, this significance test was used to determine whether the sample mean was significantly higher than the population mean, after ascertaining that there were no univariate outliers, z-scores in excess of an absolute value of 2.5 (Kirk, 1995). The sample mean was not significantly higher than the population mean (z=0.295, p<0.38). The graduate students in this sample did not score significantly higher on the SAT II: Writing Test, Part B than the typical high school senior whose scores enter into the norm group. Demographic Variables and SAT II: Writing Test Scores Statistical tests were used to determine whether significant differences in SAT II: Writing Test scaled scores existed for selected demographic variables. This was done in an exploratory manner, with the goal of understanding whether significant differences existed based on these selected variables. There was no specific theoretical framework used to test formalized hypotheses. Instead, these variables were selected based on a deductive review of the literature. The demographic variables we examined were undergraduate major, type of degree program in which participants were enrolled, gender, whether individuals participated individually or as part of a class , and type of doctorate degree (Ed.D. or Ph.D.). Table 2 contains descriptive statistics related to the significant findings. A description of each test and findings follow, and the results of the statistical tests are summarized in Table 3. Table 2 Descriptive Statistics Corresponding to Significant Effects for Demographic Variables Independent Variable n M SD Gender Female 63 60.86 9.90 Male 33 56.15 10.53 Participation Method Individual 24 65.22 9.34 Class 73 57.46 9.91 Degree Type Ed.D 36 56.75 9.02 Ph.D. 35 63.26 10.65 Note: For table 2, one person did not answer the gender question and the sample size for degree type is 71 because out of the 97 participants, 79 respondents indicated they were pursuing a doctorate in higher education and eight of those did not indicate the type of doctorate. We used parametric tests of group differences, namely between groups Analyses of Variance and independent samples t-tests depending upon the number of groups, to evaluate group differences. The assumptions of the techniques were met (e.g., Kirk, 1995). Specifically, the assumption of homogeneity of variance was evaluated and found to be tenable using the Levene statistic (Levene, 1960). Data were screened for outliers by group, using a z-score of 2.5 as a cutoff (Kirk, 1995). Furthermore, we had no reason to suspect any violation of the assumption of independence of observations since each participant completed their own SAT II writing test and there was no common intervention designed to alter the writing test scores. While there was a mild departure from normality for the writing test score, namely the previously noted negative skewness, mild departures from normality are deemed acceptable, especially when the assumption of homogeneity of variance has not been violated (Stevens, 2002). Finally, we chose to evaluate all null hypotheses at an alpha level of .05 and did not correct for Type 1 error since the analyses are considered exploratory and should be validated in a separate sample in future research. With respect to undergraduate degrees, the undergraduate majors of the 97 participants were coded into four broad categories. These categories were as follows: English (n=10), Math/Science (n=17) Social Science (n=48) Education (n=16). There were six participants who indicated that they had majored in “Business” as undergraduates. This group was too small to include in the analysis. An analysis of variance revealed there were no significant differences among the four groups (see Table 3). Participants were grouped into three categories in terms of their current program: those getting a Master’s in Higher Education (n=26), those getting a Doctorate in Higher Education (n=53) and those getting a Doctorate in another field (n=15). There were only 3 participants who indicated that they were receiving a Master’s Degree in a program outside of Higher Education and these observations were eliminated from the analysis. Analysis of variance revealed no significant differences among the three groups (see Table 3). We did find significant differences in writing test scores for gender with females having a higher average score than males (see Table 2). Furthermore, those who participated on an individual basis had significantly higher test scores than those who completed the instruments as part of a class2. Finally, participants who were pursuing an Ed.D. had significantly lower scores than participants pursuing a Ph.D. Table 3 Results of Tests for Differences in SAT II: Writing Test, Part B Scores by Demographic Variables Independent Variable Test Statistic Results Probability Effect Size Undergraduate Major F(3, 87) = 1.01 .392 0.000 Academic Program F(2, 91) = 1.95 .148 0.020 Gender t(94) = 2.17 .033 0.048 Participation (individual vs. class) t(95) = -3.32 .001 0.104 Degree type (Ed.D. vs. Ph.D.) t(69) = -2.78 .007 0.101 Note: Effect sizes are ω2 (see e.g., Kirk, 1995) for F values and Cohen’s d for t values (Cohen, 1988). Discussion and Implications for Future Research Review of the Research Questions In an attempt to gain information regarding graduate student writing proficiency, two main questions guided this study. How do graduate students compare to high school seniors on a standardized, norm-based measurement of writing proficiency? Is writing proficiency correlate with specific demographic variables? The series of analyses presented here helped to both answer these questions and to raise questions for future research. When reviewing the literature to form the basis for this study, it became clear that while there is great deal of interest in assessment of student writing, there is very little research available that addresses graduate student writing specifically. While educators agree that writing proficiency is an important skill for graduate students, there seems to be a discrepancy between expectation and reality (Hahs, 1998; Torrance, Thomas, & Robinson, 1999; Mullen, 2006; Lavelle & Bushrow, 2007; & Kamler, 2008). To wit, graduate students are expected to write like 2 In fact, those who completed the study individually did score significantly higher than the SAT II Norm Sample (z = 3.043, p < .05). See discussion section for more comments on this finding. experts, but graduate student writing quality elicits cries of despair and exasperation from faculty. This basic idea was the driving force behind this study. Is the state of graduate student writing as depicted by disgruntled faculty a myth or of anecdotal nature, or is there indeed a real issue with regard to graduate student writing proficiency? Writing Proficiency To recap, the graduate students in this sample did not score significantly higher on the SAT II: Writing Test, Part B than the typical college-bound, high school seniors whose scores constitute the norm. Implications. This finding is interesting for several reasons, some clear, some more ambiguous. Several assumptions predicated this study. For the purpose of clarity in this study, “better writers” is taken to mean more proficient, and regardless of personal definitions of the term “proficiency,” there is arguably an understanding that college students will be “better writers” when they graduate than they were prior to matriculation (North, 1996; Abate-Vaughn, 2007; & Knudson et al., 2008). It follows that this would seem to be a reasonable expectation on the part of graduate faculty – reasonable, yet unmet. The outcomes of a study by the American Association of Colleges and Universities (as cited in Abbate-Vaughn, 2007, p.52) “revealed that a dismal 11% of college seniors are able to write at the ‘proficient’ level while holding the belief that college was contributing to their skills in writing and other areas.” By requiring a bachelor’s degree as a prerequisite to admission to graduate study, it may be safe to conclude that graduate admissions committees are incorrectly assuming that an undergraduate degree has adequately prepared students for the rigors of graduate level writing since most graduate programs are writing intensive. The discovery that the graduate students sampled in this study did not in fact score significantly higher than college-bound high school seniors seems to justify faculty complaints that graduate students cannot write on a satisfactory level. Presumably, graduate faculty expects their students to write better than high school seniors. If the sample population used in this study is representative of graduate students in Higher Education courses across the United States, then the conclusion reached is that after four years of undergraduate study, and possibly even after some graduate study in the case of those with master’s degrees, the graduate students sampled in this study were no more proficient writers than average college-bound high school seniors. This finding begs many questions. Namely, are the skills required to produce writing proficiency not adequately addressed in the undergraduate curriculum? Or is there simply not enough emphasis placed on writing proficiency as a prerequisite for admission to graduate school? Does this lack of emphasis on screening lead to the admission of graduate students who are inadequately prepared to produce the level of writing expected in graduate school? Also, is it possible that the faculty assumption that entering graduate students will be proficient writers is in error or unfair? Is it possible that graduate faculty should expect to teach writing skills to their students? Writing Proficiency and Demographics Significance testing was done for writing proficiency scores and selected demographic variables. While overall this testing proved to be fairly unspectacular, it did unearth a few points of interest. Undergraduate major was considered as one variable with a potential relationship to writing proficiency score. Although a relatively small n for three of the four major categories limits the usefulness of this information, it was not surprising to find that the study participants who were English majors as undergraduates did, as a group, have the highest scores on the SAT II: Writing Test. However, it should be noted, that the English majors did not score significantly better than the other participants. Two Independent Samples T-Tests did prove to have significant results. First, females scored significantly higher than males on the writing test. This may exemplify the theory of gender’s role in language ability in general. Females typically score higher than males on tests of verbal ability (Cohen, Swerdlik, & Phillips, 1996). Perhaps this verbal ability translates to writing proficiency. This might also be the result of studying students who were, for the most part, Higher Education graduate students. As a social science, Higher Education programs are typically writing intensive, and may attract and admit students from a social science background. Females are more likely to be found in education and the social sciences than in other areas of study (Pascarella & Terenzini, 1991). Forty-eight of the ninety-seven participants in this study were social science majors as undergraduates, and the participants were predominantly female (65%). A loosely drawn conclusion then might be, that more females are undergraduate social science majors, they get more practice writing as undergraduates, they seek writing intensive graduate programs because of a comfort level with writing, and this translates to an overall higher writing proficiency. Secondly, it was discovered that the participants in this study who were pursuing a Ph.D. scored significantly higher on the SAT II: Writing Test than those participants who were pursuing an Ed.D. Perhaps this is simply a function of the type of student who pursues the Ph.D. versus the Ed.D. This finding might indicate that the more applied degree, the Ed.D., attracts students with less writing experience, interest, background, or skill. Also, it is a possibility that Ed.D programs have a different set of criteria for admission. For example, admission committees may look less upon writing proficiency as a significant factor in their decision to admit a student to graduate study in Ed.D. programs. The most bothersome finding from this portion of the study involved a comparison of participants who completed the writing test on an individual basis versus those who completed the test as part of a class. The participants who did the test on an individual basis scored significantly higher than those who participated in a classroom setting and significantly higher than the SAT II normative sample. This is troublesome because of the nature of the writing test. The SAT II: Writing Test, Part B, is a timed, 60-item, multiple-choice test. Participants in the classroom setting were monitored for time (40 minutes) and use of reference materials (not allowed). However, individual participants were not monitored. Instead, they were asked to sign a Statement of Honesty indicating that they would observe the conditions of the test. The most reasonable explanation for this significant difference is that individuals who agreed to participate in the study had a particular interest or propensity for writing and were therefore somehow more proficient writers than their cohorts who were recruited as a part of a group. Additionally, based on these “possible” characteristics the sub-sample of 24 participants may not represent the typical graduate student. Limitations Convenience Sample. Aside from the possibility of a Type II error, there is the fact that a convenience sample was used. In some cases, this might result in the findings being questionable. However, for this study this would seem to be illogical in light of the sample population’s failure to score significantly better than the population mean on a test of writing proficiency. All the participants were volunteers. Students who volunteered to complete the instruments on an individual basis comprised 24.7% of the sample. The remaining 75.3% of the sample population were recruited by graduate faculty and completed the instrument in a classroom setting. If it is assumed that the students who volunteered to participate in the study did so because of an affinity for writing, or because of a personal interest in the topic, then it would be more likely that a significant difference would have been found between the sample population and the norm population, with the sample population scoring significantly higher than the norm group. Instrument Limitations. A second issue regarding the validity of the findings centers on the use of the SAT II: Writing Test, Part B as the means of measuring writing proficiency for the sample population. This instrument tests for three major components or subscales of English proficiency: grammar, usage, diction (choice of words), and idiom; correctness and effectiveness of expression (inclusive of grammar, word choice, sentence construction, and punctuation); and organization and development relative to clarity and coherence. A second assumption of this study, knowledge of the technical aspects of English grammar translates into proficient writing or production of quality text, ties into the use of this particular instrument. Additionally, this assumption allows for the conclusion that the sample population, as reflected by mean score on the instrument, is no more adept at writing than college-bound high school students. However, if there is fault in this assumption, then there is fault in this conclusion. More simply put, when measuring the technical aspects of writing proficiency, is it safe to assume that the SAT II, Writing Test, Part B measures writing proficiency or just technical proficiency? Is there incongruence between technical knowledge of English grammar and usage and actual generation of quality text? A limitation of this study was that Part A of the SAT II: Writing Test was not used. Part A consists of an essay assignment. Test takers have twenty minutes to write an essay on an assigned topic. Two independent readers, who are experienced high school or college teachers, score these essays. Test takers receive a composite score “calculated by combining the multiple- choice score with a weighted writing sample score” (Inside SAT II: Subject tests, 1999, p. 5). The logistics and the subjective nature of the scoring made the use of Part A an impossibility for this study. However, this merits discussion. The following question arises: does technical knowledge of English grammar and usage necessarily translate to writing proficiency? In other words, is a multiple-choice test that measures technical skills (i.e., grammar, usage, diction (choice of words), and idiom; correctness and effectiveness of expression (inclusive of grammar, word choice, sentence construction, and punctuation); and organization and development relative to clarity and coherence) able to provide information about writing proficiency as it pertains to production of text when not combined with an essay test? It is hard to accept that graduate students are no more proficient at writing than college-bound seniors. Having completed an undergraduate education, students assuredly gained practice at writing, and by virtue of experience must surely possess some competence as writers (Biggs, et. al, 1999). However, perhaps they have not gained significant knowledge of the technical aspects of writing. If this is the case, then by administering Part A, of the SAT II: Writing Test, a different picture of graduate student writing proficiency might emerge. Generation of text is qualitatively a different task than proving knowledge of the technical aspects of writing on a multiple-choice test. To summarize, if both Parts A and B of the SAT II: Writing Test had been used with the sample population, and a composite score obtained for the study participants, then perhaps the sample population would have scored significantly higher than the norm base for writing proficiency. Interestingly, when asked what grade they had received on their most recent writing assignment, 88.5% of the study participants indicated that they had received an “A” (Singleton- Jackson, Lumsden, & Newsom, 2009). This information lends itself to three conclusions: 1) there is some plausibility to the idea that technical knowledge does not affect or relate to writing proficiency with regard to producing text (e.g., research papers, writing assignments), 2) professors are complaining about student writing, but perpetuating the problem by not giving low marks for poorly written papers, or 3) the reported data does not give an accurate depiction of the respondents typical writing assignment grade. However, it should be noted that when the composite score for Parts A and B of the SAT II: Writing Test is calculated, “the weighting makes the writing sample constitute only one-third of the combined score” (Inside SAT II: Subject tests, 1999, p. 5). While the SAT documentation does not provide an explicit reason for this weighting, from examining validity coefficients – correlations between the SAT II: Writing Test and freshmen writing performance – it would seem to be due to the fact that the multiple- choice portion of the test is a more valid predictor of actual writing performance. Therefore, it is in all likelihood, reasonable to assume that the SAT II: Writing Test, Part B alone provides an accurate picture of a test taker’s overall writing ability, viz. technical knowledge and ability to generate text. Implications for Future Research and Practice This study has spurred a number of ideas for future research studies in the area of writing proficiency among graduate students. Further, this study may prove to affect practice with regard to how writing is dealt with by graduate admissions committees, programs, and faculty with regard to preliminary assessments, curriculum, writing assignments, and thesis and dissertation supervision. An obvious idea for future research would be to replicate the study, but include graduate students from a wide variety of disciplines instead of exclusively studying students from Higher Education graduate courses. Also, obtaining a larger n of either exclusively Higher Education students, or graduate students from all disciplines could prove to be extremely interesting. This study used the SAT II: Writing Test, Part B to assess graduate student writing proficiency. There is also a Part A to this instrument which requires essay writing. The addition of Part A, an applied writing task, in future studies might yield a different picture of the writing proficiency of graduate students. A future study of this nature, regardless of choice of instrument(s), could be improved if the data collection environment was held consistent. It was discovered that the data were harder to collect than originally predicted. This gave rise to a change in procedure that necessitated data being collected from graduate students on an individual basis and in a classroom setting. A future study could plan for this contingency and include a plan to only collect data from proctored groups. While it did not affect the overreaching finding that the sample population did not score significantly higher on the SAT II: Writing Test, Part B than the norm group, the fact that the participants who did the test on an individual basis scored significantly higher than those who participated in a classroom setting gives pause. It would be important to find out if this was a chance finding particular to this study, or if the testing environment gives rise to performance differences. We might trust the scores of those who completed the SAT II in a class setting more than the scores of individual test-takers as being more representative of Higher Education graduate students’ proficiency indicators. Other variations might include a professor-focused study. Graduate school professors could be tested for writing proficiency. Additionally, it would be interesting to determine to what degree professors can actually articulate rules regarding grammar, usage, diction, idiom; correctness and effectiveness of expression; and organization and development relative to clarity and coherence. By doing this it might be possible to glean to what degree graduate faculty are prepared to actually teach writing skills to graduate students. It is one thing to grade for content, but an altogether different undertaking to teach writing as a part of a graduate course. A study along these lines could also include an attitudes survey that aims to discover graduate faculty attitudes toward teaching graduate students to write. A related idea could involve surveying professors about their opinions regarding the current quality of graduate student writing. Following the collection of these data, a follow-up questionnaire could be sent to the same professors asking about the grades they are typically giving on writing assignments. The purpose of this would be to follow up on the idea of inconsistency existing between graduate student writing proficiency and the grades received on writing assignments. With regard to implication for future practice, there are several possibilities for changes to graduate education. Graduate admissions committees may find it useful to more formally assess the writing proficiency of incoming graduate students, not necessarily as a tool to be used for denial of admission but to identify for students deficiencies in writing skills that need to be addressed with coursework or in conjunction with on-campus writing resources. Also, as discussed in the introduction, expectations that graduate students should possess writing proficiency exist, thus this study may provide an impetus for programs to put in place more proactive measures to help students attain the needed writing skills required for graduate level study. Required courses or mandatory workshops to address graduate student writing issues may be warranted should graduate programs feel that there are significant writing issues among their students. Graduate students might also use this study as a basis for exploring mentor-type programs wherein more expert graduate student writers provide direction, resources, and support for fellow students requiring writing assistance. Additionally, increased understanding of the reality of graduate student writing ability, based on assessments or other measures, could assist faculty as they guide students through theses and dissertations. The findings from this study indicated that the graduate students studied were no more proficient at writing than the typical college-bound, high school senior. The fact that these study participants all had at least a bachelor’s degree did not seem to result in them being better writers than students who had not yet matriculated. It could prove useful to follow this idea through graduate school to see to what degree if any a graduate education results in increased writing proficiency. A longitudinal study could be developed so that entering graduate students could be tested for writing proficiency. A follow-up study could then be done involving testing for writing proficiency at the end of the students’ course of study. This would allow for an evaluation of how much or if writing proficiency changes during the course of graduate study. Conclusion This study began with the idea that graduate student writing was a topic worthy of assessment. The findings of this exploratory study seem to confirm that initial belief. Stevenson (2006) echoes the importance of graduate student writing proficiency by asserting that there is no question that graduate students need to know how to write. We can conclude, however, that “[g]raduate students are novice researchers and writers who must be initiated into the culture of academic writing” (Mullen, 2006, p.30). This study tells us that there is reason to believe that being a graduate student does not equate with writing proficiency adequate for graduate level study. In his preface to Writing To Learn, William Zinsser indicates that he wrote this book “to try to ease two fears that American education seems to inflict on all of us in some form. One is the fear of writing” (The other being a fear of subjects that do not come easily or naturally) (Zinsser, 1988, p. vii).While fear of writing may be pervasive, it does not have to be continuous. In an attempt to initiate graduate students into the writing community of scholars, we can proceed with dispassionate study of the process and product of writing. By doing so we can discover information to reduce the stress of writing for graduate students, who are our future creators of reports, articles, chapters, books, and various other written documents that can educate, advance, and inspire both scholars and the populace. To reiterate a previous point, graduate school is a writing-intensive experience, justifiably so. Increasing understanding of writing as a critical skill for success in graduate school can reduce some of the fear that accompanies that intensity. As long as good writing remains a mystery that is accessible to only a chosen few, then fear of writing will persist. While academic standards and faculty expectations concerning writing proficiency should assuredly remain high, they need not be unattainable. If it is possible to make the road to scholarship less stressful by easing the fear of writing, both educators and students will likely have a greater appreciation of the experience of higher education. By ascertaining to what degree graduate students can write, faculty and administrators may glean a better understanding of how to help their students use writing to transition from student to scholar. References Abbate-Vaughn, J. (2007). The graduate writing challenge: A perspective from an urban teacher education program. Action In Teacher Education, 29, 51-60. Anderson, B., & Ekstrom R. (1994, September). A project to examine the graduate education admissions process and the role of GRE scores in this process: A report on phase I and a proposal for phase II (proposal to the Graduate Record Examination Board). Princeton, NJ : Educational Testing Service. Biggs, J., Lai, P., Tang, C., & Lavelle, E. (1999). Teaching writing to ESL graduate students: A model and an illustration. British Journal of Educational Psychology, 69, 293-306. Brown, R. (2004). Rhetoric in psychology personal statements. Written Communication, 21, 242- 260. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. Cohen, R.J., Swerdlik, M.F., & Phillips, S. (1996). Psychological testing and assessment: An introduction to tests and measures. New York: McGraw Hill. Cuthbert, D. & Spark, C. (2008). Getting a GRiP: Examining the outcomes of a pilot program to support graduate research students in writing for publication. Studies in Higher Education,33,77-88. Dervarics, C. (2006). Before you exit. Diverse Issues in Higher Education, 23(14). Retrieved November 10, 2008, from http://www.diverseeducation.com/ DiPierro, M. (2007) Excellence in doctoral education: Best practices. College Student Journal, 41, 368-388. Dunbar, S.B., Koretz, D.M., & Hoover, H.D. (1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4, 289-303. Educational Testing Service (1999-2000). Taking the SAT II: Subject Tests. Princeton, NJ; The College Board. ERIC Clearinghouse on Higher Education (August 10, 2000). [On-line]. Available: www.eriche.org/resource/public.html Flateby, T.L. (2005). Maximizing campus responsibility for the writing process. About Campus, January-February, 22-25. Gall, M.D., Borg, W.R., & Gall, J.P. (1996). Educational research: An introduction. White Plains, NY: Longman Publishers. Hadjioannou, X., Shelton-Rankie, N., Fu, D., & Dhanarattigannon, J. (2007). The road to a doctoral degree: Co-travelers through a perilous passage. College Student Journal, 41, 160-177. Handbook for the SAT® Program (1999-2000). Princeton, NJ; The College Board. Inside SAT® II: Subject Tests (1999). Princeton, NJ; The College Board. Hahs, D. L. (1998, November). Creating “good” graduate students: A model for success. Paper presented at the annual meeting of the Mid-South Educational Research Association, New Orleans, LA. Howell, D. C. (2002). Statistical methods for psychology (5th ed.). Belmont, CA: Wadsworth. Kamler, B. (2008). Rethinking doctoral publication practices: Writing from and beyond the thesis. Studies in Higher Education, 33, 283-294. Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences. Pacific Grove, CA: Brooks/Cole. Knudson, R. E., Zitzer-Comfort, C., Quirk, M., & Alexander, P. (2008). The California State University Early Assessment Program. The Clearing House,81, 227-31. Lavelle, E. & Bushrow, K. (2007) Writing approaches of graduate students. Educational Psychology, 27, 807-822. Levene, H. (1960). Robust tests for the equality of variance. In I. Olkin (ed.), Contributions to probability and statistics. Palo Alto, CA: Stanford University Press. Michigan State University, (1999). Improving proficiency in scientific writing: Graduate student writing groups. [On-line]. Available: http://writing.msu.edu/content/grad.html#groups. Retrieved May 2000. Mullen, C. (2006, Fall). Best writing Practices for graduate students: Reducing the discomfort of the blank screen. Kappa Delta Pi Record, 43 (1), 30-35. Newman, S. (2004). Tales of the professional imaginary: Personal statements for medical school at Johns Hopkins, 1925 to the present. Issues in Writing, 15, 31-55. North, S. M. (1996). Upper-division assessment and the postsecondary development of writing abilities. In E.M. White, W.D. Lutz, & S. Kamusikiri (Eds.), Assessment of writing: Politics, policies, practices (pp. 148-157). New York, NY: The Modern Language Association of America. Olson, D.R. (1977). From utterance to text: The bias of language in speech and writing. Harvard Educational Review, 47 (3), 257-281. Pascarella, E.T., & Terenzini, P.T. (1991). How college affects students. San Francisco: Jossey- Bass. Powers, D.E. & Fowles M.E. (1997). The personal statement as an indicator of writing skill: A cautionary note. Educational Assessment, 4 (1), 75-87. Powers, D.E., Fowles, M.E., & Willard, A.E. (1994). Direct assessment, direct validation? An example from the assessment of writing. Educational Assessment, 2 (1), 89-100. Samraj, B. & Monk, L. (2008). The statement of purpose in graduate program applications: Genre structure and disciplinary variation. English for Specific Purposes 27, 193–211. SAT® II: Subject Tests (2000). Princeton, NJ; The College Board. Scardamalia, M., & Bereiter, C. (1986). Research on written composition. In Handbook of research on teaching, 3rd ed., ed. M. C. Wittrock, 778-803. New York: Macmillan. Singleton-Jackson, J., Lumsden, D.B., & Newsom, R. (2009) [General writing experiences of graduate students]. Unpublished raw data. Sternberg, R.J. (1996). Successful intelligence. New York, NY: Simon & Schuster. Sternberg, R.J., Ferrari, M., Clinkenbeard, P., & Grigorenko, E.L. (1996). Identification, instruction, and assessment of gifted children: A construct validation of a triarchic model. Gifted Child Quarterly, 40, 129-137. Sternberg, R.J., & Williams, W.M. (1997). Does the graduate record examination predict meaningful success in the graduate training of psychologists? American Psychologist, 52 (6), 630-641. Stevens, J. P. (2002). Applied multivariate statistics for the social sciences (4th ed.). Mahwah, NJ: Lawrence Erlbaum. Stevenson, C. N. (2006). Writing, teaching of. In Encyclopedia of educational leadership and administration, vol. 2, ed. F. W. English, 1080-81. Thousand Oaks, CA: Sage. Torrance, M., Thomas, G.V., & Robinson, E.J. (1999). Individual differences in the writing behaviour of undergraduate students. British Journal of Educational Psychology, 69, 189-199. Wasley, P. (2008, July 11). Portfolios are replacing qualifying exams as a step on the road to dissertations. Chronicle of Higher Education, 54 (44), A8-A8. White, E.M. (1994). Teaching and assessing writing: Recent advances in understanding, evaluating, and improving student performance (2nd ed.). San Francisco,CA: Jossey-Bass Publishers. Zinsser W. (1988). Writing to learn. New York, NY: Harper & Row, Publishers.