Microsoft Word - Weaver-Barnard.docx Journal of Open, Flexible and Distance Learning, 23(1) 33 Source Use by Second-Year Psychology Students in Online Distance Learning Nancy Evans Weaver, Open Polytechnic of New Zealand Estelle Barnard, Open Polytechnic of New Zealand Abstract Literacy skills and behaviours of university-level students are of wide interest today. The research reported here fits in this tradition. It examines second-year psychology students’ source use in a course taught through online distance learning (ODL). The number of sources used increased across assignments. Most used were the textbook, instructor sources, and librarian-selected sources. The mark achieved increased with number of sources, instructor sources, and librarian-selected sources. Compared with previous research on first- year students, instructor and librarian-selected sources were frequently referenced by both sets of students, and source use related positively to their marks. This research expands the literature by comparing beginning and more advanced students; by replicating previous results; and by examining two assignments from ODL students. Keywords: citation analysis; source use; psychology students; online distance learning Overview and rationale This research is an attempt to further understand and measure the literacy skills and behaviours of university-level students, an important and ongoing concern in librarianship and the academic world today. As Napier, Parrott, Presley, and Valley (2018) point out, those working at the coal face—such as instructors and librarians—know very well that “a gap exists between where instructors would like their students to perform and where those students actually are” (p. 120) in terms of ability to find, interpret, and incorporate into writing the most relevant academic literature. Simply stated, university-level students need to understand information literacy if they are to achieve academically. The Association of College and Research Librarians (ACRL) noted in their framework that true information literacy involves “the set of integrated abilities encompassing . . . the discovery . . . , understanding . . . , and use of information . . . .” (American Library Association, 2015, Introduction, para. 5). One aim of the framework is to enable librarians, instructors, and other professionals to work together to “connect information literacy with student success” (ALA, 2015, Introduction, para. 6). The research reported here focuses on the use of information by students and its relation to student success in their course, and so it fits in this tradition. One tool that has been used to gauge student use of information is citation analysis. In this method, in-text citations (brief acknowledgements of sources in a paper at the point where information is used) and/or source lists (either bibliographies, which list all sources consulted; or references, which list all sources cited), are collected from written work (see Burton, 2010, for these definitions). Source variables (e.g., number of sources, type of sources, origin of sources) are then measured, and sometimes other variables are also considered (e.g., student or faculty author, before or after library instruction). The research described here uses citation analysis and Weaver, N. E., Barnard, E. 34 builds on earlier work by the same research team (Weaver & Barnard, 2015) to measure source use by psychology students at the Open Polytechnic of New Zealand (OPNZ). In that earlier work the researchers analysed the reference pages from two assignments (L1 and L2), both assignments of beginning-level psychology students, to determine their source use. These students all studied by distance, through print course materials and an active online campus platform. The number and type of sources used were analysed, and relationships between source use and the mark achieved were examined; differences between the first and second assignments were also noted. In the research described here, the focus changed to look at students at the next level of study. These are second-level students (equivalent to the middle year of university study in systems where a bachelor’s degree takes 3 years to complete) from the same psychology department; they were enrolled in a course for which the prerequisite is the course studied by these researchers in 2015. These students seemed, therefore, to be a natural focus for the next study—they are taking one possible step on from the introductory course that was researched initially. Again, the number and type of sources used in the two required assignments, the relationships between source use and the mark achieved, and differences between the first and second assignments were examined. This research expands the literature in three ways. It adds to the small number of studies of source use by students in online distance learning (ODL); it examines two successive assignments, again not often included; and, by calling on the 2015 findings, it allows a comparison of source use by students at two levels of study in the same programme and seeks to test whether those initial findings can be confirmed. The study had three aims: to investigate the number and types of sources used by these second-level students; to statistically test relationships between variables; and to determine whether source use differs between first-level and second- level students. Literature review Analysing student writing Citation analysis can be used to examine any type of writing which includes citations and/or source lists and, in, particular it has been widely used in research on student writing. A variety of studies published in early 2015 was examined in an earlier paper (see Weaver & Barnard, 2015, for details) and is recapped now for new readers before moving on to more recently published analyses of student writing and source use. In earlier studies, citations have been analysed to compare print and online materials (e.g., Knight-Davis & Sung, 2008; De Groote, 2008), to measure differences between student and faculty use of sources (e.g., Watson, 2010), and to evaluate effectiveness of library instruction (e.g., Clark & Chinburg, 2010; McClure, Cooke, & Carlin, 2011). Of most relevance, Ludovico and Wittig (2015) reported that first-year university students tend to “use resources which they perceive to be most scholarly” (p. 34). In their study, these were mostly books, followed by journal articles. Other researchers (e.g., Davis & Cohen, 2001; Carbery & Leahy, 2015) also found that students use a variety of resource types (books, journal articles, electronic resources, etc.). The sources students use can be affected by such variables as library training (e.g., Rafferty, 2013) and advice and recommendations from their instructors (Davis, 2003; McClure, Cooke, & Carlin, 2011). That last variable—the influence of the instructor—was also prominent in the results from Jones and Allen (2012), who examined student satisfaction and success using group discussions. They found that beginning students relied heavily on handouts from the Journal of Open, Flexible and Distance Learning, 23(1) 35 instructor, or instructor-assigned readings, and that more advanced students move on from there to use sources such as journal articles that they find for themselves. Building on these reports, the researchers’ 2015 study aimed to extend the literature by filling some gaps: Many published studies looked at only one writing example from each student so could not test whether source use changed with experience in the course; sample sizes were often quite small; and students studying by ODL were often not included in the study. In the researchers’ first study (Weaver & Barnard, 2015) two successive assignments were collected from students; a course with large enrolment was studied to provide a larger sample; and ODL students were the focus. Main results included that the most referenced sources for these beginning-level students were relevant to the assignment and provided by the instructor (echoing the findings of Davis, 2003; McClure et al., 2011, as well as what Jones & Allen, 2012, found in their discussion groups). Journal articles, the textbook, and the instructor’s assignment-relevant material were most popular. Source use did change across assignments, though, with the influence of the instructor seeming higher on the first assignment, while the use of library and other sources went up on the second assignment. It was also found that some source variables were related to marks: The number of sources used overall (in particular the number of instructor sources used), and the use of one type of librarian-selected source, all correlated positively with the mark the student achieved; that is, more of any of those linked to higher marks. Several relevant studies have been published since 2015. Datig (2016), who used a combination of focus groups and citation analysis to study source use by undergraduate social science students near the end of their studies, found that journal articles and books were widely used in completing essays. Especially interesting was her finding that “professors and other mentors” affected the sources students chose to use (p. 74). One of the participants in that study put it this way: “There is a kind of famous paper by this guy whose name I don’t remember but it’s really famous. He [the mentor] recommended I read it . . . .” (p. 75). Lantz, Insua, Armstrong, and Pho (2016) also looked at more than one assignment from each student. Their students, too, were in their first year of study (although in an English course rather than psychology). These researchers also used citation analysis, but they applied it to bibliographies rather than reference pages. They found heavy use of journal articles (especially in the first assignment), that more sources were used in the second assignment, and that use of some types of sources increased in the second assignment (in this instance, reference sources such as encyclopaedias, and websites). From outside academia, a marketing research project conducted by a textbook publisher is also relevant (SAGE Publishing, 2017). In a survey of 190 psychology students at a sample of United Kingdom universities, SAGE found that students are heavily influenced in the course resources they buy (and perhaps then go on to use in assignments) by recommendations from their lecturer; lecturers have a major influence on the resources students use that are not on the specific course reading list; and almost all students now access and use ebooks and other online resources. These recent results fit well with, or complement, the current researchers’ 2015 findings. They also suggest other ways that research might move forward: Many studies now examine web use and e-sources as well as print (e.g., Lantz et al., 2016), and many supplement citation analysis with more “ethnographic” approaches, such as having students maintain a diary of how they pursue research or by including observation of student research behaviour (Dunne, 2016). In the study reported in this paper, some of these issues were addressed while keeping some strengths from the 2015 methodology (e.g., examining two assignments from each student to allow analysis of any change with experience in the course). Weaver, N. E., Barnard, E. 36 Context and hypothesis of this study This research was undertaken at the Open Polytechnic of New Zealand (OPNZ), New Zealand’s largest ODL institution with some 30,000 students from across New Zealand and around the world. All courses have strong online support and enrichment, and some also provide print-based materials and require students to purchase a textbook. The library holds print and online sources, all library training is online, and librarians work with instructors to select and make resources available to students. Links to librarians are embedded on online course pages. Our research team consists of a librarian and a psychology lecturer. The team have worked together for more than 10 years, including on 73212 Thought, Memory & Language, the course studied here. It’s a second-level course focusing on cognitive psychology. The prerequisite for this course is 73195 General & Applied Psychology, which is the course studied earlier (Weaver & Barnard, 2015). All of the currently tested students had therefore successfully completed the course last examined. Further, because these 72212 students come from the population of 73195 students, they “look like the students studied earlier (although they now have more psychology and course writing experience). This group is therefore especially appropriate for further study. Like 73195, 73212 requires students to run experiments and write up results in two laboratory reports (L1 and L2) using American Psychological Association (APA) format. In 73212, students are reminded how to cite and reference, which is a skill they began to master in 73195 and other first-level courses. For each assignment 5% of the overall mark is allocated to citing and referencing. In all assignments, students were required to use at least three primary sources; in later offerings of the course that requirement went up to a minimum of five primary sources for the second assignment. (Students were also permitted to use their textbooks or other secondary sources, but those did not count towards the primary sources.) Three hypotheses were evaluated: 1. Any resource provided or recommended by the instructor would be frequently used. This hypothesis is based on the literature and on the 2015 study. As a corollary to this first hypothesis, it was expected that the pattern of use of instructor vs. library and other sources might change between L1 and L2, as it did in the 2015 study. 2. Assignments for which students used more sources would receive higher marks than assignments for which students used fewer sources. This seems logical because students who put in more work on an assignment (and one measure of this might be using more sources) would be expected to do better, and it also was part of the 2015 findings. 3. The use of sources from the instructor or the library (i.e., the sources provided or recommended by the academic professionals associated with the course) would be positively related to the mark achieved. As a corollary, it was expected that there would not be a relation to mark for other sources (i.e., those found some other way by the student). Again, these hypotheses reflect the 2015 results and the literature. Note: No specific hypotheses were made about differences between Level 5 students, as revealed in the earlier study, and the current Level 6 students. There is little literature that reports on differences over time or experience between comparable student groups so the comparison here was exploratory rather than designed to test specific hypotheses. It was of interest to test whether the 2015 results would be replicated here, or if these more advanced students would show a different pattern of resource use. For example, in 2015 the less advanced students made little use of resources they found themselves (i.e., sources other than those provided by or recommended by course instructors or librarians). Would this “other” source use go up among these more advanced students, who are probably more experienced in the use of the psychology literature? Would they make less use of instructor and librarian-selected sources? As well as testing Journal of Open, Flexible and Distance Learning, 23(1) 37 hypotheses, the researchers wanted to answer these questions by repeating the study—this time with Level 6 students. Method Participants Participants were recruited from OPNZ students enrolled in 73212 Thought, Memory & Language in the years 2013, 2014, 2015, 2016, and 2017. Based on previous research (Weaver & Barnard, 2015), this course was chosen for two reasons: (1) It is a higher-level course (OPNZ Level 6 is equivalent to second-year study at a university) than the beginning-level course (73195 General and Applied Psychology) studied previously; and (2) its prerequisite is 73195. These 73212 students therefore resemble those studied earlier, having themselves been students in that beginning level course at one time, and were considered to be a good comparison sample. Since the previous research was completed, OPNZ has reviewed its policy on researchers using student documents. In the 2015 study, the researchers were able to sample all submitted student assignments without specific permission from students. Because 73195 has a large number of enrolments, quite a large sample (n=124) of student work was available from a single course offering. Under the new policy, however, researchers had to ask students for permission to use their assignments: Students had to respond to an emailed request and specifically grant permission for the researchers to examine their work. As expected, only some students replied to the requests. In addition, as an upper-level course, 73212 enrols far fewer students than does the large introductory 73195. Overall, 69 students were enrolled in this course across the 5 years, and 55 of these submitted both assignments. All 55 were contacted, and 36 (65%) gave permission to analyse their assignments. Materials Materials for analysis were the reference pages from both assignments (L1 and L2) of these 36 students. All students submitted both assignments, and all of them chose to submit online. E- copies of the 72 reference pages were collected. Procedure The proposed research was reviewed and approved by the OPNZ Ethics Committee. Names and contact details of all students who submitted both assignments in the chosen years (n = 55) were obtained from course records. Each student was then emailed with information about the planned research and a consent form, and those who gave permission were asked to respond with a “yes” by email. Those who did not reply were emailed a second time, and telephone messages were left. Overall, 36 of the 55 students (65%) gave permission. Their reference pages were collected from the OPNZ online repository of student assignments and printed for ease of coding. The students’ marks were also collected. Analysis proceeded as described below. All participants were asked if they wished to receive the results at the study’s conclusion; those who so wished were sent a summary of the results. Weaver, N. E., Barnard, E. 38 Results Student identity was removed from the collected pages to protect anonymity and replaced with a code. Researchers saw only these codes during analysis. The following details were collected from each reference page. • Total number of sources referenced (possible range of values: whole numbers from 0 upward). • Whether or not the set textbook was referenced (0 or 1). • Whether or not the learning guide was referenced (0 or 1). (This guide is a set of printed and online course modules given to all enrolled students; it focused on specific course topics.) • Whether or not the writing guide was referenced (0 or 1). This guide is a required book that teaches APA writing style. • Number of additional instructor sources referenced (whole numbers from 0 upward). These are journal articles, textbook chapters, and other materials specifically recommended by the instructor. Most relate to a particular assignment and are linked on the online course page. Others are mentioned by the instructor in the learning guide, on the course page, or originate in another way from the instructor. • Number of librarian-selected sources referenced (whole numbers from 0 upward). These are books, articles, chapters, and other materials relevant to the course in general and chosen by the psychology librarian in consultation with the instructor. They are linked to the online course page and to the course on the library link. • Number of sources of other origin referenced (whole numbers from 0 upward). These are any other referenced sources excluding all the above (i.e., sources obtained in other ways by the student). Print sources were examined separately from online sources in this category. Descriptive statistics were calculated to summarise the data. Statistical tests were then run to evaluate specific hypotheses. All tests were evaluated at the .05 probability level. Journal of Open, Flexible and Distance Learning, 23(1) 39 Descriptive statistics of source use The mean and the range from lowest to highest (or the percentage of use for those variables with possible values of only 0 or 1) were calculated for L1 and L2 for all source types (see Table 1). Table 1 Source type and use in L1 and L2 Source variable Mean or % Range of values obtained Number of sources used L1 L2 Types of sources used Textbook L1 L2 Learning guide L1 L2 Writing guide L1 L2 Additional instructor sources L1 L2 Librarian-selected sources L1 L2 Other origin: Print L1 L2 Other origin: Online L1 L2 6.64 7.86 94% 83% 2.8% 2.8% 0% 0% 1.44 2.03 2.64 3.14 .53 .47 1.06 1.36 2 to 20 4 to 30 0 or 1 0 or 1 0 or 1 0 or 1 0 or 1 0 or 1 0 to 3 0 to 5 0 to 10 0 to 13 0 to 4 0 to 7 0 to 7 0 to 6 Analyses of sources used In the first assignment (L1), the mean number of sources used was 6.64. This increased to 7.86 for L2. For L1 the least number of sources used by a student was 2 and the most was 20. On L2 the least was 4 and the most was 30. (Median and mode were also calculated and showed no unexpected extremes: The median was 6 for L1 and 7 for L2; the mode was 7 for both assignments.) To test the significance of the difference between the means, a paired t-test was run on the data from the 36 participants. Because it was hypothesised that the L2 mean would be larger than the L1 mean, a one-tailed test was run. The difference was significant, t(35) = 2.27, p <.05. Students used significantly more sources for their second assignment than for their first assignment. The types of sources used were then analysed. Weaver, N. E., Barnard, E. 40 Instructor sources The first source type examined was sources that the researchers believed students would think of as originating from the instructor; that is, sources the instructor told them to purchase or use, or that the instructor provided or referred them to. This group includes the textbook; the learning guide; the writing guide; and other materials (e.g., journal articles relevant to a specific assignment) originating from the instructor, which are labelled “additional instructor sources” in Table 1. The results of the analysis are that some of these instructor sources were heavily referenced in the assignments, while others were not referenced at all. The textbook was mentioned by almost everyone: In L1, the textbook was referenced by 94% of students; in L2 it was also very frequently referenced (83%). The learning guide and the writing guide were much less frequently referenced: About 3% of students referenced the learning guide in L1 and in L2; no one referenced the writing guide in either assignment. Of most interest was the use of the additional instructor sources because these sources were used frequently in the 2015 research with beginning-level students (see next section for more comparison with the earlier study). In the current research, the number of additional instructor sources varied slightly in the years studied because the online course page was dynamic, the modules in the learning guide were edited and updated, and the instructor added and subtracted resources over the 5 years. As a rough summary, there were usually three specific sources available as links on the course page for each assignment, and a number of other sources were mentioned by the instructor in her online posts, in module-linked resources on the course page, and in the modules themselves. Any source that came from one of these origins was labelled an additional instructor resource. How often were these additional instructor sources used? Most students did reference these sources: Of the 36 students, 29 (81%) referenced at least one of them in L1, and 33 (92%) referenced at least one in L2. The mean number of additional instructor sources used in L1 was 1.44, ranging from 0 to 3 sources; the mean in L2 was 2.03, ranging from 0 to 5 sources. A one- tailed paired t test was run to see if the difference was significant, and it was: t(35) = 2.33, p < .05. More of the additional instructor sources were used for L2 than for L1. Librarian-selected sources Librarian-selected sources consisted of books, journal articles, and other resources collected by the OPNZ psychology librarian in consultation with the course instructor; they are relevant for the course but not necessarily for any particular assignment, and are described to students as “background” reading or “additional” material. These sources can be accessed from the online course page or via the general library link if the student searches by course code (73212). As with the instructor sources, this list is dynamic and changes over the years; a typical year from those sampled would have a list of about 18 sources. These librarian-selected sources were referenced by most students in both assignments. In L1, 32 of the 36 students (89%) referenced at least one of them, and 34 students (94%) referenced at least one in L2. The mean number of librarian-selected sources used in L1 was 2.64, ranging from 0 to 10 sources. The mean number of librarian-selected sources used in L2 was 3.14, ranging from 0 to 13. This difference is, however, not significant: t(35) = 1.125, p = .13. Sources of other origin Any source listed on a student’s reference page that did not clearly come from the instructor sources or the librarian-selected sources was categorised as “other”. Relatively few students used this type of source: For L1, 21 of the 36 students (58%) used another source; for L2, 27 of the 36 students (75%) did so. Journal of Open, Flexible and Distance Learning, 23(1) 41 Print and online sources were examined separately in this category. First, print and online sources were compared within each assignment. The result was that for both assignments students used fewer print sources of this type than online sources. In L1, a mean of .53 print sources were used (ranging from 0 to 4 sources) and 1.06 online sources (0 to 7). This difference was significant: t(35) = 2.22, p < .05. In L2, a mean of .47 print sources were used (0 to 7) and 1.36 online sources (0 to 6). This difference was also significant: t(35) = 3.33, p < .05. Second, L1 and L2 were compared within source type. For print sources, about the same number were used for the two assignments: In L1, the print mean was .53 (0 to 4), while in L2 the print mean was .47 (0 to 7). This difference is not significant. For online sources, somewhat more were used in the second assignment than in the first assignment: In L1, the online mean was 1.06 (0 to 7), while it was 1.36 (0 to 6) in L2, but this difference is also not significant. Analyses of sources used: Comparison with earlier data In this section the new Level 6 data is compared with the data collected earlier on Level 5 students. (For full details of the previous research, see Weaver & Barnard, 2015.) The first analysis was of number of sources used. Level 6 students used a mean of 6.64 sources for L1 and a mean of 7.86 sources for L2, which was a significant increase. In the previous study of Level 5 students, the same pattern was observed, and it was again significant, but these beginning Psychology students used fewer sources for both assignments than did the Level 6 students: a mean of 3.62 sources for L1 and 5.34 sources for L2. Types of sources used (instructor sources, librarian-selected sources, other sources) by the two groups of students were then compared. Instructor sources Most of the earlier students used the textbook for L1 and L2 (59% and 79%), but the rate of use was higher for the Level 6 students in the current study (94% and 83%). Both groups of students used the learning and writing guides very infrequently (1% to 16% for Level 5 students, 0% to 2.8% for Level 6). Overall, the textbook is a popular source to reference by a clear majority of students regardless of level of study, while neither the learning guide nor the writing guide is a well-used source. For additional instructor sources, the closest comparable resource to those analysed for these Level 6 students is a set of journal articles provided to the Level 5 students by their instructors. In the earlier research these instructor sources were well used, especially in the first assignment. The most popular of the three articles was referenced by 83% of Level 5 students in L1 and 43% in L2. In the current analysis of Level 6 students’ assignments, 81% referenced an additional instructor source for L1, and 92% for L2. So all students do use these additional instructor sources frequently, but Level 5 students tended to decrease their use in the second assignment while the Level 6 students tended to increase their use. Librarian-selected sources The mean number of librarian-selected sources used by the Level 6 students was 2.64 in L1 and 3.14 in L2; there was a trend to higher use in the second assignment but it was not significant. Level 5 students used librarian-selected sources less frequently (an average across the two types of .055 used on L1 and .57 L2) but, again, the trend was for an increased use for L2. Other sources Relatively fewer Level 6 students used “other” origin sources than used sources from the instructor or the library. Within an assignment, students used significantly fewer print than online Weaver, N. E., Barnard, E. 42 sources of this “other” type. Across the assignments, print sources used were about the same for L1 but there was a trend towards using more online sources than print in L2. The research on Level 5 students did not separate print from online “other” sources. Combined, relatively few students used other sources but more did so for L2 (means were .38 on L1 and 1.72 for L2). Analyses of marks For the first analysis of marks, the mean mark and range of marks for each assignment were calculated for the Level 6 students. On L1 the mean mark achieved was 71.39 out of 100 possible marks (a letter grade of B+ on the OPNZ scale), and scores ranged from 54 to 92. On L2 the mean mark achieved was 75.42 (A-), with a range from 55 to 92. The increase was small but significant: In a one-tailed paired t test, t(35) = 3.04, p < .05. Most students therefore increased their mark when they moved on to L2. Variables that might be related to marks were then analysed. Overall number of sources as related to marks On both L1 and L2 there was a positive relationship between number of sources on the reference page and the mark achieved. For L1, R = .27, p = .059. For L2, R =.35, p < .05. Using a significance level of .05, assignments with more sources tended to get higher marks on both L1 and L2, significantly so on L2. Type of source as related to marks In the instructor sources, almost all students referenced the textbook and almost no students referenced the learning guide or the writing guide, so no further testing was done. For the additional instructor sources, on the first assignment, R = .29, p < .05, so there was a significant positive relationship. On the second assignment, R = .24, p = .07. To summarise, using more of the additional instructor sources did relate positively to marks, but significantly so only on L1. Did the use of librarian-selected sources relate to marks? As with the additional instructor sources, the answer is “yes” but significantly so for only one of the assignments: For L1, R = .26, p = .059, while for L2, the relationship was stronger, with R = .39, p < .05. Using more librarian- selected sources did, therefore, relate positively to marks, but this was significant on only the second assignment. Did the use of sources from some other origin relate to marks? The use of print sources was analysed separately from the use of online sources. For print, there was no significant relationship on either assignment. For online, the result was the same, with no significant relationship to marks on either assignment. Analyses of mark achieved: Comparison with earlier data Open Polytechnic Level 6 students achieved a mean mark on L1 of 71.39 (54 to 92), while the mean mark for L1 for the Level 5 students was 64.47 (18 to 91). On L2, the mean for Level 6 was 75.42 (55 to 92) while the mean for Level 5 was 62.06 (17 to 91). Overall, the more advanced students tended to do better. When tested statistically, the drop from L1 to L2 was small but significant for Level 5 students; the increase from L1 to L2 for the Level 6 students was also significant. Journal of Open, Flexible and Distance Learning, 23(1) 43 Overall number of sources as related to marks On both L1 and L2—for both sets of students—there tended to be a small positive correlation between number of sources on the reference page and the mark achieved. For the Level 6 students, this positive relationship was significant for the second assignment; for the Level 5 students it was significant for both assignments (at the .05 level). Type of source as related to marks Was the use of particular types of sources related to marks? For the Level 6 students, use of the additional instructor sources tended to relate positively to marks, significantly so for the L2 assignment. For the Level 5 students, more use of instructor-supplied journal articles tended to relate to a higher mark achieved, significantly so for the L1 assignment. For librarian-selected sources, more sources used related to higher marks, significantly so on L2, for both the Level 6 and the Level 5 students. For both L1 and L2 and for both groups of students, there was no significant relationship between the use of other sources and the marks. Comparison across years Finally, note needs to be made that in the analyses reported above for the Level 6 students, the researchers summed across 5 years of the course. Numbers enrolled each year were low, and reduced further when data could be collected from only those students who gave permission. Hence, analysis for any individual year would have been based on very few individuals. Basing conclusions on just a few participants seemed more problematic than summing across years. In addition, while aspects of the course did change over the years, assignments stayed much the same. Many of the sources provided to students also stayed the same, so it seemed reasonable to combine years. Still, it is good to do at least a cursory examination of possible differences across years. Summing across years, students used more sources on L1 (mean = 6.64) than on L2 (mean = 7.86). Was this true of all the years? This pattern did hold for 4 of the 5 years tested. What types of sources were used? Almost all students from all years (34 of the 36 students) used the textbook; almost no students used the learning guide (1 of 36 students) or the writing guide (0 of 36 students). Summed across the years, students used more of the additional instructor sources for L2 (mean = 2.03) than L1 (mean = 1.44). Year by year, that same pattern of a higher mean for L2 than L1 held for 4 of the 5 years. Was there more difference across years for the other two types of sources? No. More of the librarian-selected sources were used for L2 than L1 (means of 3.14 vs. 2.64) when summed across years, and that pattern held for 4 of the 5 years tested. For the other sources, summed across years, about the same number were used on both assignments for print sources but somewhat more used for L2 than L1 for online sources. Year by year, print use varied on L1 compared with L2, but online use did go up for L2 compared with L1 in 4 of the 5 years. As to marks, the overall mean mark across years was 71.4 on L1 and 75.4 on L2. For L1, all years were within 4.9 marks of that overall mean. For L2, all years were within 6.2 marks of the overall L2 mean. In 4 of the 5 years, L2 marks were higher than L1 marks. Weaver, N. E., Barnard, E. 44 Discussion Data summary and evaluation of hypotheses What sources do Level 6 students studying psychology through ODL use in their assignments, and how does source use relate to their marks? The most used sources came from the course experts (i.e., the instructor and the library), including the textbook that instructors encouraged students to buy, additional resources provided by the instructor, and the course-related library resources. Students used sources they found for themselves less often. As for marks, there was a significant increase in overall marks from L1 to L2. The mark achieved went up with the overall number of sources referenced, the number of additional instructor sources referenced, and the number of librarian-selected sources referenced, although this change was not always significant for both assignments. The use of sources the students found in other places was not significantly related to their marks. The data across the 5 years of course offerings were quite consistent, with all major measures repeated in at least 4 of the 5 years. These data generally support the three hypotheses. First, students did often reference recommended sources from the instructor and from the library; they were less likely to reference sources they found elsewhere. Second, the number of sources referenced related positively to marks, significantly so in one of the assignments. Third, in particular, the number of instructor and library-recommended sources related positively to marks for at least one assignment, while the number of other sources the students found did not relate to their marks. No specific hypotheses had been formed about how these Level 6 students would compare with the Level 5 students tested earlier, but an important goal of this research was to allow that comparison. The result overall was that while Level 5 students tended to use fewer sources and receive lower marks than the Level 6 students, their patterns of source use were quite similar. Students studied across the two levels frequently referenced sources recommended or provided by their instructors; all used at least some types of librarian-selected sources quite often; there was less use of sources they found in other places; both groups of students used more sources on L2 than on L1. As to relation to mark achieved, again the pattern was similar across both groups of students: Students who used more sources tended to achieve higher marks; those who specifically used more of particular instructor and librarian-selected sources tended to achieve higher marks; the use of other sources found by the student did not relate to marks. There were some other interesting results. First, even these Level 6 students made heavy use of their textbook. Why would even these more advanced students reference their textbook so often when it is a secondary source and they know the assignment emphasised using primary sources? One possible reason is that textbooks are expensive in New Zealand, so a student who purchases one might want to use it as much as possible. In fact, most of our students do purchase the textbook. In this time of easy (and often free) online access, it might seem surprising that students would purchase an expensive text. But, remember first that OPNZ students study by distance and are located all around the country and sometimes overseas, so they are unlikely to meet classmates face to face and so cannot share books easily. Second, OPNZ Library policy is that textbooks are never loaned to students. Could the textbook be obtained elsewhere without buying? Some data were collected to try to answer this question. Over the years tested in this research, two editions of the textbook were published. The 2013 edition is not held by any public library in New Zealand and only by three polytechnics/universities, from which students can borrow only if they are enrolled at those institutions. It is not available as an ebook. The 2016 edition is held in print at only one New Zealand university; as an ebook it is available only for enrolled students at that university. Older editions can be downloaded (illegally) but the editions used in the course for the years tested are harder to obtain in New Zealand. Journal of Open, Flexible and Distance Learning, 23(1) 45 A second interesting finding is the occurrence of differences between L1 and L2. Many of these differences seem to make sense: The overall number of sources used increased for the second assignment; the use of particularly relevant instructor sources and librarian-selected sources increased; the use of the textbook decreased. Except for the textbook (which went down in their L2), those patterns also held for Level 5 students in the earlier study. Finally, it was a pleasant surprise to find that many of the tested results achieved statistical significance. Because of OPNZ’s requirement that students must give permission before their assignments are analysed (whereas in the earlier research free sampling of all submitted assignments was permitted), it was expected there would be far fewer assignments to study and that this might affect significance levels. But, in fact, there was quite a high return rate (65%), and as reported in Results, many findings achieved significance. Almost all results that did not achieve significance were at least in the predicted direction. It is acknowledged that requesting student permission to study their assignments is a good protection of participant rights and, in this research at least, it did not unduly affect significance. The findings reported here relate well to previous research. First, they replicate for Level 6 students much of what was found for Level 5 students (Weaver & Barnard, 2015). Both sets of students frequently referenced sources from course experts (the instructor, the library), more so than they referenced other sources they found for themselves. Use of sources in general and of most of those from the experts went up for L2 as compared with L1. The number of sources used overall and the number of sources used from these experts related positively to marks, while the use of other sources did not. These results also replicate those from other researchers. Jones and Allen (2012) found “an emphasis on the using of sources that were provided by the tutor” (p. 102) in their study of psychology students. Similarly, Davis and colleagues (e.g., Davis & Cohen, 2001; Davis, 2003) also found the instructor influenced the number and quality of sources cited. Lantz et al. (2016) found that up to 30% of students consulted their textbooks (in a course where, like the OPNZ course studied here, the instructor encouraged students to use the required texts). Librarian- selected sources, in particular those relevant to the specific course or assignment, were often referenced by OPNZ students and by those studied by other researchers: Rafferty (2013) found that the most-cited sources related to the specific course. In case studies that included psychology students, Calkins and Kelley (2007), echo other researchers in noting that “many undergraduates are unable to discriminate between credible and noncredible sources” (p. 151). This isn’t a surprise, but it is a possible reason why students rely so heavily on materials that are recommended or provided by experts. The increase in the use of sources, and in particular the use of additional instructor sources and librarian-selected sources, may have occurred for several reasons. Pragmatically, students were told in the assignment to use more sources for their second report so, indeed, you would expect that to happen. Secondly, students should gain from the experience of information search and source use in L1, and from the feedback given on that first report when it is returned to them. (Indeed, one of the authors, who is the course instructor and marks some of the first assignments, often writes specific encouragement to increase source use on the second assignment.) The increased number of sources may also reflect students’ growing “scholarliness” and obligation to use more sources to develop that “scholarliness”. In fact, Jones and Allen (2012) report that their psychology students realise that expectations increase as they progress in their studies. As one of their students put it, “They expect a lot more referencing, a lot more books used, and you can’t just use one book and get all your information from there you have to look for journals, books, loads” (unnamed informant quoted by Jones & Allen, 2012, p. 105). Lantz et al. (2016) also found increased use of at least some types of sources on the second assignment, echoing the current result. Weaver, N. E., Barnard, E. 46 Limitations, implications and future research There are, of course, cautions and limitations to this research. Data were collected from students in only one course, in one discipline, at one institution, and all students were using a combination of print and online resources to study by distance. In addition, researchers were able to study only those assignments from students who gave permission. It could be suggested that this sample favours students who do well in the course or who are at least interested enough to continue their involvement with it. For most of the analyses, data were summed across 5 years of the course. While the researchers believe this does fairly reflect most years, discrepancies may have been overlooked. What are the implications of this research for instructors and for librarians? In earlier research with beginning-level students (Weaver & Barnard, 2015), it was found that instructor recommendations of sources seemed to be most influential on source use early in the course, with librarian-selected sources increasing in frequency of use later. The researchers suggested then that instructors might help most at the start of a course, when students seemed to be relying on their recommendations, and that librarians might intervene a bit later in a course, after the first assignment, when students seemed more willing to move beyond instructor recommendations. In the present study with more advanced students, there was more frequent use of library resources for both assignments than had occurred by the beginning students, and instructor sources were also well used for both assignments. The overall number of sources used was also higher for the more advanced students. Clearly, students are making more use of the librarian-selected sources (as well as the instructor sources) by the time they reach the second assignment—perhaps they are encouraged by feedback received on the first assignment or by being more familiar with the resources later in the course. The recommendation now might be that students should be helped earlier to move beyond the sources provided to find more of their own sources—this may be the focus needed for instructor and librarian intervention with these more advanced students. A second implication of this research, both here with the Level 6 students and earlier with Level 5 students, is that the use of sources beyond those from instructors or the library seems to have less effect on the mark achieved. This result is difficult to interpret because the number of such other sources used was so small (i.e., no effect may have been obvious because so little data could be analysed). Perhaps that very fact—that the students studied here reference few sources beyond those recommended by the instructor and the library—is the most important aspect of this finding. This may lead future course leaders to modify the assignment to clarify that students should find more of their own sources (at least for the second assignment), and it may lead the instructor and the librarian to work together to help students take that next step towards scholarly behaviour and scholarly writing. In 2015 the researchers proposed to continue their work by analysing a different course and, in particular, a higher-level course. That is the research reported here. They also suggested that they might then go on changing their method, moving from citation analysis to a more qualitative approach. This change of tactic is echoed in several recent studies by other researchers: Dunne (2016) used “ethnographic” research that followed five undergraduates in their “research journey” by having them record a diary of their information-seeking behaviour, by observing them as they researched, and by conducting post-research interviews. Datig (2016) combined citation analysis with focus groups to examine research by advanced undergraduates. Journal of Open, Flexible and Distance Learning, 23(1) 47 There are interesting questions to be studied in what might be called the cognition of source use. How do students think about an assignment and then decide how to find sources? Are they influenced not just by instructors and librarians but also by practicalities such as how long an article or chapter is, or how easy it is to find or to read? When they find a source for themselves, how do they determine whether it is of a high enough academic quality and whether it is truly relevant to the assignment? Interviews, questionnaires or other ethnographic methods seem to be the best strategies to begin answering these questions. Conclusion After researchers completed an earlier study of Level 5 psychology students, two assignments were collected from students at the next level of study. The number and type of sources referenced and the marks achieved were examined, and three hypotheses were evaluated: that students would use recommended sources from the course professionals, probably more often than sources they found elsewhere; that students who used more sources overall would achieve higher marks; and that students who used more instructor and more librarian-selected sources (but probably not sources that they found elsewhere) would tend to achieve higher marks. All these hypotheses were supported on at least one assignment of the two studied. Current results were also compared with those obtained earlier from beginning-level students. There were many similarities: in the type of sources used most frequently (preference for sources from instructors and the library over those found elsewhere), in the change from the first to the second assignment (increasing source use), and in the relationships between source use and marks (more sources related to higher marks; more sources from instructor and library related to higher marks). Comparisons were made with the literature on citation analysis and student writing, some implications for instructors and for librarians were considered, and ideas for future research were suggested. Acknowledgement We thank our managers and colleagues for their support, Dr. Zlatko Kovacic and Dr. Lincoln Hely for statistics advice, and our students for their enthusiasm and effort in completing their course assignments. Special thanks to those students who gave permission for us to analyse their course work in this research project. References American Library Association. (2015). Framework for information literacy for higher education. Retrieved from http://www.ala.org/acrl/standards/ilframework Burton, L. J. (2010). An interactive approach to writing essays and research reports in psychology (3rd ed.). Milton, Australia: Wiley. Calkins, S., & Kelley, M. R. (2007). Evaluating internet and scholarly sources across the disciplines: Two case studies. College Teaching, 55(4), 151–156. https://doi.org/10.3200/CTCH.55.4.151-156 Carbery, A., & Leahy, S. (2015). Evidence-based instruction: Assessing student work using rubrics and citation analysis to inform instructional design. Journal of Information Literacy, 9(1), 74–90. https://doi.org/10.11645/9.1.1980 Clark, S., & Chinburg, S. (2010). Research performance in undergraduates receiving face to face versus online library instruction: A citation analysis. Journal of Library Administration, 50, 530–542. https://doi.org/10.1080/01930826.2010.488599 Weaver, N. E., Barnard, E. 48 Datig, I. (2016). Citation behavior of advanced undergraduate students in the social sciences: A mixed-method approach. Behavioral & Social Sciences Librarian, 35(2), 64–80. https://doi.org/10.1080/01639269.2016.1214559 Davis, P. M. (2003). Effect of the web on undergraduate citation behaviour: Guiding student scholarship in a networked age. Portal: Libraries and the Academy, 3(1), 41–51. https://doi.org/10.1353/pla.2003.0005 Davis, P. M., & Cohen, S. A. (2001). The effect of the web on undergraduate citation behavior 1996–1999. Journal of the American Society for Information Science and Technology, 52(4), 309–314. Retrieved from http://hdl.handle.net/1813/2557 De Groote, S. L. (2008). Citation patterns of online and print journals in the digital age. Journal of the Medical Library Association, 96(4), 362–369. https://doi.org/10.3163/1536- 5050.96.4.012 Dunne, S. (2016). How do they research? An ethnographic study of final year undergraduate research behaviour in an Irish university. New Review of Academic Librarianship, 22(4), 410– 429. https://doi.org/10.1080/13614533.2016.1168747 Jones, S., & Allen, J. (2012). Evaluating psychology students’ library skills and experiences. Psychology Teaching Review, 18(2), 94–115. Retrieved from http://eric.ed.gov/?id=EJ991414 Knight-Davis, S., & Sung, J. S. (2008). Analysis of citations in undergraduate papers. College & Research Libraries, 69(5), 447–458. https://doi.org/10.5860/crl.69.5.447 Lantz, C., Insua, G. M., Armstrong, A. R., & Pho, A. (2016). Student bibliographies: Charting research skills over time. Reference Services Review, 44(3), 253–265. https://doi.org/10.1108/RSR-12-2015-0053 Ludovico, C., & Wittig, C. (2015). A universe of information, one citation at a time: How students engage with scholarly sources. Journal of Library & Information Services in Distance Learning, 9, 30–39. https://doi.org/10.1080/1533290X.2014.946343 McClure, R., Cooke, R., & Carlin, A. (2011). The search for the skunk ape: Studying the impact of an online information literacy tutorial on student writing. Journal of Information Literacy, 5(2), 26–45. https://doi.org/10.11645/5.2.1638 Napier, T., Parrott, J., Presley, E., & Valley, L. (2018). A collaborative, trilateral approach to bridging the information literacy gap in student writing. College & Research Libraries, 79(1), 120–145. https://doi.org/10.5860/crl.79.1.120 Rafferty, R. S. (2013). The impact of library instruction: Do first-year medical students use library resources specifically highlighted during instructional sessions? Journal of the Medical Library Association, 101(3), 213–217. https://doi.org/10.3163/1536-5050.101.3.011 SAGE Publishing. (2017). Student marketing research project 2017, produced by BAM Agency (Technical Report). London, England: Author. Watson, A. P. (2010). Tips for conducting citation analysis in an academic setting. Mississippi Libraries, 74(1), 14–16. Retrieved from http://www.misslib.org/Resources/Documents/MLarchive/ML2010Spring.pdf Journal of Open, Flexible and Distance Learning, 23(1) 49 Weaver, N. E., & Barnard, E. (2015). A citation analysis of psychology students’ use of sources in online distance learning. Journal of Library & Information Services in Distance Learning, 9(4), 312–329. https://doi.org/10.1080/1533290X.2015.1127865 Biographical notes Nancy Evans Weaver Contact via Estelle Barnard: Estelle.Barnard@openpolytechnic.ac.nz Nancy is a cognitive psychologist, recently retired as Principal Lecturer in Psychology at the Open Polytechnic of New Zealand. She has worked in face-to-face and distance learning for 40 years in colleges, polytechnics and universities in the United States and New Zealand. Nancy is a member of the American Psychological Association. Her teaching and research interests centre on memory and knowledge representation, perception, applied cognition and ergonomics, and the psychological aspects of grief. Estelle Barnard estelle.barnard@openpolytechnic.ac.nz Estelle is a librarian at the Open Polytechnic of New Zealand and has been a librarian in various roles over the years. Her interests are providing online learning support, information literacy instruction and research assistance to Faculty. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. Weaver, N. E., & Barnard, E. (2019). Source use by second-year psychology students in online distance learning. Journal of Open, Flexible and Distance Learning, 23(1), [33–49.].