Literature Review Journal of Learning Development in Higher Education ISSN: 1759-667X Issue 6: November 2013 Developing criteria to assess graduate attributes in students’ work for their disciplines Kate Chanock La Trobe University, Australia Abstract After two decades, efforts to integrate the development and assessment of ‘graduate attributes’ into discipline curricula remain slow, uneven, and fraught with difficulties. Scholars have identified political, cultural and practical reasons for academics’ resistance to this requirement, including ‘lack of ownership and shared understanding of how to teach and assess graduate attributes’ (Radloff et al., 2008). Along with Barrie (2007) and de la Harpe and David (2012), Radloff et al. (2008) have argued that ‘academic staff beliefs are critical and fundamental to any attempts at developing students’ graduate attributes’. This article suggests that, rather than trying to change these beliefs via top-down mandates to adopt institutional attributes, it may make sense instead to start from academics’ beliefs and see what attributes they suggest are actually integral to their cultures of enquiry. I reflect on such a process in the context of developing criteria and standards for assessing graduate ‘capabilities’ across the three years of a BA degree, in which a Faculty working party tried to tease out what we meant by ‘good writing’ into easily applicable criteria with authentic meaning(s) across our varied disciplines. Critical thinking proved fundamental to the meanings identified. Keywords: graduate attributes; generic skills; integrated; assessment; criteria; standards; critical thinking. Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines Introduction More than two decades after universities in Australia were persuaded to add the development of ‘graduate attributes’ (or ‘skills’ or ‘capabilities’) to their educational mission, scholars who monitor this process continue to find that it is slow, uneven, and fraught with difficulties (Crebert, 2002; Radloff et al., 2008; Green et al., 2009; Jones, 2009a; Hughes and Barrie, 2010). Although graduate attributes have been defined as ‘the qualities, skills and understandings a university community agrees its students should develop during their time with the institution’ (Bowden et al., 2000), the creation of policies and regimes of compliance needed to secure that agreement suggest scepticism rather than consensus among the ‘university community’. Scholars have identified political, cultural and practical reasons for academics’ resistance to teaching and assessing graduate attributes, including ‘lack of ownership and shared understanding of how to teach and assess graduate attributes’ (Radloff et al., 2008). In concert with Barrie (2007) and de la Harpe and David (2012), Radloff et al. (2008) argue that ‘academic staff beliefs are critical and fundamental to any attempts at developing students’ graduate attributes, since academic staff are the custodians of the curriculum and the ones who determine what is taught and assessed’. I would like to suggest that, rather than trying to change these beliefs by integrating attributes designated by ‘institutional senior leaders, government and industry’ (de la Harpe and David, 2012, p.494), it may make sense instead to start from academics’ beliefs and see what attributes they suggest are actually integral to their cultures of enquiry. In this article, I will reflect on such a process in a Faculty of Humanities and Social Sciences in the context of developing criteria and standards for assessing graduate ‘capabilities’ across the three years of the BA degree. For this purpose, a working group was formed including lecturers from different disciplines, an Academic Developer, and an Academic Language and Learning adviser (myself), to try to tease out what we meant by ‘good writing’ into easily applicable criteria with ‘authenticity to the practices of a discourse community’ (Hughes and Barrie, 2010, p.331). This article will consider what authenticity may mean in this regard, and how general such criteria can be without sacrificing authenticity. Journal of Learning Development in Higher Education, Issue 6: November 2013 2 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines Ownership and authenticity The problem of ‘ownership’ to which scholars have alluded has both political and intellectual dimensions. Jones (2009a, p.186) suggests that ‘one explanation for the degree of resistance to the notion of generic attributes is that they have become associated with a managerialist culture of audit and control’, and this seems to have been a factor also in Crebert’s (2002) experience at Griffith University and that of Lowe and Marshall (2004) at Murdoch. Similarly, James et al. (2004, Online) observed of the University of Wollongong that ‘for some, the concept of graduate attributes has been taken on reluctantly and suspiciously as the unwanted child of a union among business, government, and education sectors’. Moreover, because graduate attributes come from outside the discipline communities, ‘they can be viewed as…generalised statements without substance’; Jones (2009a, p. 186) was told that ‘lists of generic attributes were added to subject outlines by some academic staff in response to departmental policy rather than because they were seen as having intrinsic value’. This seems to chime with the findings of the National Graduate Attributes Project (GAP, 2013), that compliance was often confined to mapping where graduate attributes were taught and assessed in the course of a degree, rather than seeking to develop these further (Hughes and Barrie, 2010, p.329). Academic developers, nonetheless, seem optimistic that this is a problem of clarity rather than substance (Crebert, 2002, p.4; de la Harpe and David, 2012). ‘Most important’, in de la Harpe and David’s (2012, p.505) view, ‘is the need to provide opportunities for academic staff to unpack graduate attributes so that they can better understand them’ (c.f. Lowe and Marshall, 2004). Understanding, it is generally thought, would bridge the gap between lecturers’ beliefs that graduate attributes are important, and their neglect of these in their teaching (Jones, 2009a; de la Harpe and David, 2012, p.493). Differences in meaning However, it seems that the terminology of graduate attributes is not just unclear but polysemous. Since the 1980s, investigations into the cultures of enquiry of a range of disciplines, looking at the assumptions and values underlying their work, have revealed Journal of Learning Development in Higher Education, Issue 6: November 2013 3 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines significant (but often tacit) differences in their epistemologies (e.g. Bazerman, 1981; Jones, 2009a; 2009b; Moore, 2011). Although they share a common set of terms such as ‘problem’, ‘argument’, ‘evidence’, and ‘critical thinking’, whose meanings overlap from discipline to discipline, this commonality can conceal differences in their beliefs about the status of knowledge (given or constructed, certain or contested); the scope of criticism (confined to issues of logic, or extending to issues of ethics and even, possibly, taste); what qualifies a point as relevant; what counts as evidence; and what ought to be included in ‘context’. Anna Jones, whose doctoral research was focused on such differences, has found that ‘generic skills or attributes’ are not ‘a set of discrete, measurable, technical skills that transcend disciplines’ (2009b, p.93), but instead ‘are highly context-dependent, and are shaped by the disciplinary epistemology in which they are conceptualised and taught’ (2009b, pp.85–6). Jones found, in particular, that the meaning of ‘problem solving’ or ‘critical thinking’ varied across disciplines she studied. Law lecturers understood critical thinking as ‘the examination of an argument, its evidence and logic; the examination of assumptions; a discussion of the social context of an issue, an awareness of ethical issues; and a questioning of received wisdom’ and for them, ‘problem solving is the practical application of critical thinking, often to ‘real world’ problems and so is particularly concerned with outcomes’ (Jones, 2009a, p.181). In medicine, though lecturers thought that critical thinking was important, they worried that it could ‘destroy students’ idealism or their sense of agency’ in clinical situations and expose students to ‘nasty’ rebukes for challenging a supervisor (Jones, 2009a, p.185). In physics, lecturers saw the content taught to undergraduates as ‘fairly incontestable’, so students were not required to be critical until fourth year or later (Jones, 2009a, p.181-182). By contrast, for lecturers in history, ‘critical thinking, analysis and communication…are an integral and central part of the epistemology of the discipline…and mastering them is much of what it means to be a historian’; in this discipline, ‘critical thinking is understood as an examination of evidence, the ability to understand complexity and ambiguity, an awareness of political and ideological dimensions and a questioning of received wisdom’ (Jones, 2009a, p.179). At the same time, however, when their institution introduced critical thinking and communication as generic attributes, historians resisted the pressure to teach them explicitly. ‘They are given rather cursory attention’, Jones (2009a, p.179) tells us, ‘since they are viewed as bolted on to subject outlines in a way that demonstrates lip service Journal of Learning Development in Higher Education, Issue 6: November 2013 4 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines rather than involving any substantive acknowledgement that these attributes are part of the discipline itself’. Across the disciplines she studied, in fact, Jones (2009a, p.186) found that ‘when academic staff consider the nature of their discipline and their own research practices, they consider attributes such as critical thinking, analysis, problem solving and communication to be central. However, in their teaching, generic attributes are often viewed as separate from the central business of the discipline’. Although this discrepancy seems bizarre, it may be understandable in light of the artificiality that characterises uncontextualised ‘practice’ of generic skills. For example, critical thinking may be presented generically in terms of syllogisms and/or common logical fallacies. These have little relevance for history, say, or English literature, but this is not because these disciplines do not practise critical thinking. Lecturers in humanities and social sciences are already assessing critical thinking every time they mark a substantial assignment, for many assignments are set for the purpose of engaging students in thinking critically: about themselves, their textual practices, their professional practices, the world, or all of these. In addressing these assignments, students must use critical thinking to identify what the question is about, what problems are entailed in answering it, what sources are relevant and reliable, what evidence is most compelling, and how it can be shaped into a reasoned text. The students’ decisions about all these things are embodied in the work they submit, and are always evaluated, whether the marker thinks of them as critical thinking, or good writing, or sound research, or (most accurately) all three. Most assessment that goes beyond testing recall or recognition, therefore, includes some evaluation of critical thinking – most often, of a holistic and impressionistic kind recorded in a global comment or a tick against a term like ‘critical analysis’ or ‘relevance’. Such marking, however, can conceal considerable variation in the meaning and weighting that different lecturers give to this component (as well as confusion in students’ minds about what it might mean). It seems desirable, therefore, to try to identify what critical thinking (or any other ‘attribute’) means in the in the contexts in which it is assessed most often. Implications for development of criteria and standards However, as Green et al. (2009, p.18) have noted, while universities ‘need to provide evidence of…outcomes through the application of appropriate criteria and standards’, at the same time ‘there appears to be considerable confusion over how graduate attributes Journal of Learning Development in Higher Education, Issue 6: November 2013 5 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines should be defined, what these attributes look like within each discipline, [and] how they should be taught, assessed and evaluated’. They point to the gulf between the apparent simplicity of the attributes listed on universities’ websites (interested readers will find a wealth of examples at http://www.itl.usyd.edu.au/projects/nationalgap/resources/gamap/introduction.htm on the GAP project’s website), and the reality of the ways of thinking they purport to describe. From a policy and planning perspective, it makes sense to deal with each identified attribute as a separate component. Yet…different attributes tend to develop in relation to one another…[and in] accord with the conventions and norms of each discipline…Consequently, those charged with implementation must balance policy- driven concerns of quantification and categorisation with the awareness that for graduate attributes to be truly ‘embedded’ in curricula, knowledge and literacy need to be seen as inextricably linked. (Green et al., 2009, p.21) This is the challenge of developing assessment criteria and standards of a kind that enable transparency and consistency in marking and can guide teachers and students alike as to what must be demonstrated in a particular piece of work. ‘I know it when I see it’ is a time- honoured but unsatisfactory approach to marking aspects of performance that are (for good reasons) difficult to separate from the whole. Criteria must articulate what we see, in language that is comprehensible to students and staff. Moreover, these criteria should accommodate the range of ways that critical thinking is used in the relevant disciplines, for it is this discipline-specific way of thinking that markers hope to see in students’ work for their subjects (Hammer and Green, 2011; Moore, 2011). Finally, the criteria must reflect what can reasonably be expected at each level of students’ apprenticeship in the discipline, as they develop the subject knowledge that enables them to take an increasingly critical approach to their material (Hammer and Green, 2011). The context La Trobe University has opted to assess its ‘graduate capabilities’ at the level of its Faculties (in the Australian sense of organisational units, rather than teaching staff), thus aiming at a level between generic and discipline-specific. The decision to assess graduate capabilities as demonstrated in students’ work for the disciplines, rather than outside their Journal of Learning Development in Higher Education, Issue 6: November 2013 6 http://www.itl.usyd.edu.au/projects/nationalgap/resources/gamap/introduction.htm Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines discipline subjects, accords with conceptions of graduate attributes that Barrie (2007, pp.441-445) has characterised as facilitating ‘translation’ (application of disciplinary knowledge to unfamiliar problems) and/or ‘enabling’, that is, ‘abilities that infuse university learning and knowledge’; it also accords with beliefs that students develop these attributes through the content and process of disciplinary teaching and engagement with the learning experiences in their courses. The decision was made in the context of an institutional program of curriculum reform known as the Design for Learning (La Trobe University, 2009). Along with a process of ensuring constructive alignment of teaching and assessment of subject content with intended learning outcomes (Biggs, 1996), the university nominated six graduate capabilities to be developed over the course of every degree. At the institutional level, the capabilities were stated in starkly generic terms: writing, speaking, teamwork, research/enquiry, critical thinking, and creative problem- solving (in the last few months, a somewhat more elaborate framework has been promulgated, at http://www.latrobe.edu.au/teaching/la-trobe-framework/graduate- capabilities). From there, it was the responsibility of each Faculty to articulate what the capabilities meant in the constellation of disciplines for which it was responsible, and to embed the learning, practice, and assessment of each capability in designated subjects at each stage of the course: ‘cornerstone’ (first year), ‘mid-point’ (second year), and ‘capstone’, shortly before graduation at the end of third year (La Trobe University, 2012). This process was consistent with Barrie’s (2004) view that graduate capabilities ‘are not a set of additional outcomes requiring an additional curriculum – rather, they are outcomes that can be reasonably expected from the usual higher education experience’ (p.263). As such, it ought to be possible to derive an assessment of them from the assignments already set in the disciplines. It was accepted that the meaning and practice of the capabilities might well vary across the Faculties – Science, Technology and Engineering; Health Sciences; Education; Law and Management; and Humanities and Social Sciences – and that discipline lecturers were the people with the appropriate expertise to discern those meanings and describe those practices. If the criteria were to adequately reflect their contexts of use, the perspectives of staff from different disciplines were needed. In addition, in the working party tasked with this effort in the Faculty of Humanities and Social Sciences, representatives from some of the Faculty’s disciplines (media, philosophy, politics, history, and visual arts and design) were joined by an Academic Developer with an institutional perspective and a knowledge of assessment, and an Academic Language and Learning Journal of Learning Development in Higher Education, Issue 6: November 2013 7 http://www.latrobe.edu.au/teaching/la-trobe-framework/graduate-capabilities http://www.latrobe.edu.au/teaching/la-trobe-framework/graduate-capabilities Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines adviser (me). My perspective was needed as my work across all of the Faculty’s disciplines highlights commonalities and differences in the requirements of assignments across the Faculty, and involves efforts to render the tacit knowledge of discipline lecturers explicit and comprehensible to students. In the literature on graduate attributes, staff in this role are usually mentioned in connection with remedial or add-on teaching outside the disciplines (e.g. Barrie, 2007, p.455), but our work with individual students, in particular, gives us insights into what they find unexplained or confusing, which can usefully inform discussions in the disciplines. The process In constructing a format for our criteria, we adopted a grid model similar to those used in the Research Skill Development for Curriculum Design and Assessment (RSD, 2009) or the Association of American Colleges and Universities (AACU, 2012). This structure accommodates descriptors organised by levels (down the side) and standards (across the top). Thus, users can see how each capability is supposed to be manifested in assignments, and how performance is differentiated as either falling short, or meeting, or exceeding expectations at each level of study. We produced a table for each capability, but as ‘speaking’ and ‘teamwork’ were easily separated from the rest, I will not discuss those here. Summary description Standard and Level Exceeded Met Not Met Cornerstone . Midpoint Capstone This template is simple, but the process of arriving at descriptors was not. It revealed a gulf between the commonplace belief in distinct and context-independent skills, and the holistic thinking of discipline academics, for whom writing, critical thinking, inquiry and research, and problem solving were inseparable. This was apparent in the working party’s Journal of Learning Development in Higher Education, Issue 6: November 2013 8 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines first attempt at a summary description of writing, where the mechanical aspects (italicised below) received short shrift: Writing in humanities and social science disciplines requires you to engage with other people’s arguments in order to develop, test and communicate your own. In doing so, you will learn about the writing conventions that structure the different ways disciplines organise and communicate knowledge. You will learn how to acknowledge, use and reference different kinds of texts and sources of knowledge. You will learn how to read, assess and work with arguments and evidence in order to apply those skills to your own writing. To write well in the humanities and social sciences, you will need to ensure that you respond to the question with a clear and coherent argument, that the writing is free of grammar and spelling errors, that you have used sources and evidence appropriately and that it is accurately referenced. Every sentence in that description is imbued with critical thinking, and it became clear that, for lecturers in this Faculty at least, writing is argument. Good writing is good because it presents a cogent, informed argument, and however the nature of argument might differ across the disciplines, it was the touchstone for all. However, we could hardly produce standards that would require markers to evaluate the same thing under four different capabilities, so we had to ask the question: if we take information and argument out of writing, what is left? We teased out descriptors for writing that ‘met’ the Faculty’s standards at cornerstone level, focusing on clarity, structure, correctness, and integration: • Most ideas are expressed clearly. • The writing is mostly coherently structured. • The work contains a few errors in grammar, punctuation, and/or spelling, but these do not obscure the meaning. • The sources and evidence are mostly integrated with the writing. The use of sources was then separated out from our description of writing, and assigned to ‘Inquiry/research’, yielding the following descriptors: Journal of Learning Development in Higher Education, Issue 6: November 2013 9 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines • The work shows adequate understanding of what the question/topic is asking. • The minimum required reading has informed the response to this task. • Ideas/information necessary to the task have been considered. • Information used is mostly credible and relevant. • An attempt has been made to formulate a relevant argument/position. • The piece includes most elements of the task. • Most sources are accurately attributed and appropriately referenced. Problem-solving was particularly difficult to differentiate from critical thinking, as they are virtually identical in humanities and social sciences. Perhaps in more vocationally-oriented fields, problem-solving would foreground the practical application of critical thinking to decide what action should be taken, or perhaps the focus would be on the selection of ‘tools’, such as mathematical operations, legal precedents, or protocols for patient care, to deal with any particular case. In humanities and social sciences, however, the problems are largely questions of interpretation, and solutions take the form of insights that bear upon those questions. Perhaps it would work to reserve problem solving for questions that ask ‘what should be done?’ However, in some disciplines, such as history, that question makes no sense (similarly, Jones [2009a, p.179], was told that problem solving was not important in teaching history ‘unless it is in the form of ‘discuss the causes of the French Revolution’). Our goal was to produce standards that work for any discipline, so we corralled the elements of critical thinking that had most to do with the application of tools, in this case conceptual ones as well as methods, and assigned these to problem-solving: • Given a problem, the piece of work shows some grasp of what needs to be understood. • It identifies some of the difficulties with understanding the problem. • It applies some relevant concepts/methods from the discipline or other relevant context of discussion. • It suggests a plausible solution to the problem. Again, these describe what is needed for the standard to be ‘met’ at cornerstone level. Higher level standards were created by repeating the ‘exceeded’ criteria from the previous level and adding (in bold) appropriate new ones. Journal of Learning Development in Higher Education, Issue 6: November 2013 10 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines This, then, enabled us to formulate a description of critical thinking and a set of descriptors that was no longer bound up with the other three capabilities (although, of course, in any text they remain inextricably bound up in practice). Once shorn of its expression and its application, critical thinking emerged as a combination of relevance, reasoning, an open but sceptical disposition, consideration of context, and some degree of awareness about subjectivity, as outlined in the brief defining paragraph that introduced the table: Critical thinking in humanities and social science disciplines is a habit of mind: an awareness that knowledge is constructed through a process of asking questions, considering relevant information, and articulating a reasoned explanation. You will learn to consider the ideas you encounter with an open mind; to continually question how they are applied; to appreciate explanations that have been arrived at carefully and thoroughly, and to identify any problems or limitations they may have. You will learn to apply the same care and critical evaluation to your own work. This will be shown in: (i) your selection and presentation of evidence; (ii) your awareness of your own assumptions and those of others; (iii) your recognition of how context shapes people’s different perspectives; and (iv) your evaluation of the strengths and limitations of competing explanations, including your own. For the table, we began (at ‘Cornerstone: met’) with four basic requirements shared by all our disciplines (as far as we could tell), leaving it up to each to know what they would consider to be ‘relevant’, ‘coherent’, ‘appropriate’, and ‘sufficient’. Then we added descriptors for the most salient ways that critical thinking could be expected to develop in sophistication and rigour over the course of a degree, prefacing these with ‘(where applicable)’ in recognition of the likelihood that different disciplines place differing emphasis on one or another of these, and/or expect them to develop at different stages over the course of study. Standard and Level Exceeded Met Not Met Cornerstone • All points are relevant. • All evidence is appropriate. • The reasoning is coherent. Most points are relevant. • Most evidence is appropriate. • The reasoning is Most points are not relevant. • Most evidence is inappropriate. • The reasoning lacks Journal of Learning Development in Higher Education, Issue 6: November 2013 11 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines • And sufficient to arrive at the conclusion. • (Where applicable) the work questions assumptions of others. • (Where applicable) The work recognises how context shapes perspectives. coherent. • And sufficient to arrive at the conclusion. coherence. • And is insufficient to arrive at the conclusion. At ‘midpoint’, where we specified the work to be In a task requiring somewhat complex reasoning, we added ‘(Where applicable) the work questions own assumptions’, and at ‘capstone’, for ‘a task requiring complex reasoning’, we added ‘The work acknowledges the limits of its own position’ and ‘The work considers alternatives beyond those suggested by the assignment’. Our inclusion of context and subjectivity here (as distinct from objectivity) underlines, for me, the context-specific character of ostensibly generic ways of thinking that has been brought out by scholars such as Jones (2009a; 2009b). While we were not in a position to produce standards for each discipline, our Faculty-level descriptors do attest to an orientation to knowledge that is characteristic of disciplines across the humanities and social sciences. These tend to share a view of knowledge as constructed and contingent, situated and social, and best understood, therefore, by recovering the contexts of time, place, and culture that have produced it. (‘Constructed’, here, should not be taken to mean ‘invented’; rather than suggesting that nothing can be known about the world, it means that what we know depends on what we ask and where we go for answers, and that those things occur within a social context.) Conclusion The development of standards for assessing capabilities at the Faculty level was intended to make marking more consistent and feedback more informative. At the same time, however, it revealed something about the meaning of critical thinking for humanities and social sciences. As we tried to separate the capabilities for purposes of description, it was clear that lecturers in these fields evaluate a piece of work as an informed argument rather than an array of skills and information. ‘Problem solving’ may be the motivation, ‘teamwork’ Journal of Learning Development in Higher Education, Issue 6: November 2013 12 Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines the process, ‘writing’ or ‘speaking’ the mode of expression, and ‘inquiry/research’ the means of substantiation, while ‘critical thinking’ suffuses all of these; but the resulting artefact is apprehended and appreciated as a whole: as argument. When our graduates disperse into the workplace, it seems likely that they will approach their work with an expectation, common to the fields of enquiry in which they have been learning, that knowledge is socially constructed, variable across time, place, and culture, and thus contingent and subject to change. If so, this is a valuable preparation for ‘an unknown future’ (Barnett, 2004); whether it will ‘fit’ our graduates for particular workplace cultures is another question, given that brevity, simplicity, and decisiveness can be virtues at work, if not in essays. What can this reflection add to recent efforts to elucidate the slow pace and uneven extent of integration of graduate attributes into discipline curricula? I think we have found a level at which graduate attributes can be described in ways that are internally persuasive for their users. We achieved ownership, in organisational terms, at the Faculty level, and in intellectual terms, at a level of specificity partway between the disciplines that make up the Faculty and the more generic level at which the capabilities were originally designated. Our experience bears out the view that capabilities have specific, and different, meanings for different discipline areas. At the same time, it suggests that commonalities may be seen by staff who work across a cluster of disciplines, who can help find a form of words that is meaningful to all of them. References Association of American Colleges and Universities (2012) VALUE: Valid Assessment of Learning in Undergraduate Education. Critical thinking. Available at: http://www.aacu.org/value/rubrics/index.cfm (Accessed: 5 July 2013). Barnett, R. (2004) ‘Learning for an unknown future’, Higher Education Research and Development, 23(3), pp. 247-260. Barrie, S. (2004) ‘A research-based approach to generic graduate attributes policy’, Higher Education Research and Development, 23(3), pp. 262-275. Journal of Learning Development in Higher Education, Issue 6: November 2013 13 http://www.aacu.org/value/rubrics/index.cfm Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines Barrie, S. (2007) ‘A conceptual framework for the teaching and learning of generic graduate attributes’, Studies in Higher Education, 32(4), pp. 439-458. Bazerman, C. (1981) ‘What written knowledge does: three examples of academic discourse’, Philosophy of the Social Sciences, 11, pp. 361-87. Biggs, J. (1996) ‘Enhancing teaching through constructive alignment’, Higher Education, 32, pp. 347-64. Bowden, J., Hart, G., King, B., Trigwell, K. and Watts, O. (2000) Generic capabilities of ATN university graduates, Canberra: Australian Government Department of Education, Training and Youth Affairs. Crebert, G. (2002) ‘Institutional research into generic skills and graduate attributes: Constraints and dilemmas’, 2nd International Lifelong Learning Conference. Yeppoon, Queensland 16-19 June. Central Queensland University Press, pp. 135- 142. de la Harpe, B. and David, C. (2012) ‘Major influences on the teaching and assessment of graduate attributes’, Higher Education Research and Development, 31(4), pp. 493- 510. GAP (2013) The National Graduate Attributes Project: integration and assessment of graduate attributes in curriculum. The University of Sydney Institute for Teaching and Learning. Available at: http://www.itl.usyd.edu.au/projects/nationalgap/introduction.htm (Accessed: 5 July 2013). Green, W., Hammer, S. and Star, C. (2009) ‘Facing up to the challenge: why is it so hard to develop graduate attributes?’, Higher Education Research and Development, 28(1), pp. 17-29. Hammer, S. and Green, W. (2011) ‘Critical thinking in a first year management unit: the relationship between disciplinary learning, academic literacy and learning progression’, Higher Education Research and Development, 30(3), pp. 303-315. Journal of Learning Development in Higher Education, Issue 6: November 2013 14 http://www.itl.usyd.edu.au/projects/nationalgap/introduction.htm Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines Hughes, C. and Barrie, S. (2010) ‘Influences on the assessment of graduate attributes in higher education’, Assessment and Evaluation in Higher Education, 35(3), pp. 325- 334. James, B., Lefoe, G. and Hadi, M. (2004) ‘Working ‘through’ graduate attributes: a bottom- up approach’, 27th HERDSA Annual Conference. Miri, Sarawak 4-7 July. Available at: http://ro.uow.edu.au/asdpapers/7 (Accessed: 5 July 2013). Jones, A. (2009a) ‘Generic attributes as espoused theory: the importance of context’, Higher Education, 58(2), pp. 175-191. Jones, A. (2009b) ‘Redisciplining generic attributes: the disciplinary context in focus’, Studies in Higher Education, 34(1), pp. 85-100. La Trobe University (2009) Design for learning: curriculum review and renewal at La Trobe University. Available at: http://www.latrobe.edu.au/ctlc/assets/downloads/dfl/DFL- booklet.pdf . (Accessed: 5 July 2013). La Trobe University (2012) Graduate capabilities for undergraduate students policy. Available at: http://www.latrobe.edu.au/policy/documents/graduate-capabilities- policy.pdf (Accessed: 5 July 2013). Lowe, K. and Marshall, L. (2004) ‘Plotting renewal: pushing curriculum boundaries using a web based graduate attribute mapping tool’, in Atkinson, R., McBeath, C., Jonas- Dwyer, D. and Phillips, R. (eds.) Beyond the comfort zone: proceedings of the 21st ASCILITE conference. Perth, 5-8 December, pp. 548-557. Available at: http://www.ascilite.org.au/conferences/perth04/procs/lowe-k.html (Accessed: 5 July 2013). Moore, T. (2011) ‘Critical thinking and disciplinary thinking: a continuing debate’, Higher Education Research and Development, 30(3), pp. 261-274. Journal of Learning Development in Higher Education, Issue 6: November 2013 15 http://ro.uow.edu.au/asdpapers/7 http://www.latrobe.edu.au/ctlc/assets/downloads/dfl/DFL-booklet.pdf http://www.latrobe.edu.au/ctlc/assets/downloads/dfl/DFL-booklet.pdf http://www.latrobe.edu.au/policy/documents/graduate-capabilities-policy.pdf http://www.latrobe.edu.au/policy/documents/graduate-capabilities-policy.pdf http://www.ascilite.org.au/conferences/perth04/procs/lowe-k.html Chanock Developing criteria to assess graduate attributes in students’ work for their disciplines Radloff, A., de la Harpe, B., Dalton, H., Thomas, J., and Lawson, A. (2008) ‘Assessing graduate attributes: engaging academic staff and their students’, in Duff, A., Quinn, D., Green, M., Andre, K., Ferris, T. and Copeland, S. (eds.) Engaging students in assessment. Refereed proceedings of the 2009 ATN Assessment Conference. Adelaide, Australia 20-21 November. Available at: http://www.ojs.unisa.edu.au/index.php/atna/issue/ (Accessed: 5 July 2013). Research Skill Development (RSD) (2009) Research skill development for curriculum design and assessment. Available at: http://www.adelaide.edu.au/rsd/ (Accessed: 5 July 2013). Author details Associate Professor Kate Chanock is an Academic Language and Learning lecturer at La Trobe University in Melbourne, Australia. Her interests include teaching and learning disciplinary discourses; efficacy of individual consultations; and studying with a disability. Journal of Learning Development in Higher Education, Issue 6: November 2013 16 http://www.ojs.unisa.edu.au/index.php/atna/issue/ http://www.adelaide.edu.au/rsd/ Developing criteria to assess graduate attributes in students’ work for their disciplines Abstract Introduction Ownership and authenticity Differences in meaning Implications for development of criteria and standards The context The process Conclusion References Author details