PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 25 Defining Design and Technology in an Age of Uncertainty: The View of the Expert Practitioner Copyright © 2017-2018 Prism: Casting New Light on Learning, Theory and Practice http://prism-journal.blackburn.ac.uk/ ISSN 2514-5347 Vol. 1 (2): pp. 25-54 DAVID WOOFF Edge Hill University david.wooff@edgehill.ac.uk Abstract Long standing debate surrounds the position that Design and Technology holds in the English and Welsh national curriculum. Some commentators espouse alignment with Science, Technology, Engineering and Mathematics (STEM) as its natural home, whilst others argue that this stifles creativity and takes no account of the ‘designerly’ nature they consider to be a central tenet of the subject. Against this backdrop, the subject has undergone changes in both prescribed subject knowledge content and examination and assessment arrangements by which pupil progress and attainment are measured. Set against this background, the work presented here summarises a Delphi study which sought to canvass established and experienced Design and Technology teachers about how they perceive the attributes, values and unique features of the subject. The results are analysed to give a view of the subject from within the classroom. Analysis reveals that participants in the study consider the ‘uniqueness’ of the subject to prevail over the values and attributes they collectively define it by. The study moves on to discuss the findings in relation to the values and direction which underpin the policy documentation that drives and shapes the subject from a national perspective. Finally, the work concludes by highlighting several important areas worthy of further research which have emerged and could be seen as contributory to understanding the nature and essence of Design and Technology. 1. Introduction and Rationale 1.1 Introduction Design and Technology is a relatively new academic discipline, its emergence stemming back to the first iteration of the national curriculum (Department for Children Schools and Families (DCSF), 1989). Not only did this review change the subject content studied in schools, PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 26 it also saw the introduction of the General Certificate of Secondary Education series of examinations as the final school leaving qualification in place of Ordinary Level (O-Level) and Certificate of Secondary Education (CSE) qualifications. As a curriculum subject, Design and Technology is unique in that it is an education construct which does not exist outside of formal education (Bell, Wooff, McLain, & Morrison-Love, 2017), nor did it exist prior to this curriculum revision. The separate strands brought together to form Design and Technology have their foundations in well-established subjects stretching back to the turn of the 20th century and it is acknowledged that they emerged from the recognised subject of “Handicraft” (Atkinson, 1990). However, these strands have themselves undergone identity drift over the decades since they were initially established, with Woodwork and Metalwork recently combining to form ‘Resistant Materials’, Engineering Design and Technical Drawing coming together to form ‘Graphic Products’ and even cookery (or cooking) metamorphosing via home economics into ‘Food Technology’ and arguably into the vocational offshoot of ‘Catering and Hospitality’. These subjects were initially combined, along with a subject typically aligned with art and design, sewing, or needlework, which itself became ‘Textiles Technology’. Every one of these falls under the umbrella of ‘Design and Technology’ due to exhibiting an element of design, or ‘designerly’ thinking. Originally designated as a compulsory subject at the point of its inception Design and Technology lost this protected status in a later curriculum revision which saw it become non- compulsory beyond the end of Key Stage 3 (KS3) for all 11-14 year olds (Miller & McGimpsey, 2011). Subsequently it has seen a continual decline in student numbers (Turner, 2017). More recently it has been omitted from the main school performance measure, the English Baccalaureate although it can be used in another measure designed to analyse individual pupil progress over the course of their secondary education (Department for Education, 2017). The impacts of policy changes on the identity and status of Design and Technology cannot be underestimated. Initially it appeared to be a heavily over-assessed subject with multiple combinations of attainment targets and programmes of study (DCSF, 1989). These have changed, as has the prescriptive pedagogy found in earlier curriculum revisions, to leave a PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 27 single attainment target for KS3 Design and Technology (DfE, 2013). Yet despite these changes in assessment, the actual content of the Design and Technology curriculum has evolved without fully considering the wider views across industry and commerce of what the subject should be (Design and Technology Association (DATA), 2011). Debate from within the subject itself often focuses on elements of core identity including its place within STEM (Bell, 2012, 2016; Williams, 2011), its vocational nature (Miller & McGimpsey, 2011), its academic worth (Morrison, 2013), its role in creativity (Barlex, 2002; McLellan & Nicholl, 2013) and its contribution to problem solving (Hennessy & Murphy, 1999). Irrespective of the lens adopted, this lack of cohesion gives rise to ongoing battle with subject professionals to justify its curricular existence (Barlex, 2007; Owen-Jackson, 2013). This in turn gives rise to the subject’s current instability. A desire to investigate this instability and the variance of opinion which underpins it provides the rationale for this study. Through engagement with stakeholders it seeks to ask: “What actually is Design and Technology?” In the course of engaging in this study, it is anticipated that participants will further develop their own personal epistemology (Hofer, 2000) in relation to Design and Technology, even if they are not cognisant of doing so. In drawing on the earlier work of King and Kitchener (1994), Hofer suggests that the way an individual views knowledge changes with time and experience ‘moving from a fixed to a more fluid view’ (2000, p. 380). It is this fluidity which offers the mechanism by which participants can challenge and develop their personal epistemology. This presupposes that their existing epistemological beliefs are continually evolving rather than subscribing to a dualistic belief surrounding the creation of knowledge about a subject. As this is individual to each person it is logical to anticipate that each participant in the study will be at a different point on the continuum between a fixed and a fluid view about the nature of Design and Technology. In the investigation of this difference of opinion it is anticipated that some commonality will prevail, leading to a consensus relating to the attributes and values of the subject to those wishing to study it. In accepting this variance in epistemological belief, it is necessary to also consider the impact of each individual’s personal ontology. The ongoing debate into the identity of the PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 28 subject will have an impact on participants in this study. This, along with the value their own training and background has placed on the subject, will lead them to consider what Design and Technology is to each of them. Although the purpose of this study is not to challenge the beliefs of individual participants, it is likely that the reflective nature of their engagement will lead participants to question what they know about the subject itself. Design and Technology struggles to maintain its existence (DATA, 2011). The closure and diversification of routes into teaching Design and Technology do little to reinforce its standing as an ‘academic subject’ (Miller & McGimpsey, 2011). This uncertainty is further compounded by an ongoing internal struggle within the subject itself as many stakeholders debate, disagree and theorise about what the subject actually is from their own perspective (Gilbert, Boulter, & Elmer, 2000; Hardy, 2015, 2016a; Middleton, 2005; Miller & McGimpsey, 2011). In turn, this leads to differences in belief over the direction the subject should take to sustain its future. It is clear that there are different identifiable stakeholders who, from their own unique perspective, offer a valid opinion about the value and worth of Design and Technology. After considering the differences between stakeholders, in order to provide depth and rigour and derive a consensus of what is the essence of Design and Technology, it was decided to undertake an initial investigation with a research cohort comprising of experienced Design and Technology teachers. The rationale for selecting participants from the group of stakeholders comes from considering the literature around previous studies into the subject, and it is clear that there is a gap in considering the voice of experienced teachers. This is not to say that other stakeholder groups would be unable to contribute to the debate, however they would do so from a different perspective. So, with this in mind, the study set out to answer the question: From the perspective of an experienced classroom practitioner; what are the principles which define Design and Technology? Further, the answers to three Sub-Questions (SQ) were also sought: 1. What are the attributes that Design and Technology offers to pupils who study it in secondary education? 2. What values does Design and Technology instil in students who study it? PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 29 3. What are the unique features that Design and Technology offers to the curriculum that other subjects do not? 1.2 Research Design In determining a suitable approach to the research design there were a number of factors which needed to be considered (van den Akker, Gravemeijer, McKenney, & Nieveen, 2006). This reflects that the usual approach taken in the determination of a suitable methodology is to consider a range of theoretical options which could be used to answer the initial research question (Cohen, Manion, & Morrison, 2013). With this in mind, a range of methods could have been adopted to determine a consensus from a group of individuals. These include the Delphi Technique (Linstone & Turoff, 1975), Q Methodology (Müller & Kals, 2004), nominal group technique (Sample, 1984) or use of a consensus development program as developed by the United States Department of Health & Human Services (1977). Two of these methodological approaches seek to use statistical methods and/or techniques in the ordering of qualitative responses gathered from research participants, namely Delphi Methodology and Q Methodology. Each allows the determination of a rank ordering of statements, or criteria, which emerge from the study. This was seen as a highly desirable outcome, as it would enable exacting comparisons to be made in the future as the subject further evolves with policy and curriculum change. Studies using Delphi Methodology and Q Methodology can be undertaken from afar and do not require the physical presence of participants in the same location. This was seen as being highly desirable as it allowed the best research cohort possible to be drawn together, irrespective of locality, and in so doing it sought to find the best possible field of suitable participants. 1.3 Delphi Technique Assigning a defined theoretical framework to the Delphi Technique is something which has seemingly eluded those who have utilised it in the past. Indeed, this proliferates in the work of those who have sought to critique and review its use from a conceptual stance, rather than that of researcher’s actively employing it in the course of their own study (Hasson, PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 30 Keeney, & McKenna, 2000; Hsu & Sandford, 2007; McKenna, 1994; Powell, 2003; Rowe & Wright, 1999; Williams & Webb, 1994). Rather, it is more fitting to consider that the Delphi technique is efficient in employing its own underpinning theoretical framework which aligns with the aims of this study in the pursuit of its determination of a consensus of opinion of a defined group. In doing so, the method and methodological approach are defined by the application of the technique itself without need for recourse to additional theoretical frameworks. It is acknowledged that there are variants within the technique and these have developed over the years since its first inception. Van Zolingen and Klaassen (2003) present a detailed journey of the evolution of identified variant models. The noticeable feature common to these variants seems to be their truncation of the process as a means to expedite the completion of the study. However, the structure which is used here is considered the ‘Classical Delphi Technique’, which they acknowledge as offering the most rigour and affords the most accuracy in determining findings. Overall, the technique seeks to draw opinion from experts and produce an outcome which is agreeable to all. As such it must start from an accepted stance that, for the participants, multiple constructed realities exist (Pring, 2000). To commence the study with any other assertion would render the technique unusable as there would be no need to seek a consensus, and no variance in underlying opinion. It is also recognised that the derivation of a fixed definition of ‘what a subject is’ has its own limitations. Initially the process involves the generation of a list of factors for consideration and this is shared with all participants. This could lead to participants being swayed by suggestions which they did not initially conceive of themselves. Also, the emergence of a definition in response to the original research question can only really be considered to be valid at a snapshot in time. As discussed previously, the notion that every participant in the study situates their personal epistemology on a continuum stretching from rigidity (dualistic) to fluidity (non-dualistic) implies that the way they consider the formation of their knowledge will change over time. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 31 One of the original concepts emerging from early Delphi technique work was to derive a method seeking a consensus amongst an identified cohort whilst allowing the researcher to maintain a degree of neutrality (Dalkey, 1967). In doing so it is an assertion that they are positioned in such a way as to be regarded as an “uninvolved observer” (Robson, 1993), where their role is to facilitate new knowledge creation without influencing the process. In his work, investigating bias and neutrality in the Delphi technique, Hallowell (2009) highlights several challenges to this delineated notion of research as being devoid of all bias and knowledge creation taking place in a domain isolated and uninfluenced by the researcher. Drawing on a concept which presupposes that any researcher, or participant, has some vested interest in the work they are undertaking, he surmises: ‘Bias may occur when one subconsciously uses cognitive shortcuts to reach erroneous conclusions’ (p. 1495). Clearly this is undesirable in all research studies and more problematic to address than overt bias. Norris (1997) describes an ideal situation for conducting research as one without undue influence deriving from any form of ‘researcher bias’. In seeking to achieve this, research using the Delphi technique has been designed to minimise all forms of bias on the part of the researcher and the participants. But being cognisant of the work of Hallowell (2009) it is recognised that the eradication of all forms of unconscious bias is somewhat more problematic to address throughout the process. Consequently, this should be considered when undertaking any Delphi study to ensure that no undue influence is brought to bear by either the researcher or the participants themselves. Beyond issues of neutrality and bias, the Delphi technique faces additional criticisms. Hanafin undertook a systematic literature review where she considered seven different studies all employing the Delphi technique spanning a time frame of three decades. She identifies a number of areas of concern from those studies, primarily that ‘participant anonymity may lead to a lack of accountability’ (Hanafin, 2004, p.11) although this is something which could be levelled at any study collating anonymous responses. Powell (2003) goes further and comments that this may lead to everyone aspiring to conform to an average opinion and as such, results represent the ‘lowest common denominator’ (p. 378). Other limitations highlighted by Hanafin in her work are the economic limitations of the logistics PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 32 involved in undertaking a Delphi study, the determination of the number of rounds to be used, data analysis and the inference of a consensus. It was felt that effective research design would counter each of these issues. Although participants were unknown to each other, they were known and identifiable to the researcher. This was essential as it enabled the researcher to pursue responses from those who did not return them in a timely fashion. Consequently, it was never felt that participants felt a lack of accountability whilst engaging in the study. Due to the nature of the profession of the participants, and their willingness to engage in the study from its inception, it was not envisaged that people would migrate towards the ‘lowest common denominator’ and this did not appear to prevail. Given that the researcher used his own time and resources to undertake this study, there was no financial impact attributable. The number of rounds undertaken was in alignment with the work of others who had successfully undertaken Delphi studies (Duffield, 1993) and deemed their work to have reached saturation (Barbour, 2003). Issues surrounding the validity of the consensus arrived at the end of this process are addressed in the concluding part of this study. 1.4 Participant Selection The Delphi technique does not lend itself to interviews nor focus groups (Cohen et al., 2013). Indeed, the latter goes against one of the significant reasons for selecting to undertake a study utilising the Delphi technique; guaranteed participant anonymity. (Dalkey, Brown, & Cochran, 1969). Witkin and Altschuld (1995) note that sample sizes attributed to Delphi Technique studies tend to be under 50 participants and frequently in the 15 – 20 range. There is debate around the optimum number of participants who can effectively contribute to such a methodological approach. Powell (2003) goes on to surmise that effective data saturation occurs at around 20 participants. Interestingly this is often regarded as the same level at which saturation of responses occurs in phenomenographic study (Marton, 1986). Beyond 20 participants, it is believed that saturation of opinion related to response occurs and variance in thought ceases PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 33 to occur, with work beyond this point being repetitive and not being contributory to the generation of new knowledge (Mason, 2012). For these reasons it would be safe to assume that gathering further experts together beyond this apparent threshold will not yield significantly different (or better) results. Being cognisant of the need for a defined cohort size, work was undertaken to investigate the attrition rates of participants in educational research studies. Unable to find published work on this specific area, investigation of literature was widened to include fields beyond of education. In a significant piece of work in this area, Dumville, Torgerson and Hewitt (2006) identify attrition rates of between 20% - 25% in medical research studies. With this in mind, and with the initial self-nominating volunteers as participants, it was anticipated that a much lower rate of participant attrition would be applicable in this study. To this end, a sample size of 22 participants was selected, determined by considering an optimum number of participants as 20 with the addition of 10% (n=2) to reduce the impact of participant attrition. A study using the Delphi Technique requires participants to be ‘experts’ in the field of knowledge being investigated. For the purposes of this study the term ‘expert’ was determined by adherence to the following criterion: • Professional Qualification – participants selected held Qualified Teacher Status (QTS) or an equivalent qualification recognised by the DfE should they have gained qualification overseas. • Academic Qualification – participants were all graduates, holding an academic qualification linked to teaching and learning within the subject (at either undergraduate or postgraduate level). • Performance Review Outcomes – participants all attested that they were successful in their most recent performance review. • Lesson Observation Grade Data – participants were asked to demonstrate high outcomes from observed lessons; consistently being rated as good with outstanding features as a baseline measure. Ideally it was hoped that participants would be able PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 34 to demonstrate competence by being graded as outstanding across consecutive observations over a sustained period of time. • Pupil/Class Attainment Data – participants were asked to share publicly available GCSE data about the classes they had taught who had been successful. However, it is recognised that sometimes extraneous factors outside of teacher control (like attendance, pupils who start late or move schools and so on) could have an impact on this criterion. • Length of service in post – participants who had spent at least five years in post after qualification (this includes their Newly Qualified Teacher (NQT) year) were deemed to be experienced. This aligns with the notion that disillusioned teachers tend to drop out early in their career (Gurney-Reed, 2015; Marsh 2015a, 2015b). Participants were selected after self-nomination following online dissemination of an opportunity to participate. Having been shortlisted using the pre-defined criteria, along with an assurance that they understood the commitments involved, and a willingness to provide ethical consent for the work (British Educational Research Association (BERA), 2011), they were invited to participate. Initially, 102 enquiries from interested parties were received, reduced to a total of 48 who were deemed to have met the criteria enabling them to be suitable for participation. The overriding factors for exclusion from the initial sample was that respondents had less than five years teaching experience, and/or they did not provide a complete response to the data set. From this short list, a final sample of 22 people were randomly selected from those determined to be eligible by means of a number randomiser programme. Demographically, the cohort consisted of nine male and thirteen female participants. All were qualified teachers, holding the award of QTS. Participants had undergone a mix of degree and training pathways in pursuit of the award. The majority (n=13) held a Post Graduate Certificate of Education (PGCE), a lesser number (n=8) held an undergraduate degree linked to their teacher training and a single respondent (n=1) achieved qualification by means of a Certificate in Education (Cert. Ed) programme. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 35 The number of years teaching experience varied across the cohort, with the majority falling in the range of six and fifteen years, inclusive. Fig 1. Number of Years Teaching Experience. Having identified the cohort, the first round of the study was undertaken. Participants were asked three sub questions: 1. What are the attributes that Design and Technology offers to pupils who study it in secondary education? 2. What values does Design and Technology instil in students who study it? 3. What are the unique features that Design and Technology offers to the curriculum that other subjects do not? In round one, participants were encouraged to provide as many responses to each statement as they could in any format. Words, phrases and sentences were returned in different quantities by each participant and a basic thematic coding (Guest, MacQueen, & Namey, 2011) was undertaken. The purpose of the coding was to draw together similar responses and remove duplication. This was necessary due to the variance in terminology used. Whilst there was some clear repetition of terms and phrases, others could be considered to more individual to a specific participant. This could be attributed to the setting 0 1 2 3 4 5 6 7 8 9 36-40 31-35 26-30 21-25 16-20 11-15 06-10 Number of respondents N u m b e r o f Y e a rs T e a ch in g E x p e ri e n ce PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 36 in which they worked combined with their career trajectory and even influences from the examination board(s) they were familiar with. In each case, there is a specific set of vocabulary which can be linked to these variables, the combination of which leads to different levels and variance within participant responses. By using the Delphi methodology, it is possible to get participants to consider statements generated from round one in two different ways. The first being that they rank order these statements against each other; an attribute that mirrors work undertaken using Q Methodology (Müller & Kals, 2004). The second way of applying Delphi methodology is to get participants to assign an individual value of importance to each of the statements being considered, using a Likert scale. The former method is more complex for larger numbers of variables, and the latter allows independent thought and a circumstance where variables can be attributed the same value. It was decided to use the latter approach and get participants to assign a value to these statements using an eight-point Likert scale. This is more expedient for participants as each statement is considered in its own right, so easily allowing for breaks in the work. It also affords participants the opportunity of being able to rank variables as being of the same significance. In the ranking rounds of this study, an eight-point Likert scale was used with the omission of a null vale. This being the case, there were an even number of values which participants were able to use to in expressing their opinion about the significance of any statement being considered. The rationale behind using an eight-point scale is founded in the work of Tsang (2012). In that study he determined that if there is a midpoint, or an arbitrary median value, on a ranking scale, then people using that ranking scale have a propensity to opt for the mid- point, or the “not applicable”, or the neutral ground option, prevails in cases where participants do not determine a weighting towards one end of the scale. In this case the Likert scale used by participants only defined the end points on the continuum with descriptors of “Essential” (ranking highest with a value of 8) and “Desirable” (ranking lowest with a value of 1). This allowed participants to extrapolate their own meaning and interpretation for the inter range points on the scale. This approach was adopted in all PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 37 rounds of the study and against all statements requiring ranking. In order to present a cohesive approach to undertaking the investigative element of this study and improve clarity, it has been decided to consider each sub question individually. 2. Findings and Analysis 2.1 Findings Presentation of the data and subsequent analysis relevant to each sub question is considered and presented before moving on to data and analysis of the next. It is important to recognise that this approach has been undertaken to aid analysis and that the sub questions were asked together in the discrete rounds previously identified. Sub Question 1 (SQ1): What are the attributes that Design and Technology offers to pupils who study it in secondary education? This opening question elicited a range of responses, some more narrative than others, for example: “There are many, but I would say; creativity, practical application of knowledge and skills, designing and making” Participant 04 In such cases individual attributes were separated from the initial narrative to allow for ranking and focused judgements to be made in the second stage (initial ranking) following the applied methodology inherent in a Delphi technique study. After sorting and grouping of similar phrases, the initial responses to this question were: • Appeals to those who are less academically gifted • Designing real world artefacts, for real world issues • Develops designing skills • Develops product analysis skills • Empowers pupils to be autonomous • Encourages and promotes pupil autonomy • Helps develop criticality • Helps pupils develop planning skills PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 38 • It is fun and engaging • It is stimulating • Enables pupils to engage in real world problem solving • Manipulation and combination of materials to make a product • Opportunity to learn from modelling and experimentation • Opportunity to prepare for a career • Promotes creativity • Provides a challenge for all pupils, irrespective of ability • Provides synergy between thinking and doing An interesting statement which emerged is the first; “Appeals to those who are less academically gifted” as this shows a willingness for the cohort as a whole to identify that Design and Technology is, in their opinion, of lesser academic value than other subjects. Rather than challenging what is often considered to be a misconception that Design and Technology is less academically challenging then other subjects, (Eggleston, 1996; Barlex, 2007; Green, 2016) it can be construed as reinforcing this idea. By inference, it is also reinforcing the idea that the place of Design and Technology is not as an academic subject, but rather as a vocational subject. Having confirmed the initial set of statements, these were circulated to the research cohort for comment and initial ranking using the eight-point scale as previously described. Initial rankings placed the statements in the following order, the mean value attained is shown alongside each for reference (Fig. 2). Results indicate little numerical difference between statements. This can be explained when consideration is given to the method of determining the statements themselves. As they are derived from the opinions of those undertaking the research, it is not surprising that these statements are perceived as holding value by the participants. In alignment with the Delphi study approach, this list was then recirculated, alongside the statistical mean values attained. Participants were asked to reconsider, and reassess their judgements made against each of the statements in the previous round. Again, they were asked to grade these statements using the same eight-point Likert scale. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 39 Enables pupils to engage in real world problem solving 7.23 Designing real world artefacts, for real world issues 7.20 Promotes creativity 7.16 Opportunity to learn from modelling and experimentation 7.08 Encourages and promotes pupil autonomy 7.06 Develops designing skills 7.02 It is fun and engaging 6.99 It is stimulating 6.95 Helps develop criticality 6.93 Develops product analysis skills 6.90 Manipulation and combination of materials to make a product 6.88 Provides a challenge for all pupils, irrespective of ability 6.85 Appeals to those who are less academically gifted 6.81 Provides synergy between thinking and doing 6.77 Empowers pupils to be autonomous 6.72 Helps pupils develop planning skills 6.65 Opportunity to prepare for a career 6.58 Fig 2. Initial Ranking of Responses to SQ1 In considering the data for SQ1, it is apparent that there is some movement in the ranking order of descriptor statements. Not only was there a change in their relative positions, but their statistical mean has changed in every case at the second stage of ranking. It is worth highlighting that the principle purpose of the statistical mean attributed to each statement is in ranking the statement descriptors, rather than providing a definitive value for each. Previous studies which utilised a Delphi Methodological approach such as Gupta & Clarke (1996), Hasson et al. (2000), Kenney et al. (2006) observed similar occurrences. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 40 Position Position Difference Statement Descriptor Being Considered Mean Mean Difference Round 1 Round 2 Round 1 Round 2 1 1 0 Enables pupils to engage in real world problem solving 7.23 7.48 +0.25 2 2 0 Designing real world artefacts, for real world issues 7.20 7.35 +0.15 5 3 -2 Encourages and promotes pupil autonomy 7.06 7.30 +0.24 3 4 +1 Promotes creativity 7.16 7.10 -0.06 6 5 -1 Develops designing skills 7.02 7.04 +0.02 7 6 -1 It is fun and engaging 6.99 6.97 -0.02 8 7 -1 It is stimulating 6.95 6.84 -0.11 4 8 +4 Opportunity to learn from modelling and experimentation 7.08 6.83 -0.25 9 9 0 Helps develop criticality 6.93 6.80 -0.13 10 10 0 Develops product analysis skills 6.90 6.78 -0.12 11 11 0 Manipulation and combination of materials to make a product 6.88 6.70 -0.18 13 12 -1 Appeals to those who are less academically gifted 6.81 6.40 -0.41 14 13 -1 Provides synergy between thinking and doing 6.77 6.40 -0.37 12 14 +2 Provides a challenge for all pupils, irrespective of ability 6.85 6.38 -0.47 15 15 0 Empowers pupils to be autonomous 6.72 6.32 -0.40 16 16 0 Helps pupils develop planning skills 6.65 6.28 -0.37 17 17 0 Opportunity to prepare for a career 6.58 6.20 -0.38 Fig 3. Position of Statements for SQ1 after two ranking rounds. The process was then repeated for a third, and final, round. Again, the outcomes were calculated and these, along with the original statements were shared with participants who were asked to provide a value judgement against each. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 41 Position Position Difference Statement Descriptor Being Considered Mean Mean Difference Round 2 Round 3 Round 2 Round 3 1 1 0 Enables pupils to engage in real world problem solving 7.48 7.62 +0.14 2 2 0 Designing real world artefacts, for real world issues 7.35 7.55 +0.20 3 3 0 Encourages and promotes pupil autonomy 7.30 7.51 +0.21 4 4 0 Promotes creativity 7.10 7.34 +0.24 5 5 0 Develops designing skills 7.04 7.10 +0.06 6 6 0 It is fun and engaging 6.97 7.01 +0.04 7 7 0 It is stimulating 6.84 7.00 +0.16 8 8 0 Opportunity to learn from modelling and experimentation 6.83 6.94 +0.11 10 9 -1 Develops product analysis skills 6.78 6.82 +0.04 9 10 +1 Helps develop criticality 6.80 6.80 0.00 14 11 -3 Provides a challenge for all pupils, irrespective of ability 6.38 6.70 +0.32 11 12 +1 Manipulation and combination of materials to make a product 6.70 6.61 -0.09 12 13 +1 Appeals to those who are less academically gifted 6.40 6.53 +0.13 15 14 -1 Empowers pupils to be autonomous 6.32 6.42 +0.10 13 15 +2 Provides synergy between thinking and doing 6.40 6.41 +0.01 16 16 0 Helps pupils develop planning skills 6.28 6.35 +0.07 17 17 0 Opportunity to prepare for a career 6.20 6.33 +0.13 Fig 4. Position of Statements for SQ1 after the third, and final, ranking round. Final analysis of the outcomes from SQ1 illustrate that there is still movement within the statistical mean of individual statements. Lower down the listing there is also movement in the rank order between responses. However, the results reflect that consensus has been reached on the first eight statements. So, in seeking to answer SQ1, determination of the data has resulted in the following hierarchal list of considered responses: 1. Enables pupils to engage in real world problem solving PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 42 2. Designing real world artefacts, for real world issues 3. Encourages and promotes pupil autonomy 4. Promotes creativity 5. Develops designing skills 6. It is fun and engaging 7. It is stimulating 8. Opportunity to learn from modelling and experimentation The process was then repeated for the remaining two sub-questions; sub-question 2 (SQ2); what values does Design and Technology instil in students who study it? Also, for sub question 3 (SQ3); what are the unique features that Design and Technology offers to the curriculum that other subjects do not? The process for determining responses and reaching a consensus amongst the research sample was identical to that which has been articulated in the narrative for SQ1. As there was no deviance from the aforementioned process, for reasons of brevity, only the final data set will be presented following which there will be an analysis of the outcomes. Sub Question 2 (SQ2): What values does Design and Technology instil in students who study it? As in the determining of a set of consensus statements for SQ1, the first round of the study was used to elicit a set of responses from the research cohort. In a similar finding to that initially experienced in gathering the opening statements for SQ1, responses varied greatly from individual participants. These ranged from single word responses, to bullet pointed lists, to short descriptive sentences. They were typified by responses such as: “Design and Technology is a subject that applies knowledge and skills in the pursuit of realising a solution to a problem” Participant 11 “Fun, Dynamic, Active, Creative and Engaging” Participant 17 Using this set of statements, an identical process was followed to that demonstrated in the three rounds described in determining a consensus outcome for SQ1. Consequently, for the sake of brevity, the set of statements are presented here in their final ranking order, PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 43 highlighting positional moves between their final position and their subsequent ranking position determined at the end of the second round. Position Position Difference Statement Descriptor Being Considered Mean Mean Difference Round 2 Round 3 Round 2 Round 3 1 1 0 Trial and error in pursuit of solving a problem 7.72 7.81 +0.09 2 2 0 Allows pupils to develop empathy, recognising the needs of others 7.68 7.77 +0.09 3 2 -1 Develops appreciation for all 7.64 7.77 +0.13 4 4 0 Provides freedom for individuals, and groups to make decisions 7.60 7.71 +0.11 5 5 0 Effort and determination of the individual 7.58 7.64 +0.06 6 6 0 Working collegiately in teams to develop a common understanding 7.55 7.60 +0.05 7 7 0 Encourages effort and perseverance 7.53 7.58 +0.05 8 8 0 Promotes opportunity for innovation 7.50 7.55 +0.05 9 9 0 Ability to be creative 7.49 7.51 +0.02 11 10 -1 Quality, in both design and manufacture 7.41 7.45 +0.04 10 11 1 Being organised, sequential and methodical 7.45 7.43 -0.02 12 12 0 Practical ability as well as intellect and the link(s) between them 7.36 7.33 -0.03 13 13 0 Allows for forward thinking, and conceptual design 7.35 7.27 -0.08 Fig 5. Position of Statements for SQ2 after the third, and final, ranking round. Analysis of the data reveals that in the final ranking a number of respondents increased the value they placed on the top ten statements. This is evident by the rise in the statistical mean for these statements between the second and third ranking rounds. There are also two statements ranking in equal second position. This could lead to the conclusion that saturation of responses has not yet occurred with respect to SQ2. If this assumption is accurate, then it suggests that this question may have benefited from another round of ranking by participants to ensure that a consensus in the final ranking order has definitively been reached. In alignment with the work of Witkin and Altschuld (1995), PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 44 statements of equal value can be considered as being agreed, however, the caveat which is usually applied is that there is no movement in the ranking order between rounds, not that they have achieved the same mean value. Due to the limitations of potentially not reaching saturation (Barbour, 2001), despite following best practice guidelines (Duffield, 1993) it is felt that these two descriptors should still be included due to their relatively high placing in the ranking. Being cognisant of the potential limitation outlined previously, it can be said that the study has revealed that the participants have determined that the following statements provide a hierarchal set of values which provide an answer to SQ2 although they be a slight amount of uncertainty to the definitive positioning of two of the statements What values does Design and Technology instil in students who study it? 1. Trial and error in pursuit of solving a problem 2. Allows pupils to develop empathy, recognising the needs of others * 2. Develops appreciation for all * 4. Provides freedom for individuals, and groups to make decisions 5. Effort and determination of the individual 6. Working collegiately in teams to develop a common understanding 7. Encourages effort and perseverance 8. Promotes opportunity for innovation 9. Ability to be creative * Note: Statements of Equal Ranking Sub Question 3 (SQ3): What are the unique features that Design and Technology offers to the curriculum that other subjects do not? The method followed in determining a consensus to SQ3 followed the same process as that adopted for SQ1 and SQ2. Following the initial collection of responses, a set of statements were drafted and circulated to participants. Three participants requested that additional statements were added to those circulated, or that existing ones were modified so PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 45 as to include elements for consideration which they felt had been omitted from the statements they were asked to assess and evaluate. Given the complexities and subjective nature of determining what is a “unique feature” of a given subject, it is maybe not be surprising that this was the outcome of the first stage. Indeed, maybe it is more surprising that this did not happen with the previous two sub questions due to breadth and depth of experience each participant brought with them and the diverse nature of responses initially recorded. In analysing the responses to SQ3 it can be seen that a number of the statistical means attained by some of these statements were very similar, leading to the conclusion that many of the participants struggled to prioritise one statement over another (Fig 6). In so doing, one can further surmise that many of these statements attained scoring grades in the final round of ranking of either seven or eight from participants in the study. Position Position Difference Statement Descriptor Being Considered Mean Mean Difference Round 2 Round 3 Round 2 Round 3 1 1 0 Combining different materials together to make a product 7.81 7.88 +0.07 2 2 0 Applying theoretical knowledge and practical skill to develop products and make things better 7.80 7.84 +0.04 3 3 0 Practical application of other subjects (Maths, Science) 7.78 7.83 +0.05 4 4 0 Designing (and making) real solutions for real problems 7.77 7.78 +0.01 5 5 0 Designing for future needs and opportunities 7.71 7.72 +0.01 6 6 0 Problem solving; Realise there is more than one way of arriving at a solution 7.67 7.70 +0.03 8 7 -1 Promotes creativity 7.63 7.63 0 9 8 -1 Fosters innovation 7.60 7.62 +0.02 7 9 2 Allows students to develop real empathy and appreciate the needs of others 7.65 7.59 -0.06 Fig 6. Position of Statements for SQ3 after the third, and final, ranking round. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 46 Having concluded the three rounds of ranking, it can be seen that the following has been arrived at as a consensus for statements which appertain to SQ3: What are the unique features that Design and Technology offers to the curriculum that other subjects do not? 1. Combining different materials together to make a product 2. Applying theoretical knowledge and practical skill to develop products and make things better 3. Practical application of other subjects (Maths, Science) 4. Designing (and making) real solutions for real problems 5. Designing for future needs and opportunities 6. Problem solving; Realise there is more than one way of arriving at a solution There are striking similarities in these statements most noticeably between statements four and five which relate to the design aspect of the subject. It is noticeable that the research cohort wanted the inclusion of “and making” adding to the fourth statement. Interestingly in terms of the English and Welsh national curriculum this has echoes of past Attainment Targets (ATs) and the associated Programmes of Study (POS) outlined in an early version of the national curriculum orders (DfE, 1995), where designing and making were considered as separate entities. These have been subsequently been removed in a number of curriculum revisions since then including the latest variant (DfE, 2016b). Considering this, if we consider the length of teaching service (Fig 1) of the research participants, then this particular policy may have been more influential than first anticipated. Using the data in Fig 1 it is possible to extrapolate an approximate age profile for the cohort, given that teachers can only enter training following a set period of schooling. It is likely that a significant number of participants were either pupils themselves at the time this policy was enacted, or they were educators tasked with its delivery and implementation. It is therefore likely that exposure to this disaggregation of ‘Designing’ and ‘Making’ has shaped and influenced the thinking of the participants, either consciously or subconsciously. Thus, participants associate these terms with the identity of the subject. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 47 2.2 Comparative Analysis. Comparative analysis shows that there are some trends within the data which transcend the sub questions themselves: All Statements Consensus Statements No. of Initial Responses Max Mean Value Min Mean Value Range Max Mean Value Min Mean Value Range SQ1 17 7.62 6.33 1.29 7.62 6.94 0.68 SQ2 13 7.81 7.27 0.54 7.81 7.51 0.30 SQ3 9 7.83 7.39 0.44 7.88 7.59 0.29 Fig 7. Comparison of mean values and range of mean values across sub questions. The number of initial responses decreases as more questions are asked. This suggests that there is more clarity of thinking in the cohort as they progress through the study. This is in line with other studies (Clayton, 1997; Linstone & Turoff, 1975; McKenna, 1994) which have adopted a Delphi technique, attributed to deeper and more reflective thinking by research participants as the study progresses. It is also evident that the range of statistical mean values of the statements decreases as the study progresses, both when considering all statements and when considering consensus statements. This further consolidates the assertion that participants are refining their thinking as the study progresses. Having now determined consensus answers to each of the sub questions it is possible to merge the outcomes and sort these into rank order based on mean values. In combining individual consensus statements, it is possible to draw further trends from the data (Fig 8). The research cohort ranked, on the whole, unique features of the subject (SQ3) above values of the subject (SQ2), and in turn values of the subject (SQ2) above attributes (SQ1). There is no evidence that other Delphi based studies consider the relationships between sub questions in such a fashion. Consequently, there is nothing to compare this trending observation with. Only by undertaking another study adopting with same approach to design and data analysis would it be possible to see if this was more than coincidence. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 48 Consensus Descriptor Mean Value SQ 1 SQ 2 SQ 3 Combining different materials together to make a product 7.88 ✓ Applying theoretical knowledge and practical skill to develop products and make things better 7.84 ✓ Practical application of other subjects (Maths, Science) 7.83 ✓ Trial and error in pursuit of solving a problem 7.81 ✓ Designing (and making) real solutions for real problems 7.78 ✓ Allows pupils to develop empathy, recognising the needs of others 7.77 ✓ Develops appreciation for all 7.77 ✓ Designing for future needs and opportunities 7.72 ✓ Provides freedom for individuals, and groups to make decisions 7.71 ✓ Problem solving; Realise there is more than one way of arriving at a solution 7.70 ✓ Effort and determination of the individual 7.64 ✓ Enables pupils to engage in real world problem solving 7.62 ✓ Working collegiately in teams to develop a common understanding 7.60 ✓ Encourages effort and perseverance 7.58 ✓ Designing real world artefacts, for real world issues 7.55 ✓ Promotes opportunity for innovation 7.55 ✓ Ability to be creative 7.51 ✓ Encourages and promotes pupil autonomy 7.51 ✓ Promotes creativity 7.34 ✓ Develops designing skills 7.10 ✓ It is fun and engaging 7.01 ✓ It is stimulating 7.00 ✓ Opportunity to learn from modelling and experimentation 6.94 ✓ Fig 8. Combined list of consensus descriptors, sorted by mean value. PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 49 3. Conclusion In comparison to the current iteration of the National Curriculum (DfE, 2016a, 2016b) findings from this study suggest that those tasked with teaching the subject of Design and Technology view some of its underlying values, attributes and principles differently to the curriculum document which defines it. This conclusion is drawn through the derivation and inclusion of statements by the research participants in this study which are not evident in curriculum documentation. These differences allude to the fact that the subject means different things to individual stakeholder groups, so it is little wonder that those who work outside of it struggle to understand its nuances and meanings (Hardy, 2016b). Without commonality in an approach to its definition it is hard to see how the subject can move forward in the face of constantly changing policy (DfE, 2011, 2013, 2016b) and educational direction (DfE, 2016a). This is in stark contrast to the practices seen in other parts of the world where an agreed and understood definition of the subject exists (Jones & De Vries, 2009). In applying the Delphi technique to this study, the process has arrived at the point of data saturation through multiple rounds of engagement by participants. The study has enabled participants to derive a consensus of opinion in determining responses applicable to each of the research sub questions (SQ1, SQ2 & SQ3). However, it is still clear that there is some degree of overlap between responses to each of the sub questions. In considering this overlap one must determine if the three sub questions actually answer the main research question, focusing on defining what Design and Technology is? From the outset, participants were provided with the overarching research question and the sub questions and at no stage did anyone question the relationship between them. The inference drawn is that to this group of participants, the questions did indeed define what Design and Technology is, from their perspective. Whilst not asserting that the findings presented here will halt the downward trajectory of uptake in the subject from pupils, it does suggest the principles on which a subject could be formed to take over from Design and Technology should it cease to exist in the future. Burns (2014) and Steeg (2008) argue that a new or refined subject which maintains the fundamental values of Design and Technology would be valued by those currently engaged in PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 50 the delivery of Design and Technology. This study has provided evidence of what those fundamental values mean to a group of classroom practitioners. It also offers further narrative that can be considered by those working outside of the subject in trying to understand it. In seeking to further validate the findings of this work, the opinions of other stakeholders should be canvassed to see if they concur with the outcomes. Using the same methodological approach, it would be possible to undertake comparative studies using different identified research cohorts from a range of stakeholders, these could include; pupils, parents, employers, industrialists and teacher educators amongst others or indeed an amalgam of different stakeholders. Considering next steps after this study, the outcomes from it do not provide a justification for the retention of Design and Technology. Neither do they present a view of where the subject should progress too in the future. Rather, the findings presented here seek to establish a base line from which any move forward can take place. Should the same staff be involved in the evolution of the subject who participated in this research, then paying due consideration to what they value about it most will ensure cooperation and “buy-in” as it evolves. If their views are to be considered as representative of the sector, then this provides a stable platform to understand how teachers of Design and Technology define the subject. Although it would be unwise to consider this as being definitive without further work to corroborate it. References Atkinson, S., (1990). Design and Technology in the United Kingdom. Journal of Technology Education, 2(1), Retrieved November 18, 2017, from http://scholar.lib.vt.edu/ejournals/JTE/v2n1/html/atkinson.html Barbour, R. S. (2001). Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ: British Medical Journal, 322(7294), 1115. Barlex, D. (2003). Creativity in Crisis? Design & Technology at KS3 and KS 4. Design and Technology Association. Barlex, D. (Ed.). (2007). Design & Technology: For the next generation: A collection of provocative pieces, written by experts in their field, to stimulate reflection and curriculum innovation. Cliffe Co Limited. Bell, D. (2012). Perceptions of STEM; within the context of teaching D&T in secondary schools: A phenomenographically inspired study. In PATT 26 Conference; Technology Education in the 21st PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 51 Century; Stockholm; Sweden; 26-30 June; 2012 (No. 073, pp. 74-80). Linköping University Electronic Press. Bell, D. (2016). The reality of STEM education, Design and Technology teachers’ perceptions: A phenomenographic study. International Journal of Technology and Design Education, 26(1), 61-79. Bell, D., Wooff, D., McLain, M., & Morrison-Love, D. (2017). Analysing Design and Technology as an educational construct: an investigation into its curriculum position and pedagogical identity. The Curriculum Journal, 1-20. British Educational Research Association. (2011). Revised ethical guidelines for educational research (2011). British Educational Research Association. Burns, J. (2014). Teachers fear for future of Design and Technology. BBC News. Chidgey, J. (2009). Building Schools for the Future: Implications for Design and Technology. Retrieved November 18, 2017, from http://admin.futurelab.org.uk/resources/publications-reports-articles/web- articles/Web-Article1316 Clayton, M. J. (1997). Delphi: a technique to harness expert opinion for critical decision‐making tasks in education. Educational Psychology, 17(4), 373-386. Cohen, L., Manion, L., & Morrison, K. (2013). Research methods in education. Routledge. Dalkey, N. (1967). Delphi. Retrieved December 4, 2017, from https://www.rand.org/content/dam/rand/pubs/papers/2006/P3704.pdf Dalkey, N. C., Brown, B. B., & Cochran, S. (1969). The Delphi method: An experimental study of group opinion (Vol. 3). Santa Monica, CA: Rand Corporation. Department for Children, Schools and Families. (1989). The National Curriculum, Her Majesty's Stationary Office (HMSO), London, England Department for Education. (1995). Design and Technology in the National Curriculum. Retrieved November 18, 2017, from http://www.nationalstemcentre.org.uk/dl/5090b96ee99320b3db7d6b9655abd7d6a15ad73b/17260- D%20and%20T%20in%20the%20national%20curriculum%201995_v2.pdf Department for Education (2011). The Framework for the National Curriculum. Retrieved November 18, 2017, from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/175439/NCR- Expert_Panel_Report.pdf Department for Education (2013). The National Curriculum in England; Framework. Retrieved November 18, 2017, from https://www.education.gov.uk/consultations/downloadableDocs/National%20Curriculum%20consultati on%20-%20framework%20document%20(2).docx Department for Education (2015). GCSE and A level subject content consultation. Retrieved November 18, 2017, from https://www.gov.uk/government/speeches/gcse-and-a-level-subject-content-consultation Department for Education (2016a). Types of School. Retrieved November 18, 2017, from https://www.gov.uk/types-of-school/overview PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 52 Department for Education (2016b). Education and Adoption Act 2016. Retrieved November 18, 2017, from http://www.legislation.gov.uk/ukpga/2016/6/pdfs/ukpga_20160006_en.pdf Department for Education (2017). Secondary accountability measures, Guide for maintained secondary schools. Academies and free schools. Retrieved November 20, 2017, from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/659860/Secondary_a ccountability_measures_guide.pdf Design and Technology Association. (2011). We Believe in D&T Campaign. Retrieved November 18, 2017, from http://www.data.org.uk/generaldocs/Manifesto/data_manifesto.pdf Design and Technology Association. (2015). Designed and made in Britain…? Design and Technology in schools is critical to the UK’s future success. Retrieved November 18, 2017, from https://www.data.org.uk/media/1196/damib-campaign-booklet.pdf Duffield, C. (1993). The Delphi technique: a comparison of results obtained using two expert panels. International journal of nursing studies, 30(3), 227-237. Dumville, J. C., Torgerson, D. J., & Hewitt, C. E. (2006). Research methods: reporting attrition in randomised controlled trials. BMJ: British Medical Journal, 332(7547), 969. Eggleston, J. (1996). Teaching Design and Technology. Open University Press. Gilbert, J. K., Boulter, C. J., & Elmer, R. (2000). Positioning models in science education and in Design and Technology education. In Developing models in science education (pp. 3-17). Springer Netherlands. Goodman, C. M. (1987). The Delphi technique: a critique. Journal of advanced nursing, 12(6), 729-734. Green, R. (2016). Warm, Government rhetoric versus cold, school reality. D&T Practice: The Design and Technology Publication for the Profession. (1). 2-3. Guest, G., MacQueen, K. M., & Namey, E. E. (2011). Applied thematic analysis. Sage. Gupta, U. G., & Clarke, R. E. (1996). Theory and applications of the Delphi technique: A bibliography (1975– 1994). Technological forecasting and social change, 53(2), 185-211. Gurney-Read, J. (2015, October 4). Half of teachers could leave the profession in two years. The Daily Telegraph. Retrieved November 18, 2017 from http://www.telegraph.co.uk/education/educationnews/11909401/Half-of-teachers-could-leave-the- profession-in-two-years.html Hallowell, M. R. (2009). Techniques to minimize bias when using the Delphi method to quantify construction safety and health risks. In Construction Research Congress 2009: Building a Sustainable Future (pp. 1489-1498). Hanafin, S. (2004). Review of literature on the Delphi Technique. Dublin: National Children’s Office. Hardy, A. (2015). What’s D&T for? Gathering and comparing the values of Design and Technology academics and trainee teachers. Design and Technology Education: an International Journal, 20(2). Hardy, A. (2016a). An assortment box of views: different perceptions of D&T's purpose and structure. Paper presented at the Technology Education for 21st Century skills: PATT32 conference. Retrieved November 18, 2017 from https://hubl.hu.nl/session/conference-program-papers http://www.telegraph.co.uk/education/educationnews/11909401/Half-of-teachers-could-leave-the-profession-in-two-years.html http://www.telegraph.co.uk/education/educationnews/11909401/Half-of-teachers-could-leave-the-profession-in-two-years.html PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 53 Hardy, A. (2016b). What’s the point of Design and Technology? Retrieved November 18, 2017 from https://www.stem.org.uk/blog/what%E2%80%99s-point-design-and-technology Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of advanced nursing, 32(4), 1008-1015. Hennessy, S., & Murphy, P. (1999). The potential for collaborative problem solving in Design and Technology. International journal of technology and design education, 9(1), 1-36. Hofer, B. K. (2000). Dimensionality and disciplinary differences in personal epistemology. Contemporary educational psychology, 25(4), 378-405. Hsu, C. C., & Sandford, B. A. (2007). The Delphi technique: making sense of consensus. Practical assessment, research & evaluation, 12(10), 1-8. Jones, A. T., & De Vries, M. J. (2009). International handbook of research and development in technology education. SENSE Pub. Keeney, S., Hasson, F., & McKenna, H. (2006). Consulting the oracle: ten lessons from using the Delphi technique in nursing research. Journal of advanced nursing, 53(2), 205-212. King, P. M., & Kitchener, K. S. (1994). Developing reflective judgment: Understanding and promoting intellectual growth and critical thinking in adolescents and adults. San Francisco: Jossey–Bass Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method: Techniques and applications (Vol. 29). Reading, MA: Addison-Wesley. Marsh, S. (2015a). Five top reasons people become teachers–and why they quit. Guardian Newspaper. Marsh, S. (2015b). Fact or fiction? The reasons teachers choose the job–and quit. London. The Guardian Teacher Network. Marton, F. (1986). Phenomenography—a research approach to investigating different understandings of reality. Journal of thought, 28-49. Mason, M. (2010, August). Sample size and saturation in PhD studies using qualitative interviews. In Forum qualitative Sozialforschung/Forum: qualitative social research (Vol. 11, No. 3). McKenna, H. P. (1994). The Delphi technique: a worthwhile research approach for nursing?. Journal of advanced nursing, 19(6), 1221-1225. McLellan, R., & Nicholl, B. (2013). Creativity in crisis in Design & Technology: Are classroom climates conducive for creativity in English secondary schools?. Thinking Skills and Creativity, 9, 165-185. Middleton, H. (2005). Creative thinking, values and Design and Technology Education. International Journal of Technology and Design Education, 15(1), 61-71. Miller, J., & McGimpsey, I. (2011). What’s Wrong with DT. London, UK: RSA Design and Society. http://www.thersa.org/projects/design/design-‐in-‐the-‐school-‐curriculum. Morrison, N. (2013, March 23). Are some A-level subjects better than others? The Telegraph. Retrieved November 18, 2017 from http://www.telegraph.co.uk/education/9948905/Are-some-A-level-subjects- better-than-others.html PRISM 1(2) Values in Education prism-journal.blackburn.ac.uk 54 Müller, F. H., & Kals, E. (2004). Q-sort technique and Q-methodology—Innovative methods for examining attitudes and opinions. In Forum Qualitative Sozialforschung/Forum: Qualitative Social Research (Vol. 5, No. 2). Norris, N. (1997). Error, bias and validity in qualitative research. Educational action research, 5(1), 172-176. Owen-Jackson, G. (Ed.). (2013). Debates in Design and Technology Education. Routledge. Powell, C. (2003). The Delphi technique: myths and realities. Journal of advanced nursing, 41(4), 376-382. Pring, R. (2004). The philosophy of education. Bloomsbury Publishing. Robson, C. (1993). Real world research: A resource for social scientists and practitioners- researchers. Massachusetts: Blackwell Pushers. Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: issues and analysis. International journal of forecasting, 15(4), 353-375. Sample, J. A. (1984). Nominal group technique: An alternative to brainstorming. Journal of Extension, 22(2), 1- 2. Steeg, T. (2008). Makers, Hackers and Fabbers: what is the future for D&T? In E. W. L. Norman, & D. Spendlove (Eds.), Proceedings of the Design and Technology Association International Research Conference. (63-73) Turner, C., (2017, March 10). Design and Technology GCSE axed from nearly half of schools, survey finds. The Telegraph. Retrieved November 18, 2017 from http://www.telegraph.co.uk/education/2017/03/10/design-technology-gcse-axed-nearly-half-schools- survey-finds/ Tsang, K. K. (2012). The use of midpoint on Likert Scale: The implications for educational research. Hong Kong Teachers’ Centre Journal, 11(1), 121-130. United States Department of Health & Human Services. (1977). National Institutes of Health (NIH) Consensus Development program. Retrieved November 18, 2017, from https://consensus.nih.gov/ Van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (Eds.). (2006). Educational design research. Routledge. Van Zolingen, S. J., & Klaassen, C. A. (2003). Selection processes in a Delphi study about key qualifications in senior secondary vocational education. Technological forecasting and social change, 70(4), 317-340. Williams, J. (2011). STEM education: Proceed with caution. Design and Technology Education: An International Journal, 16(1). Williams, P. L., & Webb, C. (1994). The Delphi technique: a methodological discussion. Journal of advanced nursing, 19(1), 180-186. Witkin, B. R., & Altschuld, J. W. (1995). Planning and conducting needs assessments: A practical guide. Sage.