Philip, G. (2018). How to evaluate the effectiveness of teacher talk. International Online Journal of Education and Teaching (IOJET), 5(3). 497-512. http://iojet.org/index.php/IOJET/article/view/473/249 Received: 11.05.2018 Received in revised form: 20.05.2018 Accepted: 29.06.2018 HOW TO EVALUATE THE EFFECTIVENESS OF TEACHER TALK Philip Glover Süleyman Demirel University pgaglover@gmail.com Philip Glover has taught and trained teachers in Turkey, Cyprus, UK, Hungary and Serbia. He has worked in English language teaching as a manager and as a consultant with testing and in-service training projects. Copyright by Informascope. Material published and so copyrighted may not be published elsewhere without the written permission of IOJET. http://iojet.org/index.php/IOJET/article/view/473/249 mailto:pgaglover@gmail.com https://orcid.org/0000-0002-2375-3633 International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 497-512. 497 HOW TO EVALUATE THE EFFECTIVENESS OF TEACHER TALK Philip Glover pgaglover@gmail.com Abstract Talk is a key component of teaching and learning and teacher talk is an important part of how a teacher teaches. This paper discusses principles and approaches for analysing teacher talk in classroom interaction in order to investigate its effectiveness. The paper presents four reasons for analysing teacher talk: the role of interaction in the management of learning, the analysable, interpretable and explainable nature of discourse, the value of analysing effective teaching and the existence of categories of teacher talk that support the investigation of effective teaching. These categories are based on a framework of Elicitations, Responses and Descriptions from Mercer (1995). The paper also looks at issues of reliability, validity and generalisability of findings. The paper concludes that a focus on categories of teacher talk can support academic research, for the supervision and evaluation of teachers by managers, as well as for self-development purposes by teachers themselves. Keywords: classroom research, teacher talk, discourse analysis, elicitation. 1. Introduction: Classroom research, interaction and the effective management of learning Classroom research investigates classroom events to produce insights into the learning process. Classroom research is about ‘trying to understand what goes on in the classroom setting’ and categorises classroom events as ‘inputs to the classroom (the syllabus, the teaching materials etc.)’ and ‘outputs from the classroom’ (Allwright & Bailey, 1991, p. 2). Inputs are present in the classroom in the form of teaching, lesson content or materials. Interaction in teaching is a classroom input that has been widely debated and researched. Classroom interaction is encouraged because of its contribution to learning by facilitating lifelike or ‘genuine’ communication (van Lier, 1988). However, classroom interaction should not be viewed in the same way as other types of interaction. McDonough (2002, pp. 138- 139), says there is a need to ‘de-couple’ natural language learning processes from classroom interaction in order to see the classroom as an environment which has its own language with its own purposes and characteristics. A specific characteristic of classroom language is that it has a pedagogical purpose, to facilitate learning. One way of looking at this pedagogical purpose is as part of the management of learning, which involves teachers and learners in a ‘co-production’ (Allwright & Bailey, 1991, p. 25). Teachers and learners contribute to the co- production in different ways, for example by talking. Teachers make a key contribution to the management of learning through their talk. Mercer (1995, p. 1) describes communication ‘in which one person helps another to develop their knowledge and understanding’ as ‘the guided construction of knowledge’. According to Mercer the role of the teacher can be crucial to the success of learning, and he notes classroom research that identifies ways for teachers to guide learners more effectively. This guiding is described as ‘scaffolding’, which is seen as ‘a useful metaphor for the intellectual mailto:pgaglover@gmail.com Glover 498 involvement of a teacher with a learner’s efforts during joint activity’ (Mercer, 2000, p.159- 170). Teaching that uses scaffolding thus makes an important contribution to learning. A range of viewpoints have stressed the role of teaching and teachers in learning. The Russian psychologist Vygotsky, writing in the Thirties, noted the contribution to learning made by teachers (1962, 1978). Vygotsky saw the support and guidance that teachers provide in lessons through language as operating in a ‘zone of proximal development’ to facilitate learning. More recently advocates of teacher effectiveness have noted the importance of how teachers teach and have cited a range of language factors that contribute to learning such as explaining, questioning and interaction (Muijs & Reynolds, 2001). Mercer (2000) also has a view on effectiveness, summarising research that found more effective teachers display three characteristics. Effective teachers firstly use questioning to guide the development of understanding and to test knowledge. Secondly, in addition to teaching the content of the subject, they use procedures to assist problem-solving and making sense of classroom experiences. Thirdly, they support learning as a social and as a communicative process. Teachers’ contribution to learning is achieved, at least in part, through teacher talk. Teachers use talk in lessons, and ‘What is skilful and notable is the teacher’s deployment of … language to get things done in the classroom’ (Sinclair, 1982, p.10). In addition teacher talk has characteristics that are different from other types of talk: ‘Teacher talk is different from doctor talk and preacher talk and all the many other kinds of talk… (teachers’) discourse on the job is quite different’ (Sinclair, 1982, p. 11). The study of teacher talk in lessons, then, is a way of looking at teachers’ contribution to learning. One way of researching teacher talk is by analysing talk as discourse. 2. Discourse is analyzable, interpretable and explainable Classroom research into how teachers teach looks at classroom events and teachers’ actions in lessons. Analysing classroom talk, or classroom discourse, is one way of categorising teachers’ actions and ascertaining their consequences. Classroom research has looked at features of classroom discourse for different purposes. Three features of discourse are particularly relevant to studies and evaluations of how teachers teach: 1 Discourse is action through talk that is analyzable. 2 Discourse varies between people and situations and is interpretable as showing purpose. 3 Discourse requires detailed analysis and can produce social explanations. Feature 1- Discourse is action through talk that is analyzable. People use discourse in the form of talk to get things done, for example teachers use language to get learning done. Talk is viewed as ‘social action’ (Mercer, 1995, p. 67), and discourse analysis is a means of describing and interpreting this action. Studying classroom discourse can help to explain ‘what is being done in the discourse and how this is accomplished’ (Wood & Kroger, 2000, p. 95). Studying discourse can help researchers and supervisors of teachers to understand the function of different utterances or ‘what it is that people are doing in and with their talk and text’ (Wood & Kroger, 2000:7). Looking at discourse can show how people use words to ‘scaffold the performance of social activities (whether play or work or both)’ (Gee, 1999, p. 1). In the context of the language lesson International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 497-512. 499 teacher talk provides ‘scaffolding’ (Mercer, 2000, pp. 138-140) to support pedagogical purpose. Discourse is therefore analyzable with patterns that can be revealed through study: ‘Discourse can be regarded as sets of linguistic material that are coherent in organization and content and enable people to construct meaning in social contexts’ (Cohen, Manion, & Morrison, 2000, pp. 245). Patterns in talk can support a better understanding of the talk and ‘explore the organization of ordinary talk around everyday explanations and the social actions performed in them’ (Cohen, et al., 2000, pp. 298). Patterns in the teacher talk, then, enable us to describe and interpret teachers’ actions in the classroom. Feature 2- Discourse varies between people and situations and is interpretable as showing purpose. The second feature of discourse is that it is variable between individuals and situations Variations in discourse produce and reveal different viewpoints, as ‘talk constructs different versions of the world and is oriented to different functions; variability is to be expected not only between persons, but within persons’ (Wood & Kroger, 2000, p. 10). Variations in discourse can facilitate comparisons. Similarities and differences may be found between different teachers and between teachers in different situations. Discourse is also interpretable as showing purpose and is not just a ‘sequence of linguistic units’ (Schiffrin, 1994, p. 416). Discourse involves a message, a medium of communication and intersubjectivity between the participants, a ‘sharing of knowledge or experience’, and participants ‘use utterances to convey information and to lead each other towards an interpretation of meanings and intentions’ (Schiffrin, 1994, p. 386). In turn the researcher also has to interpret meanings and intentions in order to describe and explain the pedagogical purpose of teacher talk. Discourse should be viewed as a system ‘through which particular functions are realized’ (Schiffrin, 1994, p. 32), so patterns in discourse can reveal not only the form of the talk but also the purpose or function. The discourse is interpreted as evidence of the way of doing or teaching itself. Discourse is not viewed as evidence of teachers’ beliefs or ‘some internal event or attitude’ (Wood & Kroger, 2000, p. 14). In this way a discourse approach can help to focus research on how teachers teach and avoid confusion with attitudes or cognition: ‘the discourse itself is the educational reality’ (Cohen, et al., 2000, p. 300 quoting Edwards, 1991). Feature 3- Discourse requires detailed analysis and can produce social explanations A third feature of discourse is that the researcher must be careful to deal thoroughly with the data and analysis in order to be convincing. A study requires detailed analysis of empirical data: ‘discourse analysis requires a careful reading and interpretation of textual material’ (Mason, 1996, p. 200). Qualitative research should be ‘systematically and rigorously conducted’, ‘strategically conducted, yet flexible and contextual’ and ‘should involve critical self-scrutiny by the researcher, or active reflexivity’ (Mason, 1996, p. 5). Any study of classroom discourse needs to be strategically oriented, paying attention to established views of teaching, but the study also needs to be flexible, taking context into account. Discourse analysis should produce social explanations: ‘… meaningful elements in a complex… social world’, ‘… based on methods of data generation which are flexible and sensitive to the social context’ and ‘… based on methods of analysis and explanation building Glover 500 which involve understandings of complexity, detail and context’ and findings ‘… should produce social explanations which are generalisable in some way, or which have a wider resonance’ (Mason, 1996, p. 6). Discourse analysis can reveal positive or negative effects of teacher talk. Even a small-scale study of classroom discourse can show action that is repeated by many teachers and students. 3. Effective teaching The school and teacher effectiveness literature is based on international research in Britain, the United States, the Netherlands and a number of other countries (Campbell, Kyriades, Muijs, & Robinson, 2003; Creemers & Reynolds, 1996; HayMcBer, 2000; Muijs & Reynolds, 2001; Preedy, 1993; Reynolds, 1999; Reynolds, Davies, & Phillips, 1989; Reynolds & Muijs, 2000a, 2000b, 2000c, 2000d; Sammons, 1999; Sammons, Hillman, & Mortimore, 1995; Tabberer, 1994; Teddlie, 2000). The research notes the importance of teaching behaviours, identifying ‘concentration on teaching and learning’ and ‘purposeful teaching’ as factors producing effective schools (Sammons, et al., 1995, pp. 13-15). ‘Interactive teaching’ makes an important contribution to teaching and learning and involves such features as ‘direct instruction’, teachers listening and responding to students and providing clear instructions and explanations (Muijs & Reynolds, 2001, pp. 17-26). Teacher effectiveness research states that interactive teaching (Muijs & Reynolds, 2005) is more effective than teaching with an emphasis on individual work, although individual work in combination with interaction also supports learning. Whole-class teaching is thought to be effective because it ‘allows the teacher to make more contacts with each individual pupil’ and allows teachers to monitor and respond to students more quickly (Muijs & Reynolds, 2001, pp. 5). This enables teachers to adapt classroom procedures to meet student needs and keep students on task. Appropriate and varied questioning, probing for knowledge and frequent feedback are factors in interactive teaching (Reynolds & Muijs, 2000d). Nine factors contribute to effective interactive teaching: when and how often to use questioning, eliciting a pupil response, the cognitive level of questions, open and closed questions, process and product questions, what to do when a student answers correctly, what to do when a student answers correctly but hesitantly, what to do when a student answers incorrectly, what to do when a student answers partially correctly, and prompting (Muijs & Reynolds, 2005). A number of indicators contribute to effective interaction. Questioning should take up a large part of the lesson and involve ‘guided practice’. Eliciting should bring out a response from students in a positive, non-evaluative atmosphere. The cognitive level or difficulty of questions should require thought on the part of students and should be challenging but within the capacity of students to answer. Effective questioning involves asking more open questions and more process questions. Effective interaction acknowledges correct responses, responds with positive feedback to correct hesitant student utterances and provides supportive feedback to incorrect student utterances (Muijs & Reynolds, 2005). Effective teaching therefore involves prompts using cues, reminders or references to previous learning. In addition to interaction through questioning, the teacher effectiveness literature also comments on teacher talk that passes on information to students. Muijs and Reynolds (2005) cite types of information-giving talk: directing, instructing, demonstrating, explaining and illustrating, consolidating, evaluating and summarising. Effective directing involves the teacher sharing the aims of a lesson and clarifying points for particular attention. Effective instructing involves giving students information in a clear, comprehensible and structured way, and effective demonstrating involves showing or providing models. Explaining should be accurate, at an appropriate moment in the lesson and should help the students to make connections with past experiences. Consolidating should reinforce and develop points and International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 497-512. 501 encourage student reflection. Evaluating identifies student errors and summarising involves reviewing what has been done in a lesson (Muijs & Reynolds, 2005). Talk that presents information may also use questioning or prompting to support or guide learners’ understanding. There are many common elements between the teacher effectiveness literature and Mercer’s view of the ‘guided construction of learning’ (Mercer, 1995). Both are concerned with improving teaching. Both note the importance of the contribution of teachers to learning. Both note the role that discourse, especially interactive discourse, plays in the teaching and learning process. Analysis is not just a question of describing classroom discourse but also evaluating how well it achieves its purpose, and how well the ‘ways of guiding the construction of knowledge… seem suited to the kinds of learning they are supposed to encourage’ (Mercer, 1995, p. 41). These common elements provide a foundation for analysing classroom discourse and identifying the effectiveness of teacher talk. 4. Discourse analysis in language classrooms Discourse analysis has been used extensively to analyse discourse in language classrooms. Studies have used a range of different methodologies to produce insights into classroom interaction. These studies show the potential benefits of applying discourse analysis in education and applied linguistics. The studies also indicate the importance of an analytical framework to fit the purpose and context of the research. Sinclair (1982) studied classroom discourse and identified a pattern consisting of ‘Initiation’ by the teacher, ‘Response’ by students and ‘Follow-up’ by the teacher again. This is often referred to as the ‘IRF’ pattern. One of the purposes of Sinclair’s study was to help teachers ‘monitor their own performance’ (Sinclair, 1982, p. 5). The ‘IRF’ pattern was subsequently used by studies to analyse and comment on many aspects of language teaching (Cadorath & Harris, 1998; Cullen, 1998; Dinsmore, 1985; Duff, 2002; Hall, 1998; Kumaravadevilu, 1993; Nunan, 1987; Ohta, 1999; Thornbury, 1996). Several of these studies contrasted patterns in classroom talk with other types of communication (Kumaravadevilu, 1993; Nunan, 1987). They pointed out that the structure of classroom talk was dominated by one of the participants, the teacher, and this domination denied the other participants, the students, the opportunity to make language choices. This led to claims that ‘in communicative classes interaction may not be very communicative at all’ (Nunan, 1987, p.144). Others suggested that even teachers who support communicative teaching ‘can fail to create opportunities for genuine interaction’ (Kumaravadevilu, 1993, p. 13). The presence of the ‘IRF’ pattern has therefore been used to support a view that classroom interaction should replicate real-life communication in order to facilitate learning. Alternative views of classroom discourse, however, see talk in lessons in a different light, with the function of guiding and supporting learning. According to this view it is more important to understand how ‘IRF’ patterns ‘relate to the core institutional goal rather than dismissing them as undesirable or not genuine’ (Seedhouse 1996, p. 22). Discourse analysis has been used to produce a variety of insights into classroom talk and how teachers teach. For example, a study looked at issues of identity, respect and language socialisation in a school in Canada, investigating ‘the co-construction of knowledge, identity and difference in/through classroom discussion in high schools’ (Duff, 2002, p. 295). The study focused on two lessons to provide insights into relations between students and into teacher’s unsuccessful attempts to ‘make cultural connections based on their (the students’) own backgrounds, cultures and experiences’ (Duff, 2002, p. 310). Mori (2002, p. 325) looked at ‘the task designer’s intentions and ‘notions of ‘authenticity’ and ‘naturalness’ and student Glover 502 talk during a language learning task. Another study used systemic functional grammar to analyse the ‘socialisation of students to science discourse’ (Young & Nguyen, 2002, p. 348). This study compares language used by a teacher in a lesson with language in a text book. These studies point to two specific factors. Studies need to link methodology and the ‘needs of educators’ (Mori 2002, p. 341) in order to produce useful results and the approach used needs to fits the aims of the study and the context. The framework and categories described in this paper are introduced in Glover (2006) and the results were reported in Glover (2014). 5. Analysing teacher talk: Elicitation, response, description There are several requirements for a framework for the analysis of teacher talk. In order to investigate the effectiveness of teacher talk the framework needs to: O Consider pedagogical purpose focusing on the contribution of teachers to learning. O Investigate patterns that make a positive contribution to learning. O Fit the aims and context of the study. A framework based on teachers’ guidance strategies from Mercer (1995) provides a match with these requirements. Mercer’s framework is based on the concept of the ‘guided construction of knowledge’ which he puts at the heart of the combined process of teaching and learning that makes up education. Mercer looks at pedagogical purpose, especially the teacher’s contribution to learning. He notes that teachers make an important contribution through guidance strategies (Mercer, 1995). These strategies are ‘intentional, goal-directed ways of talking… which reflect the constraints of the institutional setting’. Teachers ‘use talk’ to ‘guide learning’ with three techniques: eliciting knowledge from students, responding to what the students say and describing shared classroom experiences (Mercer, 1995, p. 25-26). These techniques form the three main categories of analysis: Elicitation, Response and Description. Mercer divides eliciting, responding and describing into sub-categories that relate to different types of teacher talk in the classroom. Mercer identifies two types of Elicitation, Direct and Cued (Mercer, 1995, p. 26-32). Direct Elicitation uses questions to bring out knowledge from students. Cued Elicitations encourage student talk by providing a clue or prompt that leads to a student utterance, for example a teacher eliciting the word ‘pulse’ by saying ‘you can feel it here’(Mercer, 1995, p. 27). The Response category refers to teacher responses, not student responses, unlike the IRF pattern which puts student responses as the second step in the sequence. Teachers use Confirmation, Rejection, Repetitions, Reformulations or Elaborations (Mercer, 1995, p.32- 33). In Confirmations and Rejections a teacher may say ‘yes’ or ‘no’, ‘right’ or ‘wrong’. Repetitions draw the attention of the class to a remark thought by the teacher to be significant. Reformulations offer a revised version of what has been said. In elaborations a teacher extends a student statement. In the Describe category teachers use ‘we’ statements, Literal Recaps and Reconstructive Recaps (Mercer, 1995, p. 33-41). ‘We’ statements talk about a past experience that is relevant to the present, for example when a teacher reminds the students about something that happened in the previous lesson or reviews what has happened in the current lesson. Recaps review aspects of shared knowledge, for example, through reminders about previous lessons or drawing conclusions after a discussion or activity. Literal Recaps repeat the shared knowledge and Reconstructive Recaps add further interpretation. International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 497-512. 503 These guidance strategies contribute to learning. They provide a means of both identifying patterns in talk, and evaluating the contribution the talk makes to learning. Mercer recognises the presence of the ‘IRF’ pattern within these categories, describing it as ‘the ‘classic’ format of teacher-pupil interactions’ (Mercer 1995, p. 38). He recognises the constraints that the pattern puts on pupil contributions to lessons, but he also points out that teachers do more than just initiate and follow up student utterances. The framework using Elicitation, Response and Description categories can enable the researcher to consider pedagogical purpose and focus on the contribution of the teacher to classroom events. The framework relates to any general classroom context, so it may require adaptation to fit a language learning context in a specific place. The framework can help to identify patterns in teacher talk. Teacher effectiveness measured by analysis and interpretation of teacher talk should not be a simple matter of counting easily identified indicators. Effectiveness cannot be brought down simply to the length of teacher talking time or other numbers, and teacher effectiveness research has been criticised for becoming ‘a reductionist mode of research… replacing sociological analysis with increasingly complex statistics’ (Wrigley, 2003, p .89). Teacher effectiveness researchers themselves warn that research requires careful reflection by practitioners and: ‘a rather large “health warning” attached to any mandated set of teaching behaviours that may emanate from national policy makers’ (Muijs & Reynolds, 2005, p . 265). Instead, evaluating the effectiveness of teacher talk needs to investigate how the different categories and sub-categories of the talk combine with each other. Two key aspects of classroom discourse that feature in teacher effectiveness literature are interaction and providing information. Interaction involves questions, prompts, evaluation and correction by the teacher. Giving students information involves explanations and instructions. These aspects of classroom discourse match the Elicitation, Response and Description categories and are summarised in Tables 1 and 2: Table 1. Mercer’s elicitation and response categories, effective questioning factors and indicators of effective teaching Mercer’s Categories Effective questioning factors Indicators of effective teaching Direct and Cued Elicitations When and how often to use questioning Questioning takes up a large part of the lesson Direct and Cued Elicitations Eliciting a pupil response Questioning brings out responses from students Direct Elicitations The cognitive level of questions There are more questions that require the students to think, answers are not easy Direct Elicitations Open and closed questions More open questions are asked in effective lessons Direct Elicitations Process and product questions More questions about processes are asked Cued Elicitations Prompting Students are guided through cues, reminders or references Confirmation Responses What to do when a pupil answers correctly Correct answers are acknowledged Confirmation Responses What to do when a pupil answers correctly but hesitantly Uncertainty is clarified Repetition Responses What to do when a pupil answers incorrectly Incorrect answers are corrected or a correct answers is prompted Table 1 summarises the match between Elicitation and Response categories and teacher effectiveness criteria. The sub-categories of Direct and Cued Elicitations connect with when and how often to use questioning and eliciting a pupil response. Direct Elicitations connect Glover 504 with the cognitive level of questions, for example through open and closed questions and process and product questions. Cued Elicitations connect with prompting. Confirmation Responses relate to what a teacher does when a pupil answers correctly or correctly but hesitantly, and Rejection or Repetition Responses relate to what a teacher does when a pupil answers incorrectly. Table 2 summarises the match between Mercer’s Description category and the teacher effectiveness categories of directing, instructing, demonstrating, explaining and illustrating, consolidating, evaluating and summarising. Table 2. The description category, teacher effectiveness categories and indicators of effective teaching Category Teacher effectiveness categories Indicators of effective teaching Descriptions (recaps or ‘we statements’) Directing Teacher shares the aims of a lesson or clarifies points for particular attention. Instructing Teacher gives students information in a clear, structured way. Demonstrating Teacher shows or provides models Explaining and illustrating Teacher explanations are accurate, at an appropriate moment in the lesson and help the students to make connections with past experiences. Consolidating Teacher reinforces and develops points and encourages student reflection. Evaluating Teacher identifies student errors as positive teaching points as well as talking about student justifications of their answers. Summarising Teacher reviews what has been done in a lesson There are several different ways of gathering data for classroom research involving discourse analysis. Audio or video recordings of lessons can be used, either by listening to the recording and taking notes or by transcribing the talk (for example van Lier, 1988; Lyle, 2002; Mori, 2002). Studies have also used observation schedules such as COLT or FOCUS as described in Allwright and Bailey (1991). Observation schedules look at classroom discourse by using predetermined categories. Such schedules are useful because they provide a clear focus for observers and enable different observers to focus on the selected features of classroom events. Classroom events are complex and much happens at the same time in classrooms so the schedule helps observers focus on specific aspects. Observation schedules have some drawbacks, however. Compared to transcriptions of audio or video recordings they provide a less direct and less flexible version of classroom events. Schedules may not anticipate all relevant classroom features in advance, so a given schedule may not fit the purposes of a given study. For example, in one study ‘several significant activities were not specifically identified by either COLT or the UCLES instrument’ (Hayes & Read, 2004, p .103). Schedules produce a simplified version of an observer’s interpretation of what was said at a given moment: ‘the thirst to operationalise concepts and constructs can easily lead researchers to provide simple indicators of complex concepts’ (Cohen, et al., 2000, p. 310). Observation schedule categories and observer’s interpretation become the data rather than the actual words used by the teacher. International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 497-512. 505 The direct and flexible nature of transcribed recordings make them appropriate to studies of classroom discourse and address sampling issues. The audio recording allows transcribing and analysing all the teacher talk that occurs in a given lesson. This is a kind of ‘theoretical’ or ‘purposive’ sampling, which ‘means selecting groups or categories to study on the basis of their relevance to your research questions, your theoretical position and analytical framework, … and … the explanation or account which you are developing’ (Mason, 1996, p .93-94). Transcribed teacher talk provides a direct representation of how teachers teach. The talk is classroom action, one of the ways in which teachers perform the functions of teaching or guiding the construction of knowledge in learners. The talk shows whether and how the teacher might influence the learning process using an analytical framework based on Elicitations, Responses and Descriptions. Data from field notes, interviews and background data can also support an analysis of teacher talk. Transcribed talk can provide a detailed and even a complete picture of teacher talk in a lesson, but it cannot capture everything that happens in a lesson. The use of multiple data sources is therefore recommended as a means of providing a fuller picture of classroom events (Allwright & Bailey, 1991). 6. Discussion In the language classroom specific categories and sub-categories of teacher talk provide a way to evaluate the effectiveness of the teaching. Teacher talk data may be in the form of recordings of lessons, observation schedules or field notes. Having ascertained which types of teacher talk are present in the data or lesson, conclusions then need to be reached about the effectiveness of the talk. This can be done by a combination of quantitative and qualitative methods. Quantities to measure are the absence or overuse of specific categories and sub- categories. Qualities to identify are the sequences or combinations of the categories and sub- categories as well as other questions such as the comprehensibility, level of challenge or interactive nature of the talk. Just looking at numbers of occurrences can be deceptive, because what really makes talk effective are the qualities of that talk. A combination of both quantitative and qualitative methods is therefore likely to be beneficial to a study. Mercer’s categories and sub-categories are summarised in Table 3. Investigating these categories and sub-categories allows a researcher to look in close detail at both the quantities and qualities present in the discourse to see how a teacher teaches and to see the effectiveness of the teaching. Table 3. Mercer’s categories and sub-categories Category Sub-category Elicitations Direct Cued Responses Confirmations Rejections Repetitions Elaborations Reformulations Descriptions we statements Literal Recaps Reconstructive Recaps In terms of quantities, the presence or absence or the number of occurrences of each category or sub-category can give an impression of how a teacher teaches. For example, if a Glover 506 lesson consists mainly of Descriptions, with few or no Elicitations and Responses, then the effectiveness of the lesson is likely to be low. If the teacher spends most of the lesson giving information to the students without interaction there is no evidence in the discourse that the students are able to understand or engage with what the teacher is saying. Similarly, if the talk is heavily dependent on one type of Elicitation or Response, for example yes-no questions, which belong only to the sub-categories of Confirmations or Rejections, then the level of cognitive challenge is likely to be low. As for the qualities of the talk, the researcher can consider how the different categories and sub-categories fit together into sequences or combinations. The way a teacher follows up student utterances makes an important contribution to the effectiveness of the discourse. For example, after an elicitation, if a teacher responds to a student’s incorrect utterance simply by stating what is correct, that is likely to be less effective than a series of further elicitations that guide the student to the correct answer by encouraging self- or peer-correction. Cognitive challenge can be investigated by looking at different types of elicitations. Elicitations can be described as direct or cued. Direct elicitations are questions, of which there may be different types. The level of cognitive and linguistic challenge of questions can vary according to the type of questions. Levels of difficulty start with simple yes-no questions, questions offering options are slightly more challenging, and open or wh-questions can be more concrete or more abstract. Questions may be about processes or concepts, for example in language teaching concept questions are a technique that can be used in language classrooms to clarify structures or the meaning of vocabulary. Cued elicitations are prompts that bring talk out of learners, for example by means of the teacher starting an utterance for learners to complete, requests for the learner to say more, wait time or back-channel encouragement. Teachers can use a succession of Elicitations and Responses to encourage learners to think about the language they are using and to coax talk out of the students. When teachers use Elicitations they work with students to build knowledge together. this ‘embodies an educational process in which the pupils are neither being drawn out of themselves… nor simply being taught directly in the ‘transmission’ sense’ because when teachers use Cued Elicitations to bring talk out of students, students ‘are being inculcated into what becomes for them a shared discourse with the teacher (discourse in the broadest sense, including concepts and terminology as well as dialogue)’. This is ‘a communicative process of substantial intrinsic interest’ where ‘pupils’ knowledge is aided and ‘scaffolded’ by the teacher’s questions, clues and prompts to achieve insights that the pupils themselves initially seem incapable of’ and as a result ‘effective scaffolding reduces the learner’s scope for failure in the task, while encouraging their efforts to advance’ (Edwards & Mercer, 198,p. 143). In these ways, how a teacher combines different types of Elicitations and Responses can be seen to contribute to effective teaching. The researcher or observer should also consider other types of talk such as the Descriptions. In an English language class Descriptions are likely to involve reviews of learning from previous lessons or the current lesson, instructions concerning a task that is to be done, or explanations of language that is being studied such as grammar or vocabulary. Descriptions are likely to be less effective if they are simply a monologue where the teacher tries to pass information or knowledge to the students. Descriptions are likely to be more effective if they are combined with Elicitations and Responses which check comprehension, for example asking students to apply a rule, put a word into a sentence or repeat back instructions for a task. International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 497-512. 507 Indicators of effective teaching that may be found in Elicitations, Responses and Descriptions are as follows. Questioning takes up a large part of the lesson and brings out responses from students through the use of some questions that require the students to think such as more open questions. In effective lessons more questions about processes are asked, correct answers are acknowledged and uncertainty is clarified. Incorrect answers are corrected or a correct answer is prompted or guided through cues, reminders or references. A combination of different types of Elicitations and Responses are therefore needed to achieve effectiveness. The indicators of effective teaching that may be found in Descriptions are that at the start the teacher shares the aims of a lesson or clarifies points for particular attention, gives students information in a clear, structured way, and shows or provides models, teacher explanations are accurate, at an appropriate moment in the lesson and help the students to make connections with past experiences, the teacher reinforces and develops points and encourages student reflection and identifies student errors as positive teaching points as well as talking about student justifications of their answers. At the end of a lesson the teacher reviews what has been done in a lesson. Consideration of all these criteria build up a picture of the teacher talk in terms of its effectiveness. 7. Reliability, validity and generalizability Data analysis will involve a ‘detailed and repeated reading of the discourse against the discourse-analytic perspective’ (Wood & Kroger, 2000,p. 95). Analysis can be supported by computer software for qualitative data analysis. Measures of difference will be overall patterns in the discourse as revealed by the analytical framework, qualities of the categories and sub-categories in the framework, the presence or absence of certain types of categories or sub-categories and the frequency of occurrences. Detailed reading and analysis of the data follows a recursive process of ‘noting’, ‘collecting’, ‘thinking’ and ‘comparing’ (Seidel, 1998). coding the data according to functions and according to the categories of the analytical framework. Looking at each utterance in turn involves deciding whether words or groups of words are Elicitations, Responses or Descriptions, and if so what type of each category they are. An Elicitation could be Direct or Cued, for example. The next stage in the process involves collecting together the examples of each sub- category and sorting them into groups that appeared to be similar in terms of their function. Whilst doing this the groups that are forming, how they form into patterns and how they relate to the overall framework must be considered. At the same time the researcher checks and improves the original coding. Analysis can be supported by a suitable computer program for qualitative data analysis. Computer assisted qualitative data analysis has advantages such as speed and rigour, as well as the disadvantage that the tool itself rather than methodological principles may influence the study (Seale, 2000). Two popular computer software packages are ATLAS-ti and NUD*IST. ATLAS-ti has been reviewed as more user-friendly, simpler and easier to use, whereas NUD*IST offers more sophisticated analysis possibilities (Barry, 1998; Lewis, 1998). Atlas-ti’s strengths lie in its ‘interconnectedness and creative interface”, whereas NUD*IST is stronger on ‘sequential structure”, ‘project management’ and ‘sophisticated searching’ (Barry, 1998, p .12.1). Evidential links between the data and the effectiveness framework are crucial for the identification of effective teaching. The first means of establishing evidential links is in the reliability and accuracy of the data and the coding. The second means of establishing evidential links lies in the validity of the analysis. The third means of establishing evidential Glover 508 links lies in the generalisability of explanations. These three elements contribute to producing analyses and explanations which are convincing (Mason, 1996). Convincing claims depend on attention to reliability, validity and generalisability, and the ‘construction of explanations needs to be done with rigour, with care, and with a great deal of intellectual and strategic thinking’ (Mason, 1996, p .162). For reliability the effectiveness of teacher talk can be measured by the quantities and qualities of the discourse categories. Differences in quantities may occur in the number of occurrences of the categories and sub-categories in the analytical framework. There must be a reliable means of coding judgements as to what utterances belong to each category and sub- category. There also needs to be a coherent and reliable way of counting the categories: the amount of time spent on each category, the number of incidences of each category or the number of words used. Differences in qualities of the teacher talk may also occur. How the teacher applies the categories may be different in terms of function, pedagogical purpose, words used or content. Coding must be consistent and accurate. Grouping utterances in each category must be based on qualities that are relevant to the study. The reliability of the method also depends on the validity of data generation (Mason, 1996, p .145-146). Data in the form of transcribed teacher talk must provide a means of analysing how teachers teach (see Chapter 2.1 and 2.2 above). During analysis the data must fit the framework chosen. Then there needs to be evidential links between the data and the categories. The quantities and qualities of Elicitations, Responses and Descriptions represent judgements of how the teacher is teaching and how that teaching relates to the examination (Mason, 1996, p. 146). The use of multiple measures, viewpoints and analyses will add to the accumulation of judgements provided. However, Mason warns against expecting multiple measures to corroborate each other in providing more evidence of the same phenomenon. Different analyses imply different explanations, not more evidence for the same explanation (Mason, 1996, p .149). The value of multiple analyses will be in the additional evidence they present rather than in strengthening previous explanations. The validity of interpretation is shown by explaining and justifying the logic of methodological choices and by describing the route by which interpretation is reached, ‘continually and assiduously charting and justifying the steps through which (my) interpretations are made’ (Mason, 1996, p. 150). Finally the analysis must demonstrate that the explanations have ‘plausibility’ and ‘fruitfulness’ (Wood & Kroger, 2000, pp. 174-175) in terms of their generalisability and value (Mason, 1996, pp. 152-155). This will involve consideration of three points: 1. How the experiences of the teachers are relevant to other teachers’ experiences. Analysis must be detailed and holistic in order to show that what happens in the selected lessons can happen in other classrooms. 2. Analysis must show ‘the wider resonance or generalisability of (my) explanations’ ‘based on the rigour of my analysis’ (Mason, 1996, p. 154) by showing that explanations can produce benefits for teachers and that there are no adverse consequences for the teachers in the study. 3. The use of ‘aggregation, numbers and counting in a meaningful fashion’ (Mason, 1996, p .155) must address issues of sampling and significance. International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 497-512. 509 8. Conclusions This paper has argued that a focus on the categories of Elicitation, Response and Description in teacher talk can help to develop an understanding of how classroom interaction can be more effective. The approach outlined can be used primarily by academic research for investigating how teachers teach and its effectiveness for learning. The approach can also be adapted and used for the supervision and evaluation of teachers by managers. The categories can support teacher supervision by establishing criteria that can be agreed upon by the supervisor and the teacher. The categories also provide supervisors and teachers with common language that can make feedback sessions quicker and more productive because they facilitate discussions as in Walsh (2002, 2006). Teachers’ self-development can also be supported by having such language to assist teachers in reflecting on what happens in their own lessons. Glover 510 References Allwright, D., & Bailey, K., M (1991). Focus on the language classroom. Cambridge: Cambridge University Press. Barry, C. A. (1998). Choosing qualitative data analysis software: Atlas/ti and nudist compared. Sociological Research Online, 3(3), . Cadorath, J., & Harris, S. (1998). Unplanned classroom language and teacher training. ELT Journal, 52(3), 188- 196. Campbell, R. J., Kyriades, L., Muijs, R. D., & Robinson, W. (2003). Differential teacher effectiveness: Towards a model for research and teacher appraisal. Oxford Review of Education, 29(3), 347- 362. Cohen, L., Manion, L., & Morrison, K. (2000). Research methods in education. London: Routledge. Creemers, B. P. M., & Reynolds, D. (1996). Issues and implications of international effectiveness research. International Journal of Educational Research, 25(3), 257- 266. Cullen, R. (1998). Teacher talk and the classroom context. ELT Journal, 52(3), 179-187. Dinsmore, D. (1985). Waiting for Godot in the EFL classroom. ELT Journal, 39(4), 225-234. Duff, P. A. (2002). The discursive co-construction of knowledge, identity, and difference: a ethnography of communication in the high school mainstream. Applied Linguistics, 23(3), 289-322. Edwards, D., & Mercer, N. (1987). Common knowledge: The development of understanding in the classroom. Cambridge: Cambridge University Press. Gee, J. P. (1999). An Introduction to discourse analysis theory and method. London: Routledge. Glover, P. (2006). Examination influence on how teachers teach: A study of teacher talk (Unpublished PhD Thesis). Lancester. Glover, P. (2014). Do language examinations influence how teachers teach? International Online Journal of Education and Teaching, 1(3), 197-214. Hall, J. K. (1998). Differential teacher attention to student utterances: the construction of different opportunities for learning in the IRF. Linguistics and education, 9(3), 287- 311. Hayes, B., & Read, J. (2004). IELTS test preparation in New Zealand: Preparing students for the IELTS academic module. In L. Cheng, Y. Watanabe & A. with Curtis (Eds.), (pp. 3- 18). New Jersey: Laurence Earle Brown Associates. HayMcBer (2000). Research into teacher effectiveness: A model of teacher effectiveness. London: Department for Education and Employment. Kumaravadevilu, B. (1993). Maximizing learning potential in the communicative classroom. ELTJ, 47(1), 12-21. Lewis, B. R. (1998). ATLAS/ti and NUD·IST: A Comparative review of two leading qualitative data analysis packages. Cultural Anthropology Methods, 10(3), 41-47. http://www.socresonline.org.uk/socresonline/3/3/4.html%3e International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 497-512. 511 Lier, L. v. (1988). The classroom and the language learner:ethnography and second- language classroom research. Harlow: Longman. Lyle, S. (2002). Talking to Learn: The voices of children, aged 9–11, engaged in Role-play. Language and Education, 16(4), 303-317. Mason, J. (1996). Qualitative researching. London: Sage. McDonough, S. (2002). Applied linguistics in language education. London: Arnold. Mercer, N. (1995). The guided construction of knowledge: Talk amongst teachers and learners. Clevedon: Multilingual Matters Ltd. Mercer, N. (2000). Words and Minds: How we use language to think together. London: Routledge. Mori, J. (2002). Task design, plan, and development of Talk-in-interaction: An analysis of a small group activity in a Japanese Language Classroom. Applied Linguistics, 23(3), 289-322. Muijs, D., & Reynolds, D. (2001). Effective Teaching: Evidence and practice. London: Sage. Muijs, D., & Reynolds, D. (2005). Effective Teaching: evidence and practice (2nd ed.). London: Sage. Nunan, D. (1987). Communicative language teaching: making it work. ELT Journal, 41(2), 136-145. Ohta, A. S. (1999). Interactional routines and the socialisation of interactional style in adult learners of Japanese. Journal of pragmatics, 31, 1493-1512. Preedy, M. (1993). Managing the effective school. London: Open University. Reynolds, D. (1999). Better teachers, better schools: Pearson Publishing. Reynolds, D., Davies, R., & Phillips, D. (1989). The Cardiff programme: an effective school improvement project based upon school effectiveness research. International Journal of Educational Research, 13(7), 801- 814. Reynolds, D., & Muijs, D. (2000a). Advanced teaching methods (pp. 8). Pearson Publishing. Reynolds, D., & Muijs, D. (2000b). Effective teaching: British research (pp. 6). Pearson Publishing. Reynolds, D., & Muijs, D. (2000c). Effective teaching: international research (pp. 6). Pearson Publishing. Reynolds, D., & Muijs, D. (2000d). It's the classroom stupid (pp. 6). Pearson Publishing. Sammons, P. (1999). School effectiveness. Lisse, Exton, PA: Swets & Zeitlinger. Sammons, P., Hillman, J., & Mortimore, P. (1995). Key characteristics of effective schools. London: Office for Standards in Education. Schiffrin, D. (1994). Approaches to discourse. Oxford: Blackwell. Seale, C. (2000). Using computers to analyse qualitative data. In D. Silverman (Ed.), Doing qualitative research; a practical handbook (pp. 154-174). London: Sage. Seidel, J. V. (1998). Qualitative data analysis (pp. 15). Retrieved from www.qualisresearch.com Sinclair, J. (1982). The structure of teacher talk. Birmingham: University of Birmingham. Glover 512 Tabberer, R. (1994). School and teacher effectiveness. Slough: National Foundation for Educational Research. Teddlie, C., & R., David (2000). The international handbook of school effectiveness research. London and New York: Falmer Press. Thornbury, S. (1996). Teachers research teacher talk. ELT Journal, 50(4), 279-289. Vygotsky, L. S. (1962). Thought and language. Cambridge: M.I.T. Press, Massachusetts Institute of Technology. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge: Harvard University Press. Walsh, S. (2002). Construction or obstruction: Teacher talk and learner involvement in the EFL classroom. Language teaching research, 6(1), 3-23. Walsh, S. (2006). Talking the talk of the TESOL classroom. ELT J, 60(2), 133-141. Wood, L. a., & Kroger, R. O. (2000). Doing discourse analysis Methods for studying action in talk and text. London: Sage. Wrigley, T. (2003). Is 'School Effectiveness' anti-democratic? British Journal of Educational Studies, 51(2), 89-112. Young, R. F., & Nguyen, H. T. (2002). Modes of meaning in high school science. Applied Linguistics, 23(3), 348-372