AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell Inquiry into enhancing reflective learning in an online language course Robert Snell robsnell84@gmail.com City and Guilds (Freelance Content Editor), Great Britain Abstract In a learner-centered approach, lecturers constantly adapt their teaching methods to better suit their students’ goals and needs. City and Guilds examinations for speakers of other languages are high stakes language examinations approved by the UK Border Agency. An online examination preparation course has been developed that consists of activities and teacher presentations to help develop students’ language skills and improve examination performance. One of the aims of this course is to enhance reflective learning, using a teacher presentation of explanatory feedback after the controlled practice activity in the lessons. It is envisaged that this would provide an opportunity for students to identify gaps in language knowledge as well as re-balance any misunderstandings. The process would be similar to a situation in a classroom when students request explanatory feedback after they have completed a task. This paper presents the background to the course content, the learning framework of the online lessons and the format of the feedback presentations. Initial informal inquiries have revealed the feedback presentations are generally not currently prompting reflective feedback as evidenced by improved scores, as was originally predicted. However, all students were viewing the presentations consistently indicating that some level of knowledge consolidation maybe taking place. It may be possible that students are not fully aware of their purpose and the information contained is not always relevant to each particular individual student. Measures to address these issues are suggested and improvements to the learning experience are discussed. Keywords: online, examination preparation, reflective learning, teacher role, feedback Online language learning courses offer instant feedback and the opportunity for students to try activities more than once. One key element often lacking is explanatory feedback relating to the answers of the questions. It was envisaged that for students taking the City and Guilds online examination preparation course, motivation and completion rates could be improved by providing text and visual presentations with narration. The presentations are used to provide three areas of support for the learner: academic, practical, and emotive, as would be provided by the teacher in the classroom (White, 2003). With regard to academic support, activities carried out by the students are explained, the process of answering the questions is examined and incorrect answers are outlined carefully with regard to why they were incorrect. Encouragement (emotive support) and organisational information (practical support) about the course (length of study time etc.) are also provided. It is hoped that this will enhance the reflective element of the students' learning, improve progress and lead to greater motivation. This paper outlines an initial inquiry into an attempt at implementing teacher presentations to enhance reflective learning and summarises the how the initial findings have informed improvements to the learning experience. Initial student feedback at this stage consists of informal interviews and user statistics from a small sample of Chinese students. The examinations can be considered as one of the ‘high-stakes English tests’ alongside IELTS, TOEIC and others, approved by the UK Border Agency. The online course will soon be adopted by students preparing for the City & Guilds English examinations internationally and particularly in schools, universities and test centres in Eastern Europe and Asia (China, Malaysia, India, Pakistan, Sri Lanka and Indonesia). The City and Guilds English examinations are proficiency tests. They consist of International English for Speakers of other Languages (IESOL) and International Spoken English for speakers of other languages mailto:robsnell84@gmail.com AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell (ISESOL) examinations. The examinations are proficiency based level tests that refer to the Common European Framework of Reference for Languages: Learning, Teaching and Assessment (CEFR). The CEFR sets out to provide a common basis for the development of language syllabuses, curriculum guidelines, examinations and textbooks. There are 6 examinations, one for each level of the CEFR: A1, A2, B1, B2, C1 and C2. Each of the examinations seeks to assess the candidates’ ability to perform the can-do statements of the CEFR for each level. For example, the City and Guilds ‘Communicator’ examination is aligned to the B2 CEFR can-do descriptors. The ESOL examination consists of 3 skills: Listening, Reading and Writing. The SESOL examination assesses Speaking. The CEFR descriptors determine the design of the examination. For example: ‘can understand the way meaning is built up in a range of texts’ is a requirement for a B2 candidate. In the B2 Reading examination Part 2, the student is required to identify how meaning is built up using cohesive devices, discourse markers and referencing techniques. In this particular task the candidate is presented with a text with 6 sentences missing. They are given a separate list of 8 sentence options and are required to identify which sentences fill the gaps. To complete the task, the candidate would need to understand the referencing and discourse features of the sentences before and after each gap, as well as in the text as a whole. Therefore if the student completes the task successfully, we can say with some level of certainty that the student ‘can understand the way meaning is built up in a range of texts’ and meets this descriptor for B2 from the CEFR. Preparation course The examination preparation course is an online self-study course that provides approximately 35 hours of study time for each level. The course primarily provides an opportunity for the students to become familiar with the task types and the language performance requirements for each level. For each Part of the examination, there are 2-3 one-hour lessons focusing on key language skills and language features including: grammar structures, vocabulary items and functional phrases. The course consists of skills and language development activities, teacher presentations and mock examinations. The language features of the lesson material are determined by the language features and skills required to complete the examination tasks successfully. For example, B2 Listening Part 1 of the IESOL examination requires candidates to listen to a short dialogue and predict what is said next to continue the conversation. The candidate requires good awareness of intonation and stress patterns to do this. The activities in the lesson are focused on developing awareness of these features in order to be able to recognise the topic, speakers and context of the conversation and importantly, the emotion of the speakers. The Learning Process The course assumes that the learning process is dynamic and non-linear. Stages of learning do not follow consecutively from one another. However, it does assume that there are certain attractor states (Harshbarger, 2007). It assumes and relies on output as indicators of language ability at that point in the learner’s knowledge and application. It also assumes the learning path can be controlled and changes made to the learning path can enhance or detract from the effectiveness of the learning. The course follows a ‘noticing & sense-making, remembering and application, incorporating’ model as it is felt that this learning framework, when compared to alternative learning frameworks, leads to deeper understanding, new skills and knowledge are better integrated into previous knowledge and application of knowledge will be more fluent when required in the future. AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell Noticing and sense-making Language is presented in context with questions that require students to notice the language feature. This would help the student integrate the new skill into their previous knowledge. The learner finds something new and interesting and considers this to be true and correct (Harshbarger, 2007). Application and remembering Having introduced the language feature, the student would be asked to try out the new skill in a controlled secure environment. This would help the student try out their new understanding of the language feature with language input designed to highlight this language feature. This leans towards the Universal Grammar theory of learning (Chomsky in Cook, 1996). The input provides the basis to set the parameters based on the language feature. Although debatable, for the purposes of the course, it is also assumed that output is a form of applying language learnt and provides evidence of current state of understanding (Krashen, 1998; Swain, 1985 in Harshbarger, 2007). Allowing for processing theory and focus on practice, questions are given to the students (Cook, 1996). Incorrect answers are assumed to indicate performance errors and an incorrect assumption related to the language item being learned or a gap in learning to be filled. The feedback can help the student assess whether their assumptions were correct or help students identify their ‘gap’ (e.g. a missing vocabulary item). This process of sense-making encompasses temporary states which are then subsequently corrected and re-organised. If the student does not answer the questions correctly, then the student can re-formulate their parameters in light of this feedback and then try the activity again. This could be framed within the information processing model of McLaughlin (1983 in Cook, 1996). The aim is to help the knowledge move from controlled process of application to automatic and therefore each activity would generally contain a series of questions focused on the same language point. Incorporating Finally, at the end of the lesson, the student is faced with an examination task. This would provide motivation for learners and another opportunity to apply language features covered in the lesson. It is hoped that the knowledge would become incorporated into the learners’ identity and would be evident in their language competency from that point. In effect, one or more ‘can-do’ statements of the CEFR would better fit a description of the learner and this would hopefully be reflected in his or her examination results after taking the actual examination. Explanatory feedback – teacher presentations After the student has reached the end of the controlled practice activities, it is expected that sufficient information processing will have taken place. The students may develop cognitive disequilibrium – that is to say in this case the student applies his or her understanding of the language point and chooses an incorrect answer to a question. The incorrect answer reveals an inconsistency or an anomaly in relation to his or her understanding (Otero & Graesser, 2001). The feedback page of the online platform presents the ‘student’s answer’ and ‘correct answer’ (or model answer in the case of free writing and speaking activities). The presentation of the correct answer may be enough for some students to rebalance their understanding and to answer similar questions correctly in the future. However, in a classroom the student may then ask for an explanation with a ‘why is this option the answer, not this?’ or ‘what if I do this – is this OK?’ type of questions (from PREG model, Otero & Graesser, 2001). At this point in the lesson, the learner is presented with explanatory feedback presentation to highlight the reasons why the answers are correct from the previous activity or to highlight the language features of a model answer for productive skills. This could be referred to as the important 'reflective' point of the AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell learning process (Siemens, 2006). The aim is to consolidate understanding, as well as rebalance understanding and help students identify gaps in knowledge. It is assumed that in a classroom situation the student may seek additional assistance from the teacher to re-formulate their understanding of a language point if the questions have revealed to them an inconsistency in their understanding. The teacher explanation presentations aim to promote this classroom learning process in an online learning environment. In short, the explanatory feedback aims to prompt reflection and consolidation of learning or rebalance understanding, as well as increase motivation. At this point, encouragement is also given. The feedback pages for all activities on the platform contain a brief: ‘Well done’, ‘Try again’, ‘Excellent’ depending on the percentage of success. The teacher presentations and narration also contain the teacher saying: ‘If you got all the questions correct, well done. If you didn’t, you can try the activity again.’ It is hoped that this will provide additional motivation to the learner to repeat the activity and improve their score. The relevance of the activities in the lesson is also emphasised and the link between the language features and improved examination performance is highlighted. Initial inquiry This study is focused on the benefit of the explanatory feedback presentations presented to the student after the controlled practice activity. It was hoped that in cases where the student was unable to score 100%, the student would then view the subsequent presentation explaining why their answers were not 100% correct and then finally the student would return to the activity and improve their score. The sample of students is too small to provide conclusive findings but can provide an initial indication of student activity and reasons for that activity. The sample size was determined by the availability of data and access to students for follow-up inquiry. The initial feedback from students consists of user statistics from the learning platform from eight early adopter students and in-depth interview feedback with two students. The interviewees were asked to give reasons for his study patterns. Interview feedback Two individual students were asked about their reaction to the teacher feedback presentations. The teaching assistant was also questioned regarding any additional explanations provided to the students related to the activities of the online course. Interviewee 1 Reason for study: intending to move to UK as an investor Level: C1 Explanatory feedback presentations: average of 0.9 views per presentation (an average of ‘1’ would indicate that the student viewed all presentations exactly once, with no content skipped and no pausing) Average % score on examination-style activities: 87% Comments: works as a university professor, able to regulate learning, reflect on performance, took C1 test shortly after study completion (well-motivated) Interviewee 2 Reason for study: Professional development Level: B1 Explanatory feedback presentations: average of 0.86 views per presentation (an average of ‘1’ would indicate that the student viewed all presentations exactly once, with no content skipped and no pausing) AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell Average % score on examination-style activities: 94% Comments: currently has no intention of taking examination, course provides opportunity to develop English skills for work purposes Teaching assistant The students also have access to a teaching assistant. It was hypothesised that if the students had additional questions related to the activities that were not covered by the teacher presentations, then the student would ask the teacher assistant for an explanation. The students had access to the teaching assistant through phone calls and an online question submission system on the homepage of the course. They were not studying in the classroom as part of a blended learning course. The statistics showed that the students were looking at the presentations, although the number of views appeared to reduce rapidly to one or occasionally two views after the first unit of the course. It was also evident that the time spent viewing the presentation was on average longer than the original length of the presentation. This meant that the students were viewing the content completely and to the end of the presentation. Interviewee 2 spent approximately 175% of the original presentation length for each view (when he chose to view a presentation rather than skipping it). He explained that he paused the presentation as he listened. If he did not understand the narration clearly, he paused the narration, checked that he understood the text on the screen regardless of the narration. When he was comfortable that he understood the text, he continued listening. The emphasis placed on the text could reflect the focus on reading in Chinese schools. Listening ability is often below reading ability for many Chinese learners of English. Did the feedback presentations prompt students to try activities again and did their scores improve? The statistics revealed that of the 178 activity attempts that were followed by a feedback presentation, only 17% of the activities were attempted more than once. In the majority of cases (83% of the time), the student did not repeat the activity. Graph 3 (left): Repeats of activity preceding teacher feedback presentations Graph 4 (right): percentage scores for activities NOT repeated It’s also clear that for those activities not repeated, there was significant room to improve scores as indicated by the red slice of the pie in Graph 4. It was originally hoped that the students would view the teacher feedback presentations and then repeat the activity with an improved score. AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell Graph 5: When the activity was repeated Graph 5 shows that the students were almost always repeating the activity before viewing the presentation. The students were not repeating the activity after the feedback presentation in almost all cases. In over a quarter of the cases, the students scored 100% on the activity on the first attempt. It is understandable that the student would not repeat the activity again here. It is maybe more than a coincidence that scores of over 60% for any activity are shown as ‘completed’ on the online learning system. In only 2 cases was the score below 60 and not repeated (see Graph 4) until a score above 60 was achieved (for the activities followed by a teacher feedback presentation.) This indicates that achieving a score above 60 was important to the learners and was a key motivator behind repeating the activities. Interviewee 2 confirmed that the main motivation to repeat the activity was to improve his score and not due to the feedback presentations. He also stated that he was not sure of the benefit of the feedback presentations and in later units he began to skip them. He felt he had mastered the language skill of the lesson without the presentations and therefore did not need to view them. In this case, the lack of views could be explained by a combination of non-existent cognitive disequilibrium (he got the answers right anyway) and a lack of motivation (after all, he was not planning to take the examination). In the absence of deep goals, perhaps the learner cannot be expected to seek a deep explanation. Interviewee 1 prepared for the C1 examination by only following the online course and using no other form of assistance. The statistics indicated that she also skipped some of the presentations in later units. She indicated time pressure as the key reason. She felt that the activities themselves were the most useful and that the experience of the different question types and indication of level/requirements were the most helpful elements of the course. Given the time constraint (she only had 2 weeks to complete the whole course of approximately 40 hours of learning, while working full-time), she felt that she did not receive much benefit from the presentations. She also suggested that the presentations could be in video format to increase engagement. Both interviewees indicated that the presentations were difficult for them to understand at times. They were both very reliant on the text to ensure understanding. Given the effort required to understand the explanations, and the uncertainty over the benefit, interviewee 2 skipped some of the presentations in lessons he considered less relevant to his needs. The C1 student (a university professor of chemical engineering) indicated that time constraints were the main reason behind her decision to skip some of the feedback presentations. She was aware of the aim but AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell she did not feel that the information contained was pertinent to her needs. As a university professor, she could also be considered as a good learner in that she actively self-regulates her learning goals (Otero & Graesser, 2001). In this case, it’s possible that the explanatory feedback was additional to her learning goals. These were stated as to become familiar with the examination requirements of the level and questions types. She was familiar with students asking questions for further explanations. She recognised that the feedback was explaining some of the answers to the previous questions but she thought that not all the possible questions students might have were being covered. Regarding questions asked to the teaching assistant, there were no instances of the questions related to activities that were explained within the content of a teacher feedback presentation from the B1 or C1 course. Evidence of reflective learning The reflective learning can be divided into two stages of learner processing. For learners that had already scored 100% on the previous activity before viewing the explanatory feedback, the presentation can be said to be consolidating learning through reflection. The explanation could be moving the students (successful) process of application of the language point or skill from a controlled process to automatic. Although yet to be validated by more thorough research, the number of presentation views and length of view for the majority of the presentations suggest that the student placed some value on viewing the presentations and therefore considered them to be helpful in some way. The students were at liberty to skip all the presentations if they had wished. There were two situations when a student repeated an activity after viewing the feedback presentation. Reflective learning may have occurred, leading the student to balance an original cognitive disequilibrium or identify a gap in understanding. Case 1 Level B1 Lesson: Writing Part 3 Correcting errors Act 10 - Write the sentence with the correct capitalisation [students are presented with a sentence with incorrect capitalisation and required to rewrite the incorrect elements correctly] Act 11 – Teacher feedback presentation [Presents and confirms answers for all questions] The student attempted activity 10 and scored 0% on the first attempt. The student then viewed the feedback presentation (length of presentation: 1.23s, view time 1.29s: this is consistent with a complete view). The student had a break of nearly two hours and then attempted act 10 again. The student was able to score 66% on the second attempt. However, this activity is the same as the activity in which two students scored less than 60% without making further attempts to improve their score. In fact the average final score for the all six learners who completed this unit was 58%. This was an anomaly as most activities had a much higher average score. It seems likely that the activity itself had been constructed unfairly. As the activity required the writing of multiple sentences, it’s possible that minor typing mistakes could have led the students to consistently achieve a low score even after they had learnt and understood the language focus of the lesson. [The activity has now been improved.] Case 2 Level B1 AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell Lesson: Listening Part 4 Following a discussion Act 5 – Multiple-select activity [students are required to identify what topics are talked about by two teachers having a discussion, gist activity] Act 6 – Teacher feedback presentation [samples from the listening script are presented on the screen and the process of identifying the topic from the vocabulary is explained] Student scored 0% on three attempts and then viewed the presentation (presentation length: 1.48s, view time: 1.59s: this is consistent with complete view). The student scored 100% on the fourth attempt. In this case, as there was only one question item for the multiple-select activity, the scoring mechanism of the automated marking system meant that anything except all the items selected correctly is scored 0%. In both these cases, the student may have experienced frustration related to the construct of the activity. The student may have been confident of their mastery of the language point but were still unable to achieve the 60% score required for completion. Although it was disappointing to find evidence of poorly constructed activities in the course, the analysis has provided a method of identifying poorly constructed activities. Further developments The average number of views of the presentations and length of view indicate that for students who scored 100% on the previous activity, some consolidation of understanding could be taking place as a result of the teacher feedback presentations. This is supported by the absence of student questions directed to the teaching assistant. However, the feedback presentations have not proven as successful in enhancing reflective learning as evidenced by improved performance, as was originally predicted. Ideas to improve rate of reflective learning from teacher presentations: 1. Increase pass rate from 60% to 70% It appears the motivation to improve score is tied closely to the desire to repeat an activity. To reach a higher score of 70% the additional learning to be gained from the teacher presentations may become more necessary and useful. 2. To increase awareness of benefit by changing the title of and location of teacher feedback presentations The lesson overview presentations, feedback presentations and examination tips presentations are all labelled ‘Listen to the teacher’. This may have caused the student to fail to realise the full benefit of the teacher feedback presentations. An explicit activity title could be ‘Help from the teacher for activity 6’. Currently, in order to view the presentations related to a particular activity, the students must move on from the automated feedback pages indicating their score and the correct answers. If the student were able to view the presentation without moving away from the feedback page, the student may be more inclined to view it before repeating the activity. 3. Content – whose question is the feedback answering? The teacher explanations may be useful if it relates to the particular misunderstanding of the individual student but otherwise it may be irrelevant to his or her needs. 4. What if the students could choose the question and then watch a teacher-created video based on that question? This would make it more relevant to the students' needs and more engaging for the students. Teachers using the online course with their students could use a simple screen capture video tool and upload the video for all students to view. The students could view a list of questions, each with a related video. They would then select the video related to their own AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell question. If the question is not listed, then an explanation to answer this question could be requested from a teacher. This process would more closely mimic the question and answer process that would take place in the classroom. Ultimately, the majority of the students’ questions would have a video providing explanatory feedback. It may be possible to increase the standard number of ‘good’ questions asked by an average student in a classroom. In addition, the student has the flexibility to ask many questions without feeling that they are wasting other students’ time. 5. To increase validity of repeated activity as an indicator of learning: item banking If a student repeats the activity and answers the same questions as before it’s likely that memory of the answers and questions will assist the student in achieving a better score. To eliminate this issue, the student could be presented with different questions related to the same language point. 6. Reassess the format of the teacher presentation - too passive? 7. It’s likely that the presentation style does not prompt the active processing that is necessary for effective reflective learning to take place. The student is told how to apply the language skill to achieve the correct answer but it would be better if the student is required to form his or her own explanation first before listening to the teacher explanatory feedback. Research possibilities This initial investigation has shown that motivation for deeper learning appears could be tied to pass rate score. Students may see this as an indicator of satisfactory learning required to achieve success in the examination. Future research questions generated from this initial inquiry are: 1. Does an increase in pass rate percentage lead to an increased use of teacher presentations and improved scores on the online course activities? 2. Does viewing the teacher presentations increase learning evidenced by better results in the examination? Conclusion This paper has outlined the learning process following an initial inquiry into an attempt at enhancing reflective learning in an online language learning course. The predicted use of the presentations did not match the actual use of the early-adopter students. The inquiry has shed light on the correlation between student motivation and online language study activity. It has shown that motivation may be a key variable to determine the effectiveness of any attempt to promote reflective learning as part of an online course. This offers an interesting research possibility for the future. Further analysis of student use will hopefully improve the learning effectiveness of the course in the future. References Cook, V. (1996). Second Language Learning and Language Teaching 2 nd Edition. Edward Arnold Publishers Limited. Graesser A.C., McNamara D. S., & Van Lehn K., (2005). Scaffolding Deep Comprehension Strategies Through Point & Query, Autotutor, and iSTART. Educational psychologist, 40(4), 225-234, Lawrence Erlbaum Associates, Inc. Retrieved from: http://www.learnlab.org/uploads/mypslc/publications/graesser%20mcnamara%20vanlehn%20ep%200 5.pdf Harshbarger, B. (2007). Chaos, Complexity and Language Learning. Language Research, Bulletin, 22, Tokyo, ICU. Retrieved from: http://web.icu.ac.jp/lrb/volume-22.html http://www.learnlab.org/uploads/mypslc/publications/graesser%20mcnamara%20vanlehn%20ep%2005.pdf http://www.learnlab.org/uploads/mypslc/publications/graesser%20mcnamara%20vanlehn%20ep%2005.pdf http://web.icu.ac.jp/lrb/volume-22.html AsiaCALL Online Journal (ISSN 1936-9859) 2013 – Special Issue – AsiaCALL2012 Proceedings, Robert Snell Krashen, S. (1998). Comprehensible Output. System, 26, p175-182. Retrieved from: http://www.sdkrashen.com/articles/comprehensible_output/01.html Siemens, G (2006). Knowing Knowledge. Retrieved from: www.knowingknowledge.com Swain, M. (2007). Keynote presentation slides, Beijing Foreign Studies University 2007. Retrieved from: http://www.celea.org.cn/2007/keynote/ppt/Merrill%20Swain.pdf White, C. (2003). Language Learning in Distance Education. Cambridge University Press. Websites Council of Europe CEFR http://www.coe.int/t/dg4/Linguistic/CADRE_EN.asp City & Guilds English examinations: http://www.cityandguildsenglish.com/ City & Guilds ESOL/ISESOL online courses: www.cgetutor.com [Demo account login information available here: http://www.cityandguildsenglish.com/cat/120] UK border agency approved tests http://www.ukba.homeoffice.gov.uk/sitecontent/applicationforms/new- approved-english-tests.pdf For level alignment of City & Guilds examinations with Cambridge examinations: www.cambridgeesol.org http://www.sdkrashen.com/articles/comprehensible_output/01.html http://www.knowingknowledge.com/ http://www.celea.org.cn/2007/keynote/ppt/Merrill%20Swain.pdf http://www.coe.int/t/dg4/Linguistic/CADRE_EN.asp http://www.cityandguildsenglish.com/ http://www.cgetutor.com/ http://www.cityandguildsenglish.com/cat/120 http://www.ukba.homeoffice.gov.uk/sitecontent/applicationforms/new-approved-english-tests.pdf http://www.ukba.homeoffice.gov.uk/sitecontent/applicationforms/new-approved-english-tests.pdf http://www.cambridgeesol.org/