English Language Teaching Educational Journal ISSN 2621-6485 Vol. 5, No. 1, 2022, pp. 48-59 10.12928/eltej.v5i1.5779 http://journal2.uad.ac.id/index.php/eltej/index eltej@pbi.uad.ac.id Comparing the effectiveness of explicit EAL feedback through slideshow (text+audio) and captioned video Jonathan Harrison Nihon University, CST, 7-24-1 Narashinodai, Funabashi, Chiba 274-8501, Japan jon.harrison@nihon-u.ac.jp A R T I C L E I N F O A B S T R A C T Article history Received 20 February 2022 Revised 23 April 2022 Accepted 27 April 2022 Research seems to show that captions and subtitles are generally beneficial to learners of English as an Additional Language (EAL), but some research does contradict this. Research on slideware and slide design seems to focus more on attractiveness of slides and less on educational effectiveness. However, research on slide design and specific approaches to slide design continue to become more detailed. This study compared comprehension of explicit feedback received through either slideshow (text+audio) video or captioned video on an EAL writing task in an on-demand university setting. Results (n=163) indicated that approximately 50% of learners clicked the feedback video to advance to the quiz without viewing it completely. Of the learners with at least one full viewing (n=86), slideshow video seems to have engaged students for a longer duration than captioned video. The quiz items were easier for the slideshow video groups, and the quiz items performed better for these groups. The slideshow video groups had slightly higher means, but a significant difference between the effectiveness of slideshow video and captioned video to transmit feedback to students was not found. This is an open access article under the CC–BY-SA license. Keywords captioned video English as an Additional Language explicit feedback group feedback slideshow video How to Cite: Harrison, J. (2022). Comparing the effectiveness of explicit EAL feedback through slideshow (text+audio) and captioned video. English Language Teaching Educational Journal, 5(1), pp. 48-59. 1. Introduction During the 2020 and 2021 academic years, the number of educational institutions engaged in online education substantially increased due to the coronavirus pandemic. Video is an important component of many asynchronous, on-demand, lessons both for instructional and feedback purposes, and how instructors use video varies by course content, purpose, activity flow, access to technology, instructor technological skills and digital literacy, etc. For face-to-face instruction, slideware, such as Microsoft PowerPoint, has been popular for decades, and instructors use this for synchronous and asynchronous online instruction. Also, for synchronous and asynchronous online instruction, Zoom or other video conferencing software with recording functionality have become popular recently. With asynchronous online English as an Additional Language (EAL) instruction, learners on occasion request captions or subtitles for videos. In this report the term EAL will be used when discussing English as a Foreign Language (EFL) and English as a second language (ESL). Although there are studies comparing the effectiveness of captions and subtitles in various educational settings, comparisons specifically focusing on slideshow video (text+audio) and captioned video could not be located. Research on the effectiveness of captions and subtitles will be discussed further in the background sections as will the history of PowerPoint, some PowerPoint related research, and slide design research. In order to identify which practice is better when using video in the classroom, this report compares student performance on a quiz after students watched either a slideshow presentation with in-screen video of https://doi.org/10.12928/eltej.v5i1.5779 http://journal2.uad.ac.id/index.php/eltej/index mailto:jon.harrison@nihon-u.ac.jp http://creativecommons.org/licenses/by-sa/4.0/ http://creativecommons.org/licenses/by-sa/4.0/ http://journal2.uad.ac.id/index.php/eltej/article/view/5779 ISSN 2621-6485 English Language Teaching Educational Journal 49 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) the teacher speaking or a captioned slideware video of the teacher giving general feedback on a task regarding 10 common language mistakes on a previous task. The research questions were: 1. Do EAL learners engage longer with slideshow video or captioned video? 2. Does either format garner more views from EAL learners? 3. Which format is more effective for transmitting feedback content demonstrated through quiz scores? 4. If EAL learners are divided by quiz performance level, do all groups under both conditions perform similarly? 1.1 Background on Captioned Video We will begin with research regarding research on captions and native speakers, in this case of English. First, a differentiation between captions and subtitles must be made. Captions are in the same language as the audio, and subtitles are in a different language than the audio. Captions used in native- speaking education seem to be taboo or possibly have negative effects. Ritzhaupt et al. (2015), studied the effects of captions on native speakers using time-compressed video at 3 speeds of accelerated playback. A negative significant difference was found, and a significant difference was found toward normal video speed. These results indicate that learners may be able to accelerate video speeds to 1.5 times the normal speed, but will probably be less satisfied with the experience. Mayer (2021) suggested not adding captions that repeat spoken words. His article is on how to design effective instructional videos using evidence-based principles that are grounded in cognitive theories of learning and instruction. These principles include multimedia (present words and graphics), coherence (avoid extraneous material in slides and script), signaling (highlight key material), redundancy (do not add captions that repeat the spoken words), spatial contiguity (place printed text next to corresponding part of graphic), temporal contiguity (present corresponding visual and verbal material at the same time), segmenting (break a complex slide into progressively presented parts), pre-training (provide pre-training in the names and characteristics of key concepts), modality (present words as spoken text), personalization (use conversational language), voice (use appealing human voice), image (do not display static image of instructor's face), embodiment (display gesturing instructor), and generative activity (add prompts for generative learning activity). However, Liu (2018, 2019) responded to Mayer’s earlier work arguing that because Mayer’s theory and principles were developed based on empirical studies of native English-speaking students, they may not be applicable to EAL students. Specifically, Liu found issues with the modality and redundancy principles that involve text and audio saying that they could become compromised in the EAL context as learners have difficulty fully comprehending the English text and audio. Liu’s 2018 study sought to identify EAL learners' optimum input mode (graphics and audio, graphics and text, or graphics, audio and text) and to test whether the modality and redundancy principles also applied to their multimedia learning. Results indicated that when it came to knowledge retention, no statistically significant differences were found in EAL students' learning outcomes from the three input modes. These findings demonstrated that Mayer's modality and redundancy principles did not apply to content knowledge and vocabulary learning for EAL students when certain multimedia learning principles were followed. Some research supports that captioned video increases EAL learner performance. Hayati and Mohmedi (2011) studied the effect of English captions, Persian subtitles, and no captions or subtitles on listening comprehension of 90 EAL learners. Groups of learners watched 6 5-minute episodes of a DVD under 1 of these conditions, and after each viewing session, tests were administered to examine listening comprehension rates. The English caption group outperformed the Persian subtitles group at a “considerably higher level,” and the Persian subtitles group outperformed the no subtitle group at a “substantially higher level.” Arndt (2014, 2018) in research comparing vocabulary acquisition via blog post reading and watching YouTube video blogs reported that the caption group significantly outperformed the non-caption group. Ashcroft, Garner, and Hadingham (2018) suggest that subtitles lead to incidental vocabulary learning, demonstrated through recall, after a single viewing of a movie. Ketabi and Sadeghi (2020) performed research with captioned and non-captioned video in regard to the comprehension of idiomatic expressions. In this research both groups saw the video twice, and the group that viewed video with captions performed significantly higher than the group without captions. Zhang and Zou (2021) performed a literature review of 41 Social Sciences Citation Index journal 50 English Language Teaching Educational Journal ISSN 2621-6485 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) articles regarding multimedia input, including text images, audio, animation, and captions/subtitles in EAL learning and concluded that the effectiveness of three features were reported most frequently: audio-plus-animation-plus-captions/subtitles in developing vocabulary and grammar knowledge; audio-plus-animation in promoting listening comprehension; and text-plus-audio in facilitating reading comprehension and called for more research in these areas. These reports seem to demonstrate that for language learning captioned video leads to better comprehension of listening comprehension, idiomatic expressions, incidental vocabulary learning and vocabulary acquisition. In French and Spanish as an additional language, similar results have been put forth by a number of researchers. Fievez et. al (2020) studied incidental vocabulary learning with captions and subtitles with French learners, and results indicated that both subtitle and caption groups outperformed the control group in meaning recognition group but only the caption group outperformed the control group on meaning recall tests. Camacho Velez and Pozo Estévez (2021) in their research on vocabulary development concluded that audiovisual materials with captions have a positive effect on the development of second language vocabulary. Cintrón-Valentín and García-Amaya (2021) investigated the effect of captioned video on grammar and vocabulary through form-focused instruction. In this research a randomized control design was used to investigate the effect of captioned media on the learning of vocabulary and grammar. Through four data-collection sessions, participants (n = 369) were presented with a grammar-lesson video and a multimodal video with one of three captioning formats: textually enhanced target vocabulary, textually enhanced target grammar, or no captioning. Results showed strong immediate effects of captioning on target vocabulary and on some, but not all, of the target-grammar structures. According to the authors, learning of some grammatical structures is more conducive to captioning than others. These studies suggest that for language learning captioned video was more conducive to vocabulary development and learning some, but not all targeted grammar structures. Other research suggests that captioned video is better than non-captioned video, but results may vary by language of the caption and language proficiency level. Winke, Gass, and Sydorenko (2010) concluded via results from t-tests and two-way ANOVA, that captioned videos were more effective than non-captioned videos. However, they deduced that for performance on aural vocabulary tests captioning of the first showing of the videos was more effective. Yet, results varied by language of the captions, for Spanish and Russian, captioning of the first showing was generally more effective than captioning of the second, and for Arabic and Chinese, captioning the second showing seemed to be more effective. Learners revealed in interviews that they used captions to increase their attention, improve processing, reinforce previous knowledge, and analyze language. Some learners stated that they used captions as a crutch, for support, as they got through the videos. Further research by Gass et al. (2019) used eye-tracking to gather data and describe how captions help people learn during captioned-video watching. This work confirmed previous work in which captions generally promoted second language comprehension, and it also discussed the possible role of working memory with results showing “The two working memory groups went in opposite directions: the high groups reduced their caption reading time, whereas the low-working memory groups increased their reading time” (p. 97). This corresponded with Desjarlais (2017) who summarized first language research on multimedia learning and suggested that individual differences would account for variance in information processing during multimedia learning. These differences include working memory. The above research suggests that different proficiency groups may use captions in different ways. Regarding the benefits of subtitles, only a single study was found. Pujadas and Muñoz (2020) investigated caption and subtitle use for TV dramas viewed by secondary school EAL learners, and came to the seemingly obvious conclusion that subtitles led to significantly higher content comprehension than captions. Regarding improvements in pronunciation, results were mixed. Wisniewska and Mora (2020) studied if extended exposure to captioned videos would affect adult second language pronunciation, and tests were run to determine effects on speech processing skills (segmentation, speed of lexical access, and sentence processing), phonological accuracy in perception (ABX discrimination), and production (accentedness ratings). All control and non-control groups showed benefits in speech segmentation and speech processing skills irrespective of viewing mode. For phonological accuracy in perception, no significant differences were detected. In production, a focus on phonetic form improved pronunciation only in the absence of captions, whereas captioned viewing led to pronunciation gains as long as there was no focus on phonetic form. Results, therefore, indicate that ISSN 2621-6485 English Language Teaching Educational Journal 51 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) improvements in pronunciation can take place either with captions or without captions when learners’ attention is focused on pronunciation. As with some other studies, the authors believed cognitive overload may be the reason no benefits were obtained when attention was directed to pronunciation when using captions. There is research that suggests captions and subtitles have no significant effect. Gordon (2020) added captions to recorded lectures for EAL learners in an e-learning environment and found no significant effects. Korucu-Kis (2021), researching captioning as a scaffold for L2 listening to discover if the dual coding of aural and written stimuli may lead to more comprehensible input and in-depth processing, stated that the effectiveness of captioning does not have a significant impact of listening. This narrative literature review concluded that captions do not necessarily lead to improved comprehension, and caption effectiveness is influenced by learner, material, measurement, task, and L1/L2 characteristics. Kruger et al. (2014) in a study on eye movement found that in terms of attention distribution, subtitle language and comprehension, the language or presence/absence of subtitles did not have any significant impact on comprehension of a work discussed in a recorded lecture. However, the three groups in this study distributed their visual attention resources differently in tests that were indicators of short term and long term retention of knowledge respectively. The findings for one of the groups suggested that captions resulted in a higher retention of knowledge in the longer term. 1.2 Background on Slideware and Slide Design Research specifically comparing slideshow video with captioned video was limited or not able to be found in searches perhaps because slideshows are multi-modal including text, images, audio, video, etc. Therefore, a brief, limited summary of some of the research in the field will be given in chronological order. From PowerPoints debut in 1987, slideshows, overwhelmingly PowerPoint slideshows, became standard for face-to-face classroom situations, conferences, and business meetings. It was, and perhaps still is, generally accepted that slideshows support lectures, but the support is more related to aesthetics, attractiveness and entertainment than to effective learning. Although dated, Bartsch and Cobern (2003) explored the effectiveness of PowerPoint versus overhead transparency use in lectures. Results indicated that during semester transparencies were preferred but by end of term preferred PowerPoint and that students performed worse on quizzes when PowerPoint presentations included non-text items such as pictures and sound effects. In a second study participants were shown PowerPoint slides that contained only text, contained text and a relevant picture, or contained text with a picture that was not relevant. Students performed worse on recall and recognition tasks and had greater dislike for slides with pictures that were not relevant. They concluded that PowerPoint could be beneficial, but irrelevant material could interfere with learning. Craig (2006), in a well-researched and witty article, decries the lack of studies between 1987 and 2006 (less than 20) on the effectiveness of PowerPoint and the mountain of articles touting PowerPoint’s catchiness and entertainment value. He states, “Generally, the available studies lack substance and internal and external validity and adopt rather constrained characterizations of the concept effectiveness” (p. 149). During this time, it appears that researchers wanted to say that a product, in this case PowerPoint slideware, could create a statistically significant difference in learning; however, slideshow software has many different aspects and uses. Therefore, educators have been using it in many different ways that are complex and difficult to quantify. However, some researchers and designers realized this and have done research and offered advice on how to design better slides. A couple of these authors were Duarte (2008) and Reynolds (2012), and an internet search will offer quite a selection of books on presentation and slide design. Even now, ten years later the research focuses on a mix of aesthetics, attention-holding, and educational aspects. LeFebvre (2022) investigated two different PowerPoint slide designs for multimedia learning, presentation and teleprompter. Differences in participant fixations were assessed via eye tracking measures. Participants demonstrated greater fixation counts for teleprompter slides, measures of aesthetic liking evidenced that slides incorporating imagery resulted in more pleasurable learning experiences, and visually-based slides influenced more reflective learning and greater activation of information processing. One case study in scientific presentation slide design from this era is that of the assertion-evidence approach which uses a single sentence and a highly relevant graphic on each slide to ensure attention is drawn to the most important points in the presentation. This case demonstrates how researchers realized that slideware needed a slide design approach that would lead to educational gains for the 52 English Language Teaching Educational Journal ISSN 2621-6485 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) audience (and the presentation creator) and have tried to bring about change in the way presenters communicate information to audiences. Research on this approach seems to focus on audience comprehension and recall as well as slideshow presenter depth of content understanding. Garner et al. (2011) wrote about assertion-evidence slide design and how it may lead to better comprehension and recall of more complex topics within presentations. In this experiment two groups of roughly 55 audience members each watched a 6-minute presentation with either a topic-subtopic or an assertion- evidence slide structure, and they were tested immediately afterward and then again one week later. The authors reported that on comprehension items and items related to retention of more complex concepts, learners using assertion-evidence slides achieved higher scores than learners from topic- subtopic slides. This reportedly occurred on both tests, and some items achieved statistical significance. The authors point out that although learners of topic-subtopic slides viewed more written information during the presentation, those learners did not understand and remember that material. The authors go one to reiterate the importance of that point, referring to “additional benefits” of the assertion-evidence approach, and citing themselves (Garner et al., 2009; Alley, 2003) for theoretical support that presenters creating a presentation with this approach will develop “a more focused and overall stronger presentation” than if they used the traditional topic-subtopic approach. Garner and Alley (2016) compared “open” student-created slide presentations with assertion-evidence structured slide presentations, approximately equal groups, for 120 undergraduate engineers. They argue that students usually create topic-subtopic structure slides dictated by the default settings of PowerPoint or other slideware and support that indicating that over 80 percent of the “open” group created topic- subtopic slides. An unannounced post-test of comprehension was given 1 day later, and results revealed a statistically significant advantage (p < 0.05) for participants who created assertion-evidence slides. It appears that the assertion-evidence structure slide creation led to a statistically significant increase in the presenter’s understanding of the content, which seems to support earlier research by these authors. Independent research on assertion-evidence slide design audience comprehension and recall indicated significant gains for both slideware formats but better retention of information. Root Kustritz (2014) reported on a study of third-year veterinary students in a required theriogenology diagnostics course who blindly self-selected to attend either a presentation with PowerPoint slides in a traditional format or one with PowerPoint slides in the assertion-evidence format. Students took a pre-test, a take- home assignment, an online post-test, and another online post-test after one month to evaluate retention. Groups did not differ on pre-test, assignment, or post-test scores, and both groups showed significant gains from pre-test to post-test and from pre-test to retention test. However, the traditional group showed a significant decline from post-test to retention test, while the assertion-evidence group did not. Miraldi (2021) published a dissertation investigating the reasons that the assertion-evidence approach has not been more widely implemented. Miraldi writes, “Despite the theoretical guidance from cognitive psychology and multimedia learning, the common practice of plugging text into the default template of PowerPoint persists throughout educational and professional settings” (p. 14). Later, Gaskin (2012) is cited, but I will include more of the quote here. Gaskin, the co-inventor of PowerPoint, reflects on PowerPoint saying, “Since the defaults can easily be changed (any presentation made in PowerPoint can be set as the default style), and a single default can be augmented with unlimited libraries of templates constructed in any style desired, I’ve often wondered myself why users don’t change and replace the defaults more often.” (p. 417) Miraldi’s results indicated that two innovation attributes, compatibility and trialability, were significantly and positively associated with implementation. First, re-invention was a significant moderating variable, and, second, that workplace norms play a significant role in the diffusion of this innovation. In other words, it is hard to get people to rally around new slide design formats that are not defaults, and it is hard to change office/education/scientific community culture. In EAL research, some studies indicate that content comprehension is better when PowerPoint is used and that learners prefer PowerPoint with audio to other means of on-demand instruction. Gordani and Khajavi (2020) investigated the effect of PowerPoint-supported lectures on immediate comprehension and longer term retention of content by EAL university students. Participants (n=69) were randomly assigned into three groups: A. PowerPoint-supported lectures with slides’ contents being read out, B. PowerPoint-supported lectures with slides at propositional level, and C. oral presentation with no multimedia. During each of the twelve 1-hour sessions, the students were ISSN 2621-6485 English Language Teaching Educational Journal 53 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) presented with a lesson followed by a posttest of immediate recall, and they sat for the 1-month delayed posttest after the last session. Results indicated that learners’ comprehension improved significantly both immediately and after a 1-month delay when they are provided with PowerPoint- supported lectures with slides at propositional level. The authors offer a caveat stating that for efficiency, the content of the slides “must be selected and designed with great care so that key terms, issues and main ideas are covered. Copying course material into slides and including too much explanations and details will have an adverse effect on students’ retention and recall.” Oh (2021) in a report on blended learning writes that students preferred on-demand instruction (non-real-time online instruction) to real-time online instruction. Regarding content presentation, the students preferred class video containing a PowerPoint presentation and the instructor’s audio explanation to real-time instruction via Zoom, PowerPoint and instructor’s face, and PowerPoint only. This is by no means a complete review of the literature, but the gist of the research indicates there some researchers are vaguely researching PowerPoint while others are doing more detailed research on how aspects of slideware, slide design, and approaches to presentation making affect educational outcomes. The latter approach is where future researcher should concentrate their efforts. 2. Methods 2.1 Setting and Participants This study was conducted at a private university in Japan. Students (n=163) in 8 courses of English Communication II completed all three stages of an introduction task. Due to the coronavirus situation, all of the lessons concerned with this project were in an online, on-demand format on a Moodle LMS. 2.2 Research Instruments A quantitative design was used with 2 videos acting as variables and a quiz to measure the results. Students watched either a video of a PowerPoint slideshow with text on the slides recorded in Zoom which showed the teacher speaking in the upper-right corner (without captions) or a video of the teacher speaking with captions (no slides). There was not a control group with only audio and no written aspect as from an ethical standpoint all students were to receive the same content. Therefore, both videos had the same content and the same length. The common mistakes detailed in the videos are given in Table 1, which also includes the time each point began and the total time spent on each point. The quiz on the common English errors discussed in the videos contained 2 items for each of the 10 common errors. Contact the author for details. Table 1. Points of General Feedback and Duration Order Feedback Point Start time Duration Opening comments 0:00 1:32 1 Paragraph writing vs. sentence writing 1:33 1:40 2 Use of spaces after punctuation 3:13 1:33 3 Use of "in" for affiliation, e.g. I am in the x department. 4:46 1:23 4 General capitalization: first word, abbreviations 6:09 2:24 5 "What x do you like?" vs. "What do you like x?" 8:33 1:06 6 Name order in English and Japanese 9:39 2:41 7 Name capitalization and use of title and punctuation 12:20 1:23 8 Comma use before conjunctions when joining sentences 13:43 1:24 9 Use of "because" when joining sentences 15:07 1:01 10 Plurals/Use of S 16:08 1:12 Closing comments 17:20 0:20 Total 17:40 17:40 54 English Language Teaching Educational Journal ISSN 2621-6485 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) 2.3 Student Tasks, Data Collection Procedures, and Data Anaalysis The student task was to be completed in 3 stages (1 stage per lesson for 3 lessons). The first part of Lesson 1 focused on self-introductions, and meeting others, and asking questions, and the second part of the lesson focused on health and health during the pandemic. One of the final tasks in the lesson was for each student to post 1 new discussion in a forum. The directions were: “Introduce yourself to the other people in the course. Write or speak at least 5 good self-introduction sentences, and end with a question to the other students.” Scoring criteria and instructions on how to add a discussion, add audio and videos files, etc. were also given. The second stage of the task was in Lesson 2 where students were asked to reply to 2 other students’ questions with responses of at least 3 sentences each. Again, scoring criteria and instructions on how to reply were given. During this stage, the teacher edited student self-introductions from the first stage by putting asterisks where errors occurred and offering private feedback on posts that were lacking in amount of content or off-topic. The third stage of the project was the review and feedback stage. First, students were instructed to watch a 17-minute 40-second video of general feedback on common mistakes. Half (4 courses) of the students watched a video of a PowerPoint slideshow, and the other half of the students (4 courses) watched a video of the teacher speaking with captions. Second, students were asked to review their posted discussion and the replies from other students to their discussion questions and to notice the asterisked marked areas. Finally, students were asked to review the video again before taking a 1- attempt online quiz which consisted of 20 items. Regarding data collection, data was downloaded at the beginning of the sixth week. Therefore, students who had not completed the tasks due to late registration, absences, etc. were excluded from this study. Data from the quiz was analyzed to check for differences between the slideshow group and the caption group. An item facility analysis and item discrimination analysis were also conducted for each group to compare how the items performed for both groups. 3. Findings and Discussion 3.1 Results for All Participants Results for all students, shown in Table 2, suggested that both the slideshow and caption groups were similar. The minimums and means (M) were slightly higher for the slideshow group. For example, out of 20 total points, the mean score was .4 higher for the slideshow group, but with rounding to the nearest whole number the quiz score means and standard deviations (SD), the mean quiz time, and the mean of actual views were the same. The most noticeable difference was the mean time on video was 114 seconds (approx. 2 minutes) longer for the slideshow group. The p-values were high and did not denote significance. Regarding distributions, the slideshow scores were 61% within 1 SD and 100% within 2 SDs, and the caption scores were close to normal but with 75% within 1 SD. Table 2. Comparison of Scores, Views and Time on Video by Slideshow and Caption Groups Slideshow (n=76) Captions (n=87) Variable Range M SD Range M SD t df p Quiz Score 5-19 12.20 4.21 5-20 11.78 3.84 0.65 153 0.51 Quiz Time (s) 296-1201 849 243 239-1201 831 259 0.45 160 0.65 Views (clicks) 1-4 1.51 0.7 1-4 1.53 0.74 -0.14 160 0.089 Actual Views 0-2 0.73 0.61 0-2 0.7 0.51 0.62 148 0.054 Time on Video (s) 26-3301 1010 973 928-3567 896 727 0.84 137 0.04 3.2 Results for All Participants with at least 1 Complete View One issue with this analysis was that students clicked to view the video but many students clicked only as a means to open the quiz and expedite advancement through the lesson. Therefore, a second ISSN 2621-6485 English Language Teaching Educational Journal 55 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) analysis was conducted after removal of participants with 0 actual views, a view ending before the first feedback point ended (1 min. 40 sec.). This removed exactly 45% of each group. Therefore, 42 students from the slideshow group and 48 students from the caption group remained with .1 or more views, .1 representing the viewing of 1 feedback point from the video. When students with less than 1 full view of all 10 feedback points were removed (views under the duration of the video). This removed another 5% of the slideshow group from the data set but the caption group remained the same. What remained was 50% (n=38) of the slideshow group and 55% (n=48) of the caption group who had viewed the full video at least once. The results, shown in Table 3, indicate that for the caption group the range is one point higher and the views are negligibly higher. However, for the slideshow group the mean quiz score is .15 points higher, the mean of actual views is .1 higher, and the mean time on video is higher by 605 seconds (approx. 10 mins). A glance at the raw data showed that most students in the slideshow group were taking 1-2 minutes longer for viewing and were watching the video longer the second time. The distributions of the slideshow and caption groups were nearly identical with 81% and 79% within 1 SD and 100% within 2 SDs. Table 3. Comparison of Scores, Full Views and Time on Video by Slideshow and Caption Groups Slideshow (n=38) Captions (n=48) Variable Range M SD Range M SD t df p Quiz Score 5-19 13.22 4.15 6-20 13.06 3.69 0.17 74 0.86 Quiz Time (s) 326-1201 843 222 326-1201 859 259 -0.14 83 0.89 Views (clicks) 1-3 1.49 0.68 1-4 1.56 0.77 -0.06 83 0.95 Actual Views 1-2 1.18 0.4 1-2 1.10 0.26 1.39 61 0.17 Time on Video (s) 970-3301 1501 908 928-3567 896 535 1.13 57 0.26 3.3 Item Facility and Item Discrimination The quiz had 20 items, 2 items for each common error addressed in the feedback video. Item facility (IF) analysis and item discrimination (ID) present a picture of what items were difficult for students and how high and low proficiency students perform compared to each other. The feedback video lead to a quiz which was meant to be a criterion-referenced test; however, the quiz will be evaluated as a as a norm-referenced test as pre-tests and post-tests were not given as part of this feedback exercise. For the analysis IF and ID values were calculated for the entire slide group (n=76) and caption group (n=87) as well as the full view slide group (n=38) and caption group (n=48), shown in Table 4. Considering many students did not have a full view, item facility can be used as if for a norm-reference test. The lower the IF, the more difficult the item was for students. When an IF of .6 is used as a cut off to mark “difficult” items, meaning less than 60% of students correctly answered the item, the full slide group had 10 difficult items, and the 1-view group had 6 difficult items. The caption groups had more difficult items as the full caption group had 12 and the 1-view caption group had 10 difficult items. Therefore, the slide groups seem to have had less difficulty with the items. When the IFs for all 20 items are summed, the caption groups had lower IF totals (11.79, 13.05) than the slide groups (12.01, 13.22) suggesting that in total the slide group found the items easier. Regarding the feedback items and areas where these EFL students struggled even after the feedback video, looking at the IDs in Table 4, lower IDs represent items where the item both groups struggled with defining a paragraph (1a), capitalization of proper nouns (4ab, 7ab), comma conjunction (8a), connecting sentences with “because” (9ab) and plurals (10ab). The item description indicates the degree to which an item separates the students who performed well for those who did poorly on the test as a whole. An ID of .29 or lower indicates that the item performed poorly and may need adjustment (Brown, 2005). None of the items performed poorly for all four of the groups. However, the full slide group had 3 items and the full caption group had 7 items which performed badly. The 1-view slide group had 4 poorly performing items and the 1-view caption group had 5. When the IDs for all items are summed the full slide group and 1-view slide group total ID sums are higher (9.83, 9.76) than those of the caption groups (7.50, 8.47) suggesting that the items 56 English Language Teaching Educational Journal ISSN 2621-6485 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) perform better with the slide groups. Also, the items performed better for the full slide group than the full caption group on 14 items, and they also performed better for the 1-view slide group on 13 items. Items 2b, 6a, 6b, and 9a performed better for both caption groups, but it is unclear why this is the case. Table 4. Item Facility and Item Discrimination for Groups Item Feedback Point Slide IF (n=76) Cap. IF (n=87) Slide ID (n=76) Cap. ID (n=87) Slide IF (n=38) Cap. IF (n=48) Slide ID (n=38) Cap. ID (n=48) 1a Paragraph 0.49 0.45 0.69 0.27 0.53 0.58 0.83 0.50 1b Paragraphs 0.64 0.69 0.62 0.21 0.74 0.79 0.58 0.38 2a Spaces after punctuation 0.85 0.83 0.38 0.23 0.92 0.90 0.25 0.19 2b Spaces after punctuation 0.86 0.80 0.23 0.27 0.95 0.79 -0.08 0.31 3a Statement "in" department 0.72 0.77 0.65 0.35 0.84 0.85 0.42 0.19 3b Interrogative "in" department 0.68 0.74 0.73 0.39 0.79 0.75 0.50 0.31 4a Capitalization of proper nouns 0.42 0.32 0.77 0.57 0.53 0.50 1.00 0.63 4b Capitalization of proper nouns 0.51 0.45 0.77 0.78 0.61 0.54 0.67 0.81 5a Imperitive: Tell me what x you like 0.63 0.56 0.69 0.44 0.71 0.58 0.75 0.56 5b Interrogative: What x do you like? 0.72 0.67 0.38 0.39 0.76 0.71 0.58 0.44 6a Capitalization of names, name order 0.63 0.56 0.31 0.34 0.68 0.65 0.25 0.38 6b Capitalization of names, name order 0.79 0.78 0.15 0.32 0.82 0.83 0.25 0.38 7a Capitalization of proper nouns 0.41 0.45 0.31 0.24 0.37 0.44 0.50 0.25 7b Name order, cap. of proper nouns 0.54 0.59 0.42 0.40 0.66 0.67 0.68 0.38 8a Comma conjunction 0.59 0.52 0.62 0.42 0.68 0.52 0.33 0.69 8b Comma conjunction 0.78 0.77 0.46 0.28 0.87 0.81 0.33 0.44 9a Connecting because 0.35 0.41 0.38 0.64 0.26 0.56 0.50 0.69 9b Connecting because 0.41 0.45 0.62 0.38 0.42 0.54 0.67 0.56 10a Plurals 0.41 0.43 0.42 0.32 0.45 0.46 0.42 0.13 10b Plurals 0.58 0.55 0.23 0.26 0.63 0.58 0.33 0.25 4. Discussion This research focused on how 2 groups of learners performed on a 20-item grammar quiz after watching a feedback video regarding an online writing task. To briefly discuss the results, significant differences were not found between slideware groups and caption groups. These results are similar to Liu (2018) who did not find statistically significant differences when trying to identify an optimum input mode for EAL learners. The findings in this report, therefore, add to the existing literature that captions have no significant effect (Kruger et al, 2014; Gordon, 2020; Mayer, 2021; Korucu-Kis, 2021). That being said, there are caveats in the research; for example, findings for one of the groups in research by Kruger et al, (2014) suggested that captions resulted in a higher retention of knowledge in the long term. A caveat in the findings of this study is that although significant differences were not found between slideware and caption groups, EAL learners engaged longer with slideshow video, mean scores on quizzes were slightly higher for the slideshow groups, and quiz items performed better for the slideshow groups. These results also seem to support that learners prefer or are more comfortable ISSN 2621-6485 English Language Teaching Educational Journal 57 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) with the slideshow format with audio, e.g. PowerPoint, to other means of on-demand instruction (Gordani & Khajavi, 2020), in this case captioned video. To consider this more deeply, one area requiring more inquiry is regarding the length of learner engagement with the video and the quiz results. The point of view discussed in the previous paragraph is that the learners engaged longer with the slideshow video and this is a positive factor as it may have led to slightly higher mean scores for that group. However, it must also be considered that as the differences in mean scores between sideshow and caption groups were not significantly different that the excess time spent viewing the slideshow video was perhaps inefficient as it did not lead to significantly better quiz scores. Returning to the opening point of the discussion, if captions do not hinder learners as significant differences were not found in this study or in studies by the above mentioned researchers, then educators in EAL settings should include captions as research by other researchers seems to demonstrate that for language learning captioned video leads to better comprehension of listening comprehension, idiomatic expressions, incidental vocabulary learning and vocabulary acquisition (Wink et al., 2010; Hayati & Mohmedi, 2011; Arndt, 2014, 2018; Ketabi & Sadeghi, 2020). For educators teaching pronunciation, captions do not hinder learners as long as there is not a focus on phonetic form (Wisniewska & Mora, 2020). Limitations of this study are numerous as it was a short-term study focused on feedback comprehension of a variety of grammar points and long tern retention was not considered in the research design. The study also merely focused on the comprehension of text and audio and did not include graphics or animation as in other studies. Thirdly, slideshow text and captions were compared; therefore, first language subtitles were not a variable in this study. Lastly, some research suggests that different proficiency groups use captions in different ways, but proficiency level as measured by a standardized test or other means was not a part of this study. Research (Desjarlais, 2017; Gass et al., 2019; Wisniewska & Mora, 2020) does suggest that the proficiency levels of learners should be a consideration in future research regarding captions and working memory. The findings in this report add to the existing literature, but more specific, detailed studies are needed. As stated earlier, some researchers are vaguely researching PowerPoint while others are doing more detailed research on how aspects of slideware, slide design, and approaches to presentation making affect educational outcomes. The latter approach is where future researchers should concentrate their efforts as technology continues to be developed and integrated for educational purposes. 5. Conclusion The goal of this research was to compare the effectiveness of explicit EAL feedback between slideshow video and captioned video. This research was carried out at a university in an on-demand (asynchronous) educational setting, and in this setting approximately 50% of the learners clicked the feedback video to advance to the quiz without watching it. To answer the research questions based on the learners who did partake of at least 1 full viewing of the video, Q1. EAL learners engage longer with a slideshow video (text+audio) than with a captioned video. Q2. Regarding which format garnered more views, views (clicks) and actual views were almost exactly the same for all groups. Q3. The format that was more effective for transmitting the feedback content demonstrated via quiz scores seems to be slideshow video as the mean scores are slightly higher than those of the captioned video group. However, a significance difference was not demonstrated. Q4. When EAL learners were divided by quiz performance level, the slideshow group seems to have benefited more. The results of the item facility analysis suggested that the slideware group found the items easier. The areas where these EAL students struggled even after watching the feedback video were defining a paragraph, capitalization of proper nouns, comma use before conjunctions separating sentences, connecting sentences with “because,” and use of plurals. Item discrimination analysis suggested that the items performed better with the slideshow video groups. To conclude, slideshow video seems to have engaged students for a longer duration than captioned video. The quiz items were easier for the slideshow video groups, and the items performed better for these groups. The slideshow video groups had slightly higher scores, but a significant difference between slideshow video and captioned video to transmit feedback to students was not found. 58 English Language Teaching Educational Journal ISSN 2621-6485 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) REFERENCES Alley, M. (2003). The craft of scientific presentations. Springer-Verlag. Arndt, H. (2014). English vocabulary learning through watching YouTube video blogs and reading blog Posts. [Master’s thesis, University of Oxford]. Arndt, H. L., & Woore. R. (2018).Vocabulary learning from watching YouTube videos and reading blog posts. Language Learning & Technology, 22(1), 124–142. Ashcroft, R. J., Garner, J., & Hadingham, O. (2018). Incidental vocabulary learning through watching movies. Australian Journal of Applied Linguistics, 1(3), 135-147. https://doi.org/10.29140/ajal.v1n3.89 Bartsch, R. A., & Cobern, K. M. (2003). Effectiveness of PowerPoint presentations in lectures, Computers & Education, 41(1),77-86. https://doi.org/10.1016/S0360-1315(03)00027-7 Brown, J. D. (2005) Testing in language programs: A comprehensive guide to English language assessment. McGraw-Hill ESL/ELT. Camacho Velez, A. B. & Pozo Estévez, V. A. (2021). Effects of captioning on L2 vocabulary development. [Master’s thesis, University of Central Ecuador]. Cintrón-Valentín, M., & García-Amaya, L. (2021). Investigating textual enhancement and captions in L2 grammar and vocabulary: An experimental study. Studies in Second Language Acquisition, 1(26). https://doi.org/10.1017/S0272263120000492 Craig, R.J., & Amernic, J.H. (2006). PowerPoint presentation technology and the dynamics of teaching. Innovation in Higher Education, 31, 147–160. https://doi.org/10.1007/s10755-006-9017-5 Desjarlais, M. (2017). The use of eye-gaze to understand multimedia learning. In C. Was, F. Sansosti, & B. Morris (Eds.), Eye-tracking Technology Applications in Educational Research, 122–142. Hershey, PA: IGI Global. https://doi.org/10.4018/978-1-5225-1005-5.ch007 Duarte, N. (2008). Slide:ology: The art and science of creating great presentations. O'Reilly Media. Fievez, I, Perez, M.M., Cornillie, F., & Desmet, P. (2020). Vocabulary learning through viewing captioned or subtitled videos and the role of learner- and Word-Related Factors, CALICO Journal, 37(3). https://doi.org/10.1558/cj.39370 Garner, J., Alley, M., Gaudelli, A., & Zappe, S. (2009). Common use of PowerPoint versus assertion-evidence slide structure: a Cognitive Psychology Perspective. Technical Communication, 56(4), 331−345. Garner, J. K., Alley, M., Sawarynski, L. E., Wolfe, K. L., & Zappe, S.E. (2011). Assertion-evidence slides appear to lead to better comprehension and recall of more complex concepts. ASEE Annual Conference and Exposition, Conference Proceedings. Garner, J.K., & Alley, M.P. (2016). Slide structure can influence the presenter’s understanding of the presentation’s content. International Journal of Engineering Education, 32(1)(A), 39–54. Gaskins, R. (2012). Sweating bullets: Notes about inventing PowerPoint. https://www.robertgaskins.com/powerpoint-history/sweating-bullets/gaskins-sweating-bullets-webpdf- isbn-9780985142414.pdf. Gass, S., Winke, P., Isbell, D. R., & Ahn, J. (2019). How captions help people learn languages: A working- memory, eye-tracking study. Language Learning & Technology, 23(2), 84–104. https://doi.org/10125/44684 Gordani, Y., & Khajavi, Y. (2020). The impacts of multi-modal PowerPoint presentation on the EFL students’ content knowledge attainment and retention over time. Education and Information Technologies 25, 403– 417. https://doi.org/10.1007/s10639-019-09979-z Gordon M. (2020). The effect of adding same-language subtitles to recorded lectures for non-native, English speakers in e-learning environments, Research in Learning Technology, 28. https://doi.org/10.25304/rlt.v28.2340 https://doi.org/10.29140/ajal.v1n3.89 https://doi.org/10.1016/S0360-1315(03)00027-7 https://doi.org/10.1017/S0272263120000492 https://doi.org/10.1007/s10755-006-9017-5 https://doi.org/10.4018/978-1-5225-1005-5.ch007 https://doi.org/10.1558/cj.39370 https://www.robertgaskins.com/powerpoint-history/sweating-bullets/gaskins-sweating-bullets-webpdf-isbn-9780985142414.pdf. https://www.robertgaskins.com/powerpoint-history/sweating-bullets/gaskins-sweating-bullets-webpdf-isbn-9780985142414.pdf. https://doi.org/10125/44684 https://doi.org/10.1007/s10639-019-09979-z https://doi.org/10.25304/rlt.v28.2340 ISSN 2621-6485 English Language Teaching Educational Journal 59 Vol. 5, No. 1, 2022, pp. 48-59 Harrison, J. (Comparing the effectiveness of explicit EAL feedback ….) Hayati, A., & Mohmedi, F. (2011). The effect of films with and without subtitles on listening comprehension of EFL learners. British Journal of Educational Technology, 42(1). https://doi.org/10.1111/j.1467- 8535.2009.01004.x Ketabi, S., & Sadeghi, A. (2020). The Effects of Captioned Slides on Learning of English Idiomatic Expressions among EFL Intermediate Learners. International Journal of Advanced Studies in Humanities and Social Science, 9(1), 1-20. https://doi.org/10.33945/SAMI/IJASHSS.2020.1.1 Korucu-kiş, S. (2021). On the effectiveness and limitations of captioning in L2 listening. International Journal of Modern Education Studies, 5(2), 516-536. Kruger, J.-L., Hefer, E., & Matthew, G. (2014). Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2. Journal of Eye Movement Research, 7(5). https://doi.org/10.16910/jemr.7.5.4 LeFebvre, L., Parsons, M., Entwistle, C., Boyd, R., & Allen, M. (2022). Rethinking PowerPoint slide design for multimedia learning. Journal of Educational Technology Systems. https://doi.org/10.1177/00472395221077628 Liu, Y., Jang, B. G, & Roy-Campbell, Z. (2018). Optimum input mode in the modality and redundancy principles for university ESL students' multimedia learning, Computers & Education, 127, 190-200. https://doi.org/10.1016/j.compedu.2018.08.025 Liu, Y. (2019). Multimedia input modes, the Modality Principle, and the Redundancy Principle for university ESL students’ learning (Publication No. 27670850) [Doctoral dissertation, Syracuse University]. ProQuest Dissertations Publishing. Mayer, R. E., (2021). Evidence-based principles for how to design effective instructional videos, Journal of Applied Research in Memory and Cognition, 10(2), 229-240. https://doi.org/10.1016/j.jarmac.2021.03.007 Miraldi, L.B. (2021). Using diffusion of innovations to understand factors related to the use of assertion- evidence presentation slides in the engineering workplace (Publication No. 28929435) [Doctoral dissertation, Pennsylvania State University]. ProQuest Dissertations Publishing. Oh, E., (2021). Blended instruction in online learning environments, Asia-pacific Journal of Convergent Research Interchange, 7(1), 125-134. https://doi.org/10.47116/apjcri.2020.01.12 Pujadas, G., & Muñoz, C. (2020). Examining adolescent EFL learners' TV viewing comprehension through captions and subtitles, Studies in Second Language Acquisition, 42(3), 551-575. https://doi.org/10.1017/S0272263120000042 Reynolds, G. (2012). Presentation zen: Simple ideas on presentation design and delivery. New Riders. Ritzhaupt, A., Pastore, R., & Davis, Robert. (2015). Effects of captions and time-compressed video on learner performance and satisfaction, Computers in Human Behavior, 45. https://doi.org/10.1016/j.chb.2014.12.020 Root Kustritz M. V. (2014). Effect of differing PowerPoint slide design on multiple-choice test scores for assessment of knowledge and retention in a theriogenology course. Journal of Veterinary Medical Education, 41(3), 311–317. https://doi.org/10.3138/jvme.0114-004R Wisniewska, N., & Mora, J. (2020). Can captioned video benefit second language pronunciation? Studies in Second Language Acquisition, 42(3), 599-624. https://doi.org/10.1017/S0272263120000029 Winke, P., Gass, S. & Sydorenko, T. (2010). The effects of captioning videos used for foreign language listening activities, Language Learning & Technology, 14(1), 65-86. Zhang, R., & Zou, D. (2021). A state-of-the-art review of the modes and effectiveness of multimedia input for second and foreign language learning. Computer Assisted Language Learning, 1-27. https://doi.org/10.1080/09588221.2021.1896555 https://doi.org/10.1111/j.1467-8535.2009.01004.x https://doi.org/10.1111/j.1467-8535.2009.01004.x file:///C:/Users/MuhammadWildan/Downloads/Ketabi,%20S.,%20&%20Sadeghi,%20A.%20The%20Effects%20of%20Captioned%20Slides%20on%20Learning%20of%20English%20Idiomatic%20Expressions%20among%20EFL%20Intermediate%20Learners.%20International%20Journal%20of%20Advanced%20Studies%20in%20Humanities%20and%20Social%20Science,%209(1),%201-20.%20https:/doi.org/10.33945/SAMI/IJASHSS.2020.1.1 https://doi.org/10.16910/jemr.7.5.4 https://doi.org/10.1016/j.compedu.2018.08.025 https://doi.org/10.1016/j.jarmac.2021.03.007 https://doi.org/10.1017/S0272263120000042 https://doi.org/10.1016/j.chb.2014.12.020 https://doi.org/10.3138/jvme.0114-004R https://doi.org/10.1017/S0272263120000029 https://doi.org/10.1080/09588221.2021.1896555