Microsoft Word - watson.doc Australasian Journal of Educational Technology 2008, 24(3), 258-274 Effects of a computer based beginning reading program on young children Tara Watson and Kerry Hempenstall RMIT University This study was an evaluation of a parent delivered, computer based beginning reading program with a group of 15 Kindergarten and Grade 1 students. Completing the F u n n i x program at home through a CD copy was expected to produce educationally and statistically significant improvements in phonemic awareness, letter-sound fluency, non-word decoding skills, and oral reading fluency, when compared to the outcomes for a similar group of 16 students in a wait list comparison group. Data were analysed using univariate analyses of covariance incorporating effect sizes. Statistically significant treatment effects were found for Kindergarten students in the intervention group on letter-sound fluency, oral reading fluency, non- word decoding, and phonemic awareness skills. Grade 1 students in the intervention group demonstrated significant improvement over time on letter-sound fluency, letter-name knowledge, non-word decoding and oral reading fluency; however, these improvements were not significantly greater than those for the Grade 1 comparison group. It was concluded that Kindergarten students gained the most benefits, and that at risk students may represent a fruitful target audience. The Funnix program and reading skills of Kindergarten and Grade 1 students An ongoing subject of debate in educational research is how the complex skills involved in learning to read should be taught to children. Perhaps the most definitive information to date regarding reading instruction was presented in the National Reading Panel report (National Reading Panel, 2000). The specific areas of reading instruction the Panel noted as crucial were phonemic awareness, phonics, fluency, vocabulary and comprehension. For children in pre-school and in their first year of formal schooling, the Panel found that training in phonemic awareness skills, such as blending and segmenting, provided the strongest benefits (National Reading Panel, 2000). It further recommended that conjoint phonemic awareness and phonics emphases should be taught directly, rather than incidentally, as effective instruction in both skills leads to early progress in reading and spelling. Computer delivered instruction was considered a potentially powerful tool for teaching these skills (National Reading Panel, 2000). Since that time, important national reports from Great Britain (Rose, 2006) and Australia (DEST, 2005) have presented similar recommendations. The direct instruction approach has been applied successfully to many curriculum areas and to a variety of learners, consistently elevating academic performance in comparison to other methods of instruction (Hempenstall, 2004; National Institute for Direct Instruction, 2003; Watkins & Slocum, 2003). Recently, the U.S. Department of Watson and Hempenstall 259 Education’s American Institutes for Research reviewed more than 800 studies of 22 educational models on their capacity to influence student achievement (American Institutes for Research, 2006). The report named direct instruction as one of only two models to achieve their highest rating. In reading instruction, its content and method of instruction are consistent with the recommendations of the three reports mentioned earlier (DEST, 2005; National Reading Panel, 2000; Rose, 2006). The obvious identifying feature of the direct instruction programs is the attention paid to the structure of the teaching sequence in addition to the curriculum content. The designer's brief is to avoid ambiguity in instruction; the focus is on logical-analysis principles. These principles allow the organisation of concepts according to their structure, and the communication of them to the learner through the presentation of positive and negative examples. The model also shares a number of features with behavioural approaches (e.g., reinforcement, stimulus control, prompting, shaping, extinction, fading), and with the effective teaching research (mastery learning, teacher presentation skills, academic learning time, and correction procedures). Programs are designed according to what, not whom, is to be taught. Thus, children work through the same sequence of tasks using the same teaching strategies. Individual differences are allowed for through different entry points, reinforcement, amounts of practice, and correction strategies (Gregory, 1983). There is a long history of parental assistance in the reading development of their children, although, in the last century, their role has been largely restricted to reading to their young children, and hearing the reading of their older children (Burgess, Hecht & Lonigan, 2002). There are, however, examples in which parents and other non-teachers have taken the lead role with great success (Baker, Gersten & Keating, 2000; Fitzgerald, 2001; Kraft, Findlay, Major, Gilberts & Hofmeister, 2001; Leach & Siddall, 1990). Kraft et al. (2001) evaluated the reading skills of Kindergarten level children following a home based beginning reading program before which parents undertook two hours of instructional training. They reported that parents in the study were capable of effectively teaching important skills such as letter-sound associations and blending to their children following this limited period of instruction. Leach and Siddall (1990) compared 4 different methods of tutoring: hearing reading, paired reading, pause, prompt, praise (McNaughton, Glynn & Robinson, 1981), and Teach Your Child to Read in 100 Easy Lessons (Engelmann, Haddox & Bruner, 1983) taught to parents of First Grade children with reading problems. Greater progress in reading accuracy and comprehension occurred with the use of the additional instructional strategies included in the direct instruction method. The greater the level of structure and support provided within the program, the less reliant the child reading outcomes should be upon existing parent knowledge of reading development and instruction. Computers can readily provide this structure, and also represent a colourful and attractive environment to children. The delivery of computer based lessons through a combination of media such as text, sound, graphics and video clips offers potential motivational and pragmatic advantages over purely print based interventions (Wissick & Gardner, 2000). It has been suggested that future reading instruction will be presented increasingly through computers (Martinez & McGee, 2000), and research is beginning to focus on the development of adequate instructional, hardware and software technology (such as accurate voice recognition software) to enable this change (Blok, Ootsdam, Otter & 260 Australasian Journal of Educational Technology, 2008, 24(3) Overmaat, 2002). Among the 2000 National Reading Panel’s influential recommendations was incorporating computer speech and hypertext as potentially valuable enhancements to traditional reading instruction. One recently released computer based direct instruction beginning reading program is Funnix (Engelmann, Engelmann & Seitz-Davis, 2001). It embraces both the National Reading Panel’s literacy curricula and instructional delivery recommendations. Funnix Beginning Reading and Funnix 2 are beginning reading programs designed for use within the home for beginning readers and remedial non-readers. Funnix 2 i s available for children who have completed Funnix Beginning Reading, or for those reading at about the second grade level. The approach is based upon the work of Siegfried Engelmann, who achieved significant success in reading instruction with earlier programs, such as Distar Reading, Reading Mastery, and Corrective Reading (Adams & Engelmann, 1996). These are teacher presented group programs; however, one parent based direct instruction program called Teach Your Child to Read in 100 Easy Lessons (Engelmann, Haddox & Bruner, 1983) has been published, and has received some empirical validation (Hempenstall, 2002; Leach & Siddall, 1990). It was a parent version developed out of the Reading Mastery classroom program. F u n n i x is a derivative of another recent direct instruction reading program, called Horizons (Engelmann, Engelmann & Davis, 1997), described by Silbert (2005). It offers several points of similarity with, and difference from, the 100 Lessons program. Funnix is presented on a CD, and as with the 100 Lessons program, it provides explicit training in phonological awareness and the alphabetic principle (blending and decoding). However, it also emphasises fluency, vocabulary, and comprehension, and includes written exercises to highlight the connection between reading, writing, and spelling (Funnix, 2003). One objective is for children to build oral reading fluency to about 60 words-per-minute by the end of Funnix Beginning Reading, and about 90 words per minute by the end of Funnix 2 (Engelmann & Engelmann, 2003). Oral reading fluency is recognised as a hallmark of reading progress. All skills are delivered in a cumulative and carefully sequenced order, and the active participation of a parent or other tutor is required throughout the program. The parent role is to navigate through the 30 minute lessons, and to provide feedback, corrections, and re-presentation of a sequence when errors are made. The program commences after the Funnix icon is activated. Each lesson begins with a narrated introduction; and buttons at the bottom of the screen are used for navigation. They include buttons for Exit, Pause, Repeat, Continue, Over (restart this exercise). As a program’s efficacy can be compromised by failing to follow directions, the additional structure included in the CD lesson presentation sequences in Funnix may make it somewhat more likely to be presented with fidelity than is the 100 Lessons program. The programs are available for purchase from the Funnix webpage; thus, most parents can be expected to buy them online, and employ them using only the parent instruction disk to guide them. A suggestion made by Stenhoff and Jung (2006) that libraries make copies available to parents is being trialled at the RMIT University Clinic, the site of the current study. Parents who attend the Clinic expressing concern that their young child is not progressing with literacy are provided with a copy, a demonstration of the program in action, and follow up support. Early reports from parents suggest that the dynamic nature of the computer presentation, along with the colourful graphics, offers entertainment value that increases motivation to persevere, Watson and Hempenstall 261 and also helps maintain the child’s concentration more so than does the purely text delivered Teach Your Child to Read in 100 Easy Lessons program. The early emphasis is upon the phonological underpinnings of successful reading progress. For example, oral blending tasks instruct children how to combine single syllable words into multisyllabic words (Pop . . . corn . . . Say it fast. P o p c o r n ) . Continuous oral blending (no pauses between sounds) is emphasised (mmmaaannn) because it is a prompt that promotes easier recognition of the actual word. Later, this prompt is removed and segmented series of phonemes (pauses between the phonemes) are emphasised. This sequence leads to spelling activities - sounding out words one phoneme at a time, and then writing them as letters. As research has indicated, employing letters alongside phonemic awareness activities enhances the transfer of phonological sensitivity from the purely oral to sensitivity to letter structure of the written word - the alphabetic principle. About 10% of each lesson involves introducing new concepts – the remainder reviews and applies familiar content. There is little empirical evidence available for evaluating the effectiveness of Funnix. However, the Oregon Reading First Center included it in a broad scale evaluation of beginning reading programs. In a rating scale to evaluate a program’s theoretical adequacy, Funnix received a very high rating of 93% for its instruction of phonemic awareness, 81% for phonics, and 80% for fluency (Oregon Reading First Center, 2004). Jordan (2004), in a favourable review of the theoretical adequacy of the program for the Florida Center for Reading Research, also reported on a study in which Funnix was provided to 10 pre-school children. Their scores on basic reading, letter-word identification, and story recall tasks increased from their pretest scores, but the study’s methodological limitations allow only that the results were promising, and further rigorous evaluation of the program’s effectiveness is required. The Florida Center for Reading Research also concluded that the content and instructional activities included in the Funnix program are aligned with current research (Jordan, 2004). However, these necessary elements are only the ingredients for success. Having all the right culinary ingredients doesn’t guarantee a perfect soufflé. There are other issues, such as what proportion of each ingredient is optimal, when should they be added, how much stirring and heating is necessary? Errors on any of these requirements may lead to sub-optimal outcomes. So, it is with literacy programs. Just because a program has all the elements doesn’t mean that it will be necessarily effective. Engelmann (2004) pointed to the logical error of inferring a whole based upon the presence of some or even all of its elements. It is possible to analyse the content of effective programs, and then assume incorrectly that the mere presence of those characteristics is sufficient to ensure effectiveness. Engelmann is thus critical of merely “research based” programs, that is, programs constructed only to ensure each respected component is somewhere represented. “Yet there is a big difference between a program based on such elements and a program that has itself been compared with matched or randomly assigned control groups” (Slavin, 2003, p.15). Thus empirical support is also necessary. An evaluation of the reading outcomes of children undertaking the Funnix program enables a comparison between those receiving regular class instruction and those additionally completing the reading program. It was expected that, compared to similar beginning readers in a wait list comparison group, participation in the Funnix beginning reading program would lead to a statistically and educationally significant 262 Australasian Journal of Educational Technology, 2008, 24(3) increase in phonemic awareness skills, such as blending and segmenting, letter-name knowledge, letter-sound fluency, oral reading fluency, and non-word decoding skills. A further question of interest was whether the program produced different outcomes for students in Kindergarten year, who are just beginning to receive reading instruction, compared with students in Grade 1, who already have had a year of school based reading instruction. As the program is designed for beginning readers, would the results for Grade 1 students be significantly different to those in their Kindergarten year? Method Participants Ethics approval was obtained from the Human Research Ethics Sub-Committee of the Faculty of Applied Science, RMIT University, to conduct the study. Participants in the study were families of children enrolled in their first year of formal schooling and Grade 1, who were invited to become involved in an evaluation of a new computer based reading program. A total of 40 children were recruited through advertisements placed in school newsletters and local newspapers; however, nine participants withdrew prior to its conclusion. There were 31 participants: 13 females and 18 males, aged from 4 years 11 months to 7 years 8 months (M = 5.6 years), drawn from 18 primary schools (14 State and 4 Catholic schools). Sixteen students were enrolled in Kindergarten and 15 in Grade 1 in 2004. The experimental and comparison groups each contained approximately half Kindergarten and half Grade 1 students. Materials Material from five measures of reading ability and a beginning reading program were used in this study. The reading assessment included material from the Texas Primary Reading Inventory, the Comprehensive Test of Phonological Processing (CTOPP), the Woodcock Test of Reading Mastery – Revised, the Revised Brigance Comprehensive Inventory of Basic Skills, and Project Aim. The Book and Print Awareness warm up activity from the Texas Primary Reading Inventory (Texas Education Agency, 2000) was administered as a method of developing rapport with participants, and to prepare them for upcoming activities. This activity took approximately one minute to complete and involved asking the participants five questions related to a brief passage of writing from a children’s book. The three phonemic awareness subtests from the Comprehensive Test of Phonological Processing (CTOPP; Ages 5 & 6) (Wagner, Torgesen, & Rashotte, 1999) that form the Phonological Awareness composite are the Elision, Blending Words, and Sound Matching subtests. Two subtests from the Revised Brigance Comprehensive Inventory of Basic Skills (Brigance, 1991) were included: the Names Uppercase Letters and Names Lowercase Letters subtests combine to measure Letter-Name Knowledge. This is the best single predictor of first grade reading success prior to beginning reading instruction (Torgesen, 1998). Watson and Hempenstall 263 The Project Aim Letter-Sound Fluency (Speece, Case, & Ritchey, 2000) subtest allowed participants one minute in which to provide as many letter sounds as possible from a random list of letters. By the end of Grade 1, 39.7 letters correct per minute represents performance at the 50th percentile (Speece et al., 2000). The Woodcock Test of Reading Mastery – Revised (Woodcock, 1998) Word Attack subtest required nonsense word decoding; thereby assessing knowledge and application of phonics skills. The Oral Reading Fluency (Deno, 2001) subtest allowed participants one minute in which to read as much of a graded children’s story as possible. By the end of Grade 1, 29.83 words correct per minute represents performance at the 50th percentile (Speece et al., 2000). The first author administered both the pretest and post-test assessment package to each child individually in sessions lasting from approximately 30 to 45 minutes. Once pretesting was complete in March 2004, families were allocated (on a first come first served basis) to either the experimental or wait list comparison group and informed of their group allocation. Treatment group parents were provided with a training session and an instruction sheet, along with the program guide supplied with the CD. Eight progress sheets were provided to families in order to update the researcher on progress, and to ensure that Funnix was being administered according to instructions (treatment fidelity). A Parent Evaluation Questionnaire, containing eight items concerning the perceived effects of the program on children’s reading skills, was included as a measure of the program’s social validity. The mean program duration being was 6.53 months. When families in the experimental group completed the program, all participants were re-assessed (in October 2004) using the same materials from the pretest phase. Families in the wait-list comparison group were then provided with the opportunity to complete Funnix with their children. Results Exploratory data analysis was conducted on all variables. Assumptions of normality and homogeneity of variance were assessed, and transformations applied when necessary. Tables 1 and 2 present the means, standard deviations and range of scores for the comparison and experimental groups on the main variables according to grade. Also presented are the pre-to-post effect sizes using Cohen’s (1988) d, along with corresponding 95% confidence intervals. A series of 2 x 2 between subjects factorial analyses of covariance (ANCOVA) were conducted to examine whether any differences were present between experimental and comparison participants at each year level on the five main variables. The two between subjects factors were group (experimental vs. comparison) and grade (Kindergarten and Grade 1). Pretest scores served as the covariate while post-test scores served as the dependent variable. Paired samples t-tests were also conducted to determine whether significant differences occurred from pretest to post-test for each of the four groups. 264 Australasian Journal of Educational Technology, 2008, 24(3) Table 1: Experimental vs. Comparison group: Mean scores, standard deviations, score ranges, and effect sizes for Kindergarten students Phonological awareness composite Letter-name knowledge Letter-sound fluency Word attack Oral reading fluency Range 85-96 8-46 0-13 440-449 0-12 M 89.89 28.56 5.78 441 2.11 SD 3.37 16.2 5.61 3 3.82 Range 81-106 20-50 2-46 440-484 1-34 M 96.44 43.89 21.44 457.56 15 SD 8.69 10.34 16.61 18.03 11.35 d .88 1.25 1.21 1.05 1.29 Comparison Pretest n = 9 95% CI .00-1.62 .35-2.12 .31-2.07 .15-1.77 .37-2.17 Range 83-113 12-49 1-37 440-449 0-5 M 95.57 38.71 10.14 441.29 2.29 SD 9.71 13.02 12.7 3.4 2.06 Range 98-128 46-51 22-78 466-498 11-95 M 107.29 48.43 44 481.57 41.29 SD 11.64 1.72 19.52 12.87 30.48 d 1.28 .80 1.45 3.38 1.33 Experimental Pretest n = 7 95% CI .18-2.25 -.08-1.64 .33-2.52 1.25-5.01 .37-2.18 Table 2: Experimental vs. Comparison group: Mean scores, standard deviations, score ranges, and effect sizes for Grade 1 students Phonological awareness composite Letter-name knowledge Letter-sound fluency Word attack Oral reading fluency Range 83-104 18-51 6-41 440-486 1-46 M 98.14 44.71 26.71 460.14 17.14 SD 6.94 11.94 13.1 17.62 16.53 Range 87-102 47-50 16-46 473-497 14-65 M 96.29 49 36.71 479 45.86 SD 5.79 1.15 10.13 8.79 18.69 d .51 .38 .99 1.45 2.51 Comparison Pretest n = 7 95% CI .00-1.22 -.41-1.13 .04-1.88 .32-2.5 .93-4.06 Range 76-104 45-50 18-52 440-474 2-49 M 89.75 47.88 32.88 462 19.25 SD 9.97 1.73 12.33 11.7 15.71 Range 87-100 49-51 30-56 473-497 12-120 M 93.5 50 45.63 479.38 58.25 SD 4.84 .93 8.3 8.78 33.8 d .25 1.71 1.12 1.35 1.67 Experimental Pretest n = 8 95% CI -.51-.99 .57-2.8 .19-1.99 .35-2.31 .55-2.75 Effect size d (Cohen, 1988) was calculated for each dependent variable to determine the magnitude of the observed changes. The calculation of effect size was based upon the ratio of the difference between the group means at pretest and post-test (separately for experimental and comparison groups) and the pooled standard deviation of that group. Effect sizes for the CTOPP and Word Attack subtests were calculated using the Hunter-Schmidt error correction procedure (Hunter & Schmidt, 1990), which involves dividing the calculated effect size by the square root of the test’s reliability. Cohen (1988) stated that an effect size of 0.2 is considered small, 0.5 is medium, and 0.8 as large. Effect sizes of 0.25 or above have been argued to demonstrate educational Watson and Hempenstall 265 significance (Slavin, 1990). Partial eta squared effect sizes were also obtained, the size of these effects are defined as small .01, medium .06, and large .15 (Cohen, 1988). CTOPP Phonological Awareness composite score Results for the CTOPP Phonological Awareness composite score indicated that pretest score covaried significantly with the dependent variable, F(1, 26) = 8.39, p = .008, h2 = .24. No significant grade-by-group interaction was found for the dependent variable, F(1, 26) = 1.21, p = .28, h2 = .05. A significant main effect was found for grade, F(1, 26) = 6.82, p = .015, h2 = .21, but not for group, F(1, 26) = 3.14, p = .088, h2 = .11. Follow up single degree of freedom comparisons found a significant treatment effect for Kindergarten students, F(1, 26) = 4.37, p = .047, partial h2 = .14. A large pretest to post- test effect size was found for the experimental group, d = 1.28, while a smaller effect size was found for the comparison group, d = 0.88. No such effect was found for Grade 1 students, F(1, 26) = 0.14, p = .72, partial h2 = .01. The results of a paired samples t-test indicated that no significant pretest to post-test changes were found for Grade 1 participants in the experimental group, t(7) = -1.45, p = .189, d = .25, or the comparison group, t(6) = 0.66, p = .533, d = .51. Letter-Sound Fluency Results for Letter-Sound Fluency showed that pretest scores did not covary significantly with the dependent variable, F(1, 26) = 2.99, p = .09, partial h2 = .10. A significant main effect was found for group F(1, 26) = 6.84, p = .015, h2 = .208, but not for grade, F (1, 26) = 0.01, p = .93, h2 < .01. Follow up single degree of freedom comparisons found a significant treatment effect for Kindergarten participants, F(1, 26) = 8.58, p = .007, h2 = .25. The magnitude of this effect was large, d = 1.27. Large pretest to post-test effect sizes were found for both the experimental, d = 1.45, and comparison, d = 1.21, groups. While no significant treatment effect was found for Grade 1 participants, paired sample t-tests showed significantly improved performance over time for Grade 1 students in both the experimental t(7) = -3.16, p = .016, d = 1.12, and comparison t(6) = - 2.6, p = .041, d = .99 groups. Letter-Name Knowledge Results for Letter-Name Knowledge showed that pretest scores covaried significantly with the dependent variable, F(1, 26) = 10.3, p = .004, partial h2 = .28. No significant main effect was found for group F(1, 26) = 0.35, p = .56, h2 = .013. With the pretest results partialled out no significant group-by-grade interaction was found, F(1, 26) = 0.24, p = .63, h2 < .01. Paired samples t-tests found no significant improvement in scores across time for Kindergarten participants in the experimental group, t(6) = -2.12, p = .08, d = 1.72; however comparison Kindergarten students did show significant pretest to post-test improvements, t(8) = -3.75, p = .006, d = 1.25. The opposite was found for Grade 1 participants, where a significant improvement was found for the experimental group, t(7) = 4.82, p = .002, d = 1.71, but not the comparison group, t(6) = -1.0, p = .36, d = .38. Word Attack Word Attack (non-word decoding) results showed that pretest scores did not covary significantly with post-test scores on Word Attack, F(1, 26) = 2.29, p = .14, partial h2 = 266 Australasian Journal of Educational Technology, 2008, 24(3) .08. A significant main effect was found for group F(1, 26) = 6.57, p = .017, h2 = .202. With the pretest results partialled out, a significant group by grade interaction was found, F(1, 26) = 6.89, p = .014, h2 = .21. This indicates that attaining significant results was dependent both on the grade and group of participants. Follow up single degree of freedom comparisons found a significant treatment effect for Kindergarten students, F(1, 26) = 13.85, p = .001, partial h2 = .35. Pretest to post-test effect sizes were large for Kindergarten students in both the experimental, d = 3.38, and comparison, d = 1.05 groups. A paired samples t-test revealed significant improvements in scores over time for Grade 1 participants in both the experimental, t(7) = -3.82, p = .007, d = 1.45, and comparison groups t(6) = 3.81, p = .009, d = 1.35. Oral Reading Fluency Results for Oral Reading Fluency showed that pretest scores covaried significantly with the dependent variable, F(1, 26) = 15.29, p = .001, partial h2 = .37. No significant group by grade interaction was found, F(1, 26) = 1.17, p = .29, partial h2 = .04. With the pretest results partialled out a significant main effect was found for group, F(1, 26) = 5.0, p = .034, partial h2 = .16. Follow up single degree of freedom comparisons found a significant treatment effect for Kindergarten, F(1, 26) = 5.68, p = .025, partial h2 = .18, but not for Grade 1 students, F(1, 26) = 0.66, p = .43, partial h2 = .03. Strong effect sizes were calculated for both the experimental, d = 1.33, and comparison, d = 1.29 Kindergarten participants. A paired samples t-test revealed significant improvements in scores over time for Grade 1 participants in both the experimental, t(7) = 4.72, p = .002, d = 1.67, and comparison groups t(6) = 6.63, p < .001, d = 2.51. Pretest and post-test scores in relation to test norms Table 3 presents participants’ scores on the CTOPP composite, letter-sound fluency, oral-reading fluency and non-word decoding variables according to test norms at pretest and post-test. Percentile ranks are presented for the first three variables, while an age equivalent score is provided for non-word decoding scores. Post-test scores indicate that greater gains were made by Kindergarten students in the experimental group as opposed to the comparison group. Table 3: Pretest and post-test scores in relation to test norms Phonological awareness composite (percentile rank) Letter-sound fluency (percentile rank) Oral reading fluency (percentile rank) Word attack (age equivalent) Experimental 37th 10th 10th 5.0Pretest Comparison 23rd 5th 10th 5.0 Experimental 68th 85th 78th 7.8 Kinder- garten Post-test Comparison 39th 33rd 50th 6.10 Experimental 25th 60th 56th 6.11-7.1Pretest Comparison 45th 45th 52nd 6.11 Experimental 32nd 72nd 72nd 7.7 Grade 1 Post-test Comparison 39th 45th 62nd 7.7 Grade 1 students in the comparison group demonstrated a decrease in scores on the CTOPP from pretest to post-test, while an increase was seen for the experimental group. On letter-sound fluency, comparison Grade 1 students showed no change in their scores across time; however, those in the experimental group demonstrated gains Watson and Hempenstall 267 in their performance. On both the oral-reading fluency and non-word decoding variables, Grade 1 students in the experimental group displayed greater gains than the comparison group from pretest to post-test. The Grade 1 students achieved close to the target of 60 words correct per minute. Parent feedback questionnaire Of the 15 experimental group parents, 10 completed and returned the parent feedback questionnaire. Seven parents reported that their children are reading “much more”, and one “a little more” than before the program’s introduction. One parent reported that their child is reading “the same” amount as before commencing the program, and one did not respond to the question. Parents reported that their children are now reading a wide variety of material, ranging from specific topic areas to newspapers and small chapter books. For reading at home, six parents reported that their child is now reading “much better than” before the program’s introduction and four that their child is reading “better than” before the program. Skill improvements were noticed by parents in the areas of reading speed (six), accuracy (eight), smoothness (seven), preparedness to read aloud (six), and understanding of what is read (seven). Reading enjoyment of children also increased, with four reporting their child now finds reading “much more enjoyable” than before the program’s introduction, and six finding it “more enjoyable”. The American accent of the program narrator had a “positive effect” on the progress of one of the children, a “negative effect” on four of the children and “no effect” on five of the children. Parent comments regarding the American accent indicated that some children who completed the program occasionally read with an American accent, and the accent of the narrator was difficult to understand at times. The majority of parents who responded (eight) provided comments for future planning, which included using an Australian based program. Discussion In this study, the effects of the Funnix beginning reading program on the reading skills of Kindergarten and Grade 1 students were evaluated. The significant degree of freedom comparisons indicated that Kindergarten students who completed the Funnix program made greater gains in phonemic awareness, letter-sound and oral reading fluency, and non-word decoding than did comparison Kindergarten students. The benefits of participating in the program were statistically significant, and the effect sizes were large. These findings were similar to those of Kraft et al. (2001) in which children in their first year of school showed significant improvements in their beginning reading skills after completing a parent delivered reading program at home. Tobin’s (2003) evaluation of the Horizons program, from which Funnix was devised, also found significant increases in oral reading, phonemic awareness, and nonsense word reading skills against a comparison group. Interestingly, comparison Kindergarten students also achieved large effect sizes on these four measures, and even made significantly greater improvements over time on the letter-name knowledge variable than did the experimental Kindergarten students. These findings suggest the influence of factors outside of the intervention on student progress. There are several possible reasons why this might be so. For example, 268 Australasian Journal of Educational Technology, 2008, 24(3) perhaps the school programs were sufficiently effective that they partly submerged the program effects. No information was available to substantiate this alternative. It is also possible that the relatively small group sizes introduced error variance. Grade 1 students who completed Funnix did not demonstrate statistically significant treatment effects when compared to the progress of their waitlist peers. However, significant pretest to post-test changes occurred on some variables. Grade 1 students in the experimental group showed statistically significant and large gains over time on the non-word decoding, letter-sound fluency, oral reading fluency, and letter-name knowledge variables. These scores suggest that the program may have had an effect on the students in the experimental group; however, significant gains over time were also made by comparison students on the letter-sound and oral reading fluency variables. This again suggests the influence of other variables on reading performance. It should be noted that the effect sizes for the experimental group were all substantially above a standard deviation, a result unusual in educational research. A possibility is that the parents of children in the waitlist comparison group, having committed themselves to their child’s reading progress, provided a more supportive environment or even equally effective reading instruction during this phase. Fewer statistically significant results were found for Grade 1 students on the program than for Kindergarten students; however, it is worth considering their scores across time and in comparison to test norms. Despite the lack of significance, Grade 1 participants in the experimental group made greater improvements from pretest to post-test on phonemic awareness, letter-sound fluency, and oral-reading fluency than did the comparison group; while improvements on the non-word decoding variable were similar for both groups. It is also possible that larger sample sizes would have produced statistically significant differences at this level. The data may help suggest the best target audience for undertaking Funnix, and which skills are more likely to improve for children in different year levels. Completing the Funnix program may be more beneficial to the early reading skills of Kindergarten students compared with students in Grade 1. In addition to its use with average children, Funnix participation may also provide a useful protective mechanism for Kindergarten children against poor reading outcomes. Future evaluations of the program could examine productively a wider variety of age groups and reading abilities, for example, the effects of Funnix on the reading skills of pre-school aged children. Conducting the program with pre-school aged children may serve as an inoculation to the development of later reading problems. Studying the program’s impact on at risk beginning readers may also be enlightening, given that very low pretest scores did not preclude good progress. This is a potentially valuable finding, as many at risk children currently in Reading Recovery programs fail to make the anticipated progress (Office of the Victorian Auditor General, 2003). This is a largely speculative suggestion as the current study can shed only a little light on the issue. Of great interest is why a well designed program that claims to progress children from non-readers to a “solid second-grade level” (Funnix.com, 2003) was not obviously more effective than was normal classroom participation. One possibility is that the two groups selected were not representative of the average Australian population. The groups may have comprised a largely advantaged sample, whose progress was assured regardless of effective instruction. The average rate of progress of students over the study period was above that expected of the population average, one in which 30% of students do not develop adequate reading skills (Louden et al., 2000). Due to Watson and Hempenstall 269 the large effect sizes noted in the comparison group over a relatively brief time, it is likely that a high proportion of them were advantaged students. Participant drop out may have also contributed to this sample being unrepresentative of the wider population. Those who remained in the program were children with average reading abilities at pretest. Sufficient experience and practice in early reading skills may have been provided prior to commencing the program, and possibly continued during the intervention period. So these children may have been primed already by early experiences, to learn reading with relative ease. That their parents responded to an advertisement to participate in this program may imply that they are not a representative sample of the parent population, but rather an unusually supportive one. The small sample size may have compromised the achievement of more significant results in both grades throughout this study. Given that even the insignificant results tended to favour children in the experimental group, and that a number of significant results were found in this study, it could be expected that a larger cohort would be more likely to have produced a greater number of significant results. There may have been problems with varying degrees of adherence to the program. Some parents reported the structure of Funnix to be repetitive and boring to their child. This perception, along with a lack of understanding of the value of guided practice, may have resulted in a loss of program fidelity. Whilst Funnix provides a disc dedicated to teaching parents to implement the program, further training on the rationale behind the direct instruction approach may help ensure parent motivation and compliance with program specifications. It may be possible that some parents in the study commenced Funnix with their children before completing the parent disc, or did not supervise their child for every lesson. Whilst progress sheets were used to provide information on children’s advancement through the program, there was no space to record completion of the parent disc. In addition, many families completed only the date and lesson duration columns, making it difficult to determine the level of supervision children received. Future studies might address this by providing program discs only after the parent disc has been completed and discussed. Additionally, progress sheets may include greater detail, such as the number of times per lesson that tasks needed to be repeated. Presentation of background information in a group training session may have improved parent understanding of direct instruction principles. Enabling clarification questions to be answered may help improve program adherence. One parent commented that the study should have “… assessed child progress and provided parents with resources and information on assisting children develop good reading skills and adopting comprehensive strategies”. Given that children’s progress was formally assessed twice by the examiner, through regular progress sheets, and that Funnix and the accompanying workbook were the resources provided to parents, this response indicates very little understanding of the structure of the program. Thus, it is possible that a lack of parental knowledge may have had some impact on attaining significant results. On the other hand, the program is commercially available to parents who have no support beyond that provided by the parent disc and information booklet. So, if these problems do occur, they represent a potential program weakness. To encourage and promote program fidelity, future research may include a structured observation of families while presenting a Funnix lesson to ensure accurate program delivery. 270 Australasian Journal of Educational Technology, 2008, 24(3) Child boredom may also have resulted from children not being placed optimally in the program. The Funnix website provides a placement test which can allow a child to start at varying points throughout the program depending on their level of skill; however, all children in the study began at Lesson One. There is no specific recommendation on the site that parents use the placement test (which is not included with the disc package), or the skipping function that allows faster progress when children display high levels of skill mastery. However, the placement test may have been a useful tool with Grade 1 students in the study as child boredom and lack of parent endurance may have been reduced had some children started the program at a more advanced level. It was noted that the Grade 1 students performed at average levels on the four variables examined at pretest and post-test, suggesting that perhaps they had almost achieved the level of mastery expected of the program. Funnix is designed for beginning readers, and the average Grade 1 student is no longer a beginner. The use of the placement test may help select more accurately the appropriate starting points for children who are average performers, and also that for those who are struggling with reading. It could be that the Grade 1 students may have been better suited to the follow up program Funnix 2 (Engelmann & Engelmann, 2002). There were other possible barriers to program success. Some parents reported that their children were fatigued after long school days, making the completion of Funnix in addition to reading homework a difficult task. Implementing an American derived program was also of concern to some Australian participants. Some parents reported difficulties in understanding the American accent of the narrator, and the use of unfamiliar, or ‘Americanised’, words (e.g., ‘pail’) was confusing for some children. Workbook exercises contained print rather than the cursive script that is taught in Victorian schools. Some parents criticised this aspect of the program, believing that their children became confused about which writing style they should apply. Although some aspects of the program were criticised by parents, the feedback from the parent questionnaire provided a clear indication that the program displayed good social validity. The majority of parents believed their children were reading an increased volume of text after completing the program, and all respondents believed their child’s reading abilities had increased since completing the program. Not only did parents report an improvement in reading skills, but all reported that their children were experiencing greater enjoyment from reading than before completing Funnix. Limitations to this exploratory study are obvious. For reasons of the time available for postgraduate research, there was not random allocation to the experimental and control groups (families were allocated on a first come first served basis). Additionally, the small sample size led to pretest differences between the two groups; however, the pre-test measures were treated as covariates in the analyses. These features have implications for the internal validity of the study. Additionally, the small sample size implies that the results should not be generalised from this sample to be considered indicative of the average Australian population. Funnix was delivered via computer, which removes the need for parents to follow the scripts found in other direct instruction programs, such as Teach Your Child to Read in 100 Easy Lessons (Engelmann, Haddox, & Bruner, 1983). The obvious advantages of Funnix are that it offers the opportunity for parents to take a more direct role in reading instruction than simply hearing their child read, and it is less demanding of the parent in managing a program’s presentation. It would be interesting to compare Funnix with Teach Your Child to Read in 100 Easy Lessons, a script based program with Watson and Hempenstall 271 evidence of effectiveness (Hempenstall, 2002) to determine the form of delivery most appealing to parents. Certainly, at the point of advertising the study, many parents reported being particularly interested in the computer aspect of the program; however, some of the parents and students did not appreciate either the American accent or the Repeat routine (used to produce a repeat of the task as needed for mastery). This parent feedback, along with the assessment results, suggests that there may have been benefits produced by Funnix on the reading development of at least some of these young students in some areas. A necessarily tentative conclusion to this small study is that Funnix may be most useful with children in their first year of formal schooling; however, if further studies are supportive its value may extend to those beginners or pre-school students identified as at risk for future reading difficulties, and also to older non-readers. One recommendation of the National Reading Panel (2000) was to incorporate methods of phonics instruction into teacher training programs; however, research has found that many educators are not adequately prepared to teach the foundation skills of reading (Bos, Mather, Dickson, Podhajski & Chard, 2001). The majority of the 538 pre-service and in-service teachers surveyed by Bos et al. were found to possess limited knowledge about teaching the structure of language, phoneme segmentation, and about the definition and significance of teaching phonological processes. An Australian survey by Rohl and Greaves (2005) reported that 36% of beginning primary teachers felt unprepared to teach reading and 57% unprepared to teach phonics. Senior staff were even more pessimistic - considering that 49% of these teachers were unprepared to teach reading and 65% were unprepared to teach phonics. These figures rose dramatically (77% - 89%) when the beginning teachers were confronted with diverse learners, such as those with disabilities and whose first language was not English. Further Australian research with 340 pre and in service teachers found that there was a positive attitude to, but poor knowledge of, metalinguistics in the process of learning to read (Fielding-Barnsley & Purdie, 2005). Only 54% of the sample knew what constituted a syllable, and only 24% could correctly count the number of phonemes in a word. If such concerns are not effectively addressed in teacher education as a consequence of Inquiries, such as the US National Reading Panel (2000), the Australian National Inquiry into the Teaching of Literacy (Department of Education, Science and Training, 2005), and Great Britain’s Rose Report (Rose, 2006), it is possible that self contained, evidence based programs like Funnix will become more than solely household interventions. References Adams, G, & Engelmann, S. (1996). Research on Direct Instruction: 25 years beyond DISTAR. Seattle, WA: Educational Achievement Systems. American Institutes for Research (2006). CSRQ Center Report on Elementary School CSR Models. [viewed 12 Dec 2006, verified 10 May 2008] http://www.csrq.org/CSRQreportselementaryschoolreport.asp Baker, S., Gersten, R. & Keating, T. (2000). When less may be more: A 2-year longitudinal evaluation of a volunteer tutoring program requiring minimal training. Reading Research Quarterly, 35, 494-519. 272 Australasian Journal of Educational Technology, 2008, 24(3) Blok, H., Ootsdam, R., Otter, M. E. & Overmaat, M. (2002). Computer-assisted instruction in support of beginning reading instruction: A review. Review of Educational Research, 72, 101- 130. Brigance, A. H. (1991). Revised Brigance Diagnostic Inventory of Early Development – Birth to Seven Years. Australia: Curriculum Associates. Burgess, S. R., Hecht, S. A. & Lonigan, C. J. (2002). Relations of the home literacy environment (HLE) to the development of reading-related abilities: A one-year longitudinal study. Reading Research Quarterly, 37, 408-426. Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum. Deno, S. (2001). CBM Oral Reading Fluency. Curriculum Based Measurement Technical Manual. Nashville, TN: Vanderbilt University. [viewed 12 Aug 2005, not found 10 May 2008] http://www.aimsweb.com/ DEST (Department of Education, Science and Training) (2005). Teaching reading: Report and recommendations. National Inquiry into the Teaching of Literacy. Canberra: Commonwealth of Australia. [viewed 14 Dec 2005, verified 10 May 2008] http://www.dest.gov.au/nitl/documents/report_recommendations.pdf Engelmann, S. (2004). The Dalmation and its spots. Editorial Projects in Education, 23(20), 34-35, 48. [viewed 3 Feb 2004, Education Week subscribers only] http://www.edweek.org/ew/ew_printstory.cfm?slug=20engelmann.h23 Engelmann, S. & Engelmann, O. (2003). Funnix reviewer’s guide: A consumer’s guide to evaluating supplemental and intervention reading programs. [viewed 12 Jan 2006] http://www.funnix.com/funnix/PDFs/FunnixReviewersGuide.pdf Engelmann, S. & Engelmann, O. (2002). Funnix 2: Reading. Eugene, OR: Royal Limited Partnership. Engelmann, S., Engelmann, O. & Davis, K.L.S. (1997). Horizons. Columbus, OH: SRA/McGraw- Hill. Engelmann, S., Engelmann, O. & Seitz-Davis, K. (2001). Funnix: A tutor in your computer. Eugene, OR: Royal Limited Partnership. Engelmann, S., Haddox, P. & Bruner, E. (1983). Teach your child to read in 100 easy lessons. New York: Simon & Schuster. Fielding-Barnsley, R. & Purdie, N. (2005). Teacher’s attitude to and knowledge of metalinguistics in the process of learning to read. Asia-Pacific Journal of Teacher Education, 33, 65-76. Fitzgerald, J. (2001). Can minimally trained college student volunteers help young at-risk children to read better? Reading Research Quarterly, 36(1), 28-46. Funnix.com (2003). Why teach reading at home? [viewed 25 Mar 2003] http://www.funnix.com/ Gregory, R.P. (1983). Direct Instruction, disadvantaged and handicapped children: A review of the literature and some practical implications. Part 1. Remedial Education, 18, 108-114. Hempenstall, K. (2002). Phonological processing and phonics: Towards an understanding of their relationship to each other and to reading development. Australian Journal of Learning Disabilities, 7, 4-29. Watson and Hempenstall 273 Hempenstall, K. (2004). The importance of effective instruction. In N. E. Marchand-Martella, T. A. Slocum, & R. C. Martella (Eds.), Introduction to direct instruction (pp. 1-27). Needham Heights, MA: Allyn and Bacon. Hunter, J. E. & Schmidt, F. L. (1990). Methods of meta-analysis; Correcting error and bias in research findings. Newberry Park, CA: Sage. Jordan, G. (2004). Funnix Reading Programs. [viewed 12 Aug 2005, verified 10 May 2008] http://www.fcrr.org/fcrrreports/PDF/FunnixFinal.pdf Kraft, B., Findlay, P., Major, J., Gilberts, G. & Hofmeister, A. (2001). The association between a home reading program and young children’s early reading skills. Journal of Direct Instruction, 1, 117–136. Leach, D. J. & Siddall, S. W. (1990). Parental involvement in the teaching of reading: A comparison of hearing reading, paired reading, pause, prompt, praise, and direct instruction methods. British Journal of Educational Psychology, 60, 349-355. Louden, W., Chan, L.K.S., Elkins, J., Greaves, D., House, H., Milton, M., Nichols, S., Rivalland, J., Rohl, M. & van Kraayenoord, C. (2000). Mapping the territory—Primary students with learning difficulties: Literacy and numeracy (Vols. 1-3). Canberra: Department of Education, Training, and Youth Affairs. http://www.dest.gov.au/sectors/school_education/publications_ resources/profiles/mapping_territory_primary_students_difficulties.htm Martinez, M. G. & McGee, L. M. (2000). Children’s literature and reading instruction: Past, present, and future. Reading Research Quarterly, 35, 154-169. National Institute for Direct Instruction (2003). What is direct instruction? [viewed 18 Jun 2003, verified 10 May 2008] http://www.nifdi.org/index.html#what National Reading Panel (2000). Report of the National Reading Panel: Teaching children to read. [viewed 15 May 2003 at http://www.ldonline.org/ld_indepth/reading/teaching_children_to_ read.html, verified 10 May 2008 at http://www.nichd.nih.gov/publications/nrp/smallbook.cfm] Office of the Victorian Auditor General. (2003). Improving literacy standards in government schools. [viewed 10 October 2004, verified 10 May 2008] http://archive.audit.vic.gov.au/reports_par/Literacy_Report.pdf Oregon Reading First Center (2004). Oregon Reading First Center: Review of supplemental and intervention reading programs [viewed 23 Sep 2005, verified 10 May 2008] http://reading.uoregon.edu/curricula/or_rfc_review_si.php Rohl, M. & Greaves, D. (2005). How are pre-service teachers in Australia being prepared for teaching literacy and numeracy to a diverse range of students? Australian Journal of Learning Disabilities, 10, 3-8. Rose, J. (2006). Independent review of the teaching of early reading. London: Department for Education and Skills (DfES). [viewed 20 Mar 2006, verified 10 May 2008] http://www.standards.dfes.gov.uk/rosereview/finalreport/ Silbert, J. (2005). Using direct instruction programs as intervention programs in Grades K-3. Direct Instruction News, 5(2), 16-22. [viewed 5 Mar 2008] http://74.220.207.73/~adihome1/articles/DIN_05_02_07.pdf Slavin, R. E. (2003). A reader's guide to scientifically based research. Educational Leadership, 60(5), 12-16. [viewed 16 Dec 2003, verified 10 May 2008] http://www.ascd.org/portal/site/ascd/menuitem.459dee008f99653fb85516f762108a0c/ 274 Australasian Journal of Educational Technology, 2008, 24(3) Speece, D. L., Case, L. P. & Ritchey, K. D. (2000). Normative data for letter sound fluency and oral reading fluency probes grades one through four. Department of Special Education, University of Maryland Project AIM. [viewed 11 May 2005, verified 10 May 2008] http://www.glue.umd.edu/~dlspeece/cbmreading/norms.html Stenhoff, D. M. & Jung, L. A. (2006). Funnix and Funnix 2. In Book and Software Review Associate Editor's Column, Journal of Special Education Technology, 21(4), 66-68. [viewed 5 Mar 2008] http://jset.unlv.edu/20/JSETv21n4.pdf#page=67 Texas Education Agency (2000). Texas Primary Reading Inventory. [viewed 12 Feb 2005 at http://www.tea.state.tx.us/reading/era.html, not found 10 May 2008, see http://www.tpri.org/] Tobin, K.G. (2003). The effects of the Horizons reading program and prior phonological awareness training on the reading skills of first graders. Journal of Direct Instruction, 3, 1-16. Torgesen, J .K. (1998). Catch them before they fall: Identification and assessment to prevent reading failure in young children. American Educator, 22, 32-39. [viewed 12 Feb 2003 at http://www.ldonline.org/ld_indepth/reading/torgeson_catchthem.html, verified 10 May 2008 at http://www.ldonline.org/article/225] Wagner, R. K., Torgesen, J. K. & Rashotte, C. A. (1999). Comprehensive test of phonological processing. Austin, TX: PRO-ED, Inc. Watkins, C. L. & Slocum, T. A. (2003). The components of direct instruction. Journal of Direct Instruction, 3, 75–110. Wissick, C. A. & Gardner, J. E. (2000). Multimedia or not to multimedia? That is the question for students with learning disabilities. Teaching Exceptional Children, 32, 34-43. Woodcock, R. W. (1998). Woodcock Tests of Reading Mastery - Revised. Circle Pines, MN: American Guidance Service. Tara Watson and Kerry Hempenstall, RMIT University, Melbourne. Contact person: Dr Kerry Hempenstall, Division of Psychology, School of Health Sciences, RMIT University, Plenty Road, Bundoora Vic 3083, Australia. Web: http://www.rmit.edu.au/staff/kerry_hempenstall Email: kerry.hempenstall@rmit.edu.au