Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Volume 2, Issue 2 2019 Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo Principal, Rio Plaza Elementary School Rio School District Charles Weis Assistant Professor, Educational Leadership CSU Channel Islands Introduction Learning to read may come easily to some students, but it can be a challenge for others. Reading is a foundational skill for most future formal learning; therefore, helping all students become proficient in reading in their earliest years of schooling requires providing equitable instructional opportunity (Richards, 2010). Early identification of language arts skills, accompanied by appropriate intervention designed to meet the needs of diverse learners, can result in the achievement of equitable reading outcomes (Avant, 2016; Denton, 2012). Decades of research support reading programs that include explicit and systematic reading instruction designed to meet the individual needs of students (Hughes & Dexter, 2011; National Reading Panel, 2000; Richards, 2010). Research also supports the concept that readers need direct reading instruction in addition to learning the skills required to put these concepts together (Hughes & Dexter, 2011; International Reading Association, 2002). Conducting systematic evaluations of reading intervention programs can provide specific feedback to enhance the equitable instructional opportunity and effectiveness of such reading intervention programs (Jason, 2008). In pursuit of reading equity, educators need to diagnose students’ reading needs and learning gaps, and then design and implement intervention programs to enhance reading skills (Avant, 2016; Buffman, Mattos, & Weber, 2009). The National Reading Panel of 2000 identified the five components of reading, which are phonemic awareness [the ability to hear, identify, and manipulate individual sounds], phonics [correlating sounds with letters or groups of letters], fluency [the ability to read with speed, accuracy, and proper expression], vocabulary [words recognized or used in print], and comprehension [the ability to process text, understand its meaning, and to integrate with what the reader already knows] (National Reading Panel, 2000). Each of these components is individually important, but also intricately related to Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe one another (International Reading Association, 2002). Some of these components can be taught in isolation, though students need to know that they are related to each other. When these components are utilized together, they will help students become better readers (International Reading Association, 2002). Across the country, educators recognize reading instruction inequity and realize that there are students who are not meeting grade level reading standards (Richards, 2010). In many schools, educators have designed and implemented reading intervention programs to enhance support for struggling readers. These intervention programs utilize many identified best practices selected to meet the diagnosed needs of struggling readers; they are delivered either in-class or in out-of-class centers [termed pull-out programs]. Both Response to Intervention (RtI) and Multi-Tiered System of Support (MTSS) strategies, that are the basis of the reading intervention program being evaluated, were created to blur the lines between classroom and pull-out programs by establishing a unified system that serves all students. Both of these systems are based on progressively targeted instructional support as a result of assessing the impact of early interventions (Avant, 2016; Buffman, Mattos & Weber, 2009; Rosen, 2018; Shores & Chester, 2009). Program Description The reading intervention program at the elementary school in Ventura County was created in an effort to support students who were struggling in the mainstream classroom. The goal of the program was to increase the achievement of struggling readers, and those at risk of not meeting state standards, in the area of reading and language arts. The program began in 2008, and was developed as a push-in program, wherein two credentialed teachers, along with one instructional assistant, delivered small group intervention services in the classroom. In 2012, the program moved to a pull-out system with intervention specialists serving students in a location outside of the student’s regular classroom. During this time, an intervention specialist position was created and a credentialed teacher, along with three instructional assistants, delivered intervention services in a classroom dedicated to this service. Within the first three weeks of the school year, all students in first through fifth grade were assessed in reading skills using the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) assessment to diagnose students’ reading skills. Students enrolled after the first three weeks were assessed by the intervention specialist, instructional assistants, and re-employed retired teachers. DIBELS assesses the basic reading skills of fluency, comprehension, and beginning reading skills with an online assessment system. Professional Learning Community (PLC) grade level teacher teams met with the resource specialist teacher, the principal, and the intervention specialist to review the reading assessment data and select students for either Tier 2 and Tier 3 intervention at each grade level. Students selected for Tier 2 services were provided instruction to approximately sixty students per day in groups of 4 or 5 by the Response to Intervention (RtI) team, consisting of classroom teachers, the bilingual intervention teacher, and instructional assistants. The PLC grade level teacher teams, in collaboration with the RtI team, decided on the appropriate instruction based on the assessment results. Tier 3 service was provided to thirty students by the resource specialist teacher and an Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe instructional assistant. Tier 3 was a more intensive intervention tier, wherein students worked in groups of 3 or less. Intervention sessions averaged from thirty to forty-five minute sessions four times a week. Students were assessed using the DIBELS Progress Monitoring tools every three to four weeks to evaluate the efficacy of the program. In February, the principal, Student Study Team coordinator, intervention teacher, and the resource specialist teacher met with the student’s regular classroom teacher to review the assessment data of each student to assure that every student was receiving appropriate instruction in both tiers. Tier assignments were flexible, meaning that at any point during the year, students could move from one tier to another. The duration of intervention services varied depending on how students responded to the intervention services. While some students received one round of six weeks of intervention, other students received intervention year-round. A variety of instructional strategies and materials to provide intervention services, including commercial materials such as Triumphs, the McMillan Intervention Resource Book, and the Home-School Connections, were selected from the list of available district adopted materials. Purpose of the Program Evaluation In order to address students’ reading difficulties, school districts all over the United States have implemented reading intervention programs to help meet the needs of these struggling readers (Denton, 2012). Districts have hired experts and invested in curriculum and professional development in order to provide the best services for their students. However, there have been few published program evaluations conducted in order to determine the effectiveness of such programs (Denton 2012). There is a continued need for studies designed to evaluate programs for primary-grade students who under- perform in reading to inform other schools about what works to help struggling readers (Denton, 2012). The purpose of this study was to assist stakeholders to make decisions about the future of the reading intervention program. This purpose was accomplished by assisting stakeholders to determine the quality and effectiveness of the reading intervention program to provide equity of students’ reading mastery and growth. In addition, the program evaluation helped stakeholders to make improvements to the program that allowed resources to be directed to the areas of greatest need. Methods for Conducting the Program Evaluation Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency. During this process, the person conducting the evaluation, the evaluator, is responsible for engaging the stakeholders in all steps of the process. Stakeholders are persons affiliated with the reading intervention program that have decision-making authority. Program evaluation is indispensable for school improvement (Jason, 2008). The feature differentiating program evaluations from traditional research is the continuous involvement of stakeholders in the program evaluation process to ensure their trust and confidence in the findings. The first step of this program evaluation was to create a clear description of the reading intervention program and identify the information needs of the stakeholders that will assist them in their decision- making. Interviews conducted with stakeholders and observations of the program were utilized to write Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe an accurate program description. This description was shared with the stakeholders to make corrections and verify accuracy. Evaluation Design This study used a quasi-experimental, quantitative causal-comparative design. This correlational design allowed statistical comparison of participation in the reading intervention program (independent variable) to reading mastery and growth (dependent variable) in a pre-post assessment design. Evaluation Question Development Stakeholders took part in providing feedback, modifying and, finally, approving the evaluation questions that guided the program evaluation. The program evaluation questions that guided this study were designed to comply with the needs of school district and school-site stakeholders based on the district Local Control and Accountability Plan (LCAP), the School Plan for Student Achievement (SPSA) goals, and available assessment data. The evaluation questions were: 1. Have students who received reading intervention services grown in reading skills as measured by DIBELS? 2. Have students who received reading intervention services grown in their mastery of the common core state language arts standards as measured by SBAC Language Arts/Literacy assessment? Archived Assessment Data Two reading assessments that were regularly administered and archived by staff were used. The first assessment, the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) was a set of assessments designed to assess multiple cueing systems to regularly monitor the development of early literacy and early reading skills (University of Oregon, n.d.). The reading cueing systems assessed include phonemic awareness, alphabetic principle, accuracy and fluency with text, vocabulary, and comprehension. This assessment was administered for students in 1st through 5th grade to measure growth and to determine instructional needs and placement in Tier 2 or Tier 3 intervention groups. DIBELS data are not comparable from year-to-year because the test has different versions for each grade level. Since the DIBELS assessment was administered to each student several times per year for the preceding five years, it was determined that the scores from the beginning of each school year would be compared to the scores from the end of that school year. These assessments are considered reliable and valid for focusing instructional intervention in reading and assessing growth in the designated skills. The second assessment used was the Smarter Balanced Assessment Consortia (SBAC) assessments, which was a system of computer-adaptive tests and performance tasks, which were based on the California Common Core State Standards (CCSS) for English Language Arts/Literacy (ELA) (California Department of Education, n.d.). Computer-adaptive assessments modify the difficulty of items presented to students based on how they performed on prior items. This feature of the assessment allows for more accurate assessment of reading skill mastery. For the past three years, students in grades 3 through 5 in this school have been assessed with the summative SBAC following the California accountability mandate. Since this assessment is only administered once at the end of the school year, Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe the scores from the end of one school year were compared to the scores from the end of the following school year. The only data available for analysis was for third, fourth, and fifth graders for the 2015, 2016, and 2017 school year administrations. The SBAC was designed to allow for comparison between years by creating continuously increasing scaled scores across grade levels indicating increased mastery of the common core state standards. Students who did not have the assessment data in each year were excluded from the analysis of data. This assessment was determined valid for assessing growth on the language arts/literacy Common Core State Standards and was considered reliable (California Department of Education, n.d.). Data Analysis The Statistical Package for the Social Sciences software was used to conduct multiple t-tests for paired samples to determine statistical differences between the mean (average) scores for the beginning of the school year with the end of the year on the DIBELS and end of one school year to end of the next school year for SBAC. A Cohen’s d value was calculated to determine the effect size of student growth in reading for each of the reading assessments. Cohen’s d allows for the comparison of the change in mean (average) scores to determine the magnitude of practical growth. A Cohen’s d value of 0.2 is considered a small effect size, a d of 0.5 is considered a medium effect size and a 0.8 is considered a large effect size. For this program evaluation, an effect size of 0.5 or higher is considered significant. It is important to note that 0.5 represents ½ of a standard deviation growth suggesting that the average student grew faster than approximately 17 percent of grade-mates. This amount of growth equates to the average score moving from the center of the score distribution (50 percentile) to above two-thirds of the distribution (67 percentile) (Salkind, 2017). Results The results from both assessments demonstrate that the students who received intervention services from the reading intervention program displayed significant growth in reading as measured by the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) and the Smarter Balanced Assessment Consortium (SBAC) English Language Arts/Literacy. The amount of growth varied between the tests administered and grade levels, however, there was growth at all grade levels and during all the five years of data analyzed. Effect size and t-test calculations suggest that students who participated in the school’s reading intervention program made remarkable statistically and practically significant growth in reading as measured by the DIBELS (Figures 1.1 to 1.5 and Tables 1.1-5.5 in Appendix A) with the exception of first grade in 2012. In addition, the amount of growth improved over the five years under study suggest program maturation over time. SBAC assessment of English Language Arts/Literacy standards showed statistically significant growth but far less practical growth (Figure 1.7 and Tables 6.1 to 6.2 in Appendix B). Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Figure 1.1 First Grade Pre-Post Within Year Effect Size on DIBELS Figure 1.1 shows that the first graders at the school in the year 2012 had the lowest Cohen’s d value, showing a small effect size of .24. The 2013 DIBELS assessment data shows that the growth was increasing. For the years 2014, 2015 and 2016, the data shows that there was a significant effect size. The last three years demonstrate an effect size of more than ½ standard deviation growth. This positive growth trend may be evidence of the program maturing over time. Figure 1.2 Second Grade Pre-Post Within Year Effect Size on DIBELS Figure 1.2 displays significant growth in reading for second graders on DIBELS scores across the years from 2012 to 2016. The effect size growth for second graders ranged from a low of 1.23 standard deviation growth to over 2 SD’s. The results for this grade level is tremendous and demonstrates that there is positive growth across the years at this grade level. The most significant growth occurred in 2012 with an effect size of 2.08049, then 1.587535 in 2016. However, it is important to note that the effect size was over 1.2 SD’s in all years for second graders. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Figure 1.3 Third Grade Pre-Post Within Year Effect Size on DIBELS Figure 1.3 shows third grade students making almost 2 standard deviations reading growth as measured by the DIBELS assessment for the years 2012 to 2016. Year after year, the effect size was maintained above a 1.84, demonstrating a huge impact. Figure 1.4 Fourth Grade Pre-Post Within Year Effect Size on DIBELS Figure 1.4 shows that fourth graders across the years displayed an effect size of approximately 2 to a 2.5 standard deviation growth in reading as measured by the DIBELS assessment. These fourth graders made the most growth in 2013 with an effect size of 2.47. It is clear that these fourth graders have made tremendous growth in all years analyzed. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Figure 1.5 Fifth Grade Pre-Post Within Year Effect Size on DIBELS Figure 1.5 shows that fifth graders made significant growth in all years. The effect size growth ranges from 1.25 in 2012 to a monumental growth in 2016 of 2.53. These Cohen’s d values indicate a practical significant growth in reading. Figure 1.6 SBAC 2016 and 2017 ELA/Literacy Effect Size by Grade Level Figure 1.6 shows the Cohen’s d effect size growth of 0.5617 for fourth graders between the 2016 and 2017 school years. This demonstrates a practical significant growth of ½ standard deviation. In addition, the fifth graders from the 2016-2017 school year showed an effect size growth of 0.322236, which is considered small, insignificant practical growth. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Conclusions and Recommendations In pursuit of reading equity, educators identify students’ reading needs and learning gaps, then design and implement necessary reading intervention programs to enhance reading skills. The goal of these equity-based reading intervention programs is to raise students’ skills to levels that will enable students to read to learn in the upper elementary grades. This program evaluation focused on answering two evaluation questions to assist stakeholders in decision-making about the continued operation of the reading intervention program. Evaluation Question 1: Have students who received reading intervention services grown in reading skills as measured by DIBELS? Based on the results, the students who received intervention services showed growth each year at each grade level in reading as measured by the DIBELS. The exception to this finding was first grade students who did not make significant growth in the 2012-2013 school year which was the 1st year of program implementation. The answer to evaluation question 1 is students who received intervention services showed significant statistical and practical growth in reading skills as measured by DIBELS. The DIBELS tests assess phonemic awareness, alphabetic principle, accuracy and fluency with text, vocabulary and comprehension, but not general literacy. Additionally, DIBELS scores were used to determine student placement in the intervention program tiers and the reading intervention instruction focused on three main components assessed by DIBELS: reading comprehension, fluency, and phonemic awareness/phonics. Therefore, growth on this measure was expected. The significant finding is the magnitude of growth experienced by the students in this program. Evaluation Question 2: Have students who received reading intervention services grown in the mastery of common core state language arts standards as measured by SBAC Language Arts/Literacy assessments? Based on the results, the students who received intervention services in the 2015-2017 school years showed significant growth in the Common Core State Language Arts/Literacy standards as measured by SBAC. The calculation of effect size suggested that fourth grade program participants made significant practical growth (Cohen d = .56) in the common core state language arts standards but fifth grade students did not (Cohen d = .32). The answer to Evaluation Question 2 is fourth grade students who received intervention services showed significant statistical and practical growth in reading as measured by SBAC, but fifth grade students did not demonstrate similar practical growth. These mixed findings suggest that although mean significant difference exists, practically speaking, significant growth was not realized by fifth grade students. The difference in findings between the two assessments might be explained by the difference in what each test assessed. The SBAC Language Arts/Literacy assessment is an end of the year, comprehensive assessment of reading/literacy that requires students to apply their reading skills. DIBELS assesses only basic reading and pre-reading skills that were the focus of the instruction in the reading intervention program. Additionally, the instructional content of the reading intervention program was developed before the common core state standards and SBAC assessment was implemented. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Recommendations - Implications for Practice Based on the findings of this program evaluation, it is evident that the reading intervention program in this school has had a positive impact on students’ basic pre-reading and reading skills as assessed by the DIBELS test. However, similar impact was not as great on the SBAC Language Arts/Literacy assessment scores. These outcomes only partially fulfill the promise of equitable reading instruction. The evaluator suggests that a greater emphasis should be placed on the Common Core State Standards (CCSS) in the reading intervention program curriculum and instruction. The current reading intervention program is clearly improving students’ basic pre-reading and reading skills, which must be maintained. However, higher level application of basic skills as embodied in the CCSS would result in a greater impact of the program on reading equity and the SBAC Language Arts/Literacy scores. In addition, the evaluator suggests that the intervention specialist work more closely with the fourth and fifth grade classroom teachers to support greater emphasis on CCSS Language Arts/Literacy Standards in their classroom instruction. Limitations of the Program Evaluation Although this program evaluation was conducted carefully, the evaluator recognizes that there are limitations to interpretation of the findings. The following limitations are recognized: 1. As is the case with most program evaluation designs, no control group of students eligible for, but not receiving, reading intervention services was possible. Therefore, the reading growth recorded could not be causally attributed to the intervention program. 2. This program evaluation was not able to include all students who received intervention services at the school from 2012 to 2016 because students with missing data were not included in the analysis (following t-test assumptions). 3. While the DIBELS assessment is a reading skills diagnostic tool, it was not designed for pre and post-test analysis or cross grade-level growth. By way of comparison, the SBAC is a summative assessment of comprehensive reading performance on the common core state standards. Therefore, these two tests assess different aspects of reading. The data necessary to analyze the effects of the program using SBAC results was limited because it was implemented for the first time in 2015, making only three years of data available. 4. Tier 2 and Tier 3 students received different intensity of services but there were too few students assessed to meet assumptions to run t-tests for each Tier. 5. As with most program evaluations, the findings and conclusions of this program evaluation are only applicable to the reading intervention program at the school participating in the study. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe About the Authors Brenda Morales Bravo is a principal at Rio Plaza Elementary School. She recently earned her Masters Degree in Educational Leadership from CSUCI. She is a bilingual, multicultural, and passionate advocate for students, families, and her community of Oxnard. She is the proud mother of three little girls and works towards being an exceptional role model to them and all the students at her school site. Dr. Charles Weis is an Assistant Professor of Educational Leadership at CSU Channel Islands and Co- Director of the Collaborative Online Doctorate in Educational Leadership offered jointly by CSU Channel Islands and Fresno State. He is a retired County Superintendent of Schools and current school board member for the Hueneme Elementary School Board in Port Hueneme, California. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe References Avant, D. W. (2016) Using response to intervention/multi-tiered systems of supports to promote social justice in schools. Journal for Multicultural Education, 10, 507-520. Buffman, A., Mattos, M. & Weber, C. (2009). Pyramid Response to Intervention: RTI, Professional Learning Communities, and How to Respond When Kids Don't Learn. Bloomington, IN: Solution Tree. California Department of Education (n.d.) Smarter Balanced Summative Assessments FAQs. Retreived from https://www.cde.ca.gov/ta/tg/ca/sbsummativefaq.asp Denton, C. A. (2012) Response to intervention for reading difficulties in the primary grades: Some answers and lingering questions. Journal of Learning Disabilities, 45, 232-243. Hughes, C. A. & Dexter, D. D. (2011) Response to intervention: A research-based summary. Theory into Practice, 50, 4-11. International Reading Association. (2002). Evidence-Based Reading Instruction: Putting the National Reading Panel Report into Practice. Retrieved from https://www1.nichd.nih.gov/publications/pubs/nrp/documents/report.pdf Jason, M. H. (2008). Evaluating programs to increase student achievement (2nd ed. ed.). Thousand Oaks, CA: Corwin Press. National Reading Panel (2000). National Reading Panel: teaching children to read : an evidence-based assessment of the scientific research literature on reading and its implications for reading instruction : reports of the subgroups. Washington, D.C.: National Institute of Child Health and Human Development, National Institutes of Health. Richards, C. (2010). Early Reading Intervention: Strategies and methods for teaching struggling readers. Boston, MA: Pearson/Allyn & Bacon. Rosen, P. (2018, June 2). MTSS: What you need to know. Retrieved from https://www.understood.org/en/learning-attention-issues/treatments-approaches/educational- strategies/mtss-what-you-need-to-know Shores, C. & Chester, K. 2009. Using RTI for School Improvement. Thousand Oaks, CA: Corwin Press. Salkind, N. J. 2007. Statistics for People Who (Think They) Hate Statistics. Third Edition. Thousand Oaks, CA: Sage Publications, Inc. University of Oregon Center. (n.d.). UO DIBELS Data System. Retrieved from https://dibels.uoregon.edu/assessment/dibels/index https://dibels.uoregon.edu/assessment/dibels/index Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Appendix A DIBELS Analysis Table 1.1. t-Test 2012 1st grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 71.71 24 30.428 6.211 End 86.58 24 72.020 14.701 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -14.875 61.753 12.605 -40.951 11.201 -1.180 23 .250 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. No significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 1.2. t-Test 2013 1st grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 81.35 37 24.083 3.959 End 112.14 37 78.881 12.968 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -30.784 64.684 10.634 -52.351 -9.217 -2.895 36 .006 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 1.3. t-Test 2014 1st grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 77.62 26 24.315 4.769 End 124.27 26 87.458 17.152 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -46.654 83.938 16.462 -80.557 -12.751 -2.834 25 .009 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 1.4. t-Test 2015 1st grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 80.31 16 26.076 6.519 End 118.63 16 75.784 18.946 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -38.313 70.877 17.719 -76.080 -.545 -2.162 15 .047 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 1.5. t-Test 2016 1st grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 62.64 25 31.563 6.313 End 98.20 25 69.711 13.942 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -35.560 58.063 11.613 -59.527 -11.593 -3.062 24 .005 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 2.1. t-Test 2012 2nd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 110.38 24 75.336 15.378 End 183.29 24 84.598 17.268 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -72.917 35.048 7.154 -87.716 -58.117 -10.192 23 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 2.2. t-Test 2013 2nd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 75.79 28 53.181 10.050 End 134.75 28 68.401 12.927 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -58.964 45.353 8.571 -76.550 -41.378 -6.880 27 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 2.3. t-Test 2014 2nd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 83.04 24 47.982 9.794 End 139.17 24 74.853 15.279 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -56.125 43.800 8.941 -74.620 -37.630 -6.277 23 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 2.4. t-Test 2015 2nd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 70.50 14 61.019 16.308 End 128.71 14 71.090 19.000 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -58.214 46.950 12.548 -85.323 -31.106 -4.639 13 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 2.5. t-Test 2016 2nd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 76.00 8 57.124 20.196 End 143.38 8 87.885 31.072 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -67.375 42.440 15.005 -102.855 -31.895 -4.490 7 .003 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 3.1. t-Test 2012 3rd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 137.54 13 67.804 18.806 End 261.08 13 78.715 21.832 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -123.538 66.940 18.566 -163.990 -83.087 -6.654 12 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 3.2. t-Test 2013 3rd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 161.58 24 62.217 12.700 End 267.63 24 98.841 20.176 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -106.042 53.945 11.011 -128.821 -83.263 -9.630 23 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 3.3. t-Test 2014 3rd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 120.30 20 60.843 13.605 End 213.55 20 91.167 20.386 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -93.250 47.666 10.659 -115.559 -70.941 -8.749 19 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 3.4. t-Test 2015 3rd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 116.40 15 58.468 15.096 End 217.80 15 81.975 21.166 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -101.400 54.387 14.043 -131.519 -71.281 -7.221 14 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 3.5. t-Test 2016 3rd grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 102.43 14 65.347 17.465 End 224.36 14 82.937 22.166 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -121.929 61.206 16.358 -157.268 -86.589 -7.454 13 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 4.1. t-Test 2012 4th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 137.53 19 72.509 16.635 End 278.00 19 91.879 21.078 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -140.474 62.084 14.243 -170.397 -110.550 -9.863 18 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 4.2. t-Test 2013 4th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 177.00 33 85.441 14.873 End 332.64 33 87.010 15.147 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg-End -155.636 62.886 10.947 -177.935 -133.338 -14.217 32 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 4.3. t-Test 2014 4th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 172.00 33 92.355 16.077 End 294.24 33 96.940 16.875 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -122.242 53.250 9.270 -141.124 -103.361 -13.187 32 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 4.4. t-Test 2015 4th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 156.23 43 84.288 12.854 End 277.26 43 93.019 14.185 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -121.023 61.453 9.371 -139.936 -102.111 -12.914 42 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 4.5. t-Test 2016 4th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 143.42 33 71.465 12.440 End 267.55 33 94.110 16.382 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -124.121 55.073 9.587 -143.649 -104.593 -12.947 32 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 5.1. t-Test 2012 5th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 252.11 27 74.487 14.335 End 342.78 27 101.360 19.507 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -90.667 72.242 13.903 -119.245 -62.089 -6.521 26 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 5.2. t-Test 2013 5th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 247.35 26 76.418 14.987 End 343.35 26 85.692 16.806 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -96.000 54.459 10.680 -117.996 -74.004 -8.989 25 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 5.3. t-Test 2014 5th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 288.60 42 116.570 17.987 End 366.26 42 104.257 16.087 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -77.667 72.101 11.125 -100.135 -55.198 -6.981 41 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 5.4. t-Test 2015 5th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 183.25 24 81.882 16.714 End 284.33 24 82.411 16.822 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -101.083 50.443 10.297 -122.384 -79.783 -9.817 23 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 5.5. t-Test 2016 5th grade DIBELS (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 Beg 162.52 27 70.337 13.536 End 257.41 27 75.564 14.542 Paired Differences t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 Beg- End -94.889 37.404 7.198 -109.685 -80.093 -13.182 26 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Appendix B SBAC Analysis Table 6.1. t-Test 4th grade SBAC ELA/Literacy (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 V5 23339.27 45 56.319 8.396 V8 2368.24 45 60.016 8.947 Paired Sample Tests t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 V5-V8 -28.978 51.584 7.690 -44.475 -13.480 -3.768 44 .000 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Brenda Morales Bravo; Charles Weis Allies for Education 2018, 2, 2 https://journals.library.csuci.edu/ojs/index.php/afe Table 6.2. t-Test 5th grade SBAC ELA/Literacy (Paired Sample Statistics) Mean N Std. Deviation Std. Error Mean Pair 1 V5 2363.67 40 74.684 11.809 V8 2380.48 40 76.067 12.027 Paired Sample Tests t df Sig. (2-tailed) Mean Std. Deviation Std. Error Mean 95% Confidence Interval of the Difference Lower Upper Pair 1 V5-V8 -16.800 52.128 8.242 -33.471 -.129 -2.038 39 .048 Note. t = t value. df = degrees of freedom. Sig. = p < .05 standard for significance level. Significant difference was found. Achieving Reading Equity: A Systematic Program Evaluation of a Reading Intervention Program Introduction Program Description Purpose of the Program Evaluation Methods for Conducting the Program Evaluation Results Conclusions and Recommendations Recommendations - Implications for Practice Limitations of the Program Evaluation About the Authors References Appendix A Appendix B