Évaluation de référence du Programme de Lecture Précoce en Haïti (ERP)

DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL EVALUATION REPORT Haiti Early Reading Program (ERP) Baseline Evaluation This publication was produced for review by the United States Agency for International Development by Social Impact, Inc. COVER PHOTO: GRADE 2 STUDENTS, ARTIBONITE, HAITI ANDREW CARMONA, 2017 USAID.GOV EARLY READING PROGRAM BASELINE EVALUATION FINAL REPORT | i Haiti Early Reading Program (ERP) Baseline Evaluation USAID/Haiti AID-521-C-17-00002 March 16, 2018 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Andrew Carmona, Basab Dasgupta, Amanda Stek, Joanie Cohen-Mitchell, and Euphonise Loiseau on behalf of Social Impact, Inc. Team member qualifications can be found in Annex G of this report. ii | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV CONTENTS ACRONYMS IV ACKNOWLEDGEMENTS V ABSTRACT VI EXECUTIVE SUMMARY VIII INTRODUCTION 1 METHODOLOGY 5 FINDINGS 16 CONCLUSIONS 54 ANNEX A: FULL LISTING OF REFERENCES IN REPORT 58 ANNEX B: DATA COLLECTION TOOLS 60 ANNEX C: EVALUATION METHODOLOGY 116 ANNEX D: ADDITIONAL FINDINGS 123 ANNEX F: EVALUATION STATEMENT OF WORK 127 ANNEX G: QUALIFICATIONS OF TEAM MEMBERS 136 ANNEX H: EVALUATION DISSEMINATION PLAN TEMPLATE 139 ANNEX I: DATA QUALITY ASSURANCE MEASURES 141 ANNEX J: INTER-RATER RELIABILITY SCORES OF ENUMERATORS 142 ANNEX K: CONSIDERATIONS FOR FUTURE DATA COLLECTION 144 ANNEX L: DISCLOSURE OF ANY CONFLICTS OF INTEREST 145 TABLE OF TABLES Table 1: Partners and their SOWs for the ERP Impact Evaluation ................................................................ ........ 3 Table 2: List of ERP implementers and evaluation sample schools ................................................................ ........ 6 Table 3: Planned qualitative sample size at baseline ................................................................................................ .. 9 Table 4. EGRA subtasks ................................................................................................................................ ................. 14 Table 5: Study limitations ................................................................................................................................ ............... 15 Table 6: School sample realized by school type, rural/urban, and department ................................ ................ 17 Table 7: Final EGRA sample realized, by grade and gender ................................................................ .................. 18 Table 8: Final sample realized for qualitative data collection ................................................................ ................ 18 Table 9: Covariate balance statistics after CEM matching ................................................................ ..................... 19 Table 10: EGRA scores (for all), with p-values ................................................................................................ ......... 21 Table 11: Proportion of zeros in subtasks across treatment groups ................................................................ .. 23 Table 12: Updated MDES, by EGRA sub-tasks and Grade ................................................................ .................... 25 Table 13: Basic school-level information across schools ................................................................ ....................... 26 USAID.GOV EARLY READING PROGRAM BASELINE EVALUATION FINAL REPORT | iii Table 14: School level resources and activities, by treatment and comparison ................................ ............... 27 Table 15: EGRA subtask scores from other Haiti EGRA studies ................................................................ ........ 31 Table 16: Distribution of ORF scores across grades and by CWPM categories ................................ ............. 32 Table 17: Distribution of CLPM scores across grades ................................................................ ........................... 33 Table 18: Distribution of CLSPM scores across grades ................................................................ ......................... 33 Table 19: Regression models showing the effects of various factors on ORF ................................ .................. 40 TABLE OF FIGURES Figure 1 : Sampled schools of the ERP Baseline Evaluation.................................................................................... vii Figure 2: Sample size calculations and power analysis for the ERP Impact Evaluation ................................ ...... 9 Figure 3: Decile distribution of timed learning test scores (percent of students in each category)............ 22 Figure 4: Quartile distribution of non-zero scores in timed outcomes (for all students) ............................. 24 Figure 5: School Director experience ................................................................................................ ........................ 27 Figure 6: Teacher characteristics, per school ................................................................................................ ........... 28 Figure 7: Descriptive statistics for classroom-level variables ................................................................ ................ 29 Figure 8: Language spoken by students at home, by school type and location ................................ ................. 29 Figure 9: Extent of punitive measures (as reported by students), by school type (percentage) .................. 30 Figure 10: Percent of students who eat in the morning before school ................................ .............................. 30 Figure 11: Average scores (in words) in timed subtasks, by grade and gender ................................ ................ 34 Figure 12: Proportion of zero scores in timed subtasks, by grade and gender ................................ ................ 34 Figure 13: Timed learning outcomes non-zero scores, by gender ................................................................ ...... 35 Figure 14: Average scores in non-timed subtasks, by grade and gender ................................ ............................ 36 Figure 15: Proportion of zeros in non-timed subtasks, by grade and gender ................................ ................... 36 Figure 16: Non-timed learning outcomes, non-zero scores, by gender ................................ ............................. 37 Figure 17: Timed learning outcomes, by grade and implementer ................................................................ ........ 38 Figure 18: Non-timed learning outcomes, by grade and implementer ................................ ............................... 38 Figure 19: Timed learning outcomes, by grade and type of school ................................................................ ..... 39 Figure 20: Non-timed learning outcomes, by grade and type of school ................................ ............................ 39 Figure 21: Decomposition of drivers that explain ORF performance, overall and by gender ...................... 44 Figure 22: Classroom-level variables on classroom management and phonics approaches .......................... 49 Figure 23: Classroom-level variables on classroom materials and other topics ................................ .............. 50 iv | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV ACRONYMS AIR American Institutes for Research ATL Assistant Team Leader CEM Coarsened Exact Matching CLPM Correct letter identification per minute CLSPM Correct letter sound per minute CWPM Correct word per minute DQA Data Quality Assurance EGRA Early Grade Reading Assessment EMIS Education Management Information System EPT Éducation Pour Tous EQ Evaluation Question EQUI ® Evaluation Quality, Utilization, and Impact ® ERP Early Reading Program ESS Evaluation and Survey Services ET Evaluation Team FGD Focus Group Discussion FONHEP Haitian Foundation for Private Teaching ICC Intra-class Correlation IDB Inter-American Development Bank IE Impact Evaluation IFOS Institut de Formation du Sud IP Implementing Partner IRR Inter-rater reliability JICA Japan International Cooperation Agency KII Key Informant Interview LAC/READS Latin American and Caribbean READS Program MDES Minimum Detectable Effect Size MENFP The Ministry of Education and Professional Training NGO Non-governmental Organization ODK Open Data Kit ORF Oral Reading Fluency PTA Parent-Teacher Association QA Quality Assurance QED Quasi-experimental Design RCT Randomized Control Trial SD Standard Deviation SI Social Impact, Inc. SOW Scope of Work SSME Snapshot of School Management Effectiveness ToTAL Tout Timoun Ap Li UND University of Notre Dame UNICEF The United Nations Children's Fund USAID United States Agency for International Development USAID.GOV EARLY READING PROGRAM BASELINE EVALUATION FINAL REPORT | v ACKNOWLEDGEMENTS The authors would like to wholeheartedly thank several key individuals who were critical to the successful completion of this baseline report. We thank the Education Team at USAID/Haiti, namely Christine Veverka and Ezechiel Macon, for being flexible, available, and responsive to our various questions and inquiries. We extend very deep thanks to Samuel Emieux Jean, Ulrick Jean Claude, and Charles Tardieu of IFOS, as well as the 122 supervisors and enumerators who carried out data collection for this study. This report would not have been possible without the hard work of these motivated young Haitian men and women who spent five weeks collecting data in the middle of the rainy season across Haiti. The authors would also like to express their appreciation to the ERP partners at Save the Children (Ossemer and James), AIR (Rachelle and Mohammed), UND (TJ, Kate, and Paul), and UNICEF (Mirko, Beatrice, and Vegains) for their assistance in facilitating our work. We also want to acknowledge the public-sector stakeholders at the MENFP who authorized this study and provided valuable input, in particular Harold Narcisse and the Director General, Dr. Meniol Jeune. Finally, we offer our full gratitude to the students, teachers, school directors, parents, and departmental-level MENFP officials who graciously gave their time to take learning tests, participate in observations, and give candid feedback to our team. It is for them and all the students in Haiti that we continue to work hard to ensure that every child everywhere can read. vi | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV ABSTRACT In Fall 2017, Social Impact, through its Haiti-based Evaluation Survey Services (ESS) project, conducted a baseline evaluation of the Early Reading Program (ERP), a USAID-funded intervention focused on improving reading outcomes for Grade 1 and 2 students, teaching quality, and capacity of the Ministry of Education in Haiti to implement and scale up reading programs. The ESS team designed and conducted the baseline for a pre-post impact evaluation (IE) design with statistical matching, administering Early Grade Reading Assessment (EGRA) learning tests, a student questionnaire, a school director survey, a classroom observation tool, focus group discussions with students and parents, and key informant interviews with teachers and MENFP officials across seven departments (Artibonite, Grand’Anse, Nord, Nord-Est, Ouest, Sud, and Sud-Est). The team concluded that students are generally unable to read in Creole. Further, many important factors at the household-level (eating before school, parents’ involvement in a student’s education, access to books), school-level (director’s experience, number of formally trained 1 teachers, existence of extracurricular activities, helping students to pay their tuition) and classroom-level (teachers’ use of certain phonics approaches) were associated with higher learning scores. Teachers generally managed their classrooms well (assessed by observing students’ behavior in the classroom, utilization of classroom time between start and end times of class, use of varying materials and strategies to teach), though corporal punishment was observed in 11-13 percent of classrooms. Teachers consistently lacked sufficient books and other educational materials and did not generally use best practice phonics approaches for reading. Qualitative data suggested that the MENFP lacks the capacity and resources to support schools and teachers with funding, materials, or training. 1 A formally trained teacher is one that has a professional degree from a teaching college. USAID.GOV EARLY READING PROGRAM BASELINE EVALUATION FINAL REPORT | vii Figure 1 : Sampled schools of the ERP Baseline Evaluation viii | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY CONTEXT According to 2015 estimates, 2 61 percent of Haitian adults (64 percent of males and 57 percent of females) are literate. Education in the country is plagued by constraints that range from low funding to inadequate teacher training (CIA World Fact Book, 2018). The Ministry of Education and Professional Training (MENFP) is aware of students’ poor performance in reading and has worked with the United States Agency for International Development (USAID) in recent years to implement reading interventions to address this situation. BACKGROUND In 2017, USAID recognized a need to scrutinize their ongoing investments being made to improve reading outcomes in Haiti. USAID determined that the best way to evaluate these ongoing activities was through an independent impact assessment. The impact evaluation (IE) , titled the Early Reading Project, will assess a two-year time period of intervention from multiple different ongoing activities and implementing partners already working to support early grade reading instruction in Grade 1 and Grade 2 of primary school in approximately 658 public, private, community, and semi-private schools in the departments of Artibonite, Ouest, Grand’Anse, Sud, Sud-Est, Nord, and Nord-Est. Broadly speaking, and working across these early grade reading activities in seven different departments, USAID has recognized three common objectives: (1) Improve MENFP’s capacity to implement and eventually scale up early grade reading reform; (2) Improve teacher ability to teach early grade reading; and (3) Improve students’ early grade reading outcomes. Partners participating in the Early Reading Project IE include Save the Children (STC), American Institutes for Research (AIR), the University of Notre Dame (UND), and the UN Children’s Fund (UNICEF). All of these partners use, or will use, the early grade reading and writing curriculum materials co-developed by the Ministry of Education and USAID. The French version of these materials is called “Je parle bien francais.” The Creole version of these materials is called “M’ap li net ale”. Materials should be used concurrently in all schools. The curriculum employs a scripted model of instruction for teachers and is phonics-based. In first grade, the French portion is completely oral, focusing on building children’s vocabulary and knowledge of the French language, while the Creole portion focuses on explicit reading and writing instruction for children, so they are first learning to read in a language they speak and understand. Across partners, the intervention is employed in two different models: “core” and “enhanced.” Both models include the implementation of the aforementioned co-designed (USAID and MENFP) national curriculum and teacher training and coaching in early grade reading and writing, but the enhanced model differs in other key extracurricular interventions. The “enhanced” group also contains some UNICEF-supported schools that employ a “pre-Kindergarten” reading model. Baseline data for the Early Reading Project IE was collected in November and December of 2017. Midline data will be collected during the 2018-2019 school year. Endline data will be collected in the spring of 2020. 2 CIA World Factbook, updated 1 January 2018. Obtained on 2 March 2018. USAID.GOV EARLY READING PROGRAM BASELINE EVALUATION FINAL REPORT | ix PURPOSE AND EVALUATION QUESTIONS USAID requested that Social Impact (SI)’s Haiti Evaluation and Survey Services (ESS) to design and implement an IE of their ongoing early grade reading and writing investments. The evaluation is intended to measure whether early grade reading and writing interventions have resulted in students in the target schools demonstrating greater skills in reading than their peers of the same grades in non-targeted, similar schools. The evaluation findings will inform USAID/Haiti’s future programming in early grade reading in Haitian schools. The results of the evaluation are also intended to inform Government of Haiti policy-level decisions around implementation of reading curriculum and teacher training on a national scale. The lessons learned and recommendations documented in the final endline evaluation report may also inform future programming for other donors’ activities related to early grade reading in Haiti. The evaluation’s five high-level evaluation questions (EQ) look specifically at the outcomes of (1) reading fluency among Grade 1 and 2 students, (2) teacher quality, and (3) MENFP capacity to implement and scale up reading programs. The questions are as follows: • Q1. What is the impact, if any, of the “core” ERP intervention on students’ reading outcomes in the first and second grades? Some sub-questions the evaluation team will consider in addressing this question are: • Q2. What is the relative impact, if any, on student learning, of the “enhanced” model carried out by UND and UNICEF? Do students exposed to this model acquire stronger reading skills than their peers in control and other ERP schools? What aspects of the model impact student learning the most? • Q3. What is the impact, if any, on student learning, of the “pre-reading instruction” model piloted by UNICEF in the pre-K and K levels of 120 of the ERP program schools? Do students exposed to the “pre-reading instruction” model acquire stronger reading skills than their peers in control and other ERP schools? • Q4. What is the impact of the ERP intervention, if any, on teachers’ classroom practices in early grade reading instruction? • Q5. What is the impact, if any, of the ERP program on the Ministry’s capacity to implement and eventually scale up early grade reading reform? Are the human, material, and financial management capacities at the MENFP sufficiently improved because of the ERP intervention that the Ministry could successfully expand the ERP program nationwide? DESIGN Haiti ESS followed a pre-post IE design with statistical matching. The evaluation team (ET) collected and analyzed data on equal numbers of girl and boy students in Grades 1 and 2 from participating schools and compared their performance to a control group. To successfully measure the three outcomes of interest, the ET administered Early Grade Reading Assessment (EGRA) learning tests supplemented by student questionnaires, school director surveys, classroom observations, and qualitative data collected through focus group discussions (FGDs) and key informant interviews (KIIs). At baseline, the ET administered the EGRA in Haitian Creole with one short EGRA subtask on oral French comprehension. A sample of 130 schools was randomly sampled proportionally from “core” and “enhanced” implementing partner schools to form the core and enhanced treatment samples, with 65 schools in each sample. At baseline, the ET also identified 65 suitable comparison schools from non-treated schools. After an EGRA validation workshop and school listing activity conducted in October 2017, the ET collected baseline data in November-December 2017. A midline will be conducted in April-May 2019 and an endline in April-May 2020. x | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS GENERAL The absence of an EMIS system in Haiti hampers educational research and program implementation. The availability of school-level data is essential for conducting rigorous research and for better understanding gaps in educational access throughout the country. Without regular, authoritative census data on schools nationwide, implementers, evaluators, and public-sector actors at the MENFP will be limited in their ability to conduct high quality programming and respond to the needs of communities. READING FLUENCY Disproportionate numbers of zero scores have led to a situation where EGRA scores at baseline are partially imbalanced across treatment status. We found that students in Grade 2 in the core treatment group outperform their counterparts from the enhanced and control groups significantly in almost all subtasks (timed and untimed). Variation in EGRA scores at baseline was also noted across genders, school types, and implementers. Among implementers, better reading performance was found among Save the Children schools, which are all located in the Port-au-Prince urban area. Girls score significantly better than boys across all tests (and in both grades), and public and rural schools had lower oral reading fluency scores relative to private, community, religious, and urban schools. Surveyed students have Creole language skills but do not yet have reading skills. Students in both Grades 1 and 2 scored highest in Creole letter dictation and listening comprehension, but lowest in Creole word dictation and reading comprehension. This result was to be expected in Grade 1 and somewhat in Grade 2 as well, as students begin their formal education, but students in higher grades in primary school (Grade 3 and up) should begin to show higher reading scores. This is in line with other EGRA studies in Haiti that found students were not able to read at sufficient speed (with comprehension to understand text) until Grade 3 or later. 3 The home environment plays a critical role in a student’s reading scores. At the student-level, students who eat before school, have parents involved in their education, spend time reading, have access to a library, and have books at home have higher oral reading fluency scores than students who do not. Qualitative data showed that parents are involved and invested in their children’s education (for both girls and boys). However, they are mostly unaware of PTA activities even though school directors report that nearly three in four schools have active PTAs. Most parents in FGDs noted that school fees are a major barrier to school attendance, although they are willing to go to great lengths to pay for their children’s education. While at baseline girls reported taking on a disproportionate amount of housework outside of school, a shift in community attitudes seems to suggest more girls are being allowed to go to school than before. 3 EDDATA II: Education Data for Decision Making. The Early Grade Reading Assessment (EGRA) in Haiti. June 2014. Prepared by RTI International. Contract Number EHC-E-00-04-00004-00. www.eddataglobal.org . The three studies reviewed in the EDDATA paper showed the average third grade Haitian students were able to correctly read between 23-39 correct words per minute (cwpm) in Creole; 35-60 cwpm is considered the minimum rate necessary to comprehend a basic story. USAID.GOV EARLY READING PROGRAM BASELINE EVALUATION FINAL REPORT | xi School-level factors explain the largest amount of variation in oral reading fluency for girls and boys. Regression and decomposition analysis found that a host of school-level variables have significant positive associations with reading scores. School management assisting with students’ school fees, the experience level of school directors, and number of formally trained female teachers at a school are all associated with positive oral reading fluency scores. Notably, while a minority of schools report having extracurricular programs and services, having these programs was associated with higher oral reading fluency. Furthermore, students were associated with better reading fluency when the average number of hours taught (per day at school) was found to be higher or when the school had a library. As the student-to-teacher ratio increased (more students per teacher), student reading scores decreased. Girls appeared to benefit most from more and better-trained female teachers at a school. In the classroom, use of certain phonics approaches works better than others, and seating arrangement matters. About half of teachers used a phonics approach while teaching reading, and those teachers who practiced blending sounds and decoding syllables saw better results in reading scores than those who simply asked students to repeat words aloud. Treatment of girls and boys in the classroom was gender equitable; however, regression analysis showed that seating boys and girls together was associated with significant adverse effects on reading. TEACHING QUALITY Classroom management is generally good in terms of observed students’ behavior in the classroom, utilization of classroom time; however, corporal punishment was observed in some classrooms. Availability, access to, and use of educational materials in the classroom are highly variable. Grade 1 and 2 teachers generally manage student behavior well, start and end class on time, and use a variety of teaching techniques. However, students openly claim that teachers practice corporal punishment, and the practice of corporal punishment was observed in 11-13 percent of classrooms. Teachers generally practice gender equitable treatment of students, with no major statistical differences between how girls and boys are treated in the classroom. Classroom observations noted an inconsistent use of books and a paucity of basic items such as pencils. Qualitative data from MENFP, teachers, and parents suggest a persistent scarcity of books and materials. Use of pedagogical reading approaches in class is not consistent. Only about half of teachers use a recognized phonics approach during teacher observation. As reported in teacher KIIs, teachers with more years of experience were more likely to be able to specify good phonics practices. Reading aloud to students was the most practiced phonics approach, though it did not necessarily correlate to improved reading scores. Low levels of teacher formalization probably contribute to low reading scores. Only about a quarter of teachers in sampled schools were formalized (professionally trained) and on-the-job teacher training opportunities are nonexistent or occur very rarely. Most teachers reported teaching for years at a time without being offered training. Many teachers have no formal training. MENFP acknowledges that it lacks the resources to hold regular trainings. This deficit could pose a significant barrier to improvements in reading, given the regression analysis finding that formalized teachers with more experience are associated with higher oral reading fluency scores, especially for girls. MENFP CAPACITY TO IMPLEMENT AND SCALE UP READING PROGRAMS Departmental-level MENFP lacks the financial capacity to adequately support schools. Quantitative and qualitative data combine to paint a picture in which schools do not receive books and other educational materials, teachers are sometimes delayed payment, and MENFP-supported teacher xii | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV training is not offered on a regular basis. Families must take on the burden of supporting education for their children, and when they cannot, children go to school without necessary supplies or are held back. At baseline, MENFP offices relied heavily on NGOs to support and sustain schools with materials, funds, and teacher training. Departmental-level MENFP staff appear unaware of policies or plans through which deficiencies in the educational system can be improved. Officials were able to cite problems, such as non-compliance by teachers of the unofficial language policy, high levels of unqualified teachers, and the lack of teacher training, but they could not specify mechanisms or actions through which these issues could be resolved. MENFP officials carry out monitoring visits often but many do not appear to be deeply knowledgeable about phonics/reading pedagogy. Nearly three in four schools sampled had received an MENFP visit in the past year. However, many MENFP officials interviewed could not discuss reading pedagogy in depth. This lack of knowledge could have implications for policy development and quality oversight of reading programs in the future. GENDER In general, findings around gender equity in the household and classroom were positive when it came to education. For instance, parents informed that girls are sent to school at a higher rate than in the past and are treated equally well (or punished equally) compared to boys in the classroom. In fact, qualitative FGDs with parents revealed that parents are allowing girls to go to school more than they have in the past, signaling a shift in attitudes towards girls’ education. Some discrepancies remain, however. For instance, more girls than boys are required to do household chores. Perhaps the most important finding about gender was that girls’ reading scores are significantly higher across all measures when compared to those of boys. This finding is not entirely surprising. Studies have shown that girls do better than boys in most subjects, probably due to the fact that girls study more, have more positive perceptions of school and behave better, 4 according to findings supported by KIIs and FGDs held at baseline. An interesting finding of the regression analysis showed that when a school has more and better-trained female teachers, girls have better reading scores. The finding suggests that female teachers may act as role models for girls, enabling and encouraging them to attend and participate in school. 4 Voyer, Daniel, and Susan D. Voyer. "Gender differences in scholastic achievement: A meta-analysis." Psychological bulletin 140.4 (2014): 1174. 1 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV INTRODUCTION CONTEXT According to 2015 estimates, 61 percent of adults in Haiti (64 percent of males and 57 percent of females) are literate. Nationwide literacy rates in Haiti have risen over the last two decades, but when compared globally, Haiti still ranks near the bottom at 141 out of 161 countries. Haiti’s literacy rate is well below the regional Latin America and Caribbean average of 90 percent. 5 Education in Haiti is plagued with constraints that range from low funding to inadequate teacher training. Between 2000 and 2010, the national budget line item for education declined from 17 percent to 10 percent, putting Haiti in the bottom quartile of countries ranked by public spending on education. 6 The 2010 Haiti earthquake exacerbated these problems, displacing students and damaging schools throughout the country. The Ministry of Education and Professional Training (MENFP) is aware of students’ poor outcomes in reading as well as the system-wide challenges that must be addressed to improve those outcomes. According to the national curriculum, Haitian primary students are expected to study reading in Creole for 90 hours per academic year (Grades 1 and 2) and French for 30 hours (Grade 2 only), or for 40 minutes, three out of five days of the week each school year. 7 However, reading instruction in the early grades in Haitian schools is often cursory, under-resourced, and ineffective. As a result, children score poorly on standardized reading assessments. Few properly designed instructional materials for teaching reading in the early years are available and teachers are rarely trained in the techniques proven to be most effective in teaching reading. Formative evaluation is rare and lacking in rigor; consequently, students who are struggling to learn to read are often not identified in time for appropriate intervention. System-wide, Haiti practices “automatic promotion” of students in the early primary years, and students who have not learned to read are passed up from grade to grade. Eventually, illiterate students either fail or drop out of school. 8 In recent years, MENFP has worked with USAID to implement interventions that will improve the likelihood that children will learn to read in Haitian schools. From 2011-2016, with support from USAID, MENFP implemented two reading interventions: the ToTAL and the An n ALE programs. These interventions supported 297 schools in improving reading instruction. Support included scripted reading lesson materials, which did not always arrive on time to be used appropriately in the classroom, and intervention-specific teacher training, which was not implemented sufficiently to improve teacher effectiveness in teaching reading. EGRA AND ITS USE IN HAITI The Early Grade Reading Assessment (EGRA) is an assessment that is administered orally to students in the early phases of primary school. EGRA can be used to assess Grade 1-3 student reading skills and identifies gaps or shortcomings that need to be addressed for future student success in reading. Core subtasks that make up the EGRA include letter reading, phonemic awareness, word reading, and story 5 CIA World Factbook, updated 1 January 2018. Obtained on 2 March 2018. https://www.cia.gov/library/publications/the-world-factbook/geos/ha.html 6 Suzata, Eriko. 2011. Education in Haiti: An Overview of Trends, Issues, and Plans. World Innovative Summit for Education. 7 EDDATA II: Education Data for Decision Making. The Early Grade Reading Assessment (EGRA) in Haiti. June 2014. Prepared by RTI International. Contract Number EHC-E-00-04-00004-00. www.eddataglobal.org. 8 USAID. Haiti ESS ERP Impact Evaluation Scope of Work. October 2017. 2 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV reading with comprehension questions. Other EGRA subtasks may include pre-literacy skills, basic vocabulary, listening comprehension, and sentence or non-word writing. 9 Oral reading fluency, when coupled with comprehension, can be considered a measure for basic reading competencies. Overall, EGRA provides a methodology for quickly assessing a variety of early literacy skills. In general, EGRA identifies strengths and weaknesses in student reading which in turn indicate the quality of learning taking place in schools. 10 EGRA has been used in Haiti to evaluate both reading programs and the general state of literacy. As part of the USAID EQUIP2 project implemented by Save the Children in 2007, EGRA effectively revealed a high degree of variability in reading scores among private schools in Haiti; the EGRA found no evidence that private or public schools were doing a better job of teaching reading. A study conducted by RTI in 2008-2009 used the EGRA to assess reading in Grades 1-3 in the Nippes and Artibonite departments. The assessment found that while there were no differences between male and female test-takers, there was a clear pattern of higher literacy among students who possessed books at home. Use of EGRA in the USAID PHARE project, implemented by AIR in 2009, demonstrated no significant differences in reading scores between public and private schools or between the sexes. The EGRA assessment did, however, reveal large differences in reading scores between geographic departments. In 2013, Concern International noted high levels of illiteracy in the higher grades and expanded its Saut d’Eau project to include an EGRA assessment for Grades 1-6. This example proves that EGRA allows flexibility and adaptability and facilitates tailored approaches to specific local contexts. The 2011-2016 implementation of the USAID-funded ToTAL project is the most recent use of EGRA in Haiti. The ToTAL project linked EGRA results to results from a Snapshot of School Management Effectiveness (SSME) survey, which allowed for analyzing reading outcomes in a more holistic way. In linking SSME data to EGRA results, ToTAL staff sought to help local stakeholders by identifying helpful take-away points that could be applied at the classroom, school, and district levels. PROJECT BACKGROUND THE EARLY READING PROJECT PROGRAM In 2017, USAID recognized a need to scrutinize their ongoing investments being made to improve reading outcomes in Haiti. USAID determined that the best way to assess these ongoing activities was through an independent impact evaluation (IE). The IE, titled the Early Reading Project, will analyze a two-year time period of intervention from multiple different ongoing activities and implementing partners already working to support early grade reading instruction in Grade 1 and Grade 2 of primary school in approximately 658 public, private, community, and semi-private schools in the departments of Artibonite, Ouest, Grand’Anse, Sud, Sud-Est, Nord, and Nord-Est. Broadly speaking, and working across these early grade reading activities in seven different departments, USAID has recognized three common 9 Core subtasks or common instrument components of the present EGRA testing tool consist of listening comprehension, letter identification, familiar or non-word reading, oral reading fluency, and initial or final sound identification. Other components that do not fall under core subtasks are dictation and phoneme segmentation. Earlier EGRA tools included tests for pre-literacy skills such as picture-based subtask (as those in the Peabody Picture Vocabulary Test (PPVT)), or pictures to identify knowledge of common vocabulary such as words for body parts like hand, head, or toe. Print assessments (a student’s ability to indicate where on a page to begin reading or where to read next, for example) have also been eliminated from the current EGRA (Early Grade Reading Assessment Toolkit, Second Edition, 2016). 10 RTI International. 2015. Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: United States Agency for International Development. 3 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV objectives: (1) Improve MENFP’s capacity to implement and eventually scale up early grade reading reform; (2) Improve teacher ability to teach early grade reading; and (3) Improve students’ early grade reading outcomes. Partners participating in the Early Reading Project IE include Save the Children (STC), American Institutes for Research (AIR), the University of Notre Dame (UND), and the UN Children’s Fund (UNICEF). All of these partners use, or will use, the early grade reading and writing curriculum materials co-developed by the Ministry of Education and USAID. The French version of these materials is called “Je parle bien francais.” The Creole version of these materials is called “M’ap li net ale”. Materials should be used concurrently in all schools. The curriculum employs a scripted model of instruction for teachers and is phonics-based. In first grade, the French portion is completely oral, focusing on building children’s vocabulary and knowledge of the French language, while the Creole portion focuses on explicit reading and writing instruction for children so they are first learning to read in a language they speak and understand. Across partners, the intervention is employed in two different models: “core” and “enhanced.” Both models include the implementation of the aforementioned co-designed (USAID and MENFP) national curriculum and teacher training and coaching in early grade reading and writing, but the enhanced model differs in other key extracurricular interventions ( Table 1 ). Baseline data for the Early Reading Project IE was collected in November and December of 2017. Midline data will be collected during the 2018-2019 school year. Endline data will be collected in the spring of 2020. Table 1: Partners and their SOWs for the ERP Impact Evaluation Partner Subcontractor Model Year 1 Schools Year 2 Schools Department(s) American Institutes for Research (AIR) FONHEP Core 150 150 Nord, Nord-Est, Ouest, Artibonite Save the Children N/A Core 39 39 Ouest University of Notre Dame (UND) Catholic Relief Services (CRS); CEEC Core and Enhanced: School libraries and summer camps 160 160 Grand-Anse, Sud United Nations Children's Fund (UNICEF) CARE; PLAN Core and Enhanced: Reading clubs and PTAs; Pre-Scolaire program 106 106 Nord, Nord-Est, Sud- East DEVELOPMENT HYPOTHESIS The evaluation team has synthesized the development objectives and hypothesis as follows: DEVELOPMENT OBJECTIVES: 1. Increase literacy through the provision of core skills and organizational reforms to MENFP (for example, improved coordination, consistent and effective capacity building, improved leverage among education sector actors and networks, and creation of strong feedback loops between classroom and program designers) to assist learners in 670 schools in mastering basic literacy skills. 2. Increase proportion of students who can read and demonstrate reading comprehension by the end of the primary grades as defined by the Haitian curriculum, standards, or as agreed-upon by national experts. 3. Increase proportion of students who demonstrate improvement in reading and understanding of grade level texts by the end of two grades of primary schooling. 4 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV DEVELOPMENT HYPOTHESIS: Effective implementation of USAID-supported early grade reading and writing interventions can lead to improved reading outcomes through: 1. Provision of necessary support and technical guidance to the MENFP; 2. Implementation of the USAID/MENFP early grade reading and writing curriculum in primary schools (even if only in an experimental stage); 3. Establishment of strong feedback loops between all types of schools implementing the program and the design team in Port-au-Prince. EVALUATION PURPOSE AND AUDIENCE PURPOSE USAID requested that SI’s Haiti Evaluation Support Services (ESS) design and implement an independent IE of it’s early grade reading and writing interventions across partners. The purpose of the evaluation is to assess the overall effectiveness of the program on early grade reading among Grade 1 and 2 students. The IE is intended to measure whether the ERP interventions have resulted in students in the target schools demonstrating greater skills in reading than their peers of the same grades in non-targeted, homologous schools. The evaluation findings will inform USAID/Haiti’s future programming in early grade reading in Haitian schools. The results of the evaluation are also intended to inform Government of Haiti policy-level decisions around implementation of reading curriculum and teacher training on a national scale. The lessons learned and recommendations documented in the final endline evaluation report may also inform future programming for other donors’ activities as they relate to early grade reading in Haiti. PRIMARY STAKEHOLDERS The primary stakeholders for this evaluation include the MENFP and USAID/Haiti, the implementing partners evaluated under the Early Reading Project IE; the Latin American and Caribbean (LAC)/Reads program; UNICEF; University of Notre Dame (UND); Save the Children; The United Nations Office for Project Services (UNOPS)); and other donors and entities active in Haiti’s education sector, including the World Bank, Japan International Cooperation Agency (JICA), Inter-American Development Bank (IDB), and World Vision, as well as a variety of non-governmental organizations (NGOs) and religious affiliations devoted to improving learning in Haitian primary schools. OUTCOMES OF INTEREST READING/LITERACY: In particular, the evaluation team (ET) looked at the following skills: letter- sound mastery (phonetic awareness); familiar word reading; reading connected text/oral reading fluency (ORF); reading comprehension; writing a sentence dictated to the student; and writing a sentence the student generates in response to an open verbal prompt. TEACHING QUALITY: The ET examined certain aspects of teaching quality, including classroom management, use of phonics approaches, use of educational and teaching materials, gender equity, and corporal punishment. Observing these characteristics also allowed the team to better understand how teachers applied their training in the classroom and whether that training had a beneficial effect on student performance. MENFP CAPACITY: The ET looked at the activities under the Early Reading Project targeting the MENFP and their contributions to increasing knowledge of and capacity to implement reading programs. The team also examined the existence of financial and human resources necessary for the implementation, continuation, and/or scaling up of early grade reading programs. 5 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV EVALUATION QUESTIONS USAID and the ESS team developed the following evaluation questions (EQ) to guide the IE. EQ1. What is the impact, if any, of the “core” ERP intervention on students’ reading outcomes in the first and second grades? Some sub-questions the evaluation team will consider in addressing this question are: a. How do students exposed to any ERP programming perform when compared to similar students who are not exposed to ERP programming? b. What is the marginal effect, if any, of ERP implementer on reading outcomes? c. What is the marginal effect, if any, of type of school (semi-private, public, community, religious, etc.) on reading outcomes? d. To what extent do environmental or situational factors have an impact on students’ ability to learn to read in the ERP target schools? EQ2. What is the relative impact, if any, on student learning of the “enhanced” model carried out by UND and UNICEF? Do students exposed to this model acquire stronger reading skills than their peers in control and other ERP schools? What aspects of the model impact student learning the most? EQ3. What is the impact, if any, on student learning, of the “pre-reading instruction” model piloted by UNICEF in the pre-K and K levels of 120 of the ERP program schools? Do students exposed to the “pre-reading instruction” model acquire stronger reading skills than their peers in control and other ERP schools? EQ4. What is the impact of the ERP intervention, if any, on teachers’ classroom practices in early grade reading instruction? a. Do teachers exposed to the “enhanced” ERP intervention teach reading differently and/or better than their peers in the “core” ERP schools? b. Do teachers exposed to the ERP intervention teach reading differently and/or better than their peers in the identical or close-to-identical schools without ERP interventions? c. How, if at all, do teachers exposed to the ERP intervention teach reading differently after having received the ERP intervention? EQ5. What is the impact, if any, of the ERP program on the Ministry’s capacity to implement and eventually scale up early grade reading reform? Are the human, material, and financial management capacities at the MENFP sufficiently improved because of the ERP intervention that the Ministry could successfully expand the ERP program nationwide? METHODOLOGY EVALUATION DESIGN IMPACT EVALUATION – QUASI-EXPERIMENTAL DESIGN To address the above evaluation questions, Haiti ESS will follow a pre-post IE design with statistical matching. The evaluation seeks to collect and analyze baseline, midline, and endline data on girl and boy students in Grades 1 and 2 from participating schools and compare their respective grade-level performance to a control or comparison group. Since a randomized control trial, the gold standard for definitively assigning observed outcomes to a program, was not feasible (given that the ERP schools had already been assigned by the implementing partners), a quasi-experimental design (QED) was required to best evaluate the ERP impact. Coarsened 6 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV Exact Matching (CEM) was recommended and ultimately chosen as the best QED for the Haitian context. CEM is used to identify a control group that is almost identical to the treatment groups. The evaluation design included two treatment groups (as opposed to designing two separate evaluations) to allow for isolation of the effects of enhanced treatment (“enhanced” group) over the “core” group. This approach increases the efficiency of the evaluation by providing supplemental information about the different characteristics of the two interventions. With two treatment arms, however, it becomes difficult to identify a common control group that is almost identical in all aspects to both treatment arms, particularly when the selection criteria for different treatme