(2018) Haiti Early Reading Program (ERP) Baseline Evaluation

DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL EVALUATION REPORT Haiti Early Reading Program (ERP) Baseline Evaluation This publication was produced for review by the United States Agency for International Development by Social Impact, Inc. COVER PHOTO: GRADE 2 STUDENTS, ARTIBONITE, HAITI ANDREW CARMONA, 2017 USAID.GOV EARLY READING PROGRAM BASELINE EVALUATI ON FINAL REPORT | i Haiti Early Reading Program (ERP) Baseline Evaluation USAID/ Haiti AID-521-C-17-00002 March 16, 2018 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Andrew Carmona, Basab Dasgupta, Amanda Stek, Joanie Cohen-Mitchell, and Euphonise Loiseau on behalf of Social Impact, Inc. Team member qualifications can be found in Annex G of this report. ii | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV CONTENTS ACRONYMS IV ACKNOWLEDGEMENTS V ABSTRACT VI EXECUTIVE SUMMARY VIII INTRODUCTION 1 METHODOLOGY 5 FINDINGS 16 CONCLUSIONS 54 ANNEX A: FULL LISTING OF REFERENCES IN REPORT 58 ANNEX B: DATA COLLECTION TOOLS 60 ANNEX C: EVALUATION METHODOLOGY 116 ANNEX D: ADDITIONAL FINDINGS 123 ANNEX F: EVALUATION STATEMENT OF WORK 127 ANNEX G: QUALIFICATIONS OF TEAM MEMBERS 136 ANNEX H: EVALUATION DISSEMINATION PLAN T EMPLATE 139 ANNEX I: DATA QUALITY ASSURANCE MEASURES 141 ANNEX J: INTER-RATER RELIABILITY SCORES OF ENUMERATORS 142 ANNEX K: CONSIDERATIONS FOR FUTURE DATA COLLECTION 144 ANNEX L: DISCLOSURE OF ANY CONFLICTS OF INTEREST 145 TABLE OF TABLES Table 1: Partners and their SOWs for the ERP Impact Evaluation........................................................................ 3 Table 2: List of ERP implementers and evaluation sample schools ........................................................................ 6 Table 3: Planned qualitative sample size at baseline .................................................................................................. 9 Table 4. EGRA subtasks ................................................................................................................................................. 14 Table 5: Study limitations ............................................................................................................................................... 15 Table 6: School sample realized by school type, rural/urban, and department ................................................ 17 Table 7: Final EGRA sample realized, by grade and gender .................................................................................. 18 Table 8: Final sample realized for qualitative data collection ................................................................................ 18 Table 9: Covariate balance statistics after CEM matching ..................................................................................... 19 Table 10: EGRA scores (for all), with p-values ......................................................................................................... 21 Table 11: Proportion of zeros in subtasks across treatment groups .................................................................. 23 Table 12: Updated MDES, by EGRA sub- tasks and Grade .................................................................................... 25 Table 13: Basic school- level information across schools ....................................................................................... 26 USAID.GOV EARLY READING PROGRAM BASELINE EVALUATI ON FINAL REPORT | iii Table 14: School level resources and activities, by treatment and comparison ............................................... 27 Table 15: EGRA subtask scores from other Haiti EGRA studies ........................................................................ 31 Table 16: Distribution of ORF scores across grades and by CWPM categories ............................................. 32 Table 17: Distribution of CLPM scores across grades ........................................................................................... 33 Table 18: Distribution of CLSPM scores across grades ......................................................................................... 33 Table 19: Regression models showing the effects of various factors on ORF .................................................. 40 TABLE OF FIGURES Figure 1: Sampled schools of the ERP Baseline Evaluation .................................................................................... vii Figure 2: Sample size calculations and power analysis for the ERP Impact Evaluation ...................................... 9 Figure 3: Decile distribution of timed learning test scores (percent of students in each category)............ 22 Figure 4: Quartile distribution of non-zero scores in timed outcomes (for all students) ............................. 24 Figure 5: School Director experience ........................................................................................................................ 27 Figure 6: Teacher characteristics, per school ........................................................................................................... 28 Figure 7: Descriptive statistics for classroom-level variables ................................................................................ 29 Figure 8: Language spoken by students at home, by school type and location................................................. 29 Figure 9: Extent of punitive measures (as reported by students), by school type (percentage) .................. 30 Figure 10: Percent of students who eat in the morning before school .............................................................. 30 Figure 11: Average scores (in words) in timed subtasks, by grade and gender ................................................ 34 Figure 12: Proportion of zero scores in timed subtasks, by grade and gender ................................................ 34 Figure 13: Timed learning outcomes non- zero scores, by gender ...................................................................... 35 Figure 14: Average scores in non- timed subtasks, by grade and gender............................................................ 36 Figure 15: Proportion of zeros in non -timed subtasks, by grade and gender ................................................... 36 Figure 16: Non -timed learning outcomes, non-zero scores, by gender ............................................................. 37 Figure 17: Timed learning outcomes, by grade and implementer ........................................................................ 38 Figure 18: Non -timed learning outcomes, by grade and implementer ............................................................... 38 Figure 19: Timed learning outcomes, by grade and type of school ..................................................................... 39 Figu re 20: Non -timed learning outcomes, by grade and type of school ............................................................ 39 Figure 21: Decomposition of drivers that explain ORF performance, overall and by gender ...................... 44 Figure 22: Classroom-level variables on classroom management and phonics approaches .......................... 49 Figure 23: Classroom-level variables on classroom materials and other topics .............................................. 50 iv | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV ACRONYMS AIR American Institutes for Research ATL Assistant Team Leader CEM Coarsened Exact Matching CLPM Correct letter identification per minute CLSPM Correct letter sound per minute CWPM Correct word per minute DQA Data Quality Assurance EGRA Early Grade Reading Assessment EMIS Education Management Information System EPT Éducation Pour Tous EQ Evaluation Question EQUI ® Evaluation Quality, Utilization, and Impact ® ERP Early Reading Program ESS Evaluation and Survey Services ET Evaluation Team FGD Focus Group Discussion FONHEP Haitian Foundation for Private Teaching ICC Intra- class Correlation IDB Inter- American Development Bank IE Impact Evaluation IFOS Institut de Formation du Sud IP Implementing Partner IRR Inter- rater reliability JICA Japan International Cooperation Agency KII Key Informant Interview LAC/READS Latin American and Caribbean READS Program MDES Minimum Detectable Effect Size MENFP The Ministry of Education and Professional Training NGO Non-governmental Organization ODK Open Data Kit ORF Oral Reading Fluency PTA Parent-Teacher Association QA Quality Assurance QED Quasi-experimental Design RCT Randomized Control Trial SD Standard Deviation SI Social Impact, Inc. SOW Scope of Work SSME Snapshot of School Management Effectiveness ToTAL Tout Timoun Ap Li UND University of Notre Dame UNICEF The United Nations Children's Fund USAID United States Agency for International Development USAID.GOV EARLY READING PROGRAM BASELINE EVALUATI ON FINAL REPORT | v ACKNOWLEDGEMENTS The authors would like to wholeheartedly thank several key individuals who were critical to the successful completion of this baseline report. We thank the Education Team at USAID /Haiti, namely Christine Veverka and Ezechiel Macon, for being flexible, available, and responsive to our various questions and inquiries. We extend very deep thanks to Samuel Emieux Jean, Ulrick Jean Claude, and Charles Tardieu of IFOS, as well as the 122 supervisors and enumerators who carried out data collection for this study. This report would not have been possible without the hard work of these motivated young Haitian men and women who spent five weeks collecting data in the middle of the rainy season across Haiti. The authors would also like to express their appreciation to the ERP partners at Save the Children (Ossemer and James), AIR (Rachelle and Mohammed), UND (TJ, Kate, and Paul), and UNICEF (Mirko, Beatrice, and Vegains) for their assistance in facilitating our work. We also want to acknowledge the public- sector stakeholders at the MENFP who authorized this study and provided valuable input, in particular Harold Narcisse and the Director General, Dr. Meniol Jeune. Finally, we offer our full gratitude to the students, teachers, school directors, parents , and departmental-level MENFP officials who graciously gave their time to take learning tests, participate in observations, and give candid feedback to our team. It is for them and all the students in Haiti that we continue to work hard to ensure that every child everywhere can read. vi | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV ABSTRACT In Fall 2017, Social Impact, through its Haiti-based Evaluation Survey Services (ESS) project, conducted a baseline evaluation of the Early Reading Program (ERP), a USAID-funded intervention focused on improving reading outcomes for Grade 1 and 2 students, teaching quality, and capacity of the Ministry of Education in Haiti to implement and scale up reading programs. The ESS team designed and conducted the baseline for a pre-post impact evaluation (IE) design with statistical matching, administering Early Grade Reading Assessment (EGRA) learning tests, a student questionnaire, a school director survey, a classroom observation tool, focus group discussions with students and parents, and key informant interviews with teachers and MENFP officials across seven departments (Artibonite, Grand’Anse, Nord, Nord-Est, Ouest, Sud, and Sud-Est). The team concluded that students are generally unable to read in Creole. Further, many important factors at the household- level (eating before school, parents’ involvement in a student’s education, access to books), school-level (director’s experience, number of formally trained 1 teachers, existence of extracurricular activities, helping students to pay their tuition) and classroom-level (teachers’ use of certain phonics approaches) were associated with higher learning scores. Teachers generally managed their classrooms well ( assessed by observing students’ behavior in the classroom, utilization of classroom time between start and end times of class, use of varying materials and strategies to teach), though corporal punishment was observed in 11- 13 percent of classrooms. Teachers consistently lacked sufficient books and other educational materials and did not generally use best practice phonics approaches for reading. Qualitative data suggested that the MENFP lacks the capacity and resources to support schools and teachers with funding , materials, or training. 1 A formally trained teacher is one that has a professional degree from a teaching college. USAID.GOV EARLY READING PROGRAM BASELINE EVALUATI ON FINAL REPORT | vii Figure 1: Sampled schools of the ERP Baseline Evaluation viii | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY CONTEXT According to 2015 estimates, 2 61 percent of Haitian adults (64 percent of males and 57 percent of females) are literate. Education in the country is plagued by constraints that range from low funding to inadequate teacher training (CIA World Fact Book, 2018). The Ministry of Education and Professional Training (MENFP) is aware of students’ poor performance in reading and has worked with the United States Agency for International Development (USAID) in recent years to implement reading interventions to address this situation. BACKGROUND In 2017, USAID recognized a need to scrutinize their ongoing investments being made to improve reading outcomes in Haiti. USAID determined that the best way to evaluate these ongoing activities was through an independent impact assessment. The impact evaluation (IE) , titled the Early Reading Project, will assess a two-year time period of intervention from multiple different ongoing activities and implementing partners already working to support early grade reading instruction in Grade 1 and Grade 2 of primary school in approximately 658 public, private, community, and semi-private schools in the departments of Artibonite, Ouest, Grand’Anse, Sud, Sud-Est, Nord, and Nord-Est. Broadly speaking, and working across these early grade reading activities in seven different departments, USAID has recognized three common objectives: (1) Improve MENFP’s capacity to implement and eventually scale up early grade reading reform; (2) Improve teacher ability to teach early grade reading; and (3) Improve students’ early grade reading outcomes. Partners participating in the Early Reading Project IE include Save the Children (STC), American Institutes for Research (AIR), the University of Notre Dame (UND), and the UN Children’s Fund (UNICEF). All of these partners use, or will use, the early grade reading and writing curriculum materials co-developed by the Ministry of Education and USAID . The French version of these materials is called “Je parle bien francais.” The Creole version of these materials is called “M’ap li net ale”. Materials should be used concurrently in all schools. The curriculum employs a scripted model of instruction for teachers and is phonics-based. In first grade, the French portion is completely oral, focusing on building children’s vocabulary and knowledge of the French language, while the Creole portion focuses on explicit reading and writing instruction for children, so they are first learning to read in a language they speak and understand. Across partners, the intervention is employed in two different models: “core” and “enhanced.” Both models include the implementation of the aforementioned co -designed (USAID and MENFP) national curriculum and teacher training and coaching in early grade reading and writing, but the enhanced model differs in other key extracurricular interventions. The “enhanced” group also contains some UNICEF- supported schools that employ a “pre-Kindergarten” reading model. Baseline data for the Early Reading Project IE was collected in November and December of 2017. Mid line data will be collected during the 2018- 2019 school year. Endline data will be collected in the spring of 2020. 2 CIA World Factbook, updated 1 January 2018. Obtained on 2 March 2018. USAID.GOV EARLY READING PROGRAM BASELINE EVALUATI ON FINAL REPORT | ix PURPOSE AND EVALUATION QUESTIONS USAID requested that Social Impact (SI)’s Haiti Evaluation and Survey Services (ESS) to design and implement an IE of their ongoing early grade reading and writing investments. The evaluation is intended to measure whether early grade reading and writing interventions have resulted in students in the target schools demonstrating greater skills in reading than their peers of the same grades in non -targeted, similar schools. The evaluation findings will inform USAID/Haiti’s future programming in early grade reading in Haitian s chools. The results of the evaluation are also intended to inform Government of Haiti policy- level decisions around implementation of reading curriculum and teacher training on a national scale. The lessons learned and recommendations documented in the final endline evaluation report may also inform future programming for other donors’ activities related to early grade reading in Haiti. The evaluation’s five high-level evaluation questions (EQ) look specifically at the outcomes of (1) reading fluency among Grade 1 and 2 students, (2) teacher quality, and (3) MENFP capacity to implement and scale up reading programs. The questions are as follows: • Q1. What is the impact, if any, of the “core” ERP intervention on students’ reading outcomes in the first and second grades? Some sub-questions the evaluation team will consider in addressing this question are: • Q2. What is the relative impact, if any, on student learning, of the “enhanced” model carried out by UND and UNICEF? Do students exposed to this model acquire stronger reading skills than their peers in control and other ERP schools? What aspects of the model impact student learning the most? • Q3. What is the impact, if any, on student learning, of the “pre-reading instruction” model piloted by UNICEF in the pre-K and K levels of 120 of the ERP program schools? Do students exposed to the “pre-reading instruction” model acquire stronger reading skills than their peers in control and other ERP schools? • Q4. What is the impact of the ERP intervention, if any, on teachers’ classroom practices in early grade reading instruction? • Q5. What is the impact, if any, of the ERP program on the Ministry ’s capacity to implement and eventually scale up early grade reading reform? Are the human, material, and financial management capacities at the MENFP sufficiently improved because of the ERP intervention that the Ministry could successfully expand the ERP program nationwide? DESIGN Haiti ESS followed a pre-post IE design with statistical matching. The evaluation team (ET) collected and analyzed data on equal numbers of girl and boy students in Grades 1 and 2 from participating schools and compared their performance to a control group. To successfully measure the three outcomes of interest, the ET administered Early Grade Reading Assessment (EGRA) learning tests supplemented by student questionnaires, school director surveys, classroom observations, and qualitative data collected through focus group discussions (FGDs) and key informant interviews (KIIs). At baseline, the ET administered the EGRA in Haitian Creole with one short EGRA subtask on oral French comprehension. A sample of 130 schools was randomly sampled proportionally from “core” and “enhanced” implementing partner schools to form the core and enhanced treatment samples , with 65 schools in each sample. At baseline, the ET also identified 65 suitable comparison schools from non -treated schools. After an EGRA validation workshop and school listing activity conducted in October 2017, the ET collected baseline data in November -December 2017. A midline will be conducted in April- May 2019 and an endline in April- May 2020. x | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUS IONS GENERAL The absence of an EMIS system in Haiti hampers educational research and program implementation. The availability of school- level data is essential for conducting rigorous research and for better understanding gaps in educational access throughout the country. Without regular, authoritative census data on schools nationwide, implementers, evaluators, and public-sector actors at the MENFP will be limited in their ability to conduct high quality programming and respond to the needs of communities. READING FLUENCY Disproportionate numbers of zero scores have led to a situation where EGRA scores at baseline are partially imbalanced across treatment status. We found that students in Grade 2 in the core treatment group outperform their counterparts from the enhanced and control groups significantly in almost all subtasks (timed and untimed). Variation in EGRA scores at baseline was also noted across genders, school types, and implementers. Among implementers, better reading performance was found among Save the Children schools, which are all located in the Port- au-Prince urban area. Girls score significantly better than boys across all tests (and in both grades), and public and rural schools had lower oral reading fluency scores relative to private , community, religious, and urban schools. Surveyed students have Creole language skills but do not yet have reading skills. Students in both Grades 1 and 2 scored highest in Creole letter dictation and listening comprehension, but lowest in Creole word dictation and reading comprehension. This result was to be expected in Grade 1 and somewhat in Grade 2 as well, as students begin their formal education , but students in higher grades in primary school (Grade 3 and up) should begin to show higher reading scores. This is in line with other EGRA studies in Haiti that found students were not able to read at sufficient speed (with comprehension to understand text) until G rade 3 or later. 3 The home environment plays a critical role in a student’s reading scores. At the student-level, students who eat before school, have parents involved in their education, spend time reading, have access to a library, and have books at home have higher oral reading fluency scores than students who do not. Qualitative data showed that parents are involved and invested in their children’s education (for both girls and boys). However, they are mostly unaware of PTA activit ies even though s chool directors report that nearly three in four schools have active PTAs. Most p arents in FGDs noted that school fees are a major barrier to school attendance, al though they are willing to go to great lengths to pay for their children’s education. While at baseline girls reported taking on a disproportionate amount of housework outside of school, a shift in community attitudes seems to suggest more girls are being allowed to go to school than before. 3 EDDATA II: Education Data for Decision Making. The Early Grade Reading Assessment (EGRA) in Haiti. June 2014. Prepared by RTI International. Contract Number EHC-E-00-04-00004-00. www.eddataglobal.org. The three studies reviewed in the EDDATA paper showed the average third grade Haitian students were able to correctly read between 23-39 correct words per minute (cwpm) in Creole; 35-60 cwpm is considered the minimum rate necessary to comprehend a basic story. USAID.GOV EARLY READING PROGRAM BASELINE EVALUATI ON FINAL REPORT | xi School-level factors explain the largest amount of variation in oral reading fluency for girls and boys. Regression and decomposition analysis found that a host of school-level variables have significant positive associations with reading scores. School management assisting with students’ school fees, the experience level of school directors, and number of formally trained female teachers at a school are all associated with positive oral reading fluency scores. Notably, while a minority of schools report having extracurricular programs and services, having these programs was associated with higher oral reading fluency. Furthermore, students were associated with better reading fluency when the average number of hours taught (per day at school) wa s found to be higher or when the school had a library. As the student-to-teacher ratio increased ( more students per teacher), student reading scores decreased. Girls appeared to benefit most from more and better-trained female teachers at a school. In the classroom, use of certain phonics approaches works better than others, and seating arrangement matters. About half of teachers used a phonics approach while teaching reading, and those teachers who practiced blending sounds and decoding syllables saw better results in reading scores than those who simply asked students to repeat words aloud. Treatment of girls and boys in the classroom was gender equitable; however, regression analysis showed that seating boys and girls together was associated with significant adverse effects on reading. TEACHING QUALITY Classroom management is generally good in terms of observed students’ behavior in the classroom, utilization of classroom time; however, corporal punishment was observed in some classrooms. Availability, access to, and use of educational materials in the classroom are highly variable. Grade 1 and 2 teachers generally manage student behavior well, start and end class on time, and use a variety of teaching techniques. However, students openly claim that teachers practice corporal punishment, and the practice of corporal punishment was observed in 11 -13 percent of classrooms. Teachers generally practice gender equitable treatment of students , with no major statistical differences between how girls and boys are treated in the classroom. Classroom observations noted an inconsistent use of books and a paucity of basic items such as pencils. Qualitative data from MENFP, teachers, and parents suggest a persistent scarcity of books and materials. Use of pedagogical reading approaches in class is not consistent. Only about half of teachers use a recognized phonics approach during teacher observation. As reported in teacher KIIs, t eachers with more years of experience were more likely to be able to specify good phonics practices. Reading aloud to students was the most practiced phonics approach, though it did not necessarily correlate to improved reading scores. Low levels of teacher formalization probably contribute to low reading scores. Only about a quarter of teachers in sampled schools were formalized (professionally trained) and on-the-job teacher training opportunities are nonexistent or occur very rarely. Most teachers reported teaching for years at a time without being offered training. Many teachers have no formal training. MENFP acknowledges that it lacks the resources to hold regular trainings. This deficit could pose a significant barrier to improvements in reading, given the regression analysis finding that formalized teachers with more experience are associated with higher oral reading fluency scores, especially for girls. MENFP CAPACITY TO IMPLEMENT AND SCALE UP READING PROGRAMS Departmental-level MENFP lacks the financial capacity to adequately support schools. Quantitative and qualitative data combine to paint a picture in which schools do not receive books and other educational materials, teachers are sometimes delayed payment, and MENFP-supported teacher xii | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV training is not offered on a regular basis. Families must take on the burden of supporting education for their children, and when they cannot, children go to school without necessary supplies or are held back. At baseline, MENFP offices relied heavily on NGOs to support and sustain schools with materials, funds, and teacher training. Departmental-level MENFP staff appear unaware of policies or plans through which deficiencies in the educational system can be improved. Officials were able to cite problems, such as non- compliance by teachers of the un official language policy, high levels of unqualified teachers, and the lack of teacher training, but they could not specify mechanisms or actions through which these issues could be resolved. MENFP officials carry out monitoring visits often but many do not appear to be deeply knowledgeable about phonics/reading pedagogy. Nearly three in four schools sampled had received an MENFP visit in the past year. However, many MENFP officials interviewed could not discuss reading pedagogy in depth . This lack of knowledge could have implications for policy development and quality oversight of reading programs in the future. GENDER In general, findings around gender equity in the household and classroom were positive when it came to education. For instance, parents informed that girls are sent to school at a higher rate than in the past and are treated equally well (or punished equally) compared to boys in the classroom. In fact, qualitative FGDs with parents revealed that parents are allowing girls to go to school more than they have in the past, signaling a shift in attitudes towards girls’ education. Some discrepancies remain, however. For instance, more girls than boys are required to do household chores. Perhaps the most important finding about gender was that girls’ reading scores are significantly higher across all measures when compared to those of boys . This finding is not entirely surprising. Studies have shown that girls do better than boys in most subjects, probably due to the fact that girls study more, have more positive perceptions of school and behave better, 4 according to findings supported by KIIs and FGDs held at baseline. An interesting finding of the regression analysis showed that when a school has more and better-trained female teachers, girls have better reading scores. The finding suggest s that female teachers may act as role models for girls, enabl ing and encouraging them to attend and participate in school. 4 Voyer, Daniel, and Susan D. Voyer. "Gender differences in scholastic achievement: A meta- analysis." Psychological bulletin 140.4 (2014): 1174. 1 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV INTRODUCTION CONTEXT According to 2015 estimates , 61 percent of adults in Haiti (64 percent of males and 57 percent of females) are literate. Nationwide literacy rates in Haiti have risen over the last two decades, but when compared globally, Haiti still ranks near the bottom at 141 out of 161 countries. Haiti’s literacy rate is well below the regional Latin America and Caribbean average of 90 percent. 5 Education in Haiti is plagued with constraints that range from low funding to inadequate teacher training. Between 2000 and 2010, the national budget line item for education declined from 17 percent to 10 percent, putting Haiti in the bottom quartile of countries ranked by public spending on education. 6 The 2010 Haiti earthquake exacerbated these problems, displacing students and damaging schools throughout the country. The Ministry of Education and Professional Training (MENFP) is aware of students’ poor outcomes in reading as well as the system-wide challenges that must be addressed to improve those outcomes. According to the national curriculum, Haitian primary students are expected to study reading in Creole for 90 hours per academic year (Grades 1 and 2) and French for 30 hours (Grade 2 only), or for 40 minutes, three out of five days of the week each school year. 7 However, reading instruction in the early grades in Haitian schools is often cursory, under -resourced, and ineffective. As a result, children score poorly on standardized reading assessments. Few properly designed instructional materials for teaching reading in the early years are available and teachers are rarely trained in the techniques proven to be most effective in teaching reading. Formative evaluation is rare and lacking in rigor; consequently, students who are struggling to learn to read are often not identified in time for appropriate intervention. System-wide, Haiti practices “automatic promotion” of students in the early primary years, and students who have not learned to read are passed up from grade to grade. Eventually, illiterate students either fail or drop out of school. 8 In recent years, MENFP has worked with USAID to implement interventions that will improve the likelihood that children will learn to read in Haitian schools. From 2011- 2016, with support from USAID, MENFP implemented two reading interventions: the ToTAL and the An n ALE programs . These interventions supported 297 schools in improving reading instruction. S upport included scripted reading lesson materials, which did not always arrive on time to be used appropriately in the classroom, and intervention-specific teacher training, which was not implemented sufficiently to improve teacher effectiveness in teaching reading. EGRA AND ITS USE IN HAITI The Early Grade Reading Assessment (EGRA) is an assessment that is administered orally to students in the early phases of primary school. EGRA can be used to assess Grade 1 -3 student reading skills and identifies gaps or shortcomings that need to be addressed for future student success in reading. Core subtasks that make up the EGRA include letter reading, phonemic awareness, word reading, and story 5 CIA World Factbook, updated 1 January 2018. Obtained on 2 March 2018. https://www.cia.gov/library/publications/the-world-factbook/geos/ha.html 6 Suzata, Eriko. 2011. Education in Haiti: An Overview of Trends, Issues, and Plans. World Innovative Summit for Education. 7 EDDATA II: Education Data for Decision Making. The Early Grade Reading Assessment (EGRA) in Haiti. June 2014. Prepared by RTI International. Contract Number EHC-E-00-04-00004-00. www.eddataglobal.org. 8 USAID. Haiti ESS ERP Impact Evaluation Scope of Work. October 2017. 2 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV reading with comprehension questions. Other EGRA subtasks may include pre-literacy skills, basic vocabulary, listening comprehension, and sentence or non-word writing. 9 Oral reading fluency, when coupled with comprehension, can be considered a measure for basic reading competencies. Overall, EGRA provides a methodology for quickly assessing a variety of early literacy skills. In general, EGRA identifies strengths and weaknesses in student reading which in turn indicate the quality of learning taking place in schools. 10 EGRA has been used in Haiti to evaluate both reading programs and the general state of literacy. As part of the USAID EQUIP2 project implemented by Save the Children in 2007, EGRA effectively revealed a high degree of variability in reading scores among private schools in Haiti; the EGRA found no evidence that private or public schools were doing a better job of teaching reading. A study conducted by RTI in 2008-2009 used the EGRA to assess reading in Grades 1 -3 in the Nippes and Artibonite departments. The assessment found that while there were no differences between male and female test -takers, there was a clear pattern of higher literacy among students who possessed books at home. Use of EGRA in the USAID PHARE project, implemented by AIR in 2009, demonstrated no significant differences in reading scores between public and private schools or between the sexes . The EGRA assessment did, however, reveal large differ ences in reading scores between geographic departments. In 2013, Concern International noted high levels of illiteracy in the higher grades and expanded its Saut d’Eau project to include an EGRA assessment for Grades 1- 6. This example proves that EGRA allows flexibility and adaptability and facilitates tailored approaches to specific local contexts. The 2011-2016 implementation of the USAID-funded ToTAL project is the most recent use of EGRA in Haiti. The ToTAL project linked EGRA results to results from a Snapshot of School Management Effectiveness (SSME) survey, which allowed for analyzing reading outcomes in a more holistic way. In linking SSME data to EGRA results, ToTAL staff sought to help local stakeholders by identifying helpful take-away points that could be applied at the classroom, school, and district levels. PROJECT BACKGROUND THE EARLY READING P ROJECT PROGRAM In 2017, USAID recognized a need to scrutinize their ongoing investments being made to improve reading outcomes in Haiti. USAID determined that the best way to assess these ongoing activities was through an independent impact evaluation (IE) . The IE, titled the Early Reading Project, will analyze a two-year time period of intervention from multiple different ongoing activities and implementing partners already working to support early grade reading instruction in Grade 1 and Grade 2 of primary school in approximately 658 public, private, community, and semi-private schools in the departments of Artibonite, Ouest, Grand’Anse, Sud, Sud-Est, Nord, and Nord-Est. Broadly speaking, and working across these early grade reading activities in seven different departments, USAID has recognized three common 9 Core subtasks or common instrument components of the present EGRA testing tool consist of listening comprehension, letter identification, familiar or non-word reading, oral reading fluency, and initial or final sound identification. Other components that do not fall under core subtasks are dictation and phoneme segmentation. Earlier EGRA tools included tests for pre-literacy skills such as picture -based subtask (as those in the Peabody Picture Vocabulary Test (PPVT)), or pictures to identify knowledge of common vocabulary such as words for body parts like hand, head, or toe . Print assessments (a student’s ability to indicate where on a page to begin reading or where to read next, for example) have also been eliminated from the current EGRA (Early Grade Reading Assessment Toolkit, Second Edition, 2016). 10 RTI International. 2015. Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: United States Agency for International Development. 3 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV objectives: (1) Improve MENFP’s capacity to implement and eventually scale up early grade reading reform; (2) Improve teacher ability to teach early grade reading; and (3) Improve students’ early grade reading outcomes. Partners participating in the Early Reading Project IE include Save the Children (STC), American Institutes for Research (AIR), the University of Notre Dame (UND), and the UN Children’s Fund (UNICEF). All of these partners use, or will use, the early grade reading and writing curriculum materials co-developed by the Ministry of Education and USAID . The French version of these materials is called “Je parle bien francais.” The Creole version of these materials is called “M’ap li net ale”. Materials should be used concurrently in all schools. The curriculum employs a scripted model of instruction for teachers and is phonics-based. In first grade, the French portion is completely oral, focusing on building children’s vocabulary and knowledge of the French language, while the Creole portion focuses on explicit reading and writing instruction for children so they are first learning to read in a language they speak and understand. Across partners, the intervention is employed in two different models: “core” and “enhanced.” Both models include the implementation of the aforementioned co-designed (USAID and MENFP) national curriculum and teacher training and coaching in early grade reading and writing, but the enhanced model differs in other key extracurricular interventions (Table 1). Baseline data for the Early Reading Project IE was collected in November and December of 2017. Midline data will be collected during the 2018- 2019 school year. Endline data will be collected in the spring of 2020. Table 1: Partners and their SOWs for the ERP Impact Evaluation Partner Subcontractor Model Year 1 Schools Year 2 Schools Department(s) American Institutes for Research (AIR) FONHEP Core 150 150 Nord, Nord-Est, Ouest, Artibonite Save the Children N/A Core 39 39 Ouest University of Notre Dame (UND) Catholic Relief Services (CRS); CEEC Core and Enhanced: School libraries and summer camps 160 160 Grand-Anse, Sud United Nations Children's Fund (UNICEF) CARE; PLAN Core and Enhanced: Reading clubs and PTAs; Pre-Scolaire program 106 106 Nord, Nord-Est, Sud- East DEVELOPMENT HYPOTHES IS The evaluation team has synthesized the development objectives and hypothesis as follows: DEVELOPMENT OBJECTIVES: 1. Increase literacy through the provision of core skills and organizational reforms to MENFP (for example, improved coordination, consistent and effective capacity building, improved leverage among education sector actors and networks, and creation of strong feedback loops between classroom and program designers) to assist learners in 670 schools in mastering basic literacy skills. 2. Increase proportion of students who can read and demonstrate reading comprehension by the end of the primary grades as defined by the Haitian curriculum, standards, or as agreed-upon by national experts. 3. Increase proportion of students who demonstrate improvement in reading and understanding of grade level texts by the end of two grades of primary schooling. 4 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV DEVELOPMENT HYPOTHESIS: Effective implementation of USAID-supported early grade reading and writing interventions can lead to improved reading outcomes through: 1. Provision of necessary support and technical guidance to the MENFP; 2. Implementation of the USAID/MENFP early grade reading and writing curriculum in primary schools (even if only in an experimental stage); 3. Establishment of strong feedback loops between all types of schools implementing the program and the design team in Port-au-Prince. EVALUATION PURPOSE AND AUDIENCE PURPOSE USAID requested that SI’s Haiti Evaluation Support Services (ESS) design and implement an independent IE of it’s early grade reading and writing interventions across partners. The purpose of the evaluation is to assess the overall effectiveness of the program on early grade reading among Grade 1 and 2 students. The IE is intended to measure whether the ERP interventions have resulted in students in the target schools demonstrating greater skills in reading than their peers of the same grades in non -targeted, homologous schools. The evaluation findings will inform USAID/Haiti’s future programming in early grade reading in Haitian schools. The results of the evaluation are also intended to inform Government of Haiti policy-level decisions around implementation of reading curriculum and teacher training on a national scale. The lessons learned and recommendations documented in the final endline evaluation report may also inform future programming for other donors’ activities as they relate to early grade reading in Haiti. PRIMARY STAKEHOLDERS The primary stakeholders for this evaluation include the MENFP and USAID/Haiti , the implementing partners evaluated under the Early Reading Pro ject IE; the Latin American and Caribbean (LAC)/Reads program; UNICEF; University of Notre Dame (UND); Save the Children; The United Nations Office for Project Services (UNOPS)); and other donors and entities active in Haiti’s education sector, including the World Bank, Japan International Cooperation Agency (JICA), Inter-American Development Bank (IDB), and World Vision, as well as a variety of non- governmental organizations (NGOs) and religious affiliations devoted to improving learning in Haitian primary schools. OUTCOMES OF INTEREST READING/LITERACY: In particular, the evaluation team (ET) looked at the following skills: letter- sound mastery (phonetic awareness); familiar word reading; reading connected text/oral reading fluency (ORF); reading comprehension; writing a sentence dictated to the student; and writing a sentence the student generates in response to an open verbal prompt. TEACHING QUALITY: The ET examined certain aspects of teaching quality, including classroom management, use of phonics approaches, use of educational and teaching materials, gender equity, and corporal punishment. Observing these characteristics also allowed the team to better understand how teachers appl ied their training in the classroom and whether that training had a beneficial effect on student performance. MENFP CAPACITY: The ET looked at the activities under the Early Reading Project targeting the MENFP and their contributions to increasing knowledge of and capacity to implement reading programs. The team also examined the existence of financial and human resources necessary for the implementation, continuation, and/or scaling up of early grade reading programs. 5 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV EVALUATION QUESTIONS USAID and the ESS team developed the following evaluation questions (EQ) to guide the IE. EQ1. What is the impact, if any, of the “ core” ERP intervention on students’ reading outcomes in the first and second grades? Some sub -questions the evaluation team will consider in addressing this question are: a. How do students exposed to any ERP programming perform when compared to similar students who are not exposed to ERP programming? b. What is the marginal effect, if any, of ERP implementer on reading outcomes? c. What is the marginal effect, if any, of type of school (semi-private, public, community, religious, etc.) on reading outcomes? d. To what extent do environmental or situational factors have an impact on students’ ability to learn to read in the ERP target schools? EQ2. What is the relative impact, if any, on student learning of the “enhanced” model carried out by UND and UNICEF? Do students exposed to this model acquire stronger reading skills than their peers in control and other ERP schools? What aspects of the model impact student learning the most? EQ3. What is the impact, if any, on student learning, of the “pre-reading instruction” model piloted by UNICEF in the pre-K and K levels of 120 of the ERP program schools? Do students exposed to the “pre-reading instruction” model acquire stronger reading skills than their peers in control and other ERP schools? EQ4. What is the impact of the ERP intervention, if any, on teachers’ classroom practices in early grade reading instruction? a. Do teachers exposed to the “enhanced” ERP intervention teach reading differently and/or better than their peers in the “ core” ERP schools? b. Do teachers exposed to the ERP intervention teach reading differently and/or better than their peers in the identical or close-to-identical schools without ERP interventions? c. How, if at all, do teachers exposed to the ERP intervention teach reading differently after having received the ERP intervention? EQ5. What is the impact, if any, of the ERP program on the Ministry ’s capacity to implement and eventually scale up early grade reading reform? Are the human, material, and financial management capacities at the MENFP sufficiently improved because of the ERP intervention that the Ministry could successfully expand the ERP program nationwide? METHODOLOGY EVALUATION DESIGN IMPACT EVALUATION – QUASI-EXPERIMENTAL DESIGN To address the above evaluation questions, Haiti ESS will follow a pre-post IE design with statistical matching. The evaluation seeks to collect and analyze baseline, midline, and endline data on girl and boy students in Grades 1 and 2 from participating schools and compare their respective grade-level performance to a control or comparison group. Since a randomized control trial, the gold standard for definitively assigning observed outcomes to a program, was not feasible (given that the ERP schools had already been assigned by the implementing partners), a quasi-experimental design (QED) was required to best evaluate the ERP impact. Coarsened 6 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV Exact Matching (CEM) was recommended and ultimately chosen as the best QED for the Haitian context. CEM is used to identify a control group that is almost identical to the treatment groups. The evaluation design included two treatment groups (as opposed to designing two separate evaluations) to allow for isolation of the effects of enhanced treatment (“enhanced” group) over the “core” group. This approach increases the efficiency of the evaluation by providing supplemental information about the different characteristics of the two interventions. With two treatment arms, however, it becomes difficult to identify a common control group that is almost identical in all aspects to both treatment arms, particularly when the selection criteria for different treatment groups are different. To overcome these difficulties, the ET used basic school-level characteristics (such as teacher qualifications and experience, access to infrastructure, school management indicators, and student- teacher ratio) that were easily available across the control and both treatment groups and matched two different treatment arms separately with control schools. The ET then combined (merged) these matched groups over common control schools so that each control school would correspond to both types of treatment schools. The team discarded the control schools that matched with one treatment group, but not the other. A full explanation of how CEM is used and its benefits and limitations can be found in Annex C. TREATMENT GROUPS The study includes samples from two treatment groups , which are identified based on th e intervention type: one “core ” group and one “e nhanced” group. The main differences between the two groups can be found in Table 1. The core treatment group consists of 189 schools benefitting from interventions by Save the Children (39 schools in Ouest department) and AIR (150 schools in Nord, Nord-Est, Ouest, and Artibonite departments). These schools receive the co-developed MENFP and USAID curriculum as well as teacher training in implementing the curriculum. The ESS team randomly sampled 65 schools from these 189 schools in the core treatment group to form the core treatment sample. Table 2: List of ERP implementers and evaluation sample schools Implementer Total number of schools with interventions across Years 1 and 2 Number of schools in evaluation sample: CORE Number of schools in evaluation sample: ENHANCED AIR 150 (33%) 49 (38%) -- Save the Children 39 (9%) 16 (12%) -- UND 160 (35%) -- 37 (28%) UNICEF 106 (23%) -- 28 (22%) Note: Figures in bracket indicate proportional distribution in the treatment groups and treatment sample . The sum of treatment schools does not include TBD schools during time of sampling. The enhanced treatment group consists of 269 schools benefitting from interventions implemented by UND (160 schools in Sud and Grand’Anse departments) and UNICEF (106 schools in Nord, Sud -Est, and Nord-Est departments). While all schools in the enhanced group also benefit from the core group’s 7 | EARLY READING PROGRAM BASELINE FINAL EVALUATION REPORT USAID.GOV curriculum and teacher training, several UND schools introduced additional activities, including school libraries and summer camps, and some UNICEF schools have reading clubs, PTA -focused activities, and Pre-K interventions. It is these extracurricular activities that differentiate the core from the enhanced group. A total of 65 schools were randomly selected to form the enhanced treatment sample for the evaluation. SELECTION CRITERIA F OR TREATMENT SCHOOLS: Prior to ERP intervention implementation, targeted schools were chosen by the four implementing partners based on a set of pre-determined criteria. The ET utilized the same criteria to the maximum extent possible to identify and randomly select the control group of schools. The implementing partners’ specific school selection criteria , which varied across partners, can be found in Annex C. COMPARISON/CONTROL GROUP To measure the impact of the interventions, the evaluation strategy requires the identification of a similar group of schools and students to enable comparison of the outcomes achieved by the treated group. 11 To identify a suitable comparison group from non-treated schools, the ET required access to robust school-level data. However, Haiti lacks an Education Management Information System (EMIS) from which school-level data can be easily obtained. To account for this, the ET used the same lists of schools that each implementing partner (IP) used to identify treatment schools. Lists from all four IPs were pooled to create a sampling frame that contained a total of 2,734 schools across seven departments. From th is pooled list of schools, the ET randomly selected a total of 355 schools (160 treatment and 195 control) to conduct a rapid survey of schools (known as a school listing exercise) to perform the matching. After the partial list was created, the ET conducted the school listing activity. (A detailed description of the listing activity is provided in the Data Collection Activities section.) Enumerators collected data from the 355 randomly selected treatment and control schools mentioned earlier. Data collection focused on questions related to school-selection criteria used by each implementer, as well as other questions on location, typical school demographics, basic enrollment, and existence of Grade 1 and Grade 2 students and teachers. Using CEM (explained above), d ata collected from non-treatment schools through the listing activity were then compared to the treatment schools and matched along a set of defined variables. At the end of the initial round of the CEM matching process, 65 control schools that matched the 130 treatment schools were randomly identified to form the common control group. The ET then used this total group of 195 schools to conduct the full-fledged data collection. QUALITATIVE SUPPLEMENTATION To supplement the quantitative data, SI collected qualitative data about the program ’s impacts. Whereas the quantitative data will elucidate the program’s impact, qualitative data will provide deeper insight into the mechanisms for how and why these impacts took place. At baseline, qualitative data was collected via two methods— focus group discussions (FGDs ) and key informant interviews (KIIs). FGDs and KIIs were carried out with six distinct populations: boy students, girl students, mothers of students, fathers of students, teachers at school, and MENFP departmental-level officials. Approximately six to 10 11 A comparison group is a group that is similar in the best possible way to the target group, except that it has not been exposed to the project intervention in any way . [... middle sections omitted for long document ...] USAID.GOV EARLY READING PROGRAM BASELINE EVALUATIO N | 141 ANNEX I: DATA QUALITY ASSURANCE MEASURES For every data collection activity in its portfolio, ESS and its evaluation team embed SI’s systematic quality assurance framework called Evaluation Quality, Utilization, and Impact (EQUI ®). This system is comprised of a comprehensive set of guidance documents, templates, examples, and check-points. The EQUI ® framework aggregates learning from nearly a decade of experience conducting more than 30 IEs and quantitative surveys worldwide and incorporates social science standards and best practices. The system helps evaluation teams efficiently manage survey activities and achieve ESS’s high technical quality standards. Our system includes more than ten comprehensive guidance documents for planning data collection and monitoring data quality, to ensure that these activities are carried out with the highest level of integrity. 35 To minimize data entry errors and obtain data quickly for verification, the ET utilized electronic data collection tools to collect most of the data. Specifically, IFOS enumerators were equipped with tablets pre-loaded with all quantitative survey instruments. The evaluation team programmed the tools into the tablets using Open Data Kit (ODK) software and tested each of the instruments repeatedly leading up to data collection. For the EGRA tool, the ET used the Tangerine software programmed on tablets. The ET vetted, tested, and re-tested the data collection instruments, and then trained, tested, and supervised enumerators. Additionally, the ET program med the tablets with internal quality checks - skip patterns and logic checks – and drop-down menus to ensure accuracy of certain key fields. The data uploaded to servers directly from each tablet daily and real time review. The ET used a standard protocol for data and analysis file organization to ensure consistency, efficiency, transparency, and reproducibility. It also required that data collection partners produce data quality assurance reports detailing any challenges faced during fieldwork. Specifically, IFOS supervisors identified low performing enumerators and replaced them before data collection began. Field supervisors and the ET also reviewed all data collected on tablets to ensure that enumerators were administering surveys appropriately. Lastly, IFOS and one local ET member monitored data collection and sent SI’s home office ET members weekly team progress reports on the data collection. 35All ESS’s dedicated IE staff are trained to use these guidance documents and templates, and learning is continually incorporated back into system improvements. We recognize that every large-scale survey presents unique challenges and we customize all templates to meet specific evaluation needs. Check- lists accompany each guidance document and are reviewed by at least two evaluation team members to ensure compliance with QA requirements. Moreover, by using the system’s guidance and templates to collaborate with local data collection partners, EQUI® helps SI fulfil our institutional commitment of building local capacity. USAID.GOV EARLY READING PROGRAM BASELINE EVALUATIO N | 142 ANNEX J: INTER-RATER RELIABILITY SCORES OF ENUMERATORS IRR scores of the enumerators who conducted data collection are displayed below. Enumerators had to score a minimum of 85 percent on the IRR test to be eligible for data collection; those highlighted in red were rejected. A total of three IRR tests were held at the enumerator training; these are the results from the final test. Last names of enumerators have been removed for confidentiality purposes. No Department First Name Group Sex Percentage 1 Grand'Anse Ernst EGRA M 92.2% 3 Grand'Anse Vénèse EGRA F 94.4% 4 Grand'Anse Dady EGRA M 96.7% 5 Grand'Anse Hans William SP M 96.7% 7 Grand'Anse Joane EGRA F 96.7% 8 Grand'Anse Marie Danielle EGRA F 95.6% 11 Grand'Anse Esaïe EGRA M 95.6% 12 Grand'Anse Jocelyn SP M 97.8% 13 Grand'Anse Jackson EGRA M 94.4% 14 Grand'Anse Darline EGRA F 97.8% 15 Grand'Anse Gersand Ronide EGRA F 93.3% 16 Grand'Anse Alex EGRA M 95.6% 17 Grand'Anse Jean Pricien EGRA M 98.9% 19 Grand'Anse Johnny Wickers EGRA M 95.6% 20 Grand'Anse Daniel SP M 96.7% 21 Grand'Anse Marie Erline EGRA F 93.3% 22 Grand'Anse Joe Rosano EGRA M 96.7% 23 Grand'Anse Pierre Marie SP M 94.4% 25 Grand'Anse Jean Richard EGRA M 93.3% 27 Grand'Anse Jean Bénissé D. SP M 96.7% 28 Grand'Anse Jean Rouslé EGRA M 94.4% 30 Nord/Nord-est Yolette EGRA F 97.8% 31 Nord/Nord-est Merline Claude SP F 96.7% 33 Nord/Nord-est Lisemène EGRA F 96.7% 35 Nord/Nord-est Reynald EGRA M 97.8% 38 Nord/Nord-est Sanjuannicomoises EGRA M 94.4% 36 Nord/Nord-est Sainçois EGRA M 95.6% 39 Nord/Nord-est Bendy SP M 92.2% 40 Nord/Nord-est Elise EGRA F 95.6% 41 Nord/Nord-est Monestime EGRA M 92.2% 43 Nord/Nord-est Cassandra EGRA F 92.2% 44 Nord/Nord-est Gladlyn EGRA F 91.1% 47 Nord/Nord-est Mertilus EGRA M 94.4% 49 Nord/Nord-est Vanessa SP F 96.7% 48 Nord/Nord-est Rose-Manie EGRA F 95.6% 50 Nord/Nord-est Claudine EGRA F 91.1% 51 Nord/Nord-est Gitane SP F 92.2% 53 Nord/Nord-est Evintz EGRA M 95.6% 54 Nord/Nord-est Jesula EGRA F 95.6% USAID.GOV EARLY READING PROGRAM BASELINE EVALUATIO N | 143 55 Nord/Nord-est Linsly EGRA M 96.7% Nord/Nord-est Benito SP M 57 Nord/Nord-est Rechovsky EGRA M 96.7% 58 Nord/Nord-est Edwige EGRA F 93.3% 60 Ouest/SE Rose Audine SP F 97.8% 61 Ouest/SE Marie-Micka EGRA F 96.7% 103 Ouest/SE Edzer EGRA M 90.0% 104 Ouest/SE Louis Salomon SP M 97.8% 63 Ouest/SE Stael EGRA F 93.3% 64 Ouest/SE Chilène Clergé EGRA F 83.3% 105 Ouest/SE Nathalie EGRA F 97.8% 106 Ouest/SE Sauvenel Michel SP M 96.7% 107 Ouest/SE Marcus EGRA M 93.3% 110 Ouest/SE Pierre Olsen EGRA M 93.3% 67 Ouest/SE Sandie EGRA F 97.8% 111 Ouest/SE David EGRA M 95.6% 68 Ouest/SE Menadette EGRA F 94.4% 112 Ouest/SE Dieuna EGRA F 93.3% 113 Ouest/SE Lioud EGRA M 95.6% 66 Ouest/SE Géhu Miliana EGRA F 95.6% 115 Ouest/SE Emilius Fils EGRA M 90.0% 117 Ouest/SE Dimy SP M 96.7% 118 Ouest/SE Sylvio EGRA M 85.6% 70 Ouest/SE Stevensia Sindy EGRA F 95.6% 71 Ouest/SE Avril EGRA F 96.7% 72 Ouest/SE Sony Annie SP F 95.6% 120 Ouest/SE Francklin EGRA M 91.1% 73 Sud Roseberta EGRA F 82.2% 79 Sud Medgine EGRA F 96.7% 75 Sud Mackenson EGRA M 84.4% 80 Sud Wolfie Stael EGRA F 94.4% 82 Sud Moline EGRA F 92.2% 96 Sud Micheline SP F 94.4% 85 Sud Danica EGRA F 78.9% 86 Sud Germine EGRA F 95.6% 89 Sud Delot EGRA M 96.7% 90 Sud Lidenie SP F 96.7% 92 Sud Ledens SP M 93.3% 94 Sud Pierre Max SP M 94.4% 93 Sud Marie Francoise EGRA F 92.2% 95 Sud Valentino Giovani EGRA M 92.2% 77 Sud Rooloph EGRA M 96.7% 78 Sud Ricker B. EGRA M 95.6% 98 Sud Sherlyne EGRA F 95.6% 99 Sud Imelda Tatiana EGRA F 84.4% 100 Sud Medeline EGRA F 96.7% 101 Sud Cherline SP F 93.3% 102 Sud Denise EGRA F 91.1% USAID.GOV EARLY READING PROGRAM BASELINE EVALUATIO N | 144 ANNEX K: CONSIDERATI ONS FOR FUTURE DATA COLLECTION The ET ran into several problems at baseline which hampered the team’s ability to collect high quality data. The points below act as lessons learned from baseline and will be employed at subsequent data collection periods (midline and endline) to mitigate against the risk of low data quality in certain areas. 1.Rearrange important questions in the qualitative FGD and KII guides to focus more specifically on reading barriers (students and parents). Ensure that these questions are up front and not at the end of the guides. 2.Qualitative questions should be piloted to ensure appropriateness for Grade 1 and 2 students. Questions may have been too advanced for these children. 3.Add a question about language of instruction to the C lassroom Observation Tool to capture the effects of language in the classroom. 4.Consider adding teacher quantitative questionnaires as a new tool in order to collect additional data on teachers being observed. 5.Add more questions in the Observation Tool on ERP -specific curriculum and teacher training. 6.Fix the ODK error in the Classroom Observation Tool which precluded the ability to note a teacher’s level of formal training. 7.At the conclusion of BL data collection, the ET saw high data saturation among qualitative data. By the halfway point (about 30 events out of 56), most themes began repeating. Consider reducing the number of events at midline and/or endline. 8.Only two of 14 MENFP officials interviewed for this evaluation were female. While it is unclear if this is representative of MENFP as a whole, it is important that departmental-level MENFP offices are gender balanced to the extent possible. The ET will work to the extent possible to ensure gender balance in future data collection rounds. USAID.GOV EARLY READING PROGRAM BASELINE EVALUATIO N | 145 ANNEX L: DISCLOSURE OF ANY CONFLICTS OF INTEREST NAME Title Organization Evaluation Position? Team Leader Team member Evaluation Award Number (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the 5. operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 6. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 7. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation.