USAID/HAITI LECTURE POUR TOUS : PROMOTION DE L'ÉDUCATION INCLUSIVE (RAPID) RAPPORT D'ÉVALUATION DE L'ACTIVITÉ

USAID/HAITI READING FOR ALL: PROMOTING INCLUSIVE EDUCATION (RAPID) ACTIVITY EVALUATION REPORT JANUARY 2019 DISCLAIMER- The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/HAITI Reading for All: Promoting Inclusive Education (RAPID) Activity EVALUATION REPORT January 2019 Evaluation Mechanism Number: AID-521-C-17-00002 Haiti Evaluation and Survey Services for USAID/Haiti ABSTRACT Social Impact’s Haiti Evaluation and Survey Services conducted an independent final evaluation of the United States Agency for International Development (USAID)/Haiti’s Reading for All, Promoting Inclusive Education (RAPID) activity implemented by La Société Haïtienne d'Aide aux Aveugles (SHAA). The RAPID activity goal was to increase the number of visually impaired and blind students integrated into mainstream classes. This evaluation’s purpose is to provide guidance for future USAID/Haiti programs by identifying lessons learned through an examination of the extent to which RAPID achieved intended results. RAPID’s intended outcomes included increased access to materials and resources for visually impaired and blind youth, an increased awareness of inclusive education among partnering institutions (i.e., National Confederation of Haitian Educators) and state agencies (i.e., Commission d’Adaptation Scolaire et d’Appui Social), and an increase in visually impaired and blind students accessing mainstream classrooms. SHAA integrated 197 students into mainstream classes. Challenges with recruitment impacted their successful integration of 300 targeted students. SHAA was able to produce over 120 materials in the form of Braille documents and audiobooks, however materials produced appeared to be insufficient to meet all SHAA students’ needs. SHAA also successfully developed two specialized libraries in Port-au-Prince and Cap-Haitien. The organization raised awareness of disability inclusion through partnerships and media outlets. However, SHAA teacher training only impacted student learning to some extent. TABLE OF CONTENTS ABSTRACT II TABLE OF CONTENTS III LIST OF TABLES AND FIGURES IV ACRONYMS V EXECUTIVE SUMMARY I Evaluation Purpose and Evaluation Questions i Evaluation Questions i Project Background i Evaluation Design, Methods, and Limitations i Findings and conclusions ii recommendations iv I. INTRODUCTION 1 Project Background 1 Theory of Change 1 Activities/Components 2 Evaluation Purpose and Audience 2 Evaluation Questions 2 II. EVALUATION DESIGN 2 Sampling 2 Data Collection Methods 3 Analysis 4 Limitations and Mitigation Strategies 5 III. FINDINGS AND CONCLUSIONS 5 IV. RECOMMENDATIONS 10 Recommendations for SHAA 10 Recommendations for USAID 11 ANNEX A: EVALUATION STATEMENT OF WORK 12 ANNEX B: REFERENCES AND ENDNOTES 19 ANNEX C. EVALUATION TEAM MEMBERS 20 ANNEX D: EVALUATION DESIGN MATRIX 21 ANNEX E: LIST OF PERSONS INTERVIEWED 24 ANNEX F: INFORMED CONSENT SCRIPTS & DATA COLLECTION PROTOCOLS 25 ANNEX G: DISCLOSURE OF CONFLICT OF INTEREST 39 LIST OF TABLES AND FIGURES Table 1: FGDs by Stakeholder Group and Gender 4 Table 2: Key Informant Interviews by Stakeholder Group and Gender 4 ACRONYMS CASAS CNEH Commission d’Adaptation Scolaire et d’Appui Social National Confederation of Haitian Educators ESS Evaluation and Survey Services ET Evaluation Team EQ Evaluation Question FGD Focus Group Discussion KII Key Informant Interview MENFP MOU Ministry of National Education and Vocational Training Memorandum of Understanding RAPID Reading for All: Promoting Inclusive Education SHAA La Société Haïtienne d'Aide aux Aveugles SI Social Impact TL Team Leader USAID United States Agency for International Development i | RAPID PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) activity design and conduct an independent final evaluation of the Reading for All: Promoting Inclusive Education (RAPID) activity, implemented by La Société Haïtienne d'Aide aux Aveugles (SHAA). The purpose of this report is to present the evaluation team’s (ET) findings, conclusions, and recommendations. EVALUATION PURPOSE AND EVALUATION QUESTIONS The evaluation purpose is to identify possible follow-up actions as well as guidance for similar USAID programs in Haiti and elsewhere based on lessons learned. This is based on an examination of the extent to which RAPID achieved its intended outcomes and results according to the activity’s design and expected implementation plans. This report documents RAPID activity performance, outcomes, lessons learned, and any issues observed during the evaluation process. The primary audience for this final evaluation is USAID/Haiti, which can use the evaluation findings and recommendations to inform its future education programming. Additional stakeholders may include SHAA staff, education leaders, targeted schools, and the Government of Haiti Ministry of Education. These secondary audiences may also consider evaluation findings and recommendations in their current programs and possible future collaboration with USAID/Haiti. EVALUATION QUESTIONS This evaluation answers the evaluation questions (EQs) below. When answering these EQs, the ET considered stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent has the project been effective in achieving results regarding blind and visually impaired children (i.e., integration into mainstreaming schools, access to documents and specialized libraries)? 2. To what extent has awareness been raised to promote inclusive education at the institutional (Ministry of National Education and Vocational Training (MENFP), Commission d’Adaptation Scolaire et d’Appui Social (CASAS), others), community and parental levels? 3. To what extent and in what ways has RAPID interventions (i.e., book-based clubs, training of trainers) affected the quality of teaching and learning for the blind and visually impaired? 4. Overall, what intended and unintended contributions, results and/or outcomes has the RAPID project achieved relative to improving inclusive access to educational opportunities for disadvantaged children and youth who are blind or visually impaired? PROJECT BACKGROUND Committed to the inclusion of visually impaired and blind children and youth in educational settings, SHAA provides educational support such as Braille instruction, materials transcription into Braille, and additional services aimed to increase access to schools. SHAA was awarded a USAID grant to implement the RAPID activity, interventions for which included the development of adapted educational materials in Braille, customized educational services (e.g., tutoring), the development of specialized libraries, book clubs, and computer adaptive training. EVALUATION DESIGN, METHODS, AND LIMITATIONS To examine RAPID’s impact on program participants, the ET employed qualitative data collection methods including document review, site observation, key informant interviews (KIIs), and focus group discussions (FGDs). The ET reviewed activity documents and developed an inception report, then conducted FGDs and KIIs during the fieldwork. ii | RAPID PERFORMANCE EVALUATION USAID.GOV The ET conducted eight FGDs, including two with parents – one each in the West and the Northeast. The team also held three FGDs with students – two FGDs in the West and one in the Northeast. Finally, they conducted three FGDs with teachers – one in the West and two in the North. Additionally, the ET collected data on the perspectives and experiences of four key informants involved in the RAPID activity who were identified using a purposive sampling technique. These included a school director, SHAA staff member, National Confederation of Haitian Educators (CNEH) representative, and USAID Education Office Representative. Bias: The ET was aware of potential recall, selection, response, and gender bias risks in data collection and analysis, and developed mitigation strategies to overcome these. Specifically, the ET included up to 12 participants in each FGD to triangulate responses, increase evaluation findings’ validity, and allow for more diverse input. The ET mitigated response bias by interviewing leaders and middle managers separately and holding teacher and parent FGDs separately. Child Protection Rights: RAPID included disabled children as beneficiaries, thus a major risk was possibly violating child protection rights. To minimize this, the ET hired qualified personnel experienced in working with children and provided ethical data collection training to all ET members who had direct contact with children during the evaluation implementation training. Limited availability of key informants: Transportation to the venue for FGD participants served as a barrier to participation. To mitigate this risk, more participants were invited to participate in FGDs than proposed in the evaluation design. Although the ET was able to conduct FGDs with all stakeholder groups proposed in the evaluation design, the number of FGD participants (e.g., eight to ten participants per FGD) was not met. FINDINGS AND CONCLUSIONS EQ1: TO WHAT EXTENT HAS THE PROJECT BEEN EFFECTIVE IN ACHIEVING RESULTS REGARDING BLIND AND VISUALLY IMPAIRED CHILDREN (INTEGRATION INTO MAINSTREAMING SCHOOLS, ACCESS TO DOCUMENTS AND SPECIALIZED LIBRARIES, ETC.)? During Year 1 of implementation, SHAA was able to meet its target of 100 visually impaired and blind youth integrated into mainstream classes. In Years 2 and 3, they did not meet their targets. SHAA field agents, who worked in the community, recruited most students. During Year 2 (2017), however, SHAA faced challenges recruiting potential students to be integrated. SHAA reported that the primary reasons parents offered included lack of funds for transportation to attend school and travel to SHAA’s office to receive Braille training, which is required for inclusion into mainstream classes. Given these challenges, SHAA did not meet its target integration of 100 and only integrated 30 new students into mainstream classes in Year 2. Most students who were identified and received services through RAPID were integrated into mainstream classes and remained in school for the entire year. Once students were identified and then integrated, they received support services from SHAA, which began with training to read in Braille. Students also received tutoring provided by itinerant teachers (teachers employed by SHAA). SHAA itinerant teachers, SHAA staff, and students all reported the provision of these support services. Although most students integrated through the RAPID activity in Year 1 remained in school for the entire year, parents and SHAA staff reported that parents’ inability to pay tuition served as the main factor preventing some students from remaining in school for a full year. Students and itinerant teachers reported that an increase in materials and resources was required to meet the needs of all students SHAA served. During Year 2, SHAA reported that 30 students received kits that included a digital tape recorder, a tablet, a punch game, braille paper, and an abacus counter. Students also confirmed that they received iii | RAPID PERFORMANCE EVALUATION USAID.GOV these materials, however, they also reported that the amount of materials was insufficient for the population served and inadequate to meet their needs. SHAA developed two Specialized Libraries. All students reported using the specialized libraries. The RAPID Annual report indicated that in 2016, 3,575 visually impaired, blind, and sighted students accessed the libraries, followed by 3,519 in 2017. Students reported that the specialized libraries were beneficial because they were able to participate in computer training and book clubs held at the libraries. Students recommended that SHAA increase the number of materials at the specialized libraries. EQ2: TO WHAT EXTENT HAS AWARENESS BEEN RAISED TO PROMOTE INCLUSIVE EDUCATION AT THE INSTITUTIONAL (MENFP, CASAS, OTHERS), COMMUNITY AND PARENTAL LEVELS? SHAA was able to raise awareness through partnerships and media outlets. SHAA reported establishing partnerships with other organizations with similar interests. They worked closely with CNEH to increase awareness of the needs of visually impaired and blind students and promoted inclusion primarily by providing teacher training to CNEH teachers. Specifically, CNEH reported that through SHAA facilitated instruction, teachers received training on how to be better prepared to receive and accommodate visually impaired and blind students into mainstream classes. One of SHAA’s efforts that helped them gain greater visibility is radio air time where they shared their services and promoted inclusive education for the visually impaired and blind. SHAA awareness-raising programs reached eight of the ten departments in Haiti through their interactions with other organizations and mass media. Unfortunately, SHAA’s many attempts to reach out to the MENFP were met with minimal success. Although some strides have been made (they have begun building a relationship with the CASAS, a state agency responsible for special education), they have not had any major meetings with Ministry leadership. Parents of the students appeared to have a lack of understanding of what is meant by inclusive education. While teachers and collaborating partners were aware and had a general understanding of inclusive education, students’ parents appeared to lack understanding of what is meant by inclusive education. EQ3: TO WHAT EXTENT AND IN WHAT WAYS HAS RAPID INTERVENTIONS (BOOK-BASED CLUBS, TRAINING OF TRAINERS, ETC.) AFFECTED THE QUALITY OF TEACHING AND LEARNING FOR THE BLIND AND VISUALLY IMPAIRED? SHAA provided training to CNEH Teachers. According to CNEH teachers and SHAA staff, the training content focused on accommodations that can be implemented in the classroom environment to support inclusion into mainstream classrooms. Although CNEH teachers participated in the training, they did not have the opportunity to put learning into practice because they did not have any children in their classes with vision problems or who were blind. The itinerant teachers reported, and CNEH teachers concurred, that they learned strategies to be used in the classroom. Itinerant Teachers report that the SHAA training they attended allowed them to better train teachers with SHAA students integrated in their classrooms. In contrast, SHAA’s itinerant teachers reported applying what they learned during training to impact the teaching of teachers of visually impaired and blind students. They also reported impacting student learning. Itinerant teachers reported providing classroom training for teachers who had visually impaired and blind students integrated in their classrooms (i.e., SHAA students). Students also reported receiving training from SHAA. iv | RAPID PERFORMANCE EVALUATION USAID.GOV Some students reported receiving training to read Braille. Students who received this training reported their satisfaction with it, reporting that the itinerant teachers’ tutoring helped them to do well in school. EQ. 4 OVERALL, WHAT INTENDED AND UNINTENDED CONTRIBUTIONS, RESULTS AND/OR OUTCOMES HAS THE RAPID PROJECT ACHIEVED RELATIVE TO IMPROVING INCLUSIVE ACCESS TO EDUCATIONAL OPPORTUNITIES FOR DISADVANTAGED CHILDREN AND YOUTH WHO ARE BLIND OR VISUALLY IMPAIRED? Some intended outcomes of the RAPID activity included increased access to materials and resources for visually impaired and blind youth, an increased awareness of inclusive education among partnering institutions (i.e., CNEH) and state agencies (e.g., CASAS), and an increase in visually impaired and blind students accessing mainstream classrooms. Given the lack of prior existing data on this population it is difficult to determine the extent of RAPID’s impact on the desired outcomes. The team did not identify any unintended outcomes. However, the ET identified several unanticipated barriers to implementation. Transportation and Finances served as a barrier to regular attendance. Students faced several unanticipated barriers preventing them from attending school and RAPID intervention activities regularly. Specifically, transportation was one critical barrier. Some students reported that they live far from school or the SHAA office and there were instances when they did not have transportation or money for public transportation. Another unanticipated barrier to inclusion of visually impaired and blind students into mainstream classrooms was parents’ inability to pay school tuition. RECOMMENDATIONS RECOMMENDATIONS FOR SHAA 1. Develop a recruitment strategy for identifying youth with visual impairment and blindness SHAA should consider developing a recruitment strategy for identifying youth with visual impairment and blindness. Given misunderstandings about blind children’s ability to learn, a strategy that includes a variety of means (e.g., outreach and recruitment at community churches, presentations, training, and recruitment at local schools before the start of the school year) to reach youth, would be ideal. A recruitment strategy would have increased SHAA’s likelihood of meeting their target of 100 new students integrated each year. 2. Develop a sustainable strategy for materials development SHAA should consider developing a sustainable strategy for developing materials. SHAA may want to consider reaching out to organizations that specialize in developing materials for the blind, such as The Braille Institute. They should consider obtaining additional equipment so that they can produce more Braille and audio books. 3. Identify information dissemination methods regarding available resources and materials that can be incorporated into participant program registration SHAA should identify methods to disseminate information to parents regarding available resources and materials available through SHAA (e.g., provide parents with a list of services and resources during registration, hold parent engagement nights to share information on services with parents, provide schools with a list of services to share with parents). 4. Include an educational component for students’ parents, including helping parents understand inclusive education and the role they can play in promoting and supporting it 5. Map where participants are located to identify a limited number of schools at which to centralize support services v | RAPID PERFORMANCE EVALUATION USAID.GOV SHAA should consider mapping where all their participants are located to identify a limited number of schools at which to centralize their support services. This may allow for more schools that are equipped with specialized services, and an environment that can better accommodate blind students. In areas where there is only one student in a geographic area, other solutions, such as transportation provision may be needed. 6. Establish minimum quality standards a given school must meet to be prepared to receive visually impaired or blind students SHAA should consider establishing minimum standards a school must meet to be deemed appropriate or equipped to accommodate blind students. Schools that meet the minimum standards would be selected to receive SHAA support services. SHAA should begin with the implementation of their services in two to three schools so that they can improve the implementation of their support services. While an assessment of minimum quality standards for physical access to schools was outside of the scope of this evaluation, future scale up will need to consider it. RECOMMENDATIONS FOR USAID 1. Consider piloting a model that includes one or two schools for inclusive integration of visually impaired students before scaling up to multiple schools Future programming should consider piloting a model that includes one or two schools for inclusive integration of visually impaired students before scaling up to multiple schools. This would allow USAID to establish best practices for the integration of this population into mainstream classes in Haiti before scaling up. 2. Provide support to SHAA for mapping locations and concentrations of visually impaired and blind students USAID could provide support to SHAA for mapping locations and concentrations of visually impaired and blind students. One option is to consider conducting a census focused on identifying households with blind children. 3. Facilitate important connections with leadership in the Ministry of Education as well as other high-level bilateral and multilateral agencies implementing programs in the inclusive education sphere USAID could facilitate important connections with leadership in the Ministry of Education as well as other high-level bilateral and multilateral agencies implementing programs in the inclusive education sphere. These connections would affect synergies in the areas of marketing, stakeholder coordination, policy change, and awareness raising. 1 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV I. INTRODUCTION According to the World Health Organization, there are an estimated 1.4 million visually impaired children worldwide. Approximately 500,000 children become visually impaired each year, with children from developing countries affected the most. In developing countries, the rate of blindness in children is estimated to be as high as 1.5 per 1000 population as compared to 0.3 per 1000 in industrialized countries. The literature indicates that children in poor countries are four to five times more likely to be visually impaired than those born in high-income countries. 1 In Haiti the prevalence of visually impaired or blind children is unknown due to a lack of population-level data. Estimates report that 80,000 people are visually impaired or blind in Haiti. 2 Among this population, there are estimated to be 20,000 who are school-age children of which only 1,160, or 5.8 percent attend school. Visually impaired and blind children are often deprived of basic education due to socioeconomic barriers, inaccessible schools, and discrimination and stigmatization. Overcrowding, limited qualified teachers, inadequate curricula, and social discrimination serve as major barriers to educational access and inclusion for blind children. 3 Thus, children who are visually impaired or blind may fall victim to negative outcomes including reduced quality of life (e.g., vision functioning, emotional well-being and social relationship concerns) 4 poor school attendance, and poor school performance. PROJECT BACKGROUND Established in 1952, La Société Haïtienne d'Aide aux Aveugles (SHAA) provides a range of services to individuals who are visually impaired and blind in Haiti. Committed to the inclusion of visually impaired and blind children and youth into educational settings, SHAA provides educational support such as Braille instruction, tutoring, and additional services aimed to increase access to schools. USAID awarded SHAA a 775, 503 USD grant ( Award No. AID-521-F-15-00013 under APS Local Solutions Project 521-14-000027) to implement the Reading for All: Promoting Inclusive Education (RAPID) activity. Interventions implemented under the RAPID activity went from August 28, 2015 to August 27, 2018 and were intended to expand SHAA’s population reach by increasing the number of visually impaired and blind students they integrated into mainstream classes and expanding their service provision to include the development of materials translated in Braille, teacher training, and the development of specialized libraries equipped with materials in Braille and audiobooks. SHAA was also able to hire more itinerant teachers who were tasked with providing tutoring services and teacher training. THEORY OF CHANGE The theory of change for the RAPID activity postulates that an increase in the availability of adapted educational materials (e.g., transcription into Braille), improved customized educational services, and specialized training and capacity building will lead to integration of visually impaired and blind children and youth into mainstream classrooms, which will ultimately lead to improved inclusive access to quality educational opportunities for visually impaired and blind children. The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) activity, design and conduct an independent final evaluation of the RAPID activity, implemented by SHAA (see Annex A for the evaluation Statement of Work). Before conducting the fieldwork for this evaluation, the evaluation team (ET) presented its initial findings based on a document review (see Annex B for list of documents consulted and Annex C for the ET members’ profiles) in an evaluation design, a detailed report approved by USAID/Haiti (see summary of evaluation 1 Gilbert C , Foster A. Childhood blindness in the context of Vision 2020—the right to sight. Bull World Health Organ 2001; 79:227–32; Hoyt CS , Good WV. The many challenges of childhood blindness. Br J Ophthalmol2001; 85:1145–6; Cunningham ET Jr , Lietman TM, Whitcher JP. Blindness: a global priority for the twenty-first century. Bull World Health Organ 2001; 79:180; Foster A . Cataract and “Vision 2020—the right to sight” initiative. Br J Ophthalmol 2001; 85:635–7. 2 USAID, Social Impact Request for Proposals “Reading for All: Promoting Inclusive Education (RAPID) Activity 3 SHAA (2015). La Société Haïtienne d'Aide aux Aveugles. Local Solutions Initiative: Program Description. 4 Lamoureux, E., and K. Pesudovs. 2011. Vision-specific quality-of-life research: A need to improve the quality. American Journal of Ophthalmology 151(2):195–197, e192. 2 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV design in the Evaluation Design Matrix found in Annex D ). The purpose of this report is to present the ET’s main findings, conclusions, and recommendations. ACTIVITIES/COMPONENTS The overall goal of the RAPID activity was to increase access to quality education to children who are blind and visually impaired. Activities were concentrated in two main geographical areas: Port-au-Prince and the Cap-Haitian/Ouanaminthe area. The intermediate results of the activity were to: 1. Increase integration of children who are blind and visually impaired into mainstream schools; 2. Increase availability of accessible documents for people who are blind and visually impaired; and 3. Increase access to SHAA’s specialized library services for people who are blind, visually impaired, and sighted. EVALUATION PURPOSE AND AUDIENCE The evaluation purpose was to identify lessons learned and provide guidance on possible follow-up actions for similar USAID programs in Haiti and elsewhere. This was based on an examination of the extent to which RAPID achieved its intended outcomes, and a determination of whether the expected results occurred according to the activity’s design and implementation. Based on this, the evaluation provides recommendations. This evaluation report documents the RAPID activity’s performance, outcomes, lessons learned, and any issues observed during the evaluation process as relevant to the evaluation questions. The primary audience for this final evaluation is USAID/Haiti, which can use the evaluation findings and recommendations to inform its future programming in the area of inclusive education for the special needs population. Additional stakeholders may include SHAA staff, education leaders, targeted schools, and the Government of Haiti Ministry of Education. These secondary audiences (e.g., SHAA staff, education leaders, target schools) may also consider evaluation findings and recommendations in their current programs and possible future collaboration with USAID/Haiti. EVALUATION QUESTIONS This evaluation answers the evaluation questions (EQs) below. When answering these EQs, the ET considered stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent has the project been effective in achieving results regarding blind and visually impaired children (i.e., integration into mainstreaming schools, access to documents and specialized libraries)? 2. To what extent has awareness been raised to promote inclusive education at the institutional (Ministry of National Education and Vocational Training (MENFP), Commission d’Adaptation Scolaire et d’Appui Social (CASAS), others), community and parental levels? 3. To what extent and in what ways has RAPID interventions (i.e., book-based clubs, training of trainers) affected the quality of teaching and learning for the blind and visually impaired? 4. Overall, what intended and unintended contributions, results and/or outcomes has the RAPID project achieved relative to improving inclusive access to educational opportunities for disadvantaged children and youth who are blind or visually impaired? II. EVALUATION DESIGN SAMPLING The evaluation utilized a purposive sampling technique to identify parents, students and teachers to participate in the Focus Group Discussions (FGDs). SHAA provided the ET with a list of all parents, students, and teachers involved in the interventions. Once the ET identified FGD participants, SHAA reached out to participants and extended an invitation for participation. The ET conducted a similar process for selecting teachers. Those chosen to participate in the FGDs met the following criteria: 3 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Parents had the following characteristics: a) a child that was exposed to the RAPID activity and, b) demonstrated availability and willingness to participate in the focus groups. Teachers and Students had the following characteristics: a) exposure to the RAPID activity and, b) demonstrated availability and willingness to participate in the focus groups. All participants were informed of their rights, and that participation was completely voluntary with no associated risks or benefits. The ET also conducted key informant interviews (KIIs) with a school director, SHAA staff member, National Confederation of Haitian Educators (CNEH) representative, and USAID RAPID Agreement Officer Representative. These informants provided information to address the information gaps identified and not addressed through the document review. Overall, information gathered from FGDs and KIIs provided different key stakeholders’ perspectives on the RAPID activity processes, accomplishments, and challenges. Information gathered from the USAID RAPID Agreement Officer Representative provided USAID’s perspectives on the RAPID activity’s accomplishments and challenges (see Annex E for a complete List of Persons Interviewed). DATA COLLECTION METHODS The ET employed qualitative data collection methods to examine the RAPID activity’s impact on program participants. Qualitative methods consisted of document review, KIIs, observation, and FGDs. The ET reviewed activity documents and developed an Inception Report in September 2018. The team conducted FGDs and KIIs during the evaluation fieldwork period, from December 4 – 21, 2018. The data collection protocols that guided fieldwork can be found in Annex F. The purpose of the protocols was to: (1) ensure all key issues are covered during data collection, (2) elicit rich, sometimes unanticipated, information from respondents, (3) help organize information in a form that could be usefully and efficiently analyzed, and (4) obtain the information necessary to address all the EQs. The protocols consisted of questions that addressed and were derived from the EQs, as well as from the ET’s document review, and meetings with SHAA staff. DOCUMENT REVIEW The Team Leader (TL) conducted a document review to gather preliminary information regarding the RAPID activity’s implementation, which, in turn, informed the evaluation design, data collection protocol development, and identified the data the ET gathered during evaluation activities to answer the EQs. Documents reviewed included: • SHAA proposal document; • RAPID Year 2 annual report; • RAPID Year 3 monitoring and evaluation plans; and • RAPID fact sheet. FOCUS GROUP DISCUSSIONS The ET conducted eight FGDs. Although the ET had four FGDs scheduled with parents, only two FGDs were held with parents – one in the West and one in the Northeast. Four FGDs were scheduled with students, of which ultimately, three were held – two in the West and one in the Northeast. While two FGDs were scheduled with children under the age of 18 years, only one FGD was held for this age group. 4 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Two FGDs were scheduled with teachers. 5 The ET conducted three FGDs with teachers – one in the West and two in the North. The main barrier to parent, student, and teacher participation in the FGDs was transportation to the venue. SHAA staff explained that participants who agreed to attend the did not reside close to SHAA offices in both the West and Northeast, and thus the lack of transportation made it difficult for them to attend. The ET highlighted to participants that interview content would be kept confidential and that they would omit personally identifiable information from the report (see informed consent protocol in Annex F). A total of 42 individuals were included in the FGDs, 52 percent of which were female. Table 1: FGDs by Stakeholder Group and Gender Type of Informant Number of Informants 2 Parent FGDs 8 (8 F) 3 Student FGDs 23 (13M, 10F) 3 Teacher FGDs 11 (7M, 4F) Total 42 (20M, 22F) KEY INFORMANT INTERVIEWS Additionally, the ET collected data on the perspectives and experiences of four key informants involved in the RAPID activity using a purposive sampling technique (a school director, SHAA staff member, CNEH representative, and USAID/Haiti Education Team representative). Although two school director interviews were proposed, only one director was available during the data collection week. Similarly, only one of each staff member at SHAA, CNEH, and USAID were available during the data collection. Table 2: Key Informant Interviews by Stakeholder Group and Gender Type of Informant Number of Informants 4 Key Informant Interviews 4 (2M, 2 F) Total 4 ANALYSIS All KII and F GD data were numbered and organized according to gathering sites and collection date. Participant names were not included on any recordings or notes. Instead, a reference number was used to preserve confidentiality. All notes from FGDs and KIIs were collected in electronic format. The ET took detailed notes of KIIs and FGDs, and cleaned and shared notes on an ongoing basis during fieldwork. At the end of each data collection day, the ET met to discuss data collected and any issues, all of which were addressed immediately. The TL supervised and managed systematic analysis of data captured through the FGDs and KIIs. The ET employed content analysis (e.g., coding and theme identification) using NVivo qualitative analysis software. Data were compared across stakeholder groups to assess either convergence or divergence in perspectives. The TL captured preliminary findings, conclusions, and recommendations in an Excel-based matrix that categorized analysis by EQ. The matrix: a) ensured the ET prepared a systematic and thorough response to each EQ, b) verified that preliminary analysis accounted for gender and social dimensions, c) identified any gaps where additional clarification or analysis would be necessary, and d) served as the basis for developing the evaluation report. 5 Three types of teachers were interviewed as part of the evaluation: (1) itinerant teachers (employed and trained by SHAA, who then teach CNEH teachers, regular teachers, and students themselves), (2) CNEH teachers (who are taught by itinerant teachers), and other teachers (who are also taught by itinerant teachers and who have some SHAA students in their classes). 6 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 5 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV LIMITATIONS AND MITIGATION STRATEGIES BIASES The ET was aware of several bias risks in data collection and analysis and developed mitigation strategies to overcome these. 1. Recall bias occurs when respondents cannot recall previous events, recall them inaccurately, or omit details. Thus, the accuracy of the responses may be at risk. Given that the primary data collection method for the RAPID evaluations was qualitative interviews and respondents were self- reporting, recall bias is unavoidable and will operate at some level. 2. Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. Response bias often operates within the Haitian context. Given the poverty conditions participants experience, they may tend to provide favorable responses in hopes of receiving future support. 3. Selection bias occurs when individuals or groups are selected in a way that is not random, but rather purposefully. This ensures that the results of the data are not necessarily representative of the full group. Given that the ET spoke with a relatively small group of targeted individuals during this evaluation, the results presented in this report may not be fully representative of the larger visually impaired and blind population. To overcome these biases, the ET included up to 12 participants in each FGD, as possible within the fieldwork period, to triangulate responses, increase the validity of evaluation findings, and allow for more diverse input. The ET mitigated response bias by interviewing leaders and middle managers separately and holding teacher and parent FGDs separately. In addition, the ET ensured that participants clearly understood the evaluation’s objectives and potential benefits. RAPID includes disabled children as beneficiaries, thus a major risk is the violation of child protection rights. To minimize this risk, the ET hired qualified personnel experienced working with children and provided training for all ET members who had direct contact with children during the evaluation implementation. Moreover, the ET implemented measures to prevent unauthorized access to or disclosure of participants’ personally identifiable information. Finally, to ensure participants’ protection, the evaluation design was submitted and approved by an institutional review board ethics review both within SI and in Haiti. LIMITED AVAILABILITY OF KEY INFORMANTS Parents, teachers, and students who received SHAA services were dispersed throughout Port-au-Prince and Cap-Haitian. Funds available were not within the evaluation budget to cover transportation costs for participants to attend FGDs. SHAA reported that prohibitive transportation costs to FGDs was the primary reason why teachers were unable to attend. Although, the ET conducted FGDs with all stakeholder groups proposed in the evaluation design, the number of FGD participants (e.g., eight to ten per FGD) was not met. Given the low teacher participation in the FGDs, findings should be interpreted with caution. Teachers with visually impaired and blind students integrated into the classroom did not participate in the FGDs. The perspective of these teachers would have provided data about the extent to which visually impaired and blind students were successfully integrated. In the absence of this data, the ET’s ability to comment on the extent to which the RAPID activity met its overall goal is limited. III. FINDINGS AND CONCLUSIONS This section synthesizes the evaluation’s findings and conclusions, which are presented by EQ. EQ1: TO WHAT EXTENT HAS THE PROJECT BEEN EFFECTIVE IN ACHIEVING RESULTS REGARDING BLIND AND VISUALLY IMPAIRED CHILDREN (INTEGRATION INTO MAINSTREAMING SCHOOLS, ACCESS TO DOCUMENTS AND SPECIALIZED LIBRARIES, ETC.)? 6 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV FINDINGS During Year 1 of implementation, SHAA was able to meet its target of 100 visually impaired and blind youth integrated into mainstream classes. Year 2 and 3, they did not meet their targets. SHAA field agents worked in Port-au-Prince and Cap Haitian with the purpose of meeting families with a visually impaired or blind children. During these interactions, SHAA field workers engaged in dialogue with the families and shared the services offered through SHAA. It is through this field work that most students were recruited. As the field agents identified a visually impaired or blind child or youth, they recorded the contact information and eventually established a database. Students added to the database received some services before participation in the RAPID activity. These services were expanded once the RAPID activity was implemented. The database was the main source for recruitment during Year 1 (2016) of implementation in which SHAA was able to meet its target of 100 students identified. Other recruitment strategies included radio advertisement, and solicitation on SHAA services at clinics. The radio advertisements proved to be the least effective recruitment strategy as no students were reported to be recruited via this method. During Year 2 (2017), however, SHAA faced challenges recruiting potential students for integration. SHAA reported that the primary reasons parents offered included lack of funds for transportation to attend school and travel to SHAA’s office to receive Braille training, which is required for inclusion into mainstream classes. SHAA reported that parents’ lack of funds to pay school tuition and false perceptions that their child cannot learn also served as barriers. Given these challenges, SHAA did not meet its target integration of 100 and only integrated 30 new students into mainstream classes in Year 2. The majority of students who were identified and received services through the RAPID activity were integrated into mainstream classes and remained in school for the entire year. Once students were identified and then integrated, they received support services from SHAA, which began with training to read in Braille. Students also received tutoring provided by itinerant teachers (i.e., teachers employed by SHAA) as part of support services. Through the RAPID activity, SHAA was able to expand its reach by increasing the number of students they integrated into mainstream classes and their service provision to include tutoring. While the majority of students who were identified and received services (e.g., training in Braille, tutoring) through the RAPID activity remained in school for the entire year, parents and SHAA staff reported parents’ inability to pay tuition served as the main factor preventing some students from remaining in school for a full year. Students and itinerant teachers reported that an increase in materials and resources is needed to meet the needs of all students served by SHAA. SHAA was also tasked with providing materials and resources to visually impaired and blind students. During Year 2 SHAA reported that 30 students received kits that included a digital tape recorder, a tablet, a punch game, braille paper, and an abacus counter. Students also confirmed that they received materials, however, they reported that the amount of materials was insufficient for the population served and inadequate to meet their needs. For example, university students reported that the recorder SHAA provided did not have the capacity to record all their courses and they resorted to recording courses on their phones. University students also reported that the number of materials produced by SHAA were not enough to meet the needs of all students served. They stated that it would be beneficial to have materials that can be taken home so that they can practice Braille and typing at home. This finding was echoed by itinerant teachers (i.e., teachers SHAA employed) who reported that they needed more means to produce additional Braille books for children. Itinerant teachers also explained that as they travel throughout the community to provide tutoring support and training to teachers, they do not have enough materials. The availability of materials is underscored by SHAA’s inability to implement a sustainable method to produce materials in Braille and audiobooks that can sufficiently meet participants’ needs. Through funds received from the RAPID activity, SHAA was able to purchase equipment to develop Braille materials. Despite the new equipment, itinerant teachers still report that SHAA needed equipment to produce more materials. While the new equipment allowed them to produce materials that did not exist before the 7 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV activity, the equipment does not appear to be enough to develop the material at a rate or level to meet the needs of all SHAA students. Most students reported receiving and using materials SHAA provided (e.g., abacus, tablets, recorders), however, there were a few students who reported that they never received any materials or were not fully aware of the resources SHAA offered. Students who reported not receiving any materials, reported accessing other services through SHAA such as adaptive computer service and school tuition assistance. SHAA developed Specialized Libraries. All students reported using the specialized libraries. The RAPID Annual report SHAA submitted indicates that in 2016, 3,575 visually impaired, blind, and sighted students accessed the libraries, followed by 3,519 in 2017. Students reported that they benefitted from the use of audiobooks and books written in Braille available in the libraries. Students also reported that the specialized libraries were beneficial because they were able to participate in computer training and the book clubs that are held at the libraries. SHAA staff reported that for the book clubs held at the Cap Haitian library, the team in Port-au-Prince must periodically send books. Overall, participants were aware that the specialized libraries exist. Students and itinerant teachers reported that while the specialized libraries are beneficial and accessible, materials in Braille or audio are limited. Students recommended that SHAA increase the number of materials at the specialized libraries. CONCLUSIONS SHAA was successful, to some extent, in achieving RAPID activity results. While the prevalence of visually impaired or blind children is unknown in Haiti due to a lack of population-level data, estimates report that 80,000 children are visually impaired or blind in Haiti. Thus, we know that there are children in Haiti that SHAA can recruit for services and integrate into mainstream classes. However, SHAA was faced with recruitment challenges. It appears that a lack of a recruitment strategy before the start of the RAPID activity further exacerbated successful recruitment of the targeted population. SHAA met its objective to produce materials and resources and these materials are reaching intended students. However, the amount of materials did not appear to be sufficient to accommodate all students served. The specialized libraries were achieved but are still in need of development. Additional materials need to be developed and made available at the libraries. EQ2: TO WHAT EXTENT HAS AWARENESS BEEN RAISED TO PROMOTE INCLUSIVE EDUCATION AT THE INSTITUTIONAL (MENFP, CASAS, OTHERS), COMMUNITY AND PARENTAL LEVELS? FINDINGS SHAA was able to raise awareness through partnerships and media outlets. SHAA was tasked with raising awareness to promote inclusive education at the institutional, community, and parental levels. To accomplish this task, SHAA reported establishing partnerships with other organizations with similar interests. For example, through their connection with the National Associative Network for Integration of Disabled People, SHAA was able to connect with other organizations with similar interests, doing similar work. They have been working closely with CNEH to increase awareness of the needs of visually impaired and blind students and promote inclusion primarily by providing teacher training to CNEH teachers. Specifically, CNEH reported that through instruction SHAA facilitated, teachers received training on how to be better prepared to receive students into mainstream classes. CNEH also reported that SHAA met their expectations by providing teachers with adequate training and tools that will help them to better accommodate visually impaired and blind students in their classroom. SHAA and CNEH reported that their collaboration resulted in a Memorandum of Understanding (MOU), outlining CNEH’s commitment to inclusive education and teacher training. Bringing great visibili