(2019) USAID/Haiti Lecture pour Tous : Promotion de l'Éducation Inclusive (RAPID) Rapport d'Évaluation de l'Activité

USAID/HAITI READING FOR ALL: PROMOTING INCLUSI VE EDUCATION (RAPID ) ACTIVITY E VALUATION REPORT J ANUARY 2019 DISCLAIMER- T he authors’ views expressed in this publication do not necessarily ref lect the views of the United States Agency for International Develop ment or the United States Governmen t. USAID/HAITI Reading f or All: P romoting In clusive Education (RAPID) Activity EVALUATION REPORT January 2019 Evaluation Mechanism Number: AID- 521-C-17-00002 H aiti Evaluation a nd Survey Services f or USAID/Haiti ABSTRACT Social Impact’s Haiti Evaluation and Survey Services conducted an independent final evaluation of the United States Agency for International Development (USAID)/Haiti’s Reading for All, Promoting Inclusive Education (RAPID) a ctivity implemented by La Société Haïtienne d'Aide aux Aveugles (SHAA). The RAPID a ctivity goal was to increase the number of visually impaired and blind students integrated into mainstream classes. This evaluation’s purpose is to provide guidance for future USAID/Haiti programs by identifying lessons learned through an examination of the extent to which RAPID achieved intended results. RAPID’s intended outcomes included increased access to materials and resources for visually impaired and blind youth, an increased awareness of inclusive education among partnering institutions (i.e., National Confederation of Haitian Educators) and state agencies (i.e., Commission d’Adaptation Scolaire et d’Appui Social), and an increase in visually impaired and blind students accessing mainstream classrooms. SHAA integrated 197 students into mainstream classes. Challenges with recruitment impacted their successful integration of 300 targeted students. SHAA was able to produce over 120 materials in the form of Braille documents and audiobooks, however materials produced appeared to be insufficient to meet all SHAA students’ needs. SHAA also successfully developed two specialized libraries in Port-au-Prince and Cap-Haitien . The organization raised awareness of disability inclusion through partnerships and media outlets. However, SHAA teacher training only impacted student learning to some extent. TABLE OF CONTENTS ABSTRACT II TABLE OF CONTENTS III LIST OF TABLES AND FIGURES IV ACRONYMS V EXECUTIVE SUMMARY I Evaluation Purpose and Evaluation Questions i Evaluation Questions i Project Background i Evaluation Design, Methods, and Limitations i Findings and conclusions ii recommendations iv I . INTRODUCTION 1 Project Background 1 Theory of Change 1 Activities/Components 2 Evaluation Purpose and Audience 2 Evaluation Questions 2 II . EVALUATION DESIGN 2 Sampling 2 Data Collection Methods 3 Analysis 4 Limitations and Mitigation Strategies 5 III. FINDINGS AND CONCLUSIONS 5 IV. RECOMMENDATIONS 10 Recommendations for SHAA 10 Recommendations for USAID 11 ANNEX A: EVALUATION STATEMENT OF WORK 12 ANNEX B: REFERENCES AND ENDNOTES 19 ANNEX C. EVALUATION TEAM MEMBERS 20 ANNEX D: EVALUATION DESIGN MATRIX 21 ANNEX E: LIST OF PERSONS INTERVIEWED 24 ANNEX F: INFORMED CONSENT SCRIPTS & DATA COLLECTION PROTOCOLS 25 ANNEX G: DISCLOSURE OF CONFLICT OF INTEREST 39 LIST OF TABLES AND FIGURES Table 1: FGDs by Stakeholder Group and Gender 4 Table 2: Key Informant Interviews by Stakeholder Group and Gender 4 ACRONYMS CASAS CNEH Commission d’Adaptation Scolaire et d’Appui Social National Confederation of Haitian Educators ESS Evaluation and Survey Services ET Evaluation Team EQ Evaluation Question FGD Focus Group Discussion KII Key Informant Interview MENFP MOU Ministry of National Education and Vocational Training Memorandum of Understanding RAPID Reading for All: Promoting Inclusive Education SHAA La Société Haïtienne d'Aide aux Aveugles SI Social Impact TL Team Leader USAID United States Agency for International Development i | RAPID PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMAR Y The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) activity design and conduct an independent final evaluation of the Reading for All: Promoting Inclusive Education (RAPID) activity, implemented by La Société Haïtienne d'Aide aux Aveugles (SHAA). The purpose of this report is to present the evaluation team’s (ET) findings, conclusions, and recommendations. EVALUATION PURPOSE AND EVALUATION QUESTIONS The evaluation purpose is to identify possible follow-up actions as well as guidance for similar USAID programs in Haiti and elsewhere based on lessons learned. This is based on an examin ation of the extent to which RAPID achieved its intended outcomes and results according to the activity’s design and expected implementation plans. This report documents RAPID a ctivity performance, outcomes, lessons learned, and any issues observed during the evaluation process. The primary audience for this final evaluation is USAID /Haiti, which can use the ev aluation findings and recommendations to inform its future education programming. Additional stakeholders may include SHAA staff, education leaders, targeted schools, and the Government of Haiti Ministry of Education. These secondary audiences may also consider evaluation findings and recommendations in their current programs and possible future collaboration with USAID /Haiti. EVALUATION QUESTIONS This evaluation answers the evaluation questions (EQs) below. When answering these EQs, the ET considered stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent has the project been effective in achieving results regarding blind and visually impaired children (i.e., integration into mainstreaming schools, access to documents and specialized libraries)? 2. To what extent has awareness been raised to promote inclusive education at the institutional (Ministry of National Education and Vocational Training (MENFP), Commission d’Adaptation Scolaire et d’Appui Social (CASAS), others), community and parental levels? 3. To what extent and in what ways has RAPID interventions (i.e., book-based clubs, training of trainers) affected the quality of teaching and learning for the blind and visually impaired? 4. Overall, what intended and unintended contributions, results and/or outcomes has the RAPID project achieved relative to improving inclusive access to educational opportunities for disadvantaged children and youth who are blind or visually impaired? PROJECT BACKGROUND Committed to the inclusion of visually impaired and blind children and youth in educational settings, SHAA provides educational support such as Braille instruction, materials transcription into Braille, and additional services aimed to increase access to schools. SHAA was awarded a USAID grant to implement the RAPID a ctivity, interventions for which included the development of adapted educational materials in Braille, customized educational services (e.g., tutoring), the development of specialized libraries, book clubs, and computer adaptive training. EVALUATION DESIGN, METHODS, AND LIMITATIONS To examine RAPID’s impact on program participants, the ET employed qualitative data collection methods including document review, site observation, key informant interviews (KIIs ), and focus group discussions (FGDs). The ET reviewed activity documents and developed an inception report, then conducted FGDs and KIIs during the fieldwork. ii | RAPID PERFORMANCE EVALUATION USAID.GOV The ET conducted eight FGDs, including two with parents – one each in the West and the Northeast. The team also held three FGDs with students – two FGDs in the West and one in the Northeast. Finally, they conducted three FGDs with teachers – one in the West and two in the North. Additionally, the ET collected data on the perspectives and experiences of four key informants involved in the RAPID a ctivity who were identified using a purpos ive sampling technique. These included a school director, SHAA s taff member, National Confederation of Haitian Educators ( CNEH) representative, and USAID Education Office Representative. Bias: The ET was aware of potential recall, selection, response, and gender bias risks in data collection and analysis, and developed mitigation strategies to overcome these. Specifically, the ET included up to 12 participants in each FGD to triangulate responses, increase evaluation findings’ validity, and allow for more diverse input. The ET mitigated response bias by interviewing leaders and middle managers separately and holding teacher and parent FGDs separately. Child Protection Rights: RAPID included disabled children as beneficiaries, thus a major risk wa s possibly violating child protection rights. To minimize this, the ET hired qualified personnel experienced in working with children and provided ethical data collection training to all ET members who had direct contact with children during the evaluation implementation training. Limited availability of key informants: Transportation to the venue for FGD participants served as a barrier to participation. To mitigate this risk, more participants were invited to participate in FGDs than proposed in the evaluation design. Although the ET was able to conduct FGDs with all stakeholder groups proposed in the evaluation design, the number of FGD participants (e.g., eight to ten participants per FGD) was not met. FINDINGS AND CONCLUSIONS EQ1: TO WHAT EXTENT HAS THE PROJECT BEEN EFFECTIVE IN ACHIEVING RESULTS REGARDING BLIND AND VISUALLY IMPAIRED CHILDREN (INTEGRATION INTO MAINSTREAMING SCHOOLS, ACCESS TO DOCUMENTS AND SPECIALIZED LIBRARIES, ETC.)? During Year 1 of implementation, SHAA was able to meet its target of 100 visually impaired and blind youth integrated into mainstream classes . In Years 2 and 3, they did not meet their targets. SHAA field agents , who worked in the community, recruited most students. During Year 2 (2017), however, SHAA faced challenges recruiting potential students to be integrated. SHAA report ed that the primary reasons parents offered included lack of funds for transportation to attend school and travel to SHAA’s office to receive Braille training, which is required for inclusion into mainstream classes. Given these challenges, SHAA did not meet its target integration of 100 and only integrated 30 new students into mainstream classes in Year 2. Most students who were identified and received services through RAPID were integrated into mainstream classes and remained in school for the entire year. Once students were identified and then integrated, they received support services from SHAA, which began with training to read in Braille. Students also received tutoring provided by itinerant teachers (teachers employed by SHAA). SHAA itinerant teachers, SHAA staff, and students all reported the provision of these support services. Although most students integrated through the RAPID activity in Year 1 remained in school for the entire year, parents and SHAA staff reported that parents’ inability to pay tuition served as the main factor preventing some students from remaining in school for a full year. Students and itinerant teachers reported that an increase in materials and resources wa s required to meet the needs of all students SHAA served. During Year 2, SHAA reported that 30 students received kits that included a digital tape recorder, a tablet, a punch game, braille paper, and an abacus counter. Students also confirmed that they received iii | RAPID PERFORMANCE EVALUATION USAID.GOV these materials, however, they also reported that the amount of materials was insufficient for the population served and inadequate to meet their needs. SHAA developed two Specialized Libraries. All students reported using the specialized libraries. The RAPID Annual report indicated that in 2016, 3,575 visually impaired, blind, and sighted students accessed the libraries, followed by 3,519 in 2017. Students reported that the specialized libraries were beneficial because they were able to participate in computer training and book clubs held at the libraries. Students recommended that SHAA increase the number of materials at the specialized libraries. EQ2: TO WHAT EXTENT HAS AWARENESS BEEN RAISED TO PROMOTE INCLUSIVE EDUCATION AT THE INSTITUTIONAL (MENFP, CASAS, OTHERS), COMMUNITY AND PARENTAL LEVELS? SHAA was able to raise awareness through partnerships and media outlets . SHAA reported establishing partnerships with other organizations with similar interests. They worked closely with CNEH to increase awareness of the needs of visually impaired and blind students and promoted inclusion primarily by providing teacher training to CNEH teachers. Specifically, CNEH reported that through SHAA facilitated instruction, teachers received training on how to be better prepared to receive and accommodate visually impaired and blind students into mainstream classes. One of SHAA’s efforts that helped them gain greater visibility is radio air time where they shared their services and promoted inclusive education for the visually impaired and blind. SHAA awareness -raising programs reached eight of the ten departments in Haiti through their interactions with other organizations and mass media. Unfortunately, SHAA’s many attempts to reach out to the MENFP were met with minimal success. Although some strides have been made ( they have begun building a relationship with the CASAS, a state agency responsible for special education), they have not had any major meetings with Ministry leadership. Parents of the students appeared to have a lack of understanding of what is meant by inclusive education. While teachers and collaborating partners were aware and had a general understanding of inclusive education, students’ parents appeared to lack understanding of what is meant by inclusive education. EQ3: TO WHAT EXTENT AND IN WHAT WAYS HAS RAPID INTERVENTIONS (BOOK-BASED CLUBS, TRAINING OF TRAINERS, ETC.) AFFECTED THE QUALITY OF TEACHING AND LEARNING FOR THE BLIND AND VISUALLY IMPAIRED? SHAA provided training to CNEH Teachers. According to CNEH teachers and SHAA staff, the training content focused on accommodations that can be implemented in the classroom environment to support inclusion into mainstream classrooms. Although CNEH teachers participated in the training, they did not have the opportunity to put learning into practice because they did not have any children in their classes with vision problems or who were blind. The itinerant teachers reported, and CNEH teachers concurred, that they learned strategies to be used in the classroom. Itinerant Teachers report that the SHAA training they attended allowed them to better train teachers with SHAA students integrated in their classrooms. In contrast, SHAA’s itinerant teachers reported applying what they learned during training to impact the teaching of teachers of visually impaired and blind students. They also reported impacting student learning. Itinerant teachers reported providing classroom training for teachers who had visually impaired and blind students integrated in their classrooms (i.e., SHAA students). Students also reported receiving training from SHAA. iv | RAPID PERFORMANCE EVALUATION USAID.GOV Some students reported receiving training to read Braille. Students who received this training reported their satisfaction with it , reporting that the itinerant teachers ’ tutoring helped them to do well in school. EQ. 4 OVERALL, WHAT INTENDED AND UNINTENDED CONTRIBUTIONS, RESULTS AND/OR OUTCOMES HAS THE RAPID PROJECT ACHIEVED RELATIVE TO IMPROVING INCLUSIVE ACCESS TO EDUCATIONAL OPPORTUNITIES FOR DISADVANTAGED CHILDREN AND YOUTH WHO ARE BLIND OR VISUALLY IMPAIRED? Some intended outcomes of the RAPID a ctivity included increased access to materials and resources for visually impaired and blind youth, an increased awareness of inclusive education among partnering institutions (i.e., CNEH) and state agencies (e.g., CASAS), and an increase in visually impaired and blind students accessing mainstream classrooms. Given the lack of prior existing data on this population it is difficult to determine the extent of RAPID’s impact on the desired outcomes. The team did not identify any unintended outcomes. However, the ET identified several unanticipated barriers to implementation. Transportation and Finances served as a barrier to regular attendance. Students faced several unanticipated barriers preventing them from attending school and RAPID intervention activities regularly. Specifically, transportation was one critical barrier. Some students reported that they live far from school or th e SHAA office and there were instances when they did not have transportation or money for public transportation. Another unanticipated barrier to inclusion of visually impaired and blind students into mainstream classrooms was parents’ inability to pay school tuition. RECOMMENDATIONS RECOMMENDATIONS FOR SHAA 1. Develop a recruitment strategy for identifying youth with visual impairment and blindness SHAA should consider developing a recruitment strategy for identifying youth with visual impairment and blindness. Given misunderstandings about blind children’s ability to learn, a strategy that includes a variety of means (e.g., outreach and recruitment at community churches, presentations, training, and recruitment at local schools before the start of the school year) to reach youth, would be ideal. A recruitment strategy would have increased SHAA’s likelihood of meeting their target of 100 new students integrated each year. 2. Develop a sustainable strategy for materials development SHAA should consider developing a sustainable strategy for developing materials. SHAA may want to consider reaching out to organizations that specialize in developing materials for the blind, such as The Braille Institute. They should consider obtaining additional equipment so that they can produce more Braille and audio books. 3. Identify information dissemination methods regarding available resources and materials that can be incorporated into participant program registration SHAA should identify methods to disseminate information to parents regarding available resources and materials available through SHAA (e.g., provide parents with a list of services and resources during registration, hold parent engagement nights to share information on services with parents, provide schools with a list of services to share with parents). 4. Include an educational component for students’ parents, including helping parents understand inclusive education and the role they can play in promoting and supporting it 5. Map where participants are located to identify a limited number of schools at which to centralize support services v | RAPID PERFORMANCE EVALUATION USAID.GOV SHAA should consider mapping where all their participants are located to identify a limited number of schools at which to centralize their support services. This may allow for more schools that are equipped with specialized services, and an environment that can better accommodate blind students. In areas where there is only one student in a geographic area, other solutions, such as transportation provision may be needed. 6. Establish minimum quality standards a given school must meet to be prepared to receive visually impaired or blind students SHAA should consider establishing minimum standards a school must meet to be deemed appropriate or equipped to accommodate blind students. Schools that meet the minimum standards would be selected to receive SHAA support services. SHAA should begin with the implementation of their services in two to three schools so that they can improve the implementation of their support services. While an assessment of minimum quality standards for physical access to schools was outside of the scope of this evaluation, future scale up will need to consider it . RECOMMENDATIONS FOR USAID 1. Consider piloting a model that includes one or two schools for inclusive integration of visually impaired students before scaling up to multiple schools Future programming should consider piloting a model that includes one or two schools for inclusive integration of visually impaired students before scaling up to multiple schools. This would allow USAID to establish best practices for the integration of this population into mainstream classes in Haiti before scaling up. 2. Provide support to SHAA for mapping locations and concentrations of visually impaired and blind students USAID could provide support to SHAA for mapping locations and concentrations of visually impaired and blind students. One option is to consider conducting a census focused on identifying households with blind children. 3. Facilitate important connections with leadership in the Ministry of Education as well as other high-level bilateral and multilateral agencies implementing programs in the inclusive education sphere USAID could facilitate important connections with leadership in the Ministry of Education as well as other high-level bilateral and multilateral agencies implementing programs in the inclusive education sphere. These connections would affect synergies in the areas of marketing, stakeholder coordination, policy change, and awareness raising. 1 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV I. INTRODUCTION According to the World Health Organization, there are an estimated 1.4 million visually impaired children worldwide. Approximately 500,000 children become visually impaired each year, with children from developing countries affected the most. In developing countries, the rate of blindness in children is estimated to be as high as 1.5 per 1000 population as compared to 0.3 per 1000 in industrialized countries. The literature indicates that children in poor countries are four to five times more likely to be visually impaired than those born in high- income countries. 1 In Haiti the prevalence of visually impaired or blind children is unknown due to a lack of population- level data. Estimates report that 80,000 people are visually impaired or blind in Haiti. 2 Among this population, there are estimated to be 20,000 who are school- age children of which only 1,160, or 5.8 percent attend school. Visually impaired and blind children are often deprived of basic education due to socioeconomic barriers, inaccessible schools, and discrimination and stigmatization. Overcrowding, limited qualified teachers, inadequate curricula, and social discrimination serve as major barriers to educational access and inclusion for blind children. 3 Thus, children who are visually impaired or blind may fall victim to negative outcomes including reduced quality of life (e.g., vision functioning, emotional well- being and social relationship concerns) 4 poor school attendance, and poor school performance. PROJECT BACKGROUND Established in 1952, La Société Haïtienne d'Aide aux Aveugles (SHAA) provides a range of services to individuals who are visually impaired and blind in Haiti. Committed to the inclusion of visually impaired and blind children and youth into educational settings, SHAA provides educational support such as Braille instruction, tutoring, and additional services aimed to increase access to schools. USAID awarded SHAA a 775, 503 USD grant (Award No. AID-521-F-15-00013 under APS Local Solutions Project 521- 14-000027) to implement the Reading for All: Promoting Inclusive Education (RAPID) a ctivity. Interventions implemented under the RAPID a ctivity went from August 28, 2015 to August 27, 2018 and were intended to expand SHAA’s population reach by increasing the number of visually impaired and bli nd students they integrated into mainstream classes and expanding their service provision to include the development of materials translated in Braille, teacher training, and the development of specialized libraries equipped with materials in Braille and audiobooks. SHAA was also able to hire more itinerant teachers who were tasked with providing tutoring services and teacher training. THEORY OF CHANGE The theory of change for the RAPID a ctivity postulates that an increase in the availability of adapted educational materials (e.g., transcription into Braille), improved customized educational services, and specialized training and capacity building will lead to integration of visually impaired and blind children and youth into mainstream classrooms, which will ultimately lead to improved inclusive access to quality educational opportunities for visually impaired and blind children. The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) activity, design and conduct an independent final evaluation of the RAPID activity, implemented by SHAA (see Annex A for the evaluation Statement of Work). Before conducting the fieldwork for this evaluation, the evaluation team (ET) presented its initial findings based on a document review (see Annex B for list of documents consulted and Annex C for the ET members’ profiles) in an evaluation design, a detailed report approved by USAID /Haiti (see summary of evaluation 1 Gilbert C, Foster A. Childhood blindness in the context of Vision 2020—the right to si ght. Bull World Health Organ 2001; 79:227–32; Hoyt CS, Good WV. The many challenges of childhood blindness. Br J Ophthalmol2001; 85:1145–6; Cunningham ET Jr, Lietman TM, Whitcher JP. Blindness: a global priority for the twenty-first century. Bull World Health Organ 2001; 79:180; Foster A. Cataract and “Vision 2020—the right to sight” initiative. Br J Ophthalmol 2001; 85:635– 7. 2 USAID, Social Impact Request for Proposals “Reading for All: Promoting Inclusive Education (RAPID) Activity 3 SHAA (2015). La Société Haïtienne d'Aide aux Aveugles. Local Solutions Initiative: Program Description. 4 Lamoureux, E., and K. Pesudovs. 2011. Vision -specific quality-of-life research: A need to improve the quality. American Journal of Ophthalmology 151(2):195–197, e192. 2 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV design in the Evaluation Design Matrix found in Annex D). The purpose of this report is to present the ET’s main findings, conclusions , and recommendations. ACTIVITIES/COMPONENTS The overall goal of the RAPID a ctivity was to increase access to quality education to children who are blind and visually impaired. Activities were concentrated in two main geographical areas: Port -au-Prince and the Cap-Haitian/Ouanaminthe area. The intermediate results of the activity were to: 1. Increase integration of children who are blind and visually impaired into mainstream schools; 2. Increase availability of accessible documents for people who are blind and visual ly impaired; and 3. Increase access to SHAA’s specialized library services for people who are blind, visually impaired, and sighted. EVALUATION PURPOSE AND A UDIENCE The evaluation purpose wa s to identify lessons learned and provide guidance on possible follow-up actions for similar USAID programs in Haiti and elsewhere. This wa s based on an examination of the extent to which RAPID achieved its intended outcomes, and a determination of whether the expected results occurred according to the activity’s design and implementation. Based on this, the evaluation provides recommendations. This evaluation report documents the RAPID activity’s performance, outcomes, lessons learned, and any issues observed during the evaluation process as relevant to the evaluation questions. The primary audience for this final evaluation is USAID /Haiti, which can use the evaluation findings and recommendations to inform its future programming in the area of inclusive education for the special needs population. Additional stakeholders may include SHAA staff, education leaders, targeted schools, and the Government of Haiti Ministry of Education. These s econdary audiences (e.g., SHAA staff, education leaders, target schools) may also consider evaluation findings and recommendations in their current programs and possible future collaboration with USAID /Haiti. EVALUATION QUESTIONS This evaluation answers the evaluation questions (EQs) below. When answering these EQs, the ET considered stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent has the project been effective in achieving results regarding blind and visually impaired children (i.e., integration into mainstreaming schools, access to documents and specialized libraries)? 2. To what extent has awareness been raised to promote inclusive education at the institutional (Ministry of National Education and Vocational Training (MENFP), Commission d’Adaptation Scolaire et d’Appui Social (CASAS), others), community and parental levels? 3. To what extent and in what ways has RAPID interventions (i.e., book-based clubs, training of trainers) affected the quality of teaching and learning for the blind and visually impaired? 4. Overall, what intended and unintended contributions, results and/or outcomes has the RAPID project achieved relative to improving inclusive access to educational opportunities for disadvantaged children and youth who are blind or visually impaired? II. EVALUATION DESIGN SAMPLING The evaluation utilized a purposive sampling technique to identify parents, students and teachers to participate in the Focus Group Discussions ( FGDs). SHAA provided the ET with a list of all parents, students, and teachers involved in the interventions. Once the ET identified FGD participants, SHAA reached out to participants and extended an invitation for participation. The ET conduct ed a similar process for selecting teachers. Those chosen to participate in the FGD s met the following criteria: 3 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Parents had the following characteristics: a) a child that was exposed to the RAPID activity and, b) demonstrated availability and willingn ess to participate in the focus groups. Teachers and Students had the following characteristics: a) exposure to the RAPID a ctivity and, b) demonstrated availability and willingness to participate in the focus groups. All participants were informed of their rights, and that participation was completely voluntary with no associated risks or benefits. The ET also conducted key informant interviews (KIIs) with a school director, SHAA staff member, National Confederation of Haitian Educators (CNEH) representative, and USAID RAPID Agreement Officer Representative. These informants provided information to address the information gaps identified and not addressed through the document review. Overall, information gathered from FGDs and KIIs provide d different key stakeholders ’ perspectives on the RAPID activity processes, accomplishments, and challenges. Information gathered from the USAID RAPID Agreement Officer Representative provided USAID’s perspectives on the RAPID a ctivity’s accomplishments and challenges (see Annex E for a complete List of Persons Interviewed). DATA COLLECTION METHODS The ET employed qualitative data collection methods to examine the RAPID activity’s impact on program participants. Qualitative methods consisted of document review, KIIs, observation , and FGDs. The ET reviewed activity documents and developed an I nception Report in September 2018. The team conducted FGDs and KIIs during the evaluation fieldwork period, from December 4 – 21, 2018. The data collection protocols that guided fieldwork can be found in Annex F . The purpose of the protocols was to: (1) ensure all key issues are covered during data collection, (2) elicit rich, sometimes unanticipated, information from respondents, (3) help organize information in a form that could be usefully and efficiently analyzed, and (4) obtain the information necessary to address all the EQs. The protocols consisted of questions that addressed and were derived from the EQs, as well as from the ET’s document review, and meetings with SHAA staff. DOCUMENT REVIEW The Team Leader (TL) conducted a document review to gather preliminary information regarding the RAPID activity’s implementation, which, in turn, informed the evaluation design, data collection protocol development, and identified the data the ET gathered during evaluation activities to answer the EQs. Documents reviewed included: • SHAA proposal document; • RAPID Year 2 annual report; • RAPID Year 3 monitoring and evaluation plans; and • RAPID fact sheet. FOCUS GROUP DISCUSSIONS The ET conducted eight FGDs. Although the ET had four FGDs scheduled with parents, only two FGDs were held with parents – one in the West and one in the Northeast. Four FGDs were scheduled with students, of which ultimately, three were held – two in the West and one in the Northeast. While two FGDs were scheduled with children under the age of 18 years, only one FGD was held for this age group. 4 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Two FGDs were scheduled with teachers. 5 The ET conducted three FGDs with teachers – one in the West and two in the North. The main barrier to parent, student, and teacher participation in the FGDs was transportation to the venue . SHAA staff explained that participants who agreed to attend the did not reside close to SHAA offices in both the West and Northeast, and thus the lack of transportation made it difficult for them to attend. The ET highlighted to participants that interview content would be kept confidential and that they would omit personally identifiable information from the report (see informed consent protocol in Annex F ). A total of 42 individuals were included in the FGDs, 52 percent of which were female. Table 1: FGDs by Stakeholder Group and Gender Type of Informant Number of Informants 2 Parent FGDs 8 (8 F) 3 Student FGDs 23 (13M, 10F) 3 Teacher FGDs 11 (7M, 4F) Total 42 (20M, 22F) KEY INFORMANT INTERVIEWS Additionally, the ET collected data on the perspectives and experiences of four key informants involved in the RAPID a ctivity using a purposive sampling technique (a school director, SHAA s taff member, CNEH representative, and USAID/Haiti Education Team representative). Although two school director interviews were proposed, only one director was available during the data collection week. Similarly, only one of each staff member at SHAA, CNEH, and USAID were available during the data collection. Table 2: Key Informant Interviews by Stakeholder Group and Gender Type of Informant Number of Informants 4 Key Informant Interviews 4 (2M, 2F) Total 4 ANALYSIS All KII and FGD data were numbered and organized according to gathering sites and collection date. Participant names were not included on any recordings or notes. Instead, a reference number was used to preserve confidentiality. All notes from FGDs and KIIs were collected in electronic format. The ET took detailed notes of KIIs and FGDs, and cleaned and shared notes on an ongoing basis during fieldwork. At the end of each data collect ion day, the ET met to discuss data collected and any i ssues, all of which were addressed immediately. The TL supervised and managed systematic analysis of data captured through the FGDs and KIIs. The ET employed content analysis (e.g., coding and theme identification) using NVivo qualitative analysis software. Data were compared across stakeholder groups to assess either convergence or divergence in perspectives. The TL captured preliminary findings, conclusions, and recommendations in an Excel-based matrix that categorized analysis by EQ. The matrix: a) ensured the ET prepared a systematic and thorough response to each EQ, b) verified that preliminary analysis accounted for gender and social dimensions, c) identified any gaps where additional clarification or analysis would be necessary, and d) served as the basis for developing the evaluation report. 5 Three types of teachers were interviewed as part of the evaluation: (1) itinerant teachers (employed and trained by SHAA, who then teach CNEH teachers, regular teachers, and students themselves), (2) CNEH teachers (who are taught by itinerant teachers), and other teachers (who are also taught by itinerant teachers and who have some SHAA students in their classes). 6 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 5 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV LIMITATIONS AND MITIGATION STRATEGIES BIASES The ET was aware of several bias risks in data collection and analysis and developed mitigation strategies to overcome these. 1. Recall bias occurs when respondents cannot recall previous events, recall them inaccurately, or omit details. Thus, the accuracy of the responses may be at risk. Given that the primary data collection method for the RAPID evaluations was qualitative interviews and respondents were self- reporting, recall bias is unavoidable and will o perate at some level. 2. Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. Response bias often operates within the Haitian context. Given the poverty conditions participants experience, they may tend to provide favorable responses in hopes of receiving future support. 3. Selection bias occurs when individuals or groups are selected in a way that is not random, but rather purposefully. This ensures that the results of the data are not necessarily representative of the full group. Given that the ET spoke with a relatively small group of targeted individuals during this evaluation, the results presented in this report may not be fully representative of the larger visually impaired and blind population. To overcome these biases, the ET included up to 12 participants in each FGD, as possible within the fieldwork period, to triangulate responses, increase the validity of evaluation findings, and allow for more diverse input. The ET mitigated response bias by interviewing leaders and middle managers separately and holding teacher and parent FGDs separately. In addition, the ET ensure d that participants clearly understood the evaluation’s objectives and potential benefits. RAPID includes disabled children as beneficiaries, thus a major risk is the violation of child protection rights. To minimize this risk, the ET hired qualified personnel experienced working with children and provided training for all ET members who had direct contact with children during the evaluation implementation. Moreover, the ET implement ed measures to prevent unauthorized access to or disclosure of participants’ personally identifiable information. Finally, to ensure participants’ protection, the evaluation design was submitted and approved by an institutional review board ethics review both within SI and in Haiti. LIMITED AVAILABILITY OF KEY INFORMANTS Parents, teachers, and students who received SHAA services were dispersed throughout Port-au-Prince and Cap-Haitian. Funds available were not within the evaluation budget to cover transportation costs for participants to attend FGDs. SHAA reported that prohibitive transportation costs to FGDs was the primary reason why teachers were unable to attend. Although, the ET conducted FGDs with all stakeholder groups proposed in the evaluation design, the number of FGD participants (e.g., eight to ten per FGD) was not met. Given the low teacher participation in the FGDs, findings should be interpreted with caution. Teachers with visually impaired and blind students integrated into the classroom did not participate in the FGDs. The perspective of these teachers would have provided data about the extent to which visually impaired and blind students were successfully integrated. In the absence of this data, the ET’s ability to comment on the extent to which the RAPID activity met its overall goal is limited. III. FINDINGS AND CONCLUSIONS This section synthesizes th e evaluation’s findings and conclusions, which are presented by EQ . EQ1: TO WHAT EXTENT HAS THE PROJECT BEEN EFFECTIVE IN ACHIEVING RESULTS REGARDING BLIND AND VISUALLY IMPAIRED CHILDREN (INTEGRATION INTO MAINSTREAMING SCHOOLS, ACCESS TO DOC UMENTS AND SPECIALIZED LIBRARIES, ETC.)? 6 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV FINDINGS During Year 1 of implementation, SHAA was able to meet its target of 100 visually impaired and blind youth integrated into mainstream classes. Year 2 and 3, they did not meet their targets. SHAA field agents worked in Port-au-P rince and Cap Haitian with the purpose of meeting families with a visually impaired or blind children . During these interactions, SHAA field workers engaged in dialogue with the families and shared the services offered through SHAA. It is through this field work that most students were recruited. As the field agents identified a visually impaired or blind c hild or youth, they recorded the contact information and eventually established a database. Students added to the database received some services before participation in the RAPID activity . These services were expanded once the RAPID activity was implemented. The database was the main source for recruitment during Year 1 (2016) of implementation in which SHAA was able to meet its target of 100 students identified. Other recruitment strategies included radio advertisement, and solicitation on SHAA services at clinics. The radio advertisements proved to be the least effective recruitment strategy as no students were reported to be recruited via this method. During Year 2 (2017), however, SHAA faced challenges recruiting potential students for integration. SHAA reported that the primary reasons parents offered included lack of funds for transportation to attend school and travel to SHAA’s office to receive Braille training, which is required for inclusion into mainstream classes. SHAA reported that parents ’ lack of funds to pay school tuition and false perceptions that their child cannot learn also served as barriers. Given these challenges, SHAA did not meet its target integration of 100 and only integrated 30 new students into mainstream classes in Year 2. The majority of s tudents who were identified and received services through the RAPID activity were integrated into mainstream classes and remained in school for the entire year. Once students were identified and then integrated, they received support services from SHAA, which began with training to read in Braille. Students also received tutoring provided by itinerant teachers (i.e., teachers employed by SHAA) as part of support services. Through the RAPID activity, SHAA was able to expand its reach by increasing the number of students they integrated into mainstream classes and their service provision to include tutoring. While the majority of students who were identified and received services (e.g., training in Braille, tutoring) through the RAPID activity remained in school for the entire year, parents and SHAA staff reported parents’ inability to pay tuition served as the main factor preventing some students from remaining in school for a full year. Students and itinerant teachers reported that an increase in materials and resources is needed to meet the needs of all students served by SHAA. SHAA was also tasked with providing materials and resources to visually impaired and blind students . During Year 2 SHAA reported that 30 students received kits that included a digital tape recorder, a tablet, a punch game, braille paper , and an abacus counter. Students also confirm ed that they received materials, however, they reported that the amount of materials was insufficient for the population served and inadequate to meet their needs. For example, university students reported that the recorder SHAA provided did not have the capacity to record all their courses and they resorted to recording courses on their phones. University students also reported that the number of materials produced by SHAA were not enough to meet the needs of all students served. They stated that it would be beneficial to have materials that can be taken home so that they can practice Braille and typing at home. This finding was echoed by itinerant teachers (i.e., teachers SHAA employed) who reported that they needed more means to produce additional Braille books for children. Itinerant teachers also explained that as they travel throughout the community to provide tutoring support and training to teachers, they do not have enough materials. The availability of materials is underscored by SHAA’s inability to implement a sustainable method to produce materials in Braille and audiobooks that can sufficiently meet participants’ needs. Through funds received from the RAPID activity, SHAA was able to purchase equipment to develop Braille materials. Despite the new equipment, itinerant teachers still report that SHAA needed equipment to produce more materials. While the new equipment allowed them to produce materials that did not exist before the 7 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV activity, the equipment does not appear to be enough to develop the material at a rate or level to meet the needs of all SHAA students. Most students reported receiving and using materials SHAA provided (e.g., abacus, tablets, recorders), however, there were a few s tudents who reported that they never received any materials or were not fully aware of the resources SHAA offered. Students who reported not receiving any materials, reported accessing other services through SHAA such as adaptive computer service and school tuition assistance. SHAA developed S pecialized Libraries. All students reported using the specialized libraries. The RAPID Annual report SHAA submitted indicates that in 2016, 3,575 visually impaired, blind, and sighted students accessed the libraries, followed by 3,519 in 2017. Students reported that they benefit ted from the use of audiobooks and books written in Braille available in the libraries. Students also reported that the specialized libraries were beneficial because they were able to participate in computer training and the book clubs that are held at the libraries. SHAA staff reported that for the book clubs held at the Cap Haitian library, the team in Port- au-Prince must periodically send books. Overall, participants were aware that the specialized libraries exist. Students and itinerant teachers reported that while the specialized libraries are beneficial and accessible, materials in Braille or audio are limited. Students recommended that SHAA increase the number of materials at the specialized libraries. CONCLUSIONS SHAA was successful, to some extent, in achieving RAPID activity results. While the prevalence of visually impaired or blind children is unknown in Haiti due to a lack of population- level data, estimates report that 80,000 children are visually impaired or blind in Haiti. Thus, we know that there are children in Haiti that SHAA can recruit for services and integrate into mainstream classes. However, SHAA was faced with recruitment challenges. It appears that a lack of a recruitment strategy before the start of the RAPID activity further exacerbated successful recruitment of the targeted population. SHAA met its objective to produce materials and resources and these materials are reaching intended students. However, the amount of materials did not appear to be sufficient to accommodate all students served. The specialized libraries were achieved but are still in need of development. Additional materials need to be developed and made available at the libraries. EQ2: TO WHAT EXTENT HAS AWARENESS BEEN RAISED TO PROMOTE INCLUSIVE EDUCATION AT THE INSTITUTIONAL (MENFP, CASAS, OTHERS), COMMUNITY AND PARENTAL LEVELS? FINDINGS SHAA was able to raise awareness through partnerships and media outlets. SHAA was tasked with raising awareness to promote inclusive education at the institutional, community, and parental levels. To accomplish this task, SHAA report ed establishing partnerships with other organizations with similar interests. For example, through their connection with the National Associative Network for Integration of Disabled People, SHAA was able to connect with other organizations with similar interests, doing similar work. They have been working closely with CNEH to increase awareness of the needs of visually impaired and blind students and promote inclusion primarily by providing teacher training to CNEH teachers. Specifically, CNEH reported that through instruction SHAA facilitated, teachers received training on how to be better prepared to receive students into mainstream classes. CNEH also reported that SHAA met their expectations by providing teachers with adequate training and tools that will help them to better accommodate visually impaired and blind students in their classroom. SHAA and CNEH reported that their collaboration resulted in a Memorandum of Understanding (MOU), outlining CNEH’s commitment to inclusive education and teacher training. Bringing great visibility and awareness to inclusive education, this MOU was signed at a press conference on December 7, 2016 and is one of the events that took place during SHAA’s commemoration of International Day of People with Disabilities. 8 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV SHAA also held meetings and conferences with other organizations to raise awareness, particularly with the United Nations Educational, Scientific and Cultural Organization and Sectorial Group of Education. These meetings concluded with verbal commitments to consider children with disabilities in their planning efforts. One of SHAA’s efforts that has helped them gain greater visibility is radio air time where they shared their services and promoted inclusive education for the visually impaired and blind. SHAA has been able to cover eight of the ten departments in Haiti, through their interactions with other organizations and mass media. Unfortunately, SHAA’s many attempts to reach out to the MENFP were met with minimal success. Although some strides have been made (e.g., they have begun to build a relationship with CASAS, a state agency responsible for special education), they have not had any major meetings with the Ministry leadership. SHAA reported that unsuccessful attempts to meet with MENFP are for reasons beyond their control since MENFP canceled their scheduled meetings in the past. Parents of the students appeared to have a lack of understanding of what is meant by inclusive education. While teachers and collaborating partners were aware and had a general understanding of inclusive education, students’ parents appeared to lack understanding of what is meant by inclusive education. When asked to explain it , parents reported that they did not know. They were, however, very appreciative of the services and resources their children received because of the activity. CONCLUSIONS SHAA’s efforts to raise awareness about inclusive education proved successful, to some extent, as they were able to increase their visibility in Haiti through meetings with various organizations, conferences, and the media. Radio is one of the most popular sources of news and information in Haiti – radio ownership in Haiti is high, and many people listen to the radio on their mobile phones. Given this, SHAA’s promotion of services and awareness of inclusive education has the potential to reach masses in all departments. SHAA’s efforts to raise awareness about inclusive education proved to have limited success – while SHAA increased its visibility in Haiti through meetings with various organizations, conferences, and the media, they were unable to achieve meaningful collaboration with public institutions (i.e., MENFP). SHAA lacks the human resources and time to engage in meaningful collaboration activities with similar NGOs and other institutional partners. EQ3: TO WHAT EXTENT AND IN WHAT WAYS HAS RAPID INTERVENTIONS (BOOK -BASED CLUBS, TRAINING OF TRAINERS, ETC.) AFFECTED THE QUALITY OF TEACHING AND LEARNING FOR THE BLIND AND VISUALLY IMPAIRED? FINDINGS SHAA provided training to CNEH Teachers. According to CNEH teachers and SHAA staff, the training content focused on accommodations that can be implemented in the classroom environment to support inclusion into mainstream classrooms. The training also taught teachers how to physically guide the blind and included content on the adaptation of educational pedagogy for visually impaired and blind students. CNEH teachers also reported that they received examples of blind people who read and write to debunk false beliefs or misconceptions teachers may have. Although CNEH teachers participated in the training, they did not have the opportunity to put learning into practice because they did not have any children in their classes with vision problems or who were blind. Itinerant Teachers report that the SHAA training they attended allowed them to better train teachers with SHAA students integrated in their classrooms. SHAA’s itinerant teachers reported applying what they learned during training to impact the teaching of teachers with SHAA students integrated into their classes. They also reported impacting student learning. Itinerant teachers reported that they provided classroom training for teachers who had visually impaired 9 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV and blind students integrated in their classrooms (i.e., SHAA students). The itinerant teachers reported and CNEH teachers concurred that they learned classroom strategies, such as changing the classroom environment to accommodate the student and reading out loud content that was on the board. One school director reported that after a SHAA facilitated training, one of his teachers who had a SHAA student integrated into the classroom rearranged the classroom to accommodate the student. The itinerant teachers also reported that the training they received from SHAA allowed them to be better prepared to assist students with learning. For example, itinerant teachers reported receiving instruction in Braille and online training on inclusive education. Students also reported receiving training from SHAA. Some students reported receiving training to read Braille . Students who received this training also reported their satisfaction with it , and that the tutoring received from the itinerant teachers helped them to do well in school. Parents also reported that they were satisfied with the Braille training their children received. One parent went on to report that before receiving the training in Braille, her child did not know how to write and now he can write in Braille. Another parent concurred , explaining that her child’s good school grades served as evidence that the Braille training wa s good. Students reported that while the Braille training was useful, they also recommended that SHAA consider more than five weeks to complete the training because a lot of material is packed into it. The SHAA database indicated that students served by SHAA were from 40 schools across the targeted geographic regions. CONCLUSIONS Since CNEH teachers’ training was not necessarily for those with visually impaired and/or blind students in their classrooms, the training had a muted impact on the quality of teaching and learning for the blind and visually impaired students. The training itinerant teachers received directly impacted SHAA students’ learning as they provided tutoring directly to students. The training itinerant teachers received indirectly impacted the teaching of teachers with visually impaired and blind students in their classrooms (e.g., training received by itinerant teachers allowed them to better train teachers in the classroom). EQ. 4 OVERALL, WHAT INTENDED AND UNINTENDED CONTRIBUTIONS, RESULTS AND/OR OUTCOMES HAS THE RAPID PROJECT ACHIEVED RELATIVE TO IMPROVING INCLUSIVE ACCESS TO EDUCATIONAL OPPOR TUNITIES FOR DISADVANTAGED CHILDREN AND YOUTH WHO ARE BLIND OR VISUALLY IMPAIRED? FINDINGS Some of the RAPID a ctivity’s intended outcomes included increased access to materials and resources for visually impaired and blind youth, an increased awareness of inclusive education among partnering institutions (i.e., CNEH) and state agencies (e.g., CASAS), and an increase in visually impaired and blind students accessing mainstream classrooms. Given the lack of prior existing data on th is population it is difficult to determine the extent of RAPID’s impact on the desired outcomes. The team did not identify any unintended outcomes. However, the ET identified several unanticipated barriers to implementation, which are specified below. Transportation and Finances served as a barrier to regular attendance Several unanticipated barriers prevented students from attending school and RAPID a ctivities regularly. Specifically, transportation served as a barrier. Some students reported that they lived far from school or the SHAA office and there were instances when they did not have transportation or money for public transportation. 10 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Another unanticipated barrier to inclusion of visually impaired and blind students into mainstream classrooms, was parents’ inability to pay school tuition. While SHAA was not required to make tuition payments, some parents reported that SHAA provided them with financial assistance to pay school tuition, while others reported that SHAA did not provide financial assistance for school tuition. Some parents reported that although they received support services from SHAA, when faced with financial struggles, they were unable to send their child to school. Other parent s reported that SHAA promised to help pay school tuition, but never provided any funds. Parent s who did not receive any financial assistance with school tuition reported that they wish this were part of the support services SHAA provided. SHAA staff also reported that they made promises to pay school tuition for their participants, but were unable to do so due to lack of funds. As a result, they reported that they owe some schools tuition payments. CONCLUSIONS To some extent, RAPID activities contributed to improving inclusive access to educational opportunities for children and youth who are blind or visually impaired. Baseline data, including prior school year attendance and the number of blind and visually impaired youth integrated into the targeted schools before the RAPID activities, would have provided more evidence on the extent to which RAPID activities contributed to the intended outcomes. IV. RECOMMENDATIONS RECOMMENDATIONS FOR SHAA 1. Develop a recruitment strategy for identifying youth with visual impairment and blindness SHAA should consider developing a recruitment strategy for identifying youth with visual impairment and blindness. Given misunderstandings about blind children’s ability to learn, a strategy that includes a variety of means (e.g., outreach and recruitment at community churches, presentations, training, and recruitment at local schools before the start of the school year) to reach youth, would be ideal. A recruitment strategy would have increased SHAA’s likelihood of meeting their target of 100 new students integrated each year. 2. Develop a sustainable strategy for materials development SHAA should consider developing a sustainable strategy for developing materials. SHAA may want to consider reaching out to organizations that specialize in developing materials for the blind, such as The Braille Institute. They should consider obtaining additional equipment so that they can produce more Braille and audio books. 3. Identify information dissemination methods regarding available resources and materials that can be incorporated into participant program registration SHAA should identify methods to disseminate information to parents regarding available resources and materials available through SHAA (e.g., provide parents with a list of services and resources during registration, hold parent engagement nights to share information on services with parents, provide schools with a list of services to share with parents). 4. Include an educational component for students’ parents, including helping parents understand inclusive education and the role they can play in promoting and supporting it 5. Map where participants are located to identify a limited number of schools at which to centralize support services SHAA should consider mapping where all their participants are located to identify a limited number of schools at which to centralize their support services. This may allow for more schools that are equipped with specialized services, and an environment that can better accommodate blind students. In areas where there is only one student in a geographic area, other solutions, such as transportation provision may be needed. 6. Establish minimum quality standards a given school must meet to be prepared to receive visually impaired or blind students 11 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV SHAA should consider establishing minimum standards a school must meet to be deemed appropriate or equipped to accommodate blind students. Schools that meet the minimum standards would be selected to receive SHAA support services. SHAA should begin with the implementation of their services in two to three schools so that they can improve the implementation of their support services. While an assessment of minimum quality standards for physical access to schools was outside of the scope of this evaluation, future scale up will need to consider it . RECOMMENDATIONS FOR USAID 1. Consider piloting a model that includes one or two schools for inclusive integration of visually impaired students before scaling up to multiple schools Future programming should consider piloting a model that includes one or two schools for inclusive integration of visually impaired students before scaling up to multiple schools. This would allow USAID to establish best practices for the integration of this population into mainstream classes in Haiti before scaling up. 2. Provide support to SHAA for mapping locations and concentrations of visually impaired and blind students USAID could provide support to SHAA for mapping locations and concentrations of visually impaired and blind students. One option is to consider conducting a census focused on identifying households with blind children. 3. Facilitate important connections with leadership in the Ministry of Education as well as other high-level bilateral and multilateral agencies implementing programs in the inclusive education sphere USAID could facilitate important connections with leadership in the Ministry of Education as well as other high-level bilateral and multilateral agencies implementing programs in the inclusive education sphere. These connections would affect synergies in the areas of marketing, stakeholder coordination, policy change, and awareness raising. 12 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV ANNEX A: EVALUATION STATEMENT OF WOR K STATEMENT OF WORK Performance Evaluation of the Reading for All: Promoting Inclusive Education ( RAPID) Activity I. PURPOSE OF THE EVALUATION The purpose of the contract is to conduct a final performance evaluation of the Reading for All Promoting Inclusive Education (RAPID) activity. Findings, conclusions, and recommendations from this final performance evaluation will be incorporated into USAID/Haiti’s short, medium, and long -term program planning and execution. The final performance evaluation will focus on how RAPID has been implemented, what it has achieved, and whether expected results have occurred according to the activity’s design and implementation. This final performance evaluation will also provide an opportunity to do an in-depth investigation and analyses of the issues, and to document project performance, results, impacts, lessons learned, and best practices from various stakeholder perspectives. As a critical contribution to USAID/Haiti’s knowledge management efforts, the results of this final performance evaluation will also be shared with the Haitian Society of Help for and to the Blind (SHAA) management. It is expected the findings, conclusions, and recommendations will be useful to SHAA for its own future planning. In relation to how activities are perceived, valued, and sustained moving forward, this final performance evaluation should identify areas that should be considered for continued engagement with organizations and interventions for and with people with disabilities or areas that would require strengthening before continued investment in similar interventions and activities could be considered. The evaluation must comply with the revised September 2016 USAID Evaluation Policy, available at http://www.usaid.gov/evaluation. II. SUMMARY INFORMATION Strategy/Project/Activity Name Reading for All: Promoting Inclusive Education (RAPID) Implementer Haitian Society of Aid for and to the Blind (SHAA) Cooperative Agreement/Contract # Fixed Amount Award No. AID-521-F-15-00013 under APS Local Solutions Project 521-14-000027 Total Estimated Ceiling of the Evaluated Project/Activity (TEC) $775,503 Life of Strategy, Project, or Activity August 28, 2015 – August 27, 2018 Active Geographic Regions Port-au-Prince and its surrounding areas, and the Cap- Haitien/Ouanaminthe areas Development Objective(s) (DOs) 4: Education outcomes improved USAID Office Haiti III. BACKGROUND A. DESCRIPTION OF THE PROBLEM, DEVELOPMENT HYPOTHESIS(ES), A ND THEORY OF CHANGE Around 15% of the world’s population- an estimated 1 billion people live with disabilities, according the World Health Organization (WHO). People with disabilities make up the world’s largest minority. The World Bank also estimated 20% of the world’s poorest people have some kind of disability and tend to be regarded in their own communities as the most disadvantaged. And according to UNDP, it is assumed that 80% of people with disabilities live in developing countries like Haiti. 13 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV While we lack concrete data on the overall number of persons living with disabilities in Haiti, it is estimated that one in ten Haitians live with a disability. In Haiti, based on the WHO definition, it is estimated that at least 80,000 people are blind or visually impaired. Of those 80,000 people, the population of children who are blind or visually impaired and of school-age (under 14 years old) is estimated to be 25% (20,000) of the larger visually disabled population. In Haiti, the educational situation for children who are blind or visually impaired is disconcerting. The number of children who are blind that attend school currently is only 1,160 or 5.8% of the 20,000 while the national enrollment rate of non-disabled children is 80%. Although the numbers only tell part of the story, it is widely observed in countries like Haiti that poverty, overcrowded schools, lack of qualified teachers, inaccessible public and private schools, inadequate curriculums, inaccessible or inadequate transportation and social discrimination present massive barriers to a quality and an inclusive education for children who are blind or visually impaired. SHAA is an association for the blind and visually impaired. It is one of the oldest Haitian institutions working with people with disabilities. It was created on the 23rd of February 1952 and its mission consists of preventing blindness and rehabilitating people, whose vision can no longer be restored, supporting visually impaired children and youth with their education. From a theoretical point of view and based on the experience gained over the last fifteen years, SHAA plans to ensure that identified students, who are visually impaired, according to the level of their sensory deficiency, will either be directly integrated in regular classes or through small special classes implemented in regular schools that are sufficiently motivated to welcome into their ranks children and young people with visual disabilities. With SHAA’s vision and this inclusive education experience, SHAA can motivate other people, other sections of society to reflect on the merits of an inclusive society, a society that takes into account the needs and aspirations of all its sons and daughters, disabled or not. SHAA proposes to undertake this extended RAPID activity described herein and intends to achieve the following three results: • Result one: Increased integration of children who are blind and visually impaired to mainstreaming schools. • Result two: Increased availability of documents accessible to people who are blind and visually impaired. • Result three: Increased access to SHAA's specialized library services for persons who are blind, visually impaired, and sighted. Ultimately, the final performance evaluation of the RAPID activity will test USAID/Haiti Education Office’s hypothesis stating that in order to improve inclusive access to quality educational opportunities for disadvantaged children and youth who are blind or visually impaired, the availability of adapted educational materials should be increased, the availability of customized educational services should be improved, and the integration of children and youth with disabilities into typical classrooms should be pursued with the proper supports in place. Without effective resources to support the integration of children and youth who are blind or visually impaired into mainstream classrooms and without specialized training and capacity building to scaffold the learning environment for these targeted children and youth, inclusive access to educational opportunities for children and youth who are blind or visually impaired cannot be achieved. B. SUMMARY STRATEGY/PROJECT/ACTIVITY/INTERVENTION TO BE EVALUATED The RAPID activity focuses on the promotion of inclusive education through mainstream schooling, which consists of integrating disabled children into “regular” schools alongside typically developing students because experience has shown that this approach allows access to a greater number of students with disabilities to school, and also helps foster the development of human solidarity between people who have a disability and those who do not. Integrated and inclusive education requires a series of accompanying elements that should be taken into consideration. SHAA’s RAPID activity takes these steps into careful consideration and has worked them into activity planning. 14 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV On August 28, 2015, USAID/Haiti originally awarded SHAA a two-year Fixed Amount Award (FAA) for $499,992. The planned activities included, but were not limited to, the identification of school aged children and youth who are blind or visually impaired, the teaching of rehabilitation techniques to these youth, the establishment of a production unit in the SHAA office so that materials and documents could be adapted into Braille, audio and relief imaging for various audiences, the creation of reading, debate, and socio-cultural activities clubs in the activity’s target areas and the reinforcement of their specialized library for children, youth, and adults who are visually impaired or blind in Port-au-Prince and Cap Haitien. However, in order to build on SHAA’s initial 2015- 2017 momentum, SHAA requested a cost increase and time extension to add one year to the original FAA and increased the total award amount by $297,250.The proposed intensified support under the time extension and cost increase was intended to complement ongoing activities and expand related programming to increase the access and quality of education for children who are blind and visually impaired in Port-au-Prince and Cap Haitien. On October 16, 2017, a time and cost extension were granted for a period of up to one year and increased the new total estimated cost to $775,503. Extension activities focus on the following: 1. Support for six itinerant teachers: SHAA’s itinerant teachers support their mainstreamed beneficiaries at school and at home but need additional capacity building. The itinerant teachers will receive skills to improve their ability to provide quality refresher courses, organize after school activities (also at SHAA’s main office), and help SHAA’s staff plan award ceremonies. This support also includes a communication campaign on educating parents and school administrators on what does it mean to live with a disability. 2. Support production of resources for the blind and visually impaired: The newly developed production unit will produce additional adapted materials (school textbooks, reading and academic materials, and other reading materials in braille, relief images, and audio format) for an additional 50 students, which would increase the activity total from 250 to 300 student beneficiaries. 3. Increased accessibility to adapted materials: Through use of SHAA’s specialized library in Port- au- Prince and SHAA’s computer room in Cap Haitien, SHAA’s staff will conduct training for teacher and student beneficiaries on how to use adapted materials and information technology. Adapted equipment will also be distributed to children who are blind/visually impaired. SHAA has chosen two USAID targeted corridors as geographical implementation areas for this activity: Port-au-Prince and its surrounding areas and the Cap- Haitien/Ouanaminthe areas for the following reasons: 1. Previous experience with program implementation in these regions, along with the presence of two formally established offices, in Port- au-Prince (headquarters of the central office), Cap- Haitien (seat of the office SHAA North - Northeast) 2. Ability to understand base interventions of a recently conducted pilot that achieved positive results in inclusive education. 3. The great need that has been identified for blind and visually impaired children in the northern part of the country, particularly in Cap- Haitien C. SUMMARY OF THE PROJECT/ACTIVITY MONITORING, EVALUATION, AND LEARNING (MEL) PLAN The Monitoring, Evaluation, and Learning (MEL) Plan will be made available to the evaluators. The MEL Plan includes proposed indicators for each of the three expected Intermediate Results (IR) identified for the activity. Each indicator has a Performance Indicator Reference Sheet (PIRS). There is also a Performance Reporting Table, which includes the indicators and targets for the activity. As part of this activity, SHAA will develop a system for targeting and identifying children with disabilities who are most vulnerable. The system will take into consideration various demographic factors (such as, house-hold poverty level, location 15 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV in the country, family composition and severity of the child’s disability). Additionally, various scopes of work, quarterly reports, and annual reports will also be made available to the evaluators. IV. EVALUATION QUESTIONS The evaluation will focus on four primary areas of interest regarding the performance of the RAPID project. Via the evaluation questions, USAID Haiti is interested in learning about RAPID’s performance and achieved results/outcomes. 1. MAINSTREAMING: To what extent have RAPID interventions increased integration of children and youth who are blind or visually impaired into mainstreaming schools? o Specifically, did sub-activities 1.2-1.5 contribute to increasing the integration of children and youth who are blind or visually impaired into mainstream schools? Were there any spillover effects? o Additionally, what was the result of mainstreaming for children and youth who are blind or visually impaired? 2. AWARENESS -RAISING: To what extent have RAPID interventions promoted inclusive education to stakeholders at MENFP, CASAS, other special schools/inclusive education professionals, community, and parental levels? o Specifically, did sub-activities 1.6 and 1.8 support the rights of education for all? o Additionally, what was the result of raising the awareness of the value of inclusive education to stakeholders? What was the result of these activities for children and youth who are blind or visually impaired? 3. INNOVATION: What effect, if any, did RAPID’s innovative interventions have on the quality of teaching and learning? o Specifically, did sub-activities 3.1-3.3 affect the training of trainers/teachers and the learning of end-user beneficiaries? o Additionally, what was the result of innovative interventions (i.e., sub-activities 3.1-3.3) for children and youth who are blind or visually impaired? 4. Overall, what intended and unintended contributions, results and/or outcomes has the RAPID project achieved relative to improving inclusive access to educational opportunities for disadvantaged children and youth who are blind or visually impaired (IR 4.3)? V. EVALUATION DESIGN AND METHODOLOGY It is expected that the evaluation team proposes an overall research design to address the evaluation questions and a plan for collecting and analyzing the data both qualitative and quantitative. The evaluation design should be based mainly on qualitative research methods to address the evaluation questions. Key Informant Interviews (KIIs), Focus Group Discussions (FGDs), and in- depth review of projects reports are among the techniques that the evaluation team could consider when addressing the evaluation questions. VI. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work Plan: Upon receipt of the final approved SOW, ESS shall submit within twenty working days a Work Plan to the Contracting Officer’s Representative (COR), which will include: (1) the anticipated schedule and logistical arrangements; (2) a list of the members of the evaluation team, and a local firm with key personnel delineated by roles and responsibilities with their level of effort; (3) the deliverable schedule. 2. Evaluation Design: Within two weeks of the Work Plan approval, ESS must submit to the COR an Evaluation Design (which will become an annex to the Evaluation Report). The Evaluation Design will include: (1) a detailed evaluation design matrix that links the evaluation questions in the SOW to data sources, methods, and a data analysis plan; (2) draft questionnaires and other data collection 16 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV instruments or their main features; (3) a list of potential interviewees and sites to be visited; (4) known limitations to the evaluation design; and (5) a dissemination plan. USAID/Haiti will take up to 10 business days to review and consolidate comments through the COR. Once the evaluation team receives the consolidated comments on the initial Evaluation Design and Work Plan, they are expected to revise the reports within five business days. A final budget should be attached to the evaluation design. 3. In-briefing: Prior to undertaking fieldwork, the evaluation team or research firm will have an in- briefing with the USAID/Education and ESS COR to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and to clarify any questions or logistic needs. 4. Evaluation Briefing/Presentation: The evaluation team is expected to hold a final presentation in person to discuss the summary of findings and recommendations with USAID within 20 business days after the conclusion of fieldwork. 5. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers having a bearing on the evaluation objective. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft Evaluation Report will be determined in the evaluation work plan. Once the draft Evaluation Report is submitted, USAID/Haiti will have 10 business days to review and comment, after which the ESS COR will submit consolidated comments to ESS. The evaluation team or research firm will then submit the revised final report within five business days, and again USAID/Haiti will review and send comments within five business days of its submission. 6. Final Evaluation Report: The evaluation team will be asked to take no more than 15 business days to respond/incorporate the final comments from the ESS COR and Food for Peace Team. ESS will then submit the final report to the COR. All project data and records (FGD and KII summary reports) shall be submitted in full and should be in electronic form in easily readable format, organized, and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. VII. EVALUATION TEAM COMPOSITION Haiti ESS shall sub-partner with a local Haitian firm to conduct the RAPID final performance evaluation. The selected Haitian firm should mobilize an evaluation team comprised of one Key Personnel position, a Team Leader. The firm should demonstrate proven capacity in collecting data in the relevant field and should have data analysis capabilities to perform all related data management and analysis functions. This approach is encouraged to build the local firm capacity and will also provide a Haitian perspective for the data collection and analysis. The Team Leader (TL) will be ultimately responsible for the overall management of the evaluation team, coordinating the evaluation implementation, assigning team members responsibilities and tasks, and authoring the final evaluation report in conformity with this SOW. The TL must be an experienced evaluation expert, with a documented track record of 5-10 years of experience. S/he should have a strong background in qualitative data analysis. S/he should be fluent in French/Creole and conversant in English. S/he should have at least a university Degree in Social Science: Education, Political Science, or Public Administration. The Assistant Team Leader should have proven experience in designing and implementing qualitative research methods. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. 17 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV VIII. EVALUATION SCHEDULE and estimated loe Timing (Anticipated Weeks or Duration) Scheduled Activities 4 weeks Preparation of the work plan, and evaluation design 2 Weeks USAID review of the work plan, inception report and evaluation design 1 Week Submission of the revised evaluation design, inception report and work plan; in-briefing 2 Weeks Data collection 2 Weeks Data analysis and Evaluation Briefing 2 Weeks Draft Report writing 2 Weeks USAID review of Draft Report 2 Weeks Incorporate USAID comments and prepare Final Report Estimated LOE in days by position for a team of four: Position Preparation Travel to/from Country In-Country Data Collection Finalization of Report Total LOE in days Local Team Leader 42 0 12 48 102 Facilitator 0 0 12 12 24 Note Taker 0 0 12 12 24 Totals 42 0 36 72 150 VIIII. FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2– 5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information properly identified and listed; and 18 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. • Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. X. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 6 • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person- level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action -oriented, practical, and specific. 6 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 19 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV ANNEX B: REFERENCES AND ENDNOTES 1. Gilbert C, Foster A. Childhood blindness in the context of Vision 2020—the right to sight. Bull World Health Organ 2001; 79:227–32; Hoyt CS, Good WV. The many challenges of childhood blindness. Br J Ophthalmol2001; 85:1145– 6; Cunningham ET Jr, Lietman TM, Whitcher JP. Blindness: a global priority for the twenty-first century. Bull World Health Organ 2001; 79:180; Foster A. Cataract and “Vision 2020—the right to sight” initiative. Br J Ophthalmol 2001; 85:635– 7. 2. USAID, Social Impact Request for Proposals “Reading for All: Promoting Inclusive Education (RAPID) Activity 3. SHAA (2015). La Société Haïtienne d'Aide aux Aveugles. Local Solutions Initiative: Program Description. 4. Lamoureux, E., and K. Pesudovs. 2011. Vision- specific quality- of-life research: A need to improve the quality. American Journal of Ophthalmology 151(2):195– 197, e192. 20 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV ANNEX C. EVALUATION TEAM MEMBERS Team Leader: Sandra Williams, Ph.D. Dr. Sandra Williams, the Team Lead, is of Haiti an descent and a native Creole speaker. She served as the Lead Evaluator on a number of federally funded progra ms in the United States and has expertise in conducting random assignment and quasi-experimental design program evaluation studies. She has led several eval uations and monitored and evaluated over 100 projects in which she provided monitoring reports and coaching to bring about program improvement. FGD and KII facilitator: William Michel, Ph.D. Dr. Michel William, the Facilitator, is a Haitian native and re ceived his master’s degree in Measurement and Evaluation and Doctorate Degree in Educatio nal Administration from the University of Montr eal. Dr. William has led evaluation teams for data collection and analyses throughou t Haiti . His expertise is in research design and planni ng, coordinating evaluation efforts for a number of large organizations. Notetaker: Wesly Jean Logistic Organizer: Shedna Italis. 21 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV ANNEX D: EVALUATION DESIGN MATRIX EQ Data Collection Data Source Sample Questions Analysis To what extent has the project been effec tive in achieving results regarding blind and visually impaired children (integration into mainstreaming schools, access to documents and specialized libraries, etc.)? • Site visit • Document review • Focus groups • KIIs Parents, students, teachers, School Directors COMPLETION: 1. What materials and documents were developed by SHAA for use with teachers and students who are visually impaired and blind? 2. How successful was SHAA in developing specialized libraries for visually impaired and blind children? ACCESS: 1. To what extent did SHAA reach its intended target group (i.e., children who are visually impaired and blind)? 2. To what extent did SHAA successfully integrate visually impaired and blind children into mainstream schools? 3. To what extent did students, teachers, school directors, and/or parents access documents and other materials developed by SHAA? 4. To what extent did students, teachers, school directors, and/or parents access specialized libraries developed by SHAA? Content Analyses Descriptive Analyses (if quantitative data is available) To what extent has awareness been raised to promote inclusive education at the institutional (MENFP, CASAS, others), • Focus Groups • Key Informant Interviews Parents, CNEH, teachers, School Directors 1. To what extent do individuals at the institutional (MENFP, CASAS, others), the schools (e.g., teachers, school Content Analyses 22 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV community and parental levels? directors), and parents understand inclusive education? 2. To what extent were individuals at the institutional (MENFP, CASAS, others), the schools (e.g., teachers, school directors), and parents involved with promoting inclusive education? 3. To what extent has institutions, schools (e.g. teachers, school directors), and communities change their practices to promote inclusive education? To what extent and in what ways has RAPID interventions (book-based clubs, training of trainers, etc.) affected the quality of teaching and learning for the blind and visually impaired? • Site Visit • Document review • Focus Groups • Key Informant Interviews Parents, students, teachers, School Directors 1. To what extent did students, teachers, school directors, and parents utilize the documents or materials produced by SHAA? 2. To what extent did students, teachers, school directors, and/or parents utilize the materials found in the specialized libraries? 3. What learning gains did visual impaired or blind students demonstrate after the RAPID activity was implemented? Data collected from Site Observation Tool Content Analyses Overall, what intended and unintended contributions, results and/or outcomes has the RAPID Activity achieved relative • Site Visit • Document review • Focus groups • Key Informant Interviews Parents, students, teachers, School Directors, CNEH 1. To what extent did the RAPID activity successful meet their intended objectives? 2. Did the documents, materials, and specialized libraries Content Analyses Descriptive Analyses (if quantitative data is available) T- Test (if 23 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV to improving inclusive access to educational opportunities for disadvantaged children and youth who are blind or visually impaired? meet their intended objectives? 3. To what extent did RAPID contribute to improved inclusive access to quality educational opportunities for children who are visually impaired and blind? 4. What were some unintended outcomes of RAPID activity? quantitative data is available) 24 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV ANNEX E: LIST OF PERSONS INTERVIEWED Organization/Group Names/Identity Location CNEH Madame Magalie Georges Port-au-Prince Director, SHAA Mr. Dr Michel Péan Port-au-Prince Project manager, SHAA Mr. Josué Joseph Port-au-Prince School Director, Cap-Haitien Mr. Camille Fritz Cap-Haitien Education Advisor- USAID Education Office Tamara Jacques Remote call Teachers 7 males, 4 females Port-au-Prince/Cap- Haitien Students 13 males, 10 females Port-au-Prince/Cap- Haitien Parents 8 females Port-au-Prince/Cap- Haitien 25 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV ANNEX F: INFORMED CONSENT SCRIPTS & DATA COLLECTION PROTOCOLS Informed Consent Agreement - KII (over 18 years of age) Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is conducting an evaluation of a USAID/Haiti funded project known as Reading for All: Promoting Inclusive Education (RAPID) Activity which has been implemented by The Haitian Society of Help for and to the Visually Impaired (SHAA) since 2016. The RAPID Activity aims to increase the number of children and youth that are visually impaired and blind in mainstream classes. RAPID also provided educational services and specialized library services for individuals who are visually impaired or blind. The project is expected to end in 2018. The final performance evaluation focuses on how RAPID has been implemented, what it has achieved, and whether expected results have occurred according to the activity’s design and implementation. Findings, conclusions, and recommendations from this final performance evaluation will be incorporated into USAID/Haiti’s short, medium, and long-term program planning and execution. You have been asked to participate today so that we can learn more about your experiences working with SHAA on activities related to RAPID or activities related to the visually and blind population in Haiti. These activities may have been in the form of meetings, events, conferences held in conjunction with SHAA regarding the visually impaired and blind population. You were selected for this interview because of your knowledge of the Activity and because the evaluation team believes that you can offer a unique opinion on the activity. We are speaking stakeholders who collaborated with SHAA in that aspect. We would like your honest impressions, opinions and thoughts about various issues related to this activity’s implementation and outcomes. We are independent consultants who have no affiliation with those who implemented RAPID nor do we represent the GOH. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities and services implemented under the RAPID program. The interview will take about 1 hour of your time. Although we will publish our findings in a public report, personally identifiable information will be kept confidential to the fullest extent permitted by local law, US law, and USAID policy. Nothing you tell the ET will be attributed to any individual person. Rather the report will include only a composite of all of the answers received by all of the individuals we interview. Although we may use quotes, none of the individuals interviewed will be named in the report. Only the research team, Q- Q Research, will have access to the information that you provide to us today. However, USAID may share the final report with SHAA and other relevant stakeholders. The interview will be recorded, and this consent includes permission f or the discussion to be audio- recorded. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this interview. Although this interview will not benefit you personally, we hope that our results will help improve potential future institutional capacity building programs for government institutions in Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question that you do not wish to answer. Your participation and answers to any question have no bearing on any services that you currently have access to nor any bearing on access to other services in the future. Do you have any questions at this time? [Interviewer should answer any questions] 26 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Permission to Proceed: I understand the purpose of the discussion as outlined above and understand that I can withdraw from the discussion at any time and for any reason without penalty. I agree to participate in the discussion. If you have questions or concerns please contact Antoine Wesner at (509) 3698-3754. You may also contact Social Impact headquarters staff at 1 -703-465-1884 or irb@socialimpact.com. Yes  No Signature verifying the respondent understands the purpose of the study and agrees to participate (consents): _____________________ Date _________________________ 27 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV A. KII Protocol for SHAA Staff Interview 1. Gender 2. Date: 3. Interview location: 4. Interviewer Name: 5. Primary Notetaker Name: 6. Respondent Name: 7. Respondent Title: 8. Respondent Institution: 1. Please describe how parents, students, and teachers access resources provided through the RAPID Activity? a. How were parents, students, and teachers made aware of resources? b. To what extent did parents, students, and teachers access resources? c. Were there any barriers to accessing resources? If so, what were they? 2. How did parents, students, and teachers in need of additional services, access services through RAPID? 3. To what extent did parents utilize the specialized libraries? a. Were there any barriers to accessing specialized libraries? If so, what were they? 4. Please describe the teacher trainings. a. Who participated? b. What was the level of participation? c. How many were implemented? d. What was cover during the trainings? e. Where and when did the trainings occur? 5. Please describe the train-the trainer computer sessions. a. Who participated? b. What was the level of participation? c. How many were implemented? d. What was cover during the trainings? e. Where and when did the trainings occur? 6. What were the best components of the RAPID Activity? What were some components that could have benefitted from improvement? 28 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV B. KII Protocol for School Directors 1. Gender: 2. Date: 3. Interview location: 4. Interviewer Name: 5. Primary Notetaker Name: 6. Respondent Name: 7. Respondent Title: 8. Respondent Institution: 1. Please provide an overview of the services provided to your school as a result of participation in the RAPID Activity. 2. To what extent did the services provided through the RAPID Activity meet your expectations/needs? What were the factors that contributed to and/or limited the RAPID Activity from meeting your expectations? 3. Did your teachers participate in any workshops or trainings provide through the RAPID Activity or facilitated by SHAA? a. If so, please describe the workshops or trainings . i. To what extent did the workshops or trainings meet your expectations/needs? ii. What were the factors that contributed to and/or limited the RAPID Activity from meeting your expectations? iii. To what extent did teachers implement strategies learned during teacher trainings in their classrooms? 4. Did your teachers access resources provided by SHAA or through the RAPID Activity? a. If so, what resources did they access? How did SHAA staff communicate the resources available to you and your staff? b. If your staff did not access resources, why not? What could SHAA have done to make accessing the resources easier for you and your staff? 5. Did you receive any of the educational support provided through the RAPID Activity? a. If so, what were they? To what extent did the educational support meet your expectations/needs? What were the factors that contributed to and/or limited the RAPID Activity from meeting your expectations? 6. Were there any visually impaired or blind students integrated into your school? a. If yes, how many? i. Did you receive any educational support from the RAPID Activity to ensure a successful integration of a visually impaired or blind student? ii. What educational support, if any, you did not receive that would have been beneficial for the integration of the visually impaired or blind student in your school? b. If students were not successful integrated, what are some possible reasons why? c. Are there services you did not receive from SHAA that would have been beneficial to the visually impaired and blind students integrated into your school? 7. Were you aware of the specialized libraries offered by SHAA? a. If yes, did your teachers or your visually impaired or blind student access the libraries? b. If not, why not? What could have made accessing the libraries easier for your teachers or students? 29 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV 8. What are some recommendations you have that if implemented, may improve a program like RAPID? 9. What were the best components of the RAPID Activity? What were some components that could have benefitted from improvement? 30 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV C. KII Protocol for CNEH Interview 1. Gender: 2. Date: 3. Interview location: 4. Interviewer Name: 5. Primary Notetaker Name: 6. Respondent Name: 7. Respondent Title: 8. Respondent Institution: 1. Please describe the role/mission of your organization and your main activities in Haiti. 2. Please describe your involvement (if any) with SHAA and the RAPID Activity. 3. Please describe how you were informed about the RAPID Activity. What is your understanding of inclusive education? 4. Are you involved in promoting inclusive education? Have you participated in any of the activities i.e. training, meeting, workshop, or events organized by SHAA or through the RAPID activity? 5. What is your perception of the activities implemented through the RAPID Activity? 6. To what extent did the work of SHAA and the RAPID Activity coordinate with or complement the work of your organization? 31 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV D. KII Protocol for USAID RAPID Agreement Officer Representative 1. Gender: 2. Date: 3. Interview location: 4. Interviewer Name: 5. Primary Notetaker Name: 6. Respondent Name: 7. Respondent Title: 1. To what extent do you think the RAPID Activity successfully implement the project activities? a. What were some of the strengths of the implementation of RAPID Activity activities? What were some weaknesses? b. What, if anything, might have been done better? c. Was the level of communication with you/your office adequate? d. Were activities carried out in a timely manner? e. To what extent were RAPID managers able to adapt and find solutions to unexpected challenges? 2. What factors contributed to the achievement of the activity goals and objectives? Which factors contributed to RAPID activity goals and objectives not met in Year 1 and 2? 3. What were some unintended outcomes of the RAPID Activity? 32 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Informed Consent Agreement – Teachers/Parent/Student Above 18 years old - FGD Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is conducting an evaluation of a USAID/Haiti funded project known as Reading for All: Promoting Inclusive Education (RAPID) Activity which has been implemented by The Haitian Society of Help for and to the Visually Impaired (SHAA) since 2016. The RAPID Activity aims to increase the number of children and youth that are visually impaired and blind in mainstream classes. RAPID also provided educational services and specialized library services for individuals who are visually impaired or blind. The project is expected to end in 2018. The final performance evaluation focuses on how RAPID has been implemented, what it has achieved, and whether expected results have occurred according to the activity’s design and implementation. Findings, conclusions, and recommendations from this final performance evaluation will be incorporated into USAID/Haiti’s short, medium, and long-term program planning and execution. You have been asked to participate in a focus group because you or your child (if a parent) participated in at least one RAPID Activity. We would like to learn more about your experiences with the services you received through the RAPID Activity. We are speaking stakeholders who participated in the program either as implementers or recipients of program services. We would like your honest impressions, opinions and thoughts about various aspects related to this activity’s implementation and outcomes. We are independent consultants who have no affiliation with those who implemented RAPID nor do we represent the GOH. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities and services implemented under the RAPID program. The FGD will take up to two hours of your time. Although we will publish our findings in a public report, all your answers will be kept confidential. Personally identifiable information will be kept confidential to the fullest extent permitted by local law, US law, and USAID policy. Nothing you tell the ET will be attributed to any individual person. Although we may use quotes, none of the individuals in the foc us group will be named in the report. Only the research team, Q-Q Research, will have access to the information that you provide to us today. However, USAID may share the final report with SHAA and other relevant stakeholders. As this is a group setting, to preserve confidentiality, we ask you not to share anything we discuss here today with anyone outside of this group. The FGD will be recorded and this consent includes permission for the discussion to be audio-recorded. Risks/Benefits: There are no significant risks to your participation in this focus group . You will not receive any direct benefit or compensation for participating in this focus group . Although this study will not benefit you personally, we hope that our results will help improve potential future institutional capacity building programs for government institutions in Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question that you do not wish to answer. Your participation and answers to any question have no bearing on any services that you currently have access to nor any bearing on access to other services in the future. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed: I understand the purpose of the discussion as outlined above and understand that I can withdraw from the discussion at any time and for any reason without penalty. I agree to participate in the discussion (Evaluator records if visually impaired youth verbally assents). If you have 33 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV questions or concerns, please contact Antoine Wesner at (509) 3698- 3754. You may also contact Social Impact headquarters staff at 1-703-465-1884 or irb@socialimpact.com . Yes  No Signature indicates respondent understands the purpose of the study and agrees to participate (consents): __________________________________________ _________________________ Participant Signature Date ___________________________________________ Print participant name Signature or mark of witness (for 18+ year, old visually impaired adult), verifying the respondent understands the purpose of the study and agrees to participate (consents): _____________________ Date _________________________ 34 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Informed Consent Agreement – Parent Consent for Student under 18 Years old - FGD Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is conducting an evaluation of a USAID/Haiti funded project known as Reading for All: Promoting Inclusive Education (RAPID) Activity which has been implemented by The Haitian Society of Help for and to the Visually Impaired (SHAA) since 2016. The RAPID Activity aims to increase the number of children and youth that are visually impaired and blind in mainstream classes. RAPID also provided educational services and specialized library services for individuals who are visually impaired or blind. The project is expected to end in 2018. The final performance evaluation focuses on how RAPID has been implemented, what it has achieved, and whether expected results have occurred according to the activity’s design and implementation. Findings, conclusions, and recommendations from this final performance evaluation will be incorporated into USAID/Haiti’s short, medium, and long-term program planning and execution. Your child has been selected to participate in a focus group discussion because he/she participated in at least one RAPID Activity. We would like to learn more about his/her experiences with the services received through the RAPID Activity. We are independent consultants who have no affiliation with those who implemented RAPID nor do we represent the GOH. Procedures: If you agree to allow your child to participate, he/she will participate in a FGD and will be asked to discuss his/her experience and opinion of the activities and services implemented under the RAPID program. The FGD will take up to 2 hours . Although we will publish our findings in a public report, all of responses will be kept confidential. Responses will not be attributed to any individual person. Although we may use quotes, none of the individuals in the focus group will be named in the report. However, as this is a group setting, to preserve confidentiality, we ask them not to share anything we discuss with anyone outside of this group. The FGD will be recorded and this consent includes permission for the discussion to be audio-recorded. Risks/Benefits: There are no significant risks to your child’s participation in this focus group. You will not receive any direct benefit or compensation for your child’s participation in the focus group. Although this study will not benefit you personally, we hope that our results will help improve potential future institutional capacity building programs for government institutions in Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question that you do not wish to answer. Your participation and answers to any question have no bearing on any services that you currently have access to nor any bearing on access to other services in the future. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the focus group as outlined above and understand that I can withdraw my child from the focus group at any time and for any reason without penalty. I agree to allow my child to participate in the focus group (Evaluator records). If you have questions or concerns please contact Antoine Wesner at (509) 3698- 3754. You may also contact Social Impact headquarters staff at 1-703- 465-1884 or irb@socialimpact.com. Under 18-year-old respondent verbally assents to participate. Yes  No 35 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV Signature indicates respondent understands the purpose of the study and agrees to grant permission allowing their child to participate (assents): __________________________________________ _________________________ Parent Signature Date ___________________________________________ Parent name Signature or mark of witness, verifying the respondent understands the purpose of the study and a to grant permission allowing their child to participate (assents): _____________________ Date _________________________ 36 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV A. Focus Group Guide for Parents/Students over 18 years old 1. Please provide an overview of the services you (your child/ren) received because of participation in the RAPID Activity? 2. How were you recruited for participation in the RAPID Activity? 3. What was your level of satisfaction with the recruitment process? a. What were the strengths and weaknesses of the recruitment process in terms of identifying visually impaired and blind individuals? What, if anything, might have been done better? 4. To what extent did the services provided through the RAPID Activity meet your expectations/needs? What were the factors that contributed to and/or limited RAPID Activity from meeting your expectations? 5. Did you (or your chil d/ren) access resources provided by SHAA or through the RAPID Activity? a. If so, what resources did you (your child/ren) access? How did SHAA staff and/or the school communicate the resources available to you (your child/ren)? b. If not, why not? What could SHAA have done to make accessing the resources easier for you (your child/ren)? 6. If you needed additional services through RAPID to help you (your child/ren), would you be able to access these services? a. If yes, then how? 7. Are you aware of any of the educational support provided through the RAPID Activity at your (your child/rens’) school? a. If so, what are they? What was your level of satisfaction with these services offered? 8. What are some services you (your child) did not receive from SHAA that would have been beneficial? 9. Were you aware of the specialized libraries offered by SHAA? a. If yes, did you (your child/ren) access the libraries? b. If not, why didn’t you (your child/ren) access the libraries? What could have made accessing the libraries easier for you? 10. What are some recommendations you have that if implemented, may improve a program like RAPID? 11. What were the best components of the RAPID Activity? What were some components that could have benefitted from improvement? 12. Parents Only: What is your understanding of inclusive education? a. What are your thoughts on inclusive education? b. Have you ever been involved in any activities to promote inclusive education? 37 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV B. Focus Group Guide for Students under 18 years old 1. Did you participate in any of the RAPID activities? a. If so, what did you participate in? 2. How did you get to be a part of the RAPID Activity? 3. Were the services provided through the RAPID Activity what you expected? If it was not what you expected, why not? 4. Did you access resources provided by SHAA or through the RAPID Activity? a. If so, what resources did you access? How did you find out about the resources? b. If not, why not? What could SHAA have done to make accessing the resources easier for you? 5. If you needed additional services through RAPID, would you be able to access these services? a. If yes, then how? 6. Are you aware of any of the educational support provided through the RAPID Activity at your school? a. If so, what are they? What was your level of satisfaction with these services offered at your school? 7. What are some services you did not receive from SHAA that would have been beneficial? 8. Were you aware of the specialized libraries offered by SHAA? a. If yes, did you access the libraries? b. If not, why didn’t you access the libraries? What could have made accessing the libraries easier for you? 9. What are some recommendations you have that if implemented, may improve a program like RAPID? 10. What were the best components of the RAPID Activity? What were some components that could have benefitted from improvement? 38 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV C. Focus Group Guide for Teachers 1. Please provide an overview of the services you received as a result of participation in the RAPID Activity. 2. To what extent did the services provided by through the RAPID Activity meet your expectations/needs? What were the factors that contributed to and/or limited RAPID Activity from meeting your expectations? 3. Did you participate in any workshops or trainings provide through the RAPID Activity or facilitated by SHAA? a. If so, please describe the workshops or trainings . i. To what extent did the workshops or trainings meet your expectations/needs? ii. What were the factors that contributed to and/or limited RAPID Activity from meeting your expectations? iii. To what extent did you utilize strategies learned during teacher trainings ? iv. To what extent did these strategies result in the successful integration of visually impaired or blind students into mainstream classes? b. If you did not participate, why not? What could have been done to increase the likelihood that you would have participated? 4. Did you access resources provided by SHAA or through the RAPID Activity? a. If so, what resources did you access? How did SHAA staff and/or the school communicate the resources available to you? b. If you did not access resources, why not? What could SHAA have done to make accessing the resources easier for you? 5. Did you receive any of the educational support provided through the RAPID Activity at your school? a. If so, what were they? To what extent did the educational support meet your expectations/needs? What were the factors that contributed to and/or limited RAPID Activity from meeting your expectations? 6. Were there any visually impaired or blind students integrated into your classroom? a. If yes, how many? i. Did you receive any educational support from the RAPID Activity to ensure a successful integration of a visually impaired or blind student? ii. What educational support, if any, you did not receive that would have been beneficial for the integration of the visually impaired or blind student in your class? b. Are there any services you did not receive from SHAA that would have been beneficial to the visually impaired and blind students integrated into your class? 7. Were you aware of the specialized libraries offered by SHAA? a. If yes, did you or your visually impaired or blind student access the library? b. If not, why not? What could have made accessing the library easier for you or your student? 8. What are some recommendations you have that if implemented, may improve a program like RAPID? 9. What were the best components of the RAPID Activity? What were some components that could have benefitted from improvement? 39 | USAID/HAITI RAPID ACTIVITY EVALUATION DESIGN USAID.GOV ANNEX G: DISCLOSURE OF CONFLICT OF INTEREST