Evalyasyon Final Pèfòmans Pwojè USAID Dlo ak Sanitasyon

[page 1] Cr : Voseé” FROM THE AMERICAN PEOPLE \ RTE. > = UD \\ - F3 6 à ‘À ER ! Ë } À \l | LAN nr =] PHOTO CREDIT: DAN O'NEILL, DAI HAITI Haiti Evaluation and Survey Services Project (ESS) July 21, 2022 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. [page 2] USAID/HAITI FINAL PERFORMANCE EVALUATION OF USAID WATER AND SANITATION (WATSAN) PROJECT FINAL REPORT Date: July 21, 2022 Submitted to USAID/Haiti Contract Number: AID-521-C-17-00002 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. Contact: Salima Mutima, Chief of Party 2300 Clarendon Blvd, Suite 1000 Arlington, VA 22201, USA DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. [page 3] ACKNOWLEDGEMENTS The evaluation team (ET) would like to thank the current and former staff at the prime implementer Development Alternatives International (DAI) and select implementing partners mWater, V3 Engineering, Ayiti Nexus, Centre et Formation et d’'Encadrement, and Zanmi Lasante for their participation and provision of documentation and valuable feedback. The ET also appreciates Chase’s provision of local data collection services. The ET also thanks the staff at USAID, including Raina Zantout and Marcia Urquhart Glenn, for their support and feedback throughout the entire evaluation process. Finally, the ET would like to thank the many stakeholders who spoke with us about the WATSAN/USAID activity. The support and engagement of these organizations and stakeholders proved invaluable during the evaluation process. [page 4] TABLE OF CONTENTS ABBREVIATIONS AND ACRONYMS [LL ABSTRACT VII EXECUTIVE SUMMARY VIII INTRODUCTION I BACKGROUND I HAIÏTPS WATER AND SANITATION STATUS I USAID WATSAN ACTIVITY GOAL AND OBJECTIVES 2 USAID WATSAN ACTIVITY INTERVENTIONS AND PHASES 3 PROJECT PERFORMANCE SUMMARY 4 EVALUATION QUESTIONS 5 EVALUATION QUESTION I 5 EVALUATION QUESTION 2 5 EVALUATION QUESTION 3. 5 METHODOLOGY 5 DATA COLLECTION 6 DATA ANALYSIS 9 FINDINGS AND CONCLUSIONS 12 EVALUATION QUESTION I (WATER SUPPLY). 12 EVALUATION QUESTION I (SANITATION). 24 EVALUATION QUESTION 2 29 EVALUATION QUESTION 3 35 CROSS-CUTTING ISSUES 38 RECOMMENDATIONS 42 USAID/HAITI 42 USAID IMPLEMENTERS 42 ANNEX A. SCOPE OF WORK 44 ANNEX B. DOCUMENTS REVIEWED 60 ANNEX C. KIIS/GIS/FGD EVENT LIST 63 ANNEX D. MWATER SECONDARY DATA REVIEW 67 i | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 5] ANNEX E. CTE STAFF SURVEY RESULTS 82 ANNEX F. 2021 CTE CUSTOMER SURVEY SECONDARY DATA REVIEW 90 ANNEX G. LIST OF STAKEHOLDERS CONSULTED 97 ANNEX H. WATSAN PROJECT METHODOLOGY 101 ANNEX I. DATA COLLECTION INSTRUMENTS 103 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | ii [page 6] TABLES AND FIGURES TABLE l: SUMMARY OF PERFORMANCE INDICATORS FROM THE PROJECT RESULTS FRAMEWORK 4 TABLE 2: KIIS, GIS, AND FGDS 7 TABLE 3: CTE STAFF SURVEY DATA COLLECTION 8 TABLE 4: MWATER INDICATORS AND METRIC CATEGORIES 12 FIGURE |: THE ‘“SPIRAL OF DECLINE” LEADING TO POOR PERFORMANCE (AFTER GALAITSI 2016) 13 FIGURE 2. THE DATA VALUE CHAIN MODEL FOR TURNING DATA INTO USEFUL INFORMATION 14 FIGURE 3: CTE STAFF EXPERIENCE OF USING MWATER 15 FIGURE 4: CTE EXPENSES DATA FROM MWATER (6 MONTH MOVING AVERAGE) 16 FIGURE 5: CTE REVENUE DATA FROM MWATER (6 MONTH MOVING AVERAGE) 16 FIGURE 6: CTE OPERATING RATIO DATA FROM MWATER (6 MONTH MOVING AVERAGE) 17 FIGURE 7: CTE STAFF EXPERIENCES OF USING QUICKBOOKS ACCOUNTING SOFTWARE 18 FIGURE 8: SAEP WATER SUPPLY CONTINUITY (6-MONTH MOVING AVERAGE) 19 FIGURE 9: CTE RESIDUAL CHLORINE CONFORMITY 19 FIGURE 10: FUNCTIONALITY OF CTE MANAGED KIOSKS (6 MONTH MOVING AVERAGE) 20 FIGURE I 1: HOUSEHOLD WATER ACCESS FROM HOUSEHOLD CONNECTIONS AND KIOSKS: COVERAGE BY CTE 20 FIGURE 12. CTE ACTIVE SUBSCRIBERS (6 MONTH MOVING AVERAGE) 21 FIGURE 13: CTE COLLECTION EFFICIENCIES (6 MONTH MOVING AVERAGE) 21 äüi | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 7] FIGURE 14: CTE STAFF EXPERIENCES OF USING SIGA 22 TABLE 5: EAF GRANTS AND CURRENT STATUS 30 FIGURE 15: PROJECT ISSUES TIMELINE 38 FIGURE 16: SEXUAL HARRASSMENT TRAINING 39 FIGURE 17: CTE’S MANAGEMENT OF CTE’S GENDER ISSUES BY SEX 39 FIGURE 18: CTE’S MANAGEMENT OF COMMUNITY GENDER ISSUES 40 TABLE 6: DATA OUTLIERS 67 TABLE 7: AVAILABLE DATA 68 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | iv [page 8] ABBREVIATIONS AND ACRONYMS BSFL Black Solder Fly Larvae CFET Centre de Formation et d’Encadrement Technique COP Chief of Party CTE Centre Technique d’Exploitation/Technical Centre of Exploitation DAI Development Alternatives Incorporated DCOP Deputy Chief of Party DINEPA National Directorate of Potable Water and Sanitation EAF Enterprise Acceleration Fund EQ Evaluation Question ESS Evaluation and Survey Services ET Evaluation Team ETS Ecole de Technologie Superieure FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion FSM Fecal Sludge Management GI Group Interview GOH Government of Haiti HH Household HTG Haitian Gourde IDB Inter-American Development Bank IP Implementing Partner IQR Interquartile Range IRB Internal Review Board KI Key Informant Interview LEA Latrine Emptier Association LWrs Living Water International LOP Life of Project LSPV Projet de Lits de Séchage Plantés de Végétaux MEL Monitoring, Evaluation, and Learning MFSN Mouvman Fanm Soléy No MSME Micro, Small, or Medium-Sized Enterprises MTF Mayoral Task Force NGO Non-Governmental Organization OFATMA Office d'Assurance Accidents du Travail, Maladie et Maternité ONA Office National d’Assurance-Vieillesse v | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 9] ONEPA Observatoire National de l'Eau Potable et de l'Assainissement OREPA Office Régional de l'Eau Potable et d’Assainissement SAEP Système d’alimentation d’eau potable / drinking water supply system SDG Sustainable Development Goal SI Social Impact SIEPA Système intégré d’Information sur l'Eau Potable et l’Assainissement SIGA Système Intégré de Gestion des Abonnés / CTE customer database SISKLOR Système de Surveillance du Chlore Residuel / Residual Chlorine Surveillance System SME Small and Medium Sized Enterprises TA Technical Assistance TL Team Leader USAID United States Agency for International Development UEH State University of Haiti UniQ University Quisqueya USD US Dollars WASH Water, Sanitation, and Hygiene WATSAN Water and Sanitation MWBUTF World Bank Utility Turnaround Framework USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | vi [page 10] ABSTRACT This evaluation used a mixed-methods methodology to examine the design, approaches, performance and outcomes of the United States Agency for International Development (USAID) Water and Sanitation (WATSAN) Project between December 2017 and December 2021. Project objectives included supporting access to sustainable water supply (target 250,000 people) and sanitation services (target 75,000 people) and strengthening the foundational capacity of water and sanitation service providers for sustainable service delivery. Project targets were mostly met (see Table | in the background section). Key findings for the water sector included that the Centres Technique d'Exploitation (CTES) are better able to manage their businesses using a data-driven approach via the Système intégré d'Information sur l'Eau Potable et l’Assainissement and the mWater platform. Billing and customer service are improved, though some challenges remain with continuity of CTE data. USAID’s approach and these achievements with the CTEs have provided a foundation for sustainable service delivery. Key findings for the sanitation sector included two fecal sludge management sites now operating safely with business plans in place in Les Cayes and Port-au-Prince. Moreover, the Project’s support to the Latrine Emptying Associations and small and medium sized enterprises has empowered them to deliver improved sanitation services. Coordination support to the Mayoral Task Forces on Sanitation has increased access to sanitation services. Recommendations are made on how to capitalize on the Project’s successes, including continued capacity building, improved data management and integration of data platforms, setting service delivery targets, and establishing regulatory frameworks with enforcement for the sanitation sector. vii | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 11] EXECUTIVE SUMMARY OVERVIEW AND PURPOSE United States Agency for International Development (USAID) Haiti requested that Social Impact’s (SI) Evaluation and Survey Services (ESS) activity design and conduct an independent final performance evaluation of the USAID Water and Sanitation (WATSAN) Project (the Project) implemented by (Development Alternatives Incorporated (DAI) Global, LLC, with implementing partners (IPs) V3 Engineering, Ayiti Nexus, mWater, Centre et Formation et d'Encadrement, and Zanmi Lasante across eight departments in Haiti. The purpose of this evaluation was to determine the effectiveness of the Project in achieving its objectives in order to inform future USAID/Haiti programming similar in scope or approach. The primary audience for the evaluation is USAID, with secondary audiences being Project implementing partners, the Haitian National Directorate of Potable Water and Sanitation (DINEPA), the Government of Haiti (GOH), Haitian private sector entities and donors. This report presents the evaluation methodology, findings, and conclusions for each evaluation question, followed by recommendations. EVALUATION SCOPE AND QUESTIONS The evaluation focused on the ten communes targeted over the life of the Project. For the water sector, the target communities for the evaluation were the customers and staff of the water utility management structures (the Centre Technique d'Exploitation or CTEs) and the governmental and non-governmental organizations (NGOs) supporting their service delivery at either the household or water kiosk level. Additionally, two Enterprise Acceleration Fund (EAF) grants supported water projects (water quality testing and water kiosks). For the sanitation sector, the target communities were the staff and users of the two fecal sludge management (FSM) sites at Morne a Cabri and Fonfred, Latrine Emptying Associations (LEAs), the Mayoral Task Force (MTF) on Sanitation, and small and medium sized enterprises (SMEs) involved in household latrine construction. Additionally, five EAF grants supported sanitation projects. The target populations were 250,000 people for water supply and 75,000 for sanitation. This evaluation addressed the following evaluation questions (EQs): 1. To what extent has WATSAN met its Task Order objectives to build governance capacity at multiple levels (national, regional, local) to improve sustainable water supply and sanitation service delivery? 2. How was the Enterprise Acceleration Fund utilized, and to what extent did those grants support WASH enterprises (such as micro, small, and medium sized enterprises or MSMEs, NGOs, and Bayakous) to move toward sustainable service delivery? 3. How effective was WATSAN’s approach of targeting the end of the sanitation value chain in improving the overall sanitation value chain?! ! This may also be referred to as the sanitation service chain. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | viii [page 12] METHODOLOGY The evaluation used a mixed-methods approach consisting of a document review, 55 key informant interviews (KIl) and group interviews (Gl), six focus group discussions (FGDs), 74 CTE staff surveys and secondary data review of mWater CTE indicator data, DINEPA FSM sites operational data, EAF project performance data, and a CTE customer satisfaction survey conducted in 2021. The Evaluation participants included CTE staff and managers, CTE liaison officers, CTE kiosk managers, members of the LEAs and MTFs, Observatoire National de l'Eau Potable et de l'Assainissement (ONEPA) and Office Régional de l'Eau Potable et d’Assainissement (OREPA) staff, staff of SMEs, EAF grantees, FSM managers and users, staff of implementing partners, and USAID staff. FINDINGS AND CONCLUSIONS EQI: To what extent has the Project met its Task Order to build governance capacity at multiple levels (national, regional, local) to improve sustainable water supply and sanitation service delivery? FINDINGS (WATER) Organization and Strategy Multiple respondents from the CTESs and [Ps reported that before the Project, all CTEs were insolvent and unable to plan and monitor progress, revenues, and expenditures. The | | mWater key performance indicators (see Table 4 on page 12) used by the Project CTEs have since been adopted by all 29 CTESs, including Pignon drinking water supply system (Système d’Alimentation d'Eau Potable or SAEP). The indicators have underpinned CTE business development across the five performance metrics of Organization and Strategy, Technical Operations, Commercial Operations, Financial Operations and Human Resource (HR) Management, as outlined below. Technical Operations Water supply service continuity across the Project CTESs is at between five and 20 hours/week, with Pignon at around 80 hours/week. An exception is Ouanaminthe, whose continuity in October 2020 leapt from five to 168 hours/week (24/7 supply), where it has remained since, as their management of the system improved. However, there are no established/defined water supply delivery standards for the CTESs, reference to the Sustainable Development Goals (SDGs) or identified milestones to reach them. In Ouanaminthe, use of the mWater mobile app which the Project supported means monthly meter readings were done in ten days instead of taking over a month. There remains variation in reporting between mWater indicators and CTESs in terms of regularity e.g., conformity to residual chlorine levels monitored through residual chlorine surveillance system (Système de Surveillance du Chlore Residuel or SISKLOR) was reported very irregularly in some CTESs. Repairs to CTE-managed water supply infrastructure has been limited by a lack of comprehensive and accessible technical details sufficient to locate the affected water supply infrastructure, but mWater has recently launched a mapping program to address this issue. At the kiosk level, mWVater data shows that CTE-managed kiosks have erratic and variable levels of functionality. There are challenges with the interconnectedness of the various software packages the CTESs use. ix | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 13] Commercial Operations CTE revenues have increased, the largest being Mirebalais where revenue increased from 148,000 Haitian Gourdes (HTG) in July 2019 to 805,000 HTG in January 2022 while revenues doubled for Hinche, Cap Haitien, and Belladere over the same period. CTE FGDs discussed how revenues have increased? as tariffs have been raised and there is now better billing, revenue collection, and customer service using the Système Intégré de Gestion des Abonnés (SIGA, the CTE customer database) and mWater. Key factors for CTE business sustainability and customer retention included the predictability of water supply and service continuity, together with enhanced billing and customer complaints handling. Financial Operations In 45 percent of cases, the operating ratio (expenses/revenue) was higher than 100 percent for the Project’s CTES, indicating that expenses were higher than the revenue for close to half of the reported months. À number of interviews with both the donor and the prime contractor indicated that although revenues have increased, “fear of success” was very real. CTESs are not audited, and they have concerns around how they will manage their increased revenues and how to effectively manage expenditure of those funds for future capital infrastructure projects/improvements. Some CTESs are relatively new to using QuickBooks accounting software and are not yet confident in using it. There is no network connection from QuickBooks to the SIEPA (Système Intégré d'Information sur l’Eau Potable et l’Assainissement) system. HR Management The evaluation included a survey of 74 CTE staff (Annex E) with more than 50 percent of respondents having worked at the CTE for four to five years. The survey results, when triangulated with CTE staff interviews and FGDs with a number of CTE stakeholders, found that although CTE staff capacity to use mWater and SIGA software improved over the Project period, enabling better service and billing, there Was variation in that capacity i.e. not everyone can use the data for planning purposes. See Annex E and main report for full details. CONCLUSIONS (WATER) The Project has brought data-driven planning and decision-making to each CTE for the first time. The Project enabled the CTESs to grow their customer bases and revenues and increase their business capacity. The project has significantly contributed to the sustainability of the CTESs and reversed the spiral of business decline seen before the Project started. There remains room for improvement in the areas of technical data management for operation and maintenance, consistent and timely reporting of mWater data, SISKLOR data reporting, effective management of future capital expenditure on infrastructure and for some CTESs their capacity to use QuickBooks confidently. FINDINGS (SANITATION) The Project worked with a number of stakeholders in the sanitation sector, including the FSM sites, SMEs/pit emptiers, LEAs, and MTFs. The Project was able to help the SMEs increase capacity, especially 2 All figures relating to CTE performance in this report have come from the secondary data review of mWater data. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | x [page 14] regarding management of the business (marketing, human resources, contracts, and budgets) through training and encouraged the construction of household toilets and latrines. The Project improved the infrastructure and management capacity of the Fonfred FSM and the Morne a Cabri FSM sites and supported better management practices. Both sites are currently accepting sludge from the surrounding communities, which has reduced complaints about the sites, but there is concern regarding long-term sustainability, especially regarding issues with transportation and not receiving enough sludge to be profitable. CONCLUSIONS (SANITATION) The pit-emptiers were empowered by the work of the Project. This led to them marketing their services, developing official associations, and working together. By working with multiple sanitation stakeholders, the Project was able to assist in the construction of latrines and toilets at the household level, increase the role of the MTFSs in their awareness-raising activities regarding the importance and impact of having a latrine, and increase the management and business capacities of the SMEs, LEAs, and FSMs. Through this increased capacity, they are now better able to provide more sanitation services to the communities. EQ 2: How was the enterprise acceleration fund utilized, and to what extent did those grants support WASH enterprises (such as MSME, NGOs, Bayakous) to move toward sustainable service delivery? FINDINGS (EAF) DA administered seven EAF grants during the Project, four in the sanitation sector and three in the water sector. The grants included micro credit for toilet construction, construction of 14 kiosks, research activities by University Quisqueya (UniQ) on sludge drying and testing, purchase of equipment to improve incineration services, pilot trials of black soldier fly larvae waste to value end products by SOIL and establishing a water testing laboratory at Limonade University. The kiosks provided access to water to 24,000 people, while the micro grants for latrines and toilets provided access to credit for 100 people. About half of the grantees said they did not feel involved in the Project decision-making and use of funds, and many grantees had issues with implementation timelines due to delays in the arrival and quality of equipment. CONCLUSIONS (EAF) The grants were successful in terms of increasing the number of clients for the organizations, both in terms of access to markets and the range of services offered. In addition, the lab at Limonade University can now provide water quality testing in addition to soil testing. The research activities for SOIL and UniQ proved to be a success as activities have been able to finally start. Two of the grants have ongoing issues due to delayed equipment, and one grant has an issue with equipment not functioning. The grantees shared how they were able to expand their services both in terms of the number of clients and types of services provided (but due to delays, this has not happened for all grantees). Due to delays and some challenges, it is difficult to determine if grants improved the sustainability of the organizations. xi | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 15] EQ 3: How effective was the Project’s approach of targeting the end of the sanitation value chain in improving the overall sanitation value chain? FINDINGS (SANITATION VALUE CHAIN) Before the Project, the Fonfred FSM site in Les Cayes was not operational. The Morne a Cabri site for the Port-au-Prince area was open and accepting waste, but the treatment basins were full and the site was not properly managed. The Project has enabled both sites to now provide a safe and controlled disposal area for sludge waste and illegal sludge dumping has been reduced and the end of the sanitation value chain is now stronger. The FSM sites are now operating with a business model but they are not yet profitable and a key bottleneck is insufficient sludge transport capacity. CONCLUSIONS (SANITATION VALUE CHAIN) The Project focused on the end of the sanitation value chain with the work at the FSM sites; this has ensured the communities of Les Cayes and Port-au-Prince now have à safe fecal sludge disposal facility It is important to note that while the infrastructure and management capacity of the FSM sites improved due to the Projects activities, sustainability is still an issue as the sites have issues with transportation and do not receive enough customers/sludge each month to be profitable. There are some ongoing concerns that the sites will close without further support from the Project. PRINCIPAL RECOMMENDATIONS FOR USAID/HAITI: 1. Ensure future programming supports CTESs to establish milestones for reaching defined service delivery standards that are linked to the SDGs and tracked in mWater. 2. Engage DINEPA/GOH to enable SIEPA to have better data connectivity between the various software packages for the CTESs. 3. Continue to support FSM organizations in their efforts to reach profitability/sustainability by expanding infrastructure in order to help them increase the amount of sludge they can process. In particular, support greater sludge transportation capacity from FSM users to the FSM sites. 4. Continue to support coordination among sanitation sector stakeholders to build GOH capacity in the development and enforcement of FSM regulations. PRINCIPAL RECOMMENDATIONS FOR FUTURE USAID HAITI IMPLEMENTERS: 1. Undertake a needs assessment with CTEs on how they can grow their financial and asset management capacity to support accountable capital expenditure on infrastructure. 2. Address data continuity/quality issues for data reported on mWater. 3. Support mWater’s capacity to map all water supply infrastructure to improve CTE technical operations. 4. Develop a standardized approach for all CTE-managed kiosks that includes an indicator(s) for kiosks to supplement the current single (technical) indicator of percentage functional kiosks e.g., the hours per day of operation metric used by CTESs for household connections. 5. Use EAF-type grants to support female-lead/owned organizations and projects. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | xii [page 16] INTRODUCTION The USAID WATSAN Project is a $41.8 million, five-year activity (in total) implemented in collaboration with Haiti s DINEPA with the overall goal of improving sanitation and water for all Haitians. There was a $4.9 million and six-month extension to the Project (which will now end in December 2022) to primarily focus on helping rebuild the water systems in the South following the August 14, 2021, earthquake.3 The Project reached its goals by focusing on its core values of self-reliance, resilience, and the private- sector approach. In total, the Project focused on 12 areas: eight areas hard hit by Haitis cholera outbreak (Cap-Haitien, Ouanaminthe, Pignon, Hinche, Mirebalais, Belladere, Lascahobas, and Croix des Bouquets), two areas hard hit by Hurricane Matthew in 2016 (Les Cayes and Jeremie), and now two additional areas hit by the August 14, 2021, earthquake (Aquin and Miragoane). The first five areas that were part of the Project were Cap-Haitien, Mirebalais, Les Cayes, Jeremie, and Croix des Bouquets. This evaluation assesses the Project’s implementation in the first ten communes (Aquin and Miragoane are not part of the evaluation). BACKGROUND HAITPS WATER AND SANITATION STATUS Access to water and sanitation in Haïti is the worst of any country in the Western Hemisphere. Only 66.7 percent of the population has access to basic water sources, a slight increase since 1990, when the rate was 62 percent.5 increases in services have not been able to keep pace with population growth. However, actual access rates may be lower, as many water systems’ functionality is poor. Only 37.1 percenté of the Haitian population has access to basic sanitation, a rate that has been relatively constant since 1990. Safe collection, transport, and treatment of human excreta is practically non-existent throughout Haiti. Surveys indicate that the use of improved sanitation facilities has increased from 18 to 28 percent, meaning that much of the population still relies on shared or unimproved sanitation facilities.7 While basic sanitation access rates in urban areas are higher than the national average (37.1 percent), they are still below 50 percent.8 In informal or unplanned settlements, where the poorest and most vulnerable urban populations generally live, urbanization and high localized population density may be accompanied by an increased risk of infectious disease transmission, primarily affecting the poor. The combination of a high fecal-related disease burden and inadequate infrastructure suggests that investment in expanding sanitation access in densely populated urban slums can yield important public health gains. 3 WATSAN Quarterly Report Oct-Dec 2021 4 UNICEF/WHO Joint Monitoring Programme (2020) 5 Ibid. élbid. 7 lbid. 8 Ibid. 1 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 17] It is also important to understand the impact of gender regarding access and use of water and sanitation facilities. For example, women and girls have disproportionate labor and time burdens for household water collection, management, and treatment; also, those who travel long distances to collect water are particularly vulnerable to gender-based violence. This is discussed in detail in the USAID/Haïti Strategic Framework Gender Analysis (November 2020).° In terms of water management and governance, which the USAID WATSAN activity focuses on, women are often not in decision-making roles and are not visible in both public and private sectors.!° In addition, climate change and other natural and human-made disasters as well as unplanned urbanization threaten water resources and gains made in the water supply and sanitation sector. Building the capacity of utilities and private operators to anticipate needs, plan for and finance improvements, and expand access to underserved communities on a sustainable basis is the strategy promoted by the USAID Water and Development Plan. USAID WATSAN ACTIVITY GOAL AND OBJECTIVES The objective of the USAID WATSAN Project is to build a foundation for long-term, sustainable growth in access to safe drinking water and sanitation in Haïti, where many communities suffer from high incidences of diarrheal disease. The Project’s three primary goals are to: e Help 250,000 people get access to basic or improved water; e Help 75,000 people get access to basic or improved sanitation; and e Lay the foundation for sustainable increases in access to water and sanitation across Haïti. The Project was implemented in collaboration with DINEPA, the branch of the Haitian government tasked with ensuring water and sanitation services for its citizens. At the time of this evaluation, the USAID WATSAN Project covered implementation in 10 communes, located in six departments: Nord, Nord-Est, Centre, Sud, Grand’Anse, and Ouest. USAID WATSAN initially focused on five areas: three areas hard hit by the cholera outbreak!! (Cap- Haitien, Mirebalais, and Croix des Bouquets) and two areas hard hit by Hurricane Matthew in 2016 (Les Cayes and Jeremie). During the last two years, it added five additional areas: Ouanaminthe and Hinche in 2020, and Pignon, Lascahobas, and Belladere in 2021. USAID understands that the success of its USAID WATSAN Project rests on these prioritized values. e _Self-Reliance: The institutions must be able to generate enough revenue to fully cover their operations without any subsidies. e __Resilience: Given the volatile climate in Haïti, the institutions need to design, build, operate, maintain, and upgrade systems that can withstand disruptions and economic uncertainty. ? USAID Haiti Strategic Framework Gender Analysis (November 2020) pg. 97 10 USAID Haiti Strategic Framework Gender Analysis (November 2020) pg. 98-100 11 The cholera outbreak started in October 2010. USAID provided support that helped in eliminating cholera from Haiti in 2019. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 2 [page 18] e__ Private Sector Approach: The CTESs and fecal sludge managers must adopt a private sector, customer-driven approach. The theory of change is shown in Annex H. USAID WATSAN ACTIVITY INTERVENTIONS AND PHASES The Project’s activities are organized around three closely linked components, with specific tasks for each, shown in Annex H. Component |: Increasing access to sustainable water services e Task 1.1: Water infrastructure engineering services e Task 1.2: Water infrastructure construction e Task 1.3: Technical assistance for water service providers Component 2: increased access to sustainable sanitation services e Task 2.1: Support for sanitation enterprises e Task 2.2: Waste treatment and fecal sludge management engineering services e Task 2.3: Wastewater treatment and fecal sludge management construction Component 3: Improving the enabling environment for sustainable implementation, operations, and maintenance of water and sanitation services e Task 3.1: Technical assistance to national and sub-national government structures e Task 3.2: Knowledge dissemination and learning Initially, Component | activities focused on Quick Impact Projects and the tailored technical assistance (TA) packaged for the CTES, but that expanded throughout the Project to include software packages for the CTESs, operation manuals, and support for performance management of directors. Training support to the CTEs was further developed based around the five metrics of organization and strategy, human resources management, financial management, technical operations, and commercial operations (Annex H). To help with implementation in the last quarter of Year Two, the Project moved some of its senior staff into the CTESs as well. In Year Three, the Deputy Chief of Party (DCOP), worked directly with the Cap Haitien CTE. The Senior Advisor and the Utility Specialist worked with the Les Cayes and Jeremie CTESs. The Senior Engineer worked with Mirebalais. By having these senior staff working directly with the CTESs on a daily basis, the Project was able to help the CTESs to implement the required management changes. USAID WATSAN worked with the CTESs to increase the amount of data collected, reported, and used in mWater. DINEPA tracked 11 key indicators that were self-reported by the CTESs as part of the monthly reports. Additional financial, client, and technical information is collected using SIGA, SISKLOR, and Cadastre, and also stored in the mWater database for each CTE, including a limited amount of infrastructure information. 3 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 19] Throughout the Project, there was a focus on the institutional strengthening program for the Regional Offices for Potable Water and Sanitation (OREPAs) based on the needs identified in the water and sanitation service roadmaps. Support to DINEPA focused on monitoring, especially collaboration with the ONEPA and improving their collection, monitoring, analysis, and archiving portal. Learning activities included developing a learning agenda with sector stakeholders, convening at least one learning event, and completing preliminary design of an online learning platform. Activities under the Project’s Enterprise Acceleration Fund were also essential to the Project’s activities. Initial grants and other mechanisms helped build the capacity of private sector actors working in the water and sanitation sector while also meeting their enterprise development goals and contributing to the Project’s results and impacts. PROJECT PERFORMANCE SUMMARY The key objectives of the Project were to build the governance and financial management capacity of targeted Haitian public utilities and private operators, enable 250,000 people to gain access to new or improved water services, and enable 75,000 people to gain access to basic or safelÿ managed sanitation services. The USAID WATSAN Project supported sector institutions by implementing a combination of targeted infrastructure improvements and technical assistance prescribed by the World Bank Utility Turnaround Framework. The USAID WATSAN Project also supported sector MSMEs and supported water and sanitation innovation projects through its Enterprise Acceleration Fund. The table below summarizes Project achievements against Life of Project (LOP) targets. TABLE 1: SUMMARY OF PERFORMANCE INDICATORS FROM THE PROJECT RESULTS FRAMEWORK OUTCOME INDICATORS LOP TARGET COMMENTS! Number of people receiving improved service quality from an existing basic or 250,000 Exceeded by safely managed drinking water service 24,425 Number of service providers demonstrating at least a 10 percent increase in 9 2 Remaining cost recovery Number of people receiving improved sanitation service quality from an existing 75,000 Exceeded by 1,298 “limited” or “basic” service Number of SMEs demonstrating increased sales of sanitation products and 15 0 services Percentage of staff in target sector institutions self-reporting increased ability to 50 percent Exceeded by 50 perform effectively in assigned job percent Percentage of agreed strategies/plans for improved water and/or sanitation 9 0 service delivery being implemented at national, regional and/or commune level Table | summarizes the performance outcome indicators from the Project results framework with information from March 2022. Please note that the data was not disaggregated by income group, age, disability, or sex. 12 From the QIFY22 and Q2FY22 quarterly reports USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 4 [page 20] EVALUATION QUESTIONS EVALUATION QUESTION I. To what extent has USAID WATSAN met its Task Order objectives to build governance capacity at multiple levels (national, regional, local) to improve sustainable water supply and sanitation service delivery? To answer this question, the Evaluation Team (ET) considered a number of sub-questions that looked at the CTE water supply, the kiosk water supply, and the sanitation services supported by the USAID WATSAN activity. These included: What is the impact and experience of the data management tools (mWater, SIGA, QuickBooks and Sisklor)? Has CTE customer satisfaction improved? Has the TA improved the service delivery and customer management capacity of the CTEs and led to improvement across the five metrics? For the sanitation aspect, the ET considered if the capacity of the Fonfred and Morne a Cabri FSM management teams has improved. The impact of the business support provided to the SMEs and Latrine Emptier Associations was also examined. EVALUATION QUESTION 2. How was the Enterprise Acceleration Fund utilized, and to What extent did those grants support WASH enterprises (such as MSMEs, NGOSs, and Bayakous) to move toward sustainable service delivery? To answer this question, the ET considered a number of sub-questions. Some of these looked at the services and the results from the EAF, such as what success the grants achieved and if there were any recommended modifications. The ET investigated how sustainability improved, if the business or organization expanded (in terms of services and/or customers), and the overall successes and challenges of the EAF grants. EVALUATION QUESTION 3. How effective was WATSAN’s approach of targeting the end of the sanitation value chain in improving the overall sanitation value chain? The ET understood that DINEPA and the FSM management staff will continue the work started under USAID WATSAN. When looking at the overall sanitation value chain, the ET focused on the services the FSM provided to operators and the communities. Some of the sub questions used to investigate the evaluation question included: Is the FSM operating as a business? Are clients using wastewater treatment sites? Are they satisfied? How was the sanitation value chain affected by the Project activities? How does the FSM serve the community and businesses? Has FSM improved and has there been improvement regarding sustainability? And has the opening of the FSM improved disposal of waste and improved businesses for the clients? The ET also investigated the challenges and success of the sanitation activities. METHODOLOGY As part of the evaluation, the ET reviewed USAID WATSAN annual work plans, annual reports, monitoring, evaluation, and learning (MEL) plans, and Activity descriptions and modifications. Following the kick-off meeting on December 15, 2021, the ET continued its review of USAID WATSAN Project documents, contacted DAI for additional documents and stakeholder contact information, and prepared the Inception and Evaluation Design Report. The in-briefing presentation was held with USAID on January 6, 2022, after which the ET made some adjustments to the methodology. The inception report was submitted to USAID on January 31, 2022. The ET had an introductory meeting with mWater and members of the DAI team to gain a better understanding of the Project and the mWater platform. 5 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 21] DATA COLLECTION OVERVIEW The ET used a mixed-methods evaluation design. The ET collected data on the perspectives and experiences of key stakeholders involved in USAID WATSAN and/or in the sanitation value chain using both purposeful and snowball sampling techniques. The table in Annex € shows all the data collection events undertaken by the ET. Data collection started on February 21, 2022, and ended on April 15, 2022. The data collection tools included guiding questionnaires and associated protocols for Kils (with one person), for Gls (with two to five people), for FGDs (for six people or more), and a dedicated survey tool for CTE staff. The tools were translated into French and Creole. The KIl and GI respondents and FGD participants were selected strategically from a list of key USAID, GOH, DAI, CTE, and partner staff. Respondents likely to have the most information were selected first, balanced by diversity, and supplemented with snowball sampling to fill in gaps in data sources and information Primary data collection included 55 key informant interviews (KIls) or group interviews (Gls) and six focus group discussions (FGDs) with stakeholders in all 10 communes to address EQs |, 2, and 3, and a survey of 74 CTE staff in seven communes to address EQI. The ET also conducted an extensive secondary data review of mWater CTE indicator data, DINEPA FSM sites operational data, EAF Project performance data, and a CTE customer satisfaction survey conducted in 2021. The scope of this data review was substantial for EQI (water) and looked at all mWater data collected from December 2017 to December 2021 across the Project communes for the 11 mWater indicators as well as the data from a CTE customer satisfaction survey conducted in 2021. The scope of secondary data review for EQI (sanitation), EQ2, and EQ3 was limited to DINEPA FSM sites data and (limited) performance data for the EAF Projects. There was substantially less data available for sanitation than for water supply. The evaluation faced a number of challenges related to the country context and the remote nature of data collection (see Methodological Limitations and Mitigation Strategy section below). DATA COLLECTION METHODS DOCUMENT REVIEW The ET conducted an initial review of more than 50 USAID WATSAN documents, as well as other, third-party resource material, to understand the Activity design and implementation, extract findings relevant to the EQSs, and inform the development of data collection questions and tools that appropriately supplemented, or could be cross-checked with, information in the background documents. Documents are listed and fully described in Annex D. SECONDARY DATA COLLECTION AND REVIEW The ET conducted secondary data analysis of the CTE indicators extracted for each CTE from mWater, such as revenue information, subscriber information, kiosk information, and number of staff by gender and contract type. The ET was also able to disaggregate the subscriber information and their satisfaction rating of their CTE’s services by gender from DAs CTE Client Satisfaction Survey conducted in March- April 2021. There is a delay in the approval process by OREPA, which is needed for the CTE indicator data to be publicly available. The ET followed up with the SAEPSs to ensure information was available until December 2021 at a minimum on the mWater platform. The ET had access to data from reports across USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 6 [page 22] the 10 WATSAN SAEPSs ranging between December 2018 and March 2022. The summary of available mVWater data is listed in Annex D. The Project completed a CTE customer satisfaction survey as part of a SIGA database update in March — April 2021. The survey was conducted in six communes (Cap Haitien, Mirebalais, Croix des Bouquets, Les Cayes, Jeremie, and Hinche). Using the SIGA client database, the survey attempted to reach all of its active and passive client base and managed to collect responses from a total of 10,525 clients. In addition to updating the client contact information (phone numbers, address, and GPS coordinates of the connection), the survey asked questions on client’s status (active vs. passive) and satisfaction rating of service quality. The ET used the data DAI provided to understand the client base of each CTE and their satisfaction rating. FOCUS GROUP DISCUSSIONS FGDs were undertaken to collect balanced opinions from small groups of key stakeholders, in particular those groups of individuals personally and directly involved in water or sanitation service delivery. These groups included CTE staff, kiosk managers, LEAs, and MTFs. Due to availability of interviewees and connection issues, only six FGDs were conducted of the total 24 planned (all remaining events planned as FGDs became instead Kils or Gls.) The FGDs were conducted with the LEAs and the managers for the kiosks in Cap Haitien (Living Water kiosks). Thirty-seven people participated in the FGDs, but only six of them were female. Information gathered from FGDs respond to all three evaluation questions. KEY INFORMANT AND GROUP INTERVIEWS The Kils and Gls allowed the ET to gather information for all three evaluation questions. The ET conducted 55 Kils and Gis with representatives of key stakeholder groups, including USAID, ONEPA, DAI staff (CTE management staff, EAFs, FSM management teams), FSM users (this includes companies or individual contractors that use the FSM sites but not SMESs), and other working partners. This was more than was originally planned due to challenges faced with schedule and connectivity. The number of Kils and Gls increased while the number of planned FGDs decreased. Of the participants in the Kils and Gls, 40 were female and 69 were male, for a total of 109 participants. Table 2 shows the number of each primary data collection event and the type of event that was conducted. À complete breakdown of the events including by gender is in Annex C. TABLE 2: KIIS, GIS, AND FGDS KIIS, GIS (2-5 PAX), FGDS (6 OR MORE PAX) EVENTS TYPE CTE staff 7 GI FSM staff and users 4 GI EAF grants recipients 7 KII, GI Kiosk managers 9 FGD, GI ONEPA/OREPA staff 6 KI DAI staff (COP + Project staff, CTE liaison staff) Il KII, GI 7 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 23] KIIS, GIS (2-5 PAX), FGDS (6 OR MORE PAX) EVENTS TYPE SMEs 4 FGD Pit Emptier Associations and Mayoral Task Forces 4 GI, FGD Working partners 5 KI Others | KI USAID 3 KI Total KIls/Gls 55 40F/69M Total FGD 6 6F/31M CTE staff surveys for 7 communes 74 13F/60M CTE STAFF SURVEY The ET conducted a survey of lower-level staff at seven CTEs (Cap Haitien, Mirebalais, Croix des Bouquets, Les Cayes, Jeremie, Ouanaminthe, and Hinche). These surveys were conducted by a local data collection partner in Haitian Creole. The targeted sample size for the survey was 70, with 10 respondents per CTE. When conducting the survey, the team was short three people each in Ouanaminthe and Croix des Bouquets. The local data firm then used staff from other cities to reach the target. The survey was carried out via telephone, and an Excel database was shared with the ET. The sampling for this survey was random, with the list of staff and those they were assigned to randomly generated; if someone was not available or had issues with phone connection, the surveyor proceed to the next person on the list. The plan was to have 30 percent of the surveys completed by women, but due to more male staff and connection issues, this was not possible (1 1.2 percent of survey participants were female)) TABLE 3: CTE STAFF SURVEY DATA COLLECTION DEPARTMENT CTE STAFF WOMEN MEN SURVEYED SURVEYED North Cap Haitien 22 4 9 North East Ouanaminthe 10 0 7 Center Hinche 16 | Il West Mirebalais 14 2 8 Croix des Bouquets 8 | 6 South Les Cayes 25 3 12 Grande Anse Jeremie 21 2 8 Total 116 13 6l USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 8 [page 24] DATA ANALYSIS The evaluation Team Lead (TL) oversaw and managed systematic analysis of qualitative and quantitative data. The evaluation included several data collection methods (document review, Kils, Gis, FGDs, surveys, and secondary data collection) that enabled the ET to conduct triangulation across methods and information sources. DATA ANALYSIS METHODS The ET employed several data analysis methods to identify key findings from the collected data, as well as to draw conclusions and make recommendations for Project follow-up or future potential programming. The ET captured findings, conclusions, and recommendations (FCRSs) in an Excel-based matrix that categorized analysis by EQ. The matrix: 1) ensured that the ET prepared a systematic and thorough response to each EQ; 2) compared findings across data collection sources and methods to triangulate primary and secondary data; 3) verified that analysis accounted for gender and social dimensions; 4) identified any gaps where additional clarification or analysis may be necessary; 5) clarified connections between FCRSs; and 6) served as the basis for developing the evaluation report. The Encompass Gender Specialist also participated in analysis debriefs and in reviewing the FCR to validate inclusion of sex-disaggregated data. The type of analysis depended on the data being assessed as explained below. QUANTITATIVE ANALYSIS The ET reviewed and analyzed quantitative data from mWater, the 2021 client satisfaction survey, and the primary survey of CTE lower-level staff. mWater Secondary Data Review The ET undertook a review of all available CTE mWater data through December 202 land compared this with key information from primary data collection. The findings and conclusions are discussed in the sections below for each of the five CTE performance metrics. The ET reviewed the secondary mWater data for quality and consistency and used Stata 15.0 software for data cleaning. Outliers identified were addressed for the 1 1 indicators for each CTE. The ET requested clarification on the outliers from the respective CTEs. The ET received one response from a CTE. Details on the data cleaning process are presented in Annex D. For the analysis, the ET examined trends over time across the 11 DINEPA key indicators and other key outcome indicators for each Project SAEP (number of households served, city coverage, SAEP coverage, and SAEP capacity). The analysis focused on the six-month moving averages of the indicators in order to illustrate the overall trends despite the short-term volatility/discontinuity in some of the data. CTE Client Satisfaction Survey Secondary Data Review Customer-related findings were limited to a review of the March-April 2021 SIGA database update survey, which had included with it a customer satisfaction survey of 8,987 active and 1,225 passive CTE customers across the CTEs of Cap Haitien, Les Cayes, Croix de Bouquets, Jeremie, Mirebalais, and Hinche. For the secondary data analysis the ET used Excel to tabulate descriptive statistics for each client satisfaction question asked, disaggregated by sex. The team also conducted gender analysis of the secondary data and reviewed it for any statistically significant differences between male and female client 9 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 25] responses. There were more female clients than male clients, but no other survey responses were significantly different between men and women. CTE Staff Survey For primary survey data (CTE lower-level staff survey), the ET conducted the analysis using Excel, with each step documented to allow replicability of the results. Prior to analysis, the team undertook a data cleaning process to label and format the data, verify data quality, and calculate variables required for the analysis. The analysis consisted of tabulating descriptive statistics related to the sample characteristics and key outcome indicators. The ET disaggregated descriptive statistics by CTE. In addition, the team conducted gender analysis of the staff survey data to identify any statistically significant differences between male and female staff. QUALITATIVE ANALYSIS (THEMATIC) The ET undertook systematic coding, using a developed codebook, of KI, GI, and FGD transcripts. Dedoose software was used to identify and highlight the existence of key themes and their frequencies within the data. The Dedoose data was exported to Excel where coded, transcripted data could be further analyzed to identify key themes and useful quotes. METHODOLOGICAL LIMITATIONS AND MITIGATION STRATEGY The evaluation needed to accommodate the limitations of remote data collection, and therefore the evaluation relied significantly on secondary data sources, which were plentiful and triangulated with the findings from primary data collection. Because of ongoing COVID-19 pandemic-related travel and meeting restrictions, as well as concerns for evaluation participants’ health, the team conducted all of the interviews via telephone or online conferencing. Communication was a challenge as some phone numbers provided did not work and often the connection dropped. Other significant challenges included security concerns and demonstrations that required events to be rescheduled (such as the demonstrations in Les Cayes). Some of the participants became impatient during the calls, and in some instances the event had to be stopped midway and rescheduled with smaller groups to reduce the time needed for the event. The less-than-expected availability of some respondent groups meant that some FGDs (defined as six or more participants) were undertaken with fewer people and thus are categorized as Gls. Specific limitations are discussed below. Diminished communication quality resulting from the near exclusive use of remote communication technology. The ET is aware of various challenges associated with remote data collection, such as limited internet connectivity and poor cell phone network coverage, especially among some stakeholder groups such as those working in the provinces. Poor audio quality, the effects of the absence of visual contact on non-verbal communications, and limitations on group discussion size and level of interaction were additional challenges for remotely conducted discussions. The ET mitigated these limitations to the extent possible by testing various communication tools to identify which worked best and have alternative communication technologies and sampling strategies as a back-up plan. In the case of USAID WATSAN staff, a significant proportion of key stakeholders were relatively well connected through internet and cell phone networks. The ET and the local data collection firm conducted data collection in English, French, or Haitian Creole, according to which language was USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 10 [page 26] preferred/most appropriate. The data collection tools compiled by the ET sought to be as pertinent as possible for the respective stakeholders to optimize the ease and utility of data collection. Diminished interest in participating in the evaluation by some stakeholders due to the COVID-19 and political situations. The ET also anticipated that it might be more challenging to mobilize some stakeholders during the public health emergency and poor political situation in Haiti. For example, whereas in normal times, the team may have been able to find key informants by going to their office or place of residence, this evaluation relied heavily on telephone and email. To mitigate this limitation, the team requested assistance, when necessary, from the Project, USAID, and other key informants (as appropriate) to contact some hard-to-reach stakeholders and allocated extra time to schedule and conduct data collection, considering the challenging context and competing local priorities. The USAID WATSAN Project covers many sites and institutional levels with many different activities being examined across the three EQs. The ET sought to verify and triangulate data collection sources and locations to best inform comprehensive and usable responses to each EQ. The ET allocated time to review secondary data sources in detail prior to finalizing primary data collection audiences, tools and sampling lists. Disruption/delay to data collection due to natural disaster and/or political unrest. The ET faced issues with having to reschedule events and the unpredictable availability of personnel due to demonstrations and other incidents of political unrest in the commune areas. Response, gender, and selection bias. There were challenges to getting everyone to participate in the survey and interviews due to connection issues and availability. Across all the FGDs, Kils and Gls, there were 100 male respondents and 46 female respondents (see Annex C). The ET sought wherever possible to maintain a balance between a wide range of water and sanitation stakeholders being represented in the data collection audiences and the reality that some of the service-providing stakeholders have à relative preponderance of men over women. Although 100:46 represents a relatively high degree of female participation, it is an average of all the data collection events. The CTE staff survey had just 1 1.2 percent of female respondents. Availability of data. There were significantly fewer data available for the ET regarding the activities and financial status for the FSMs and SMEs compared to the CTEs. There were limited data availability regarding the EAF grants. The ET received only six of the seven proposals and DAI completed only two evaluations as part the Project. 11 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 27] FINDINGS AND CONCLUSIONS EVALUATION QUESTION I (WATER SUPPLY). To what extent has WATSAN met its Task Order objectives to build governance capacity at multiple levels (national, regional, local!3) to improve sustainable water supply and sanitation service delivery? FINDINGS (WATER SUPPLY) Respondents from USAID, DAI, DINEPA, and CTE stakeholders indicated that before the Project, all CTEs were insolvent and unable to plan and monitor service delivery progress, revenues, and expenditures. The Project brought data-driven planning and decision-making to each CTE. The Project supported change management processes at nine CTEs and Pignon SAEP, and saw seven of nine CTEs move from Level One to Level Two on the WBUTF five-point scale.l4 This process was supported and monitored using mWater’si | key performance indicators, and this monitoring mechanism has been adopted by all 29 CTESs in Haiti, including Pignon SAËP. The mWater monitoring platform, originally intended to provide a basic monitoring framework for the 10 selected CTESs, developed substantially with support from the Project and OREPA/DINEPA, to the extent that it now underpins CTE current and future business development across the five WBUTF performance metrics: 1) Organization and Strategy, including CTE Annual Planning 2) HR Management, including CTE Staff Capacity Building 3) Financial Management, including CTE accounting and budget management tracked in mWater via the three metrics (see Table 4) using QuickBooks accounting software 4) Technical Operations, tracked in mWater via four metrics (see Table 4) with chlorine conformity data collected via SISKLOR 5) Commercial Operations, tracked in mWater via four metrics (see Table 4), which is connected to the SIGA customer database TABLE 4 MWATER INDICATORS AND METRIC CATEGORIES INDICATOR METRIC CATEGORY INDICATOR INDICATOR DEFINITION ; : : The total number of active subscribers connected to the (l Commercial Active subscribers : water system (not disaggregated by gender) 2 c jal Collection efficiency - Current Amount collected for the current period / Amount billed for ommercie (%) charges incurred during the current period * 100 3 Commercial Collection efficiency - Arrears Amount collected in arrears during the period / Amount of (%) arrears existing at the beginning of the period * 100 : Collection efficiency - Overall Total amount collected this period / Total amount billed this 4 Commercial d ou (%) period * 100 13 The evaluation data collection was naturally biased toward the local level, with a relatively greater number of local level respondents than individuals representing regional and national levels. 14 https:/documents | .worldbank.org/curated/en/515931542315166330/pdf/Water-Utility-Turnaround-Framework-A-Guide-for- Improving-Performance.pdf USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 12 [page 28] INDICATOR METRIC CATEGORY INDICATOR INDICATOR DEFINITION 5 Financial Revenue (HTG) The total amount of revenue generated by the water system (not including subsidies) 6 Financial Expenses (HTG) The total of all expenses incurred by the water system during the month 7 Financial Operating ratio (%) Total expenses / Total revenue * 100 8 Technical Total production (m3/month) qe total amount of water produced by the water system uring the month 9 Technical Service continuity The average number of hours per week that customers receive water service (168 hours in a week) : Residual chlorine conformity The percentage of residual chlorine tests that conform to 10 Technical D (%) norms 11 Technical Functional kiosks (%) # of functional kiosks / Total # of kiosks * 100 In collaboration with DINEPA, the Project managed to reverse the “spiral of decline” seen in the Project CTESs, which historically saw insolvency and chronic business underperformance by the CTESs. This decline is illustrated schematically in Figure 1. Low f revenue ) due to poor services ( Service quality decreases Users invest ) in private infrastructure ( Revenue further decreases FIGURE |: THE “SPIRAL OF DECLINE” LEADING TO POOR PERFORMANCE (AFTER GALAITSI 2016) 13 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 29] The mWater platform contribution to creating an enabling ? N environment for the CTEs was a key contributor to the Project’s water supply outputs. The mWater platform enabled the establishment of a data value chain!5 within the | | CTE’s operational practices, which is summarized in the EE FIGURE 2. THE DATA VALUE CHAIN MODEL FOR TURNING DATA INTO USEFUL INFORMATION METRIC ONE | ORGANIZATION AND STRATEGY According to CTE FGD participants, CTEs are now able to develop annual plans with monthly goals. This planning exercise is done at the beginning of each fiscal year (October-November) with SAEPA and is used to track progress on a monthly basis. The annual CTE workshop in 2021 included a competition whereby CTEs were judged against their reported mWater metrics; this was popular and very motivating for many CTEs. However, not all 1 | mWater indicators are consistently reported. Indicators on production, revenue, expenses, operating ratio, and active subscribers are reported in more than 90 percent of the monthly reports. Indicators on residual chlorine conformity (51 percent), functioning kiosks (63 percent), and collection efficiency of current bills (71 percent) were presented in less than 80 percent of the monthly reports submitted for WATSAN CTESs. In total, 18 percent of data on | 1 monthly indicators are either not reported or unavailable from the WATSAN CTE monthly report. The Project focused on local level capacity building, but not on increasing regional or national capacity or development of regulatory frameworks. While generally consistent, some outliers (e.g., a ten-fold increase in one month) and data entry errors (e.g., obvious addition of an extra “0” at the end of the reported value) were clearly observed from the mWater data and these are documented in Annex D. CTESs that had started working with the Project earlier (Cap Haitien, Les Cayes, Croix des Bouquets, Jeremie, and Mirebalais) submitted, reviewed, and had their monthly reports approved by OREPA within two months, while the CTESs that joined the Project later are taking longer to submit their data and have it approved. From the CTE staff survey, a significant proportion indicated that they were unaware of whether their CTE uses mWater (45 percent on average) or that their CTE does not use mWater (13 percent on average across CTESs). This is especially prominent for Hinche and Jeremie, where more than 80 percent of the surveyed staff noted that they do not know if their CTE uses mWater. Staff responses on their experience of using mWater varied by CTE, with 42 percent positive rating from Mirebalais and no positive ratings from Croix des Bouquets. Figure 3 summarizes the data. 15 Haiti WATSAN learning note - data driven management for water service providers USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 14 [page 30] 100% 20% 14% 75% 36% EE 43% 50 === 17% 80% 89% 31% 43% Cap Haitien Mirebalis Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets mPositive experience = Neutral experience Experience rating N/A m Dont use SIGA Don't know if CTE uses SIGA FIGURE 3: CTE STAFF EXPERIENCE OF USING MWATER CTE FGDs reported that staff capacity to use software improved, enabling better service and billing, although another stakeholder indicated that there is variation in that capacity —i.e., not everyone can use the data to plan. Multiple stakeholders confirmed that although CTESs can access SIGA data now via mWater, there remains the challenge of not having easy access to all the data used by CTESs; there is no single integrated system. METRIC TWO | HUMAN RESOURCE MANAGEMENT The Project addressed this metric by having staffing plans, staff evaluations and training, and performance-based compensation for CTE staff. The evaluation did not include a detailed assessment of individual CTE staff roles, capacity, or a learning needs analysis, but it did include a CTE staff survey of 74 staff (Annex E). More than 50 percent of staff interviewed for the CTE staff survey have worked at the CTE for four to five years, and according to the FGDs with CTE staff, people only leave their CTE for retirement or to move to different positions within DINEPA. At Jeremie CTE, 80 percent of the staff interviewed had worked at the CTE for more than six years. Almost 100 percent of CTE staff had a job description and were clear on their job expectations. The majority of CTE staff had a performance evaluation in the last year, including 100 percent of Mirebalais staff, although for Hinche the figure was just 44 percent of staff. The survey data, together with interviews and FGDs with a number of CTE stakeholders, also found that although CTE staff capacity to use mWater and SIGA software improved, enabling better service and billing, there is variation in that capacity - i.e. not everyone can use the data for planning purposes. METRIC THREE | FINANCIAL OPERATIONS The Project addressed this metric by focusing on reaching financial stability with strong accounting, budgeting, and cash flow management, and developing capital budgets for new infrastructure. These areas were tracked via three mWater indicators (revenue, expenses, operating ratio). Most CTEs increased their revenue (Mirebalais seeing the biggest growth) while a minority (Pignon SAEP, Belladere, Lascahobas) did not see a significant increase. Expenses broadly reflected revenue changes, with CTE 15 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 31] operating ratios mostly staying within the 50 to 150 percent range. A key Project achievement was enabling the selected CTESs to track, via QuickBooks and mWater, their monthly expenses and revenues, key indicator metrics that enable the CTESs to calculate their monthly operating ratio (expenses/revenue). Figures 4 and 5 show six-month rolling average data for expenses and revenues across the 10 CTESs. 2500 ? Hinche& ? Lascahobas, Belladere, & ? Ouanaminthe ; Pignon & 2000 ; î — Cap Haitien 5 5 s —— Les Cayes Ë 1,500 É : — Croix des Bouquets 5 1,000 É É —Mirebalais É 500 DA, AK — — Ouaraminthe EE ET —Belladere ° : — — Lascahobas Q a Q Q © © © © OS © SJ D d Ju D D 42 42 OR Monthly Report FIGURE 4: CTE EXPENSES DATA FROM MWATER (6 MONTH MOVING AVERAGE) 2,500 g j À Hinche & À Lascahobas, Beladere, ? Ouanaminthe H & Pignon 5 2,000 : : — Cap Haïtien 5 i i —Les Cayes © H + 5 1500 î : —— Croix des Bouquets 0 : : — jeremie E 1,000 î î — Mirebalais 5 î æ 500 a — Ouanaminthe . — TE —— palaere 0 EEE — Lascahobas POP POP OP POP EP PS PO D SO D D D D ES ON et 7 gg a 7 7 Pere Monthly Report FIGURE 5: CTE REVENUE DATA FROM MWATER (6 MONTH MOVING AVERAGE) In 45 percent of cases, the operating ratio was higher than 100 percent for the WATSAN CTESs, indicating that expenses were higher than the revenue for close to half of the reported months. Figure 6 shows a six-month rolling average operating ratio plot for the 10 CTESs. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 16 [page 32] 250 H F ? Hinche & Lascahobas, Belladere, & i Ouanaminthe Î Pignon 200 : i — Cap Haitien i —Les Cayes A ARNO PES ILSR a © 50 i Î — Ouanaminthe : —Bcladere 0 i Î —lLascahobas RE) “ Ps SR ra PE SP PE Te Monthly Report FIGURE 6: CTE OPERATING RATIO DATA FROM MWATER (6 MONTH MOVING AVERAGE) It should be noted that although this mWater data shows how revenues have increased, “fear of success” is very real. This issue was mentioned in discussions with DAI and USAID. CTESs are not currently audited and have concerns around how they will effectively manage their increased revenues and expenditure of those funds for capital infrastructure projects and improvements in the future. It appears the fear is related not to (revenue) success per se but rather to concerns CTEs have around the additional financial responsibilities and risks that come with increasing revenues - and the need to have staff capacity to manage them. The same interviews confirmed how the Project’s support to the CTEs has been a game-changer for fully decentralizing the functioning of the 10 CTESs. QuickBooks is the accounting software CTEs use to manage and track their revenues, expenses, and profits and create their financial reports. The CTE Staff Survey indicated that a large proportion of staff did not know if their CTE used QuickBooks (see Figure 7). There is no network connection from QuickBooks to the SIEPA system, and ONEPA does not yet have control over this information. DINEPA sets limits on expenditure against revenue (e.g., for | million HTG revenue, the expenditure limit is 70 percent). This monitored in QuickBooks, and some CTESs have only started using QuickBooks recently. During the FGDs with the CTES, they discussed the lack of training and support affected their ability to use QuickBooks. 17 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 33] 100% 75% 50% 43% GS 67% 71% 50% Eu 90% 89% 43% _ LL 17% PE : RES eu HIS 14% Cap Haitien Mirebalis Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience a Negative experience Experience rating N/A m Dont use Quickbooks Don't know if CTE uses Quickbooks FIGURE 7: CTE STAFF EXPERIENCES OF USING QUICKBOOKS ACCOUNTING SOFTWARE Overall, FGDs and the CTE staff survey points to CTE financial operations being well informed by revenue and expenses data, with CTE staff being generally aware of any financial management shortcomings as applicable to their CTE. Although the timeline per CTE is unknown, there is now the realistic prospect of CTEs becoming financially self-sustaining and no longer reliant on international donors. These institutional developments appear to have made a significant contribution to the sustainability of the CTESs. METRIC FOUR | TECHNICAL OPERATIONS This metric focused on the CTESs providing reliable metered water supplies to customers and minimizing non-revenue water losses. The mWater indicators of total (cubic meters) production per month, service continuity (average number of hours per week of water supply), residual chlorine conformity (the percentage of residual chlorine tests that conform to norms), and the percentage of functioning kiosks were used as metrics of CTE-managed kiosks. Figure 8 shows service continuity remained almost unchanged across all CTESs at about 5-20 hours/week with the notable exception of Pignon (around 80 hours/week), while Ouanaminthe’s continuity in October 2020 leapt from 3 to 168 hrs/week (24/7 supply) where it remained since. There was no specified hours/day water supply service delivery target for the CTESs. Interviews with implementing partners indicated that the overarching customer service priority for the CTEs was to supply water, albeit for a limited number of hours/day, based on a predictable schedule. The Ouanaminthe success was largely due to new infrastructure funded by the Inter-American Development Bank (IDB) coming into operation. This figure clearly shows that all but two of the ten CTESs are supplying an average of less than 20 hours of water per week to household connections. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 18 [page 34] ÿ 180 ÉHinche & Î Lascahobas, Belladere, & Pignon > 160 ?Ouanaminthe Î & 140 i i — Cap Haïtien E 120 À d — Les Cayes d É : — Croix des Bouquets 8 ÿ 100 . | $ H ; Î —— Jeremie z 8 © Î | — Mirebalais gg « Î ! —Hinche £5 4 ; ; — Ouinaminthe É 8 2» j H —Belladere % o s — Lascahobas 5 a Q a a © © S © S © S S S S S S 42 42 3 # S' S s SE S S re S S s ° —Pignon Monthly Report FIGURE 8: SAEP WATER SUPPLY CONTINUITY (6-MONTH MOVING AVERAGE) SISKLOR enables residual chlorine conformity to be monitored, but SISKLOR data is not regularly reported and there is no one person responsible for SISKLOR data at the CTE level. Figure 9 shows a significant variation in the amount of SISKLOR data being reported, with some CTESs’ reporting regularity being much higher than others. Some CTEs do not have a SISKLOR focal point.!é 100 2 E $ Ë 5 E Le BË SE É 5 Fi 20 $ Ë Cap Haitien Les Cayes Croix des Bouquets Jeremie Mirebalis © Hinche Ouamaminthe Beladere Lascahobas Pignon FIGURE 9: CTE RESIDUAL CHLORINE CONFORMITY The majority of CTE staff reported a positive experience with using SISKLOR (69 percent across CTESs), but not Hinche and Ouanaminthe. Sixty-seven percent of surveyed staff at Hinche reported that they were unaware of their CTE using SISKLOR. At Ouanaminthe, only 43 percent of surveyed staff shared positive ratings of SISKLOR. The Project supported CTE-managed kiosks by providing training in business management to the kiosk managers. The functionality data for CTE-managed kiosks (Figure 10) shows erratic and variable levels of functionality but FGDs with the kiosk managers indicated how the kiosks do nonetheless provide, when working, service to those without HH connections (most kiosk users are women and children). 16 Despite best efforts, the ET was unable to successfully contact the SISKLOR Director for a Ki. 19 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 35] Hinche & Ouanaminthe Lascahobas, Belladere, & Pignon __ 100 E 6 80 Î ; — ——— Les Cayes w | | ? LD —— Croix des Bouquets 5 60 ÿ | —Jeremie s | | — Mirebalais H 40 | | —Hinche $ H 4 — Ouanaminthe à ? | | — Belladere D A A A D D D D D D À AN D À D D D D —Pgron St ne D 7 a Monthly Report FIGURE 10: FUNCTIONALITY OF CTE MANAGED KIOSKS (6 MONTH MOVING AVERAGE) The ET undertook an exercise to determine the proportionate coverage of CTE household connections and kiosks across the ten communes, shown in Figure 11. It should be noted that all the data below is Cadastre based, with the exception of residential connection data for Mirebalais (Jan 2022), Belladere (Sep 2021), Lascahobas (Sep 2021), and Pignon (May 2020), which is from mWater. No data for Croix de Bouquets is shown due to data quality issues. For Mirebalais and Ouanaminthe, the total access figures are greater than the number of households in the CTE coverage area due to many households being in the catchment of more than one kiosk. 40,000 38012 35,000 & % © 30,000 27,351 5% Households in the CTE $ 3 25.000 coverage area Ë ÿ 2000 #Functional kiosk 5 $ M Residential connection ÊÈ 15:000 329 r 4810 ES Z 2 10,000 7957 . | ” Commercial connection Le F °27 4,794 500 6.10q - 3.205 2387 minstitutional connection ° 3.499 ed 3,050 | 2: D +2” BEE S s k e Q © e e 9 S & Y s S © NS # no FIGURE I 1: HOUSEHOLD WATER ACCESS FROM HOUSEHOLD CONNECTIONS AND KIOSKS: COVERAGE BY CTE METRIC FIVE | COMMERCIAL OPERATIONS The Project supported the improvement of effective customer billing and optimizing revenue collection. A lot of this support relied on day-to-day mentoring by CTE Liaison Officers rather than by bespoke training programs alone. The mWater indicators of active subscribers and collection efficiencies (current, arrears, and overall) were all used. All the CTEs showed increases in subscriber numbers, as shown in Figure 12. The dramatic increase in subscriber numbers in Ouanaminthe coincided with the start of 24/7 water supply becoming available. Other USAID and DAI staff reported how CTEs wanted to emulate Ouanaminthe; they like the idea of having greater revenues and performance-linked salaries. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 20 [page 36] Î Hinche & ; Lascahobas, Belladere, & 3500 | Ouanminthe {Pignon ———— & | s —Cap Haitien 5 2,500 i — Croix des Bouquets # 2000 Î H —Mirebalais ë | | £ 1,500 ; - ——Hinche ê H 1] î — Ouanaminthe 5 190 — "77 | - Belladere £ 500 À — è —Lascahobas ë 7] Î < 0 i —Pignon Q a Q a © © © © © © S SJ SJ SJ SJ SJ 2 2 SA °° S SP SE PP S s Monthly Report FIGURE 12. CTE ACTIVE SUBSCRIBERS (6 MONTH MOVING AVERAGE) Revenue Development Revenues increased for all the CTES, the largest being Mirebalais, which rose from 148,000 HTG in July 2019 to 805,000 HTG in January 2022. According to CTE FGD, the Mirebalais success story was due to a number of factors, including increased tariffs, increased collection, and establishing a reliable distribution schedule. Revenues doubled for Hinche, Cap Haitien, and Belladere. For Les Cayes, the post-earthquake recovery is clearly shown. The data is shown in Figure 5. Collection efficiencies data is erratic and discontinuous in places, but the data below indicates that most months, the overall collection efficiency runs between 50 and 130 percent.!7 CTE FGDs discussed how revenues have increased as tariffs have been raised and there is now better collection of monthly tariffs. Hinche, Jeremie, Croix des Bouquets, and Mirebalais show improved efficiency over time for collecting current month charges, while CTEs’ arrears collection efficiencies (see Figure 13) have largely remained the same. 300 ; ; ? Hinche& iLascahobas, Belladere, & } Ouanaminthe iPignon g 250 É H — Cap Haitien 5 | | ——Les Cayes & 200 Î | — Croix des Bouquets È 150 —Jeremie $ Da —Mirebalais se ES A % LS — g 100 DITS ET TS it ë en Si Re NE — Ouanaminthe ” FRS = î —Beladere 5 ' H — Lascahobas Q Q Q Q © © © © © © J J SJ J SJ J 12 12 —Pignon # S # ri °° ru S # r PE “ SP . PE Monthly Report FIGURE 13: CTE COLLECTION EFFICIENCIES (6 MONTH MOVING AVERAGE) 17 Collected arrears can inflate the monthly figure 21 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 37] Review of the 2021 Customer Survey data showed that less than | percent of CTE customers indicated that the reason for no longer being an active customer is due to the high price of water, although 10 percent indicated that they are no longer an active customer due to debt. Multiple stakeholders indicated to the ET that the overarching factor affecting CTE customer retention was the reliability and predictability of the water supply, not the price. For CTE-managed kiosks in Ouanaminthe, Cap Haitien, and Jeremie, Cap Haitien was the most profitable kiosk commune and the customer profile when determined using the EQUITY tool found that kiosk customers were not among the lowest income demographic!8. Billing and Customer Relations SIGA is the customer database CTEs used to manage their subscribers and was a key tool for managing the commercial and financial aspects of the CTEs’ work. More than 80 percent of CTE staff surveyed reported that their CTE uses SIGA, although their experience of using SIGA varied between CTESs, as shown in Figure 15. While every staff surveyed at Croix de Bouquets gave a positive rating of SIGA, only one staff member (8 percent) indicated a positive experience of using it. 100% 8% 25% 13% 75% mm“ 56% Cap Haitien Mirebakis Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets mPositive experience # Neutral experience Experience rating N/A # Don't use SIGA Don't know if CTE uses SIGA FIGURE 14: CTE STAFF EXPERIENCES OF USING SIGA WATSAN's support included connecting SIGA to mWater, which means that SIGA customer data can now be accessed directly via mWater. The majority of CTESs discussed in their FGDs how SIGA enables individual subscribers and their payments to be tracked on a monthly basis, and invoicing can now be done very much more quickly than before (e.g., 4,500 subscribers in a week now, rather taking more than a month (GI CTE Ouanaminthe)). CTE staff now have the billing information with which to contact any subscriber who has not paid in any given month. Customer-related data analysis was limited to a secondary data review of the 2021 SIGA database update survey, which had included with it a customer satisfaction survey of 8,987 active and 1,225 passive CTE customers across the CTEs of Cap Haitien, Les Cayes, Croix de Bouquets, Jeremie, Mirebalais and Hinche. The secondary data review (SDR) results are in Annex F. Additionally, the CTE Staff Survey also probed views on customers” opinions on tariffs and payments (see Tariffs and Collection Efficiency section below). 18 Profitability Survey by USAID Gender Specialist, FY21 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 22 [page 38] The Client Satisfaction Survey secondary data review showed that 81 percent of customers were billed in the last two months on average across the six CTEs surveyed. Over 90 percent of clients in Cap Haitien (97 percent), Les Cayes (92 percent), and Croix de Bouquets (91 percent) CTEs reported having received bills in the last two months, while Mirebalais (59 percent) and Jeremie (68 percent) clients reported a lower level of billing. In addition, issues in billing did not rank highly for customers being dissatisfied with their CTE. There was wide variation from Croix de Bouquets (58 percent) to Hinche (1 Ipercent) in response to “Are you satisfied with the CTE service provided for you?” The two main reasons for customers being passive were 1) not having water for over a year; and 2) water not being distributed on time. The billing process was also not perfect (e.g., 41 percent of Mirebalais customers had not received a bill in the last two months). The review also found that the majority of staff believe tariffs are appropriate and that clients are willing to pay for the services provided (except Hinche, where 56 percent of staff believe that the tariff is not appropriate).!° Tariffs and Collection Efficiency The household connection tariff is currently 60 HTG/cubic meter per household (100 HTG/cubic meter in Pignon) with each household typically consuming 10 cubic meters/month. À household monthly bill is therefore approximately 600 HTG/5.5 USD (1,000 HTG/9 USD for Pignon). The majority of CTE staff believe tariffs are appropriate and that clients are willing to pay for the services provided (except Hinche, where 56 percent of staff believe that the tariff is not appropriate).20 CONCLUSIONS (WATER) At the beginning of the Project, many CTEs were not functioning, had no water services, no revenue, and could not pay any bills, and some had debts. There was no system in place to reliably track revenues and expenses. The WATSAN Project succeeded in reversing this spiral of decline at the local CTE level, although there were significant variations in business performance and trajectory between CTEs. An annual CTE planning cycle was successfully established, although there is variation between CTESs in how often these plans are referenced for decision-making purposes. None of this progress would have been possible without the enabling data-driven environment of the mWater platform. Better billing and customer complaints handling means more information available for customer service, growth, and retention. However, there is variation in CTE staff awareness of mWater and in the regularity/reliability with which mWater data is collected and uploaded to the platform. This varies both between CTESs and between the | | mWater indicators, not all of which are consistently reported (in terms of regularity, continuity, and timeliness). The implications of this are that CTEs will need to support staff capacity development to optimally participate in data-driven decisions. Overall, the USAID WATSAN Project support to the CTEs was a game-changer for fully decentralizing the functioning of the 10 CTESs, and the same approach was also adopted by the other 17 CTESs that were not formally part of the WATSAN Project. This means that the WATSAN Project, although originally focusing on individual local CTES, affected regional and national water utility practices. For 19 Review of 2021 CTE Customer Survey conducted in parallel with a SIGA database customer update. 20 CTE Staff Survey 23 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 39] instance, it built governance capacity at local, regional, and national levels, although the Project did not focus on developing the national or regional regulatory environment for water supply utilities. The findings indicate that CTEs’ human resource management is a strong area. The majority of staff have a clear understanding of their roles and responsibilities, and CTE staff turnover is very low. Most CTEs have increased their revenue and improved their financial management, which is now well informed by revenue and expenses data, captured in QuickBooks. Some of the CTESs are relatively new to QuickBooks. With these increased revenues has come a “fear of success,” which relates to the need for greater financial management capacity to best manage future expenditure on capital infrastructure projects. Although the timeline per CTE is unknown, there is now the realistic prospect of CTEs becoming financially self-sustaining and no longer reliant on international donors. Many CTEs want to emulate Ouanaminthe; they like the idea of having greater revenues and performance-linked salaries. Maintaining and improving water supply standards is fundamental to CTE business sustainability. AIl but two of the CTESs provide less than 20 hours of water supply access per week, and sometimes with unpredictable regularity, which does not help retain customers. The future challenge is to achieve consistent reliability, regularity, and predictability of supply while also increasing service continuity to the (Ouanaminthe) level of 24/7 supply. Delivering on this will need a combination of capitalizing on the CTE business development capacity that the Project achieved and sufficient investment in infrastructure to make that level of service possible. Recording of infrastructure technical details sufficient to locate and repair water supply systems and networks at scale and at pace underpins the newly launched mWater infrastructure platform, which should support a reliable increase in service continuity, essential for business growth and stability. For example, following the August 2021 earthquake, emergency water bladders were deployed to priority locations rapidly identified with a survey on mWater to evaluate the status of different water sources. Although the SIGA customer database remains fundamental to CTE operation, revenue development, and customer service, there remains a need to better link together all four software packages for optimal functionality and performance. EVALUATION QUESTION I (SANITATION). To what extent has WATSAN met its Task Order objectives to build governance capacity at multiple levels (national,2! regional, local) to improve sustainable water supply and sanitation service delivery? The Project activities focused on a variety of sanitation stakeholders, including Latrine Emptier Associations, SMEs, Mayoral Task Forces on Sanitation, and FSM sites at Morne a Cabri and Fonfred. The Project’s approach was driven by the fact that providing a family with a safe toilet does not give that family safe sanitation unless there is also safely managed collection and transportation of the sludge and that sludge is safely treated and disposed. The Project activities mainly focused on the FSM in terms of addressing the issues of sludge treatment and disposal. There were some activities that supported sanitation at the household level, such as supporting SMEs in the construction of toilets. DINEPA found 21 The Projects activities in terms of sanitation focused on the local and regional levels. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 24 [page 40] that due to the absence of safe transport and treatment, the toilets would quickly fill and be abandoned. For this reason, DINEPA has forbidden subsidizing toilet construction. LATRINE EMPTIER ASSOCIATIONS The Project worked with latrine pit-emptiers and focused on training in business development, marketing, safety measures, and knowledge regarding the importance of sludge management, including visits to FSM sites in the area. In some instances, the LEAs were able to be promoted by the MTFSs, and they coordinated with the city hall especially regarding following standards for carrying out the pit emptying services and responding to the needs of larger clients such as schools and churches. During the FGDs with the LEAs, the participants shared how they felt prouder and more empowered with the work they are doing; this aspect was also shared during the Kils with the DAI staff. The Project staff are able to see how the pit-emptiers took more pride in their work, which also allows them to provide better service to a growing number of customers. SMALL AND MEDIUM SIZED ENTERPRISES The Project supported multiple SMEs throughout the Project areas. These SMESs participated both in training to help develop business management skills and a program to promote and encourage the construction of latrines at the household level. The latter was structed so that when 15 or more latrines were constructed by the SMEs, they received the first tranche of funding, and with 25 additional toilets they received the second tranche. This support included computers and office equipment. During Gls and FGDs with the SMESs, the participants discussed how the training helped improve their capacity, especially regarding management of the business, which improved marketing, human resource management, contracting, and budget preparation. INNOVATION DURING CHALLENGING TIMES During the Kils with the sanitation stakeholders, when talking about challenges, a success story emerged showing the innovation of some of the SMEs. “During Payi Lok (country lock down), it was difficult for the SMESs to reach their objectives because of travel problems. Nevertheless, they came up with a plan that surprised us; they carried out an activity called ‘Etrenne pour assainissement.” In Haïti, ‘Etrenne”’ is a time at the end of December when people share gifts. The activity ‘Etrenne assainissement’ launched in December was aimed at finding new clients and increasing business: they offered ‘specials’ for services and indeed found new clients. This initiative allowed us to see that these trainings allow them to fly on their own wings to reach their objectives despite the difficulties.”” 22 Lessons Learned in Urban Sanitation 25 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 41] MAYORAL TASK FORCES The MTF is composed of representatives of the different state authorities working in coordination with OREPA. The Project worked closely with municipal officials at Les Cayes, Jeremie, Mirebalais, and Cap- Haitien to raise awareness of the importance of basic sanitation. As per DINEPA's sanitation strategy, municipalities have a key role in managing sanitation in the communes. The Project helped the mayors and OREPA in these four areas to establish communal task forces to enable effective deliberation and to map and execute strategies to address sanitation issues.23 To help provide context for the work of the MTFSs, the aims for the MTF in Mirebalais or “Cellule de coordination pour l’assinessment de Mirebalais” were to: e Encourage owners living in the city center and its surroundings without toilets to build toilets, especially in the areas of Pylon, Lot Bo Latem, Eau Chaude, La Toilette, and Fort Anglais; e Encourage the people in charge of the markets and the city public schools to take over management of the public sanitary structures; e Take control of areas that facilitate open defecation; and e Establish a system that can encourage the community to invest in the sanitation market and use quality materials for toilet construction.24 During the Gls, the MTF discussed how the Project worked with them to help conduct sanitation surveys in order to get an understanding of the coverage of latrines/toilets in the town as well as public knowledge regarding sanitation. This informed the sanitation zoning that was undertaken as part of the Project. During Gls with the MTFs and with DAI staff, the issue of sustainability was raised, specifically how difficult it will be for the MTFs to continue with their activities once the support from the Project is over, particularly paying for the office space and organizing events. One of the MTFs shared how they carried out a survey that allowed them to learn how much of the population needs their latrine construction services. The MTF's activities have included a range of initiatives for its members including an awareness session during the COVID-19 outbreak and a special session on World Sanitation Day focusing on the importance of latrines and basic sanitation. The MTF also promulgated a communal decree, in conjunction with the town hall, concerning open defecation. They also undertook a census on the number of houses that do not have a latrine and that need the help of the MTF. PROMOTING LOCAL FRAMEWORKS One success story regarding the MTF coordination approach is in the city of Les Cayes. Like most cities in Haiti, it did not have any legal framework for sanitation. The MTF led the effort to improve sanitation in Les Cayes. However, they felt that their efforts have been hampered by a weak legal and regulatory framework. The Project provided the MTF with a consultant to review the national laws and municipal decrees governing sanitation. The consultant then worked with the MTF to develop a new municipal decree to better regulate sanitation at the municipal level. The focus of the new decree requires that à toilet be built for each house and that the type of toilet respects the sanitation zoning 23 WATSAN FY 2020 Annual Report 24 WATSAN Quarterly Report Oct-Dec FY 2022 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 26 [page 42] specifying which types of sanitation solutions are acceptable and appropriate based on the soil type, ground water table, and population density. Furthermore, the zoning specifies that all fecal sludge must be brought to the Fonfred Fecal Sludge Management Site for safe treatment. The mayor signed this decree at the end of January 2022 in a public ceremony.25 FECAL SLUDGE MANAGEMENT The majority of the Project’s activities focused on two FSM sites, both of which now have a business plan as well as an operation manual. The Project also helped finalize an agreement protocol with OREPA West and South and the Department of Sanitation to formally confirm the management structures of the FSM sites.26 By addressing the management of the FSM sites, the Project was able to address the issues at the discharge sites. For example, the Gls with the FSM staff and the FGD with MTFSs discussed that the area of the site was always considered a landfill site, either municipal landfill or dump discharge, and people complained about flies, but since the work on site with the principles and techniques being respected, people feel much safer in terms of their health. As part of the Project and to help with overall management of the two FSM sites, in December 2021, the Project signed an agreement for Fonfred through which the World Bank will finance the infrastructure works, USAID will finance the operational costs, and OREPA-South will provide the overall management? A particular quote from one the sanitation stakeholder highlights the issue with the end of Project. “The end of the Project is a major concern. We will not have enough financial resources to keep the Project alive. The site cannot pay the staff, and without the staff, without vehicles we cannot talk about a site.” During the KiIls, multiple sanitation stakeholders raised the issue of transportation and the need for both sites to have access to a greater capacity to transport fecal sludge to site. MORNE A CABRI The Project started working with the Morne a Cabri site in January 2020 and was able to increase the infrastructure and management capacity of the site’s 15 employees. The KIls discussed that the Project increased access to water at the site (which also helps reduce monthly costs for the site) and installed a lighting system. Additional work included emptying the sludge from the two anaerobic ponds, which had not been done since 201 1. The site is now able to receive sludge from the Port-au-Prince area. During the KIls with the sanitation stakeholders, it was noted that the management training helped significantly with accounting, invoicing, and customer relations. The staff shared that the site is designed to receive around 500 cubic meters of sludge per day but currently receives about 1,500 cubic meters per month. The FSM staff shared that the Project worked with OREPA West and the FSM site to organize an increase in the tipping fee from 170 to 350 and then to 500 gourdes (1.70-3.50-5.00 USD) per cubic meter and from 75 to 100 gourdes (0.75 to 1.00 USD) per drum. The Project also provided coaching to the senior staff on the importance of reaching out to the transport companies to encourage them to pay 25 WATSAN Quarterly Report Jan-March 2022 26 Draft Operations Manual 27 FY 2022- Annual Report 27 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 43] their invoices. Taken together, these changes help increase the overall revenue for the site and improve the chance for long-term sustainability. FONFRED The Project started working with the Fonfred site in March 2020. It had been abandoned in 2012, but became operational in March 2021 with nine employees. The site offers truck rental services and also treats wastewater. Currently there is a shortage of trucks, so the site is not able to keep up with consumer demand. In the Kils and FGDs, multiple sanitation stakeholders shared their concern regarding access to trucks needed to transport waste to the site. The users often have to delay services to customers because they are waiting for a truck rental. À World Bank Project to supply ten motorcycles with trailers for the site has been delayed, and FSM stakeholders shared that until this situation is resolved, timely and sufficient fecal sludge management will undoubtedly remain a big challenge. During the Kils, the FSM staff and DAI staff discussed how the Project supported FSM staff With training regarding business management and technical operations. They also shared that the site currently receives around 93 cubic meters of sludge a month; it would need to receive around 600-700 cubic meters a month to break even. MOUCHINETTE The Mouchinette site was not a key focus of the Project, but starting in 2022 the Project started working with Spanish Agency for International Development Cooperation (AECID) and USAID, in partnership with OREPA-North, to create an agreement to enable OREPA-North to acquire the necessary infrastructure and tools to commission and ensure the proper long-term operation of the fecal sludge treatment site in Cap-Haitien (similar to what was done for Fonfred and Morne a Cabri in terms of organizing the management of the site). The Mouchinette fecal sludge management site was built in 2014 as an emergency measure to treat fecal sludge generated during the Cap-Haitien Carnival. It operated for a few months and was then shut down. AECID is financing site reconstruction, two firms (Geotechsol and OZO Construction) have already signed their contracts, and the launch of the infrastructure work should start soon. The plant is a lagoon-based installation and is designed to receive 44 cubic meters/day of wastewater and sludge.28 CONCLUSIONS (SANITATION) The pit-emptiers, with the support of the Project, were empowered to organize and develop official associations, which allowed them to work together, raise their public visibility, and increase their access to the sanitation market. The LEAs in the Les Cayes area are now able to use the FSM site in Fonfred. The sustainability of the sludge disposal for the LEAs in Les Cayes will depend on how the FSM site continues to operate and if the transportation issue is addressed. Many of the SMEs became legal and registered companies with the support of the Project and were able to set up offices, which helped increase their individual visibility and access to markets. In addition, many of the SMEs were able to build a number of toilets as per the program. Having the SMEs increase their 28 WATSAN Quarterly Report Jan-March 2022 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 28 [page 44] capacity in terms of business management and increase their knowledge in terms of toilet construction allows the SMEs to continue this line of work without Project support. The MTFSs in Mirebalais and Les Cayes were particularly successful with coordination between the MTFs and LEAs and in the case of Les Cayes with the FSM site at Fonfred. These associations have a more developed level of organization and collaboration. The awareness campaigns that were organized with the Project’s support helped connect service providers with customers and increased the visibility of the MTPSs. This helped increase knowledge and use of companies for sludge disposal, which had a particularly high impact where an FSM site was available for use. The infrastructure capacity and functionality increased for both sites. For Morne a Cabri in particular, the Project helped increase access to water and electricity at the site as well as cleaning out the drying basins. The management capacity at both sites was improved with training for the staff that addressed issues and improved processes with payments, salaries, management of human resources, and technical procedures. Although the technical and management capacity of the FSM sites was improved, there are still issues regarding long term sustainability. One of the main issues regarding sustainability for the Fonfred site is the transportation issue: if businesses do not have access to trucks, they will not be able to dispose of their waste at the FSM sites. Additionally, there remain challenges with getting FSM clients to pay their bills on time and to recover payments in arrears. Both FSM staff and users shared that people in the area feel safer, because with the presence of the Project, the area is more visited. There was also an improvement in the area in terms of road infrastructure to facilitate the passage of trucks using the site, which likewise benefited the community. By working with multiple sanitation stakeholders, the Project was able to assist in the construction of latrines and toilets at the household level, increase the role of the MTFs in their awareness-raising activities regarding the importance and impact of having a latrine, and increase the management and business capacities of the SMEs, LEAs, and FSM. With the increased capacity achieved by these organizations, they are now better able to provide more sanitation services to the communities. À good example of this is the work in Les Cayes, where the Project was able to combine work on all three segments or areas of transportation (storage, emptying/transport, and treatment). Overall, with the support of the different sanitation partners (LEAs, SMEs, MTFSs, and FSM), the Project was able to increase capacity and governance, which in turn helps ensure more sustainable sanitation services were available. This is particularly true for areas served by the Fonfred and Morne a Cabri sites. EVALUATION QUESTION 2. How was the Enterprise Acceleration Fund utilized, and to What extent did those grants support WASH enterprises (such as MSMEs, NGOSs, and Bayakous) to move toward sustainable service delivery? The ET was able to interview all the organizations and companies that received an EAF grant and also interviewed key personnel from the DAI team regarding the EAF grantee activities. Seven grants were awarded during the Project: four had a focus on sanitation activities and three had a focus on water activities. In the water sector, funds were used to build water kiosks with two partners to increase access to water in the Cap Haitien area and to construct a water quality testing laboratory Within the University of Limonade. In the sanitation sector, funds supported research activities to test if 29 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 45] raising black soldier fly larvae was viable, to develop capacity regarding sludge testing and drying, to provide access to and test a microloan concept regarding the construction of toilets and latrines, and lastly to increase services for incineration of waste.2? As discussed before, gender was not an explicit focus of the Project and it was not a driving focus of the EAF grants. The DAI staff shared that the application process did not target female-owned organizations or other organizations that tend to have difficulty receiving funds. The staff also shared that many of the proposals needed to be reviewed and updated to help meet the Project’s objectives. During the Kils with EAF grantees, some respondents said the application process for financing and communication was not easy and the overall process took longer than expected. This affected the overall timeline of the Project and what was able to be completed. There was often a lot of back and forth to finalize the concept notes. For other grantees, the process was clear, especially with the support provided by DAI staff members. As part of the funding process, many of the grantees mentioned the gender mainstreaming training they had received. During Kils, it was revealed that due to delays with purchasing equipment, receiving funds, and receiving the equipment, it is difficult to determine the impact of the grants to date. At the time of the evaluation, the Projects have either not been completed or are only just finishing up. There were additional delays due to political unrest, with the associated cancellations of visits by technical experts as well as delivery of key technical training elements. There are concerns among the grantees and the WATSAN Project team regarding sustainability. Two grantees (Sanitation 509, University of Quisqueya) have not yet received equipment. A few of the grantees also have concerns about how to undertake or fund any future repairs. Although each grantee was interviewed for Evaluation Question 2, the ET received limited documentation regarding the progress of the grants or evaluations of the grants by DAI, which created an issue for the analysis for the question. Table 5 gives a general overview of the EAF grants and their current status. TABLE 5: EAF GRANTS AND CURRENT STATUS ORGANIZATION SECTOR STATUS Living Water Water - Kiosks Completed — not all kiosks function MFSN Water - Kiosks Completed — issues with water supply/quality Limonade University Water - Laboratory Delayed in receiving equipment, but the laboratory is set up Le Levier Sanitation — micro credit for toilet Only 120 of the planned 500 beneficiaries construction reached SOIL Sanitation — research on black fly Completed; able to show black fly larvae can be larvae produced 29 FY 2020 Annual Report USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 30 [page 46] ORGANIZATION SECTOR STATUS University of Quisqueya Sanitation — research on drying of Delayed in receiving equipment, only one fecal sludge deliverable received Sanitation 509 Sanitation - Incineration Delayed in receiving equipment, not yet set up LIVING WATER AND MOUVMAN FANM SOLÉY NÔ Living Water International (LW/I) is a local foundation that promotes sustainable water access, sanitation, and hygiene. Living Water’s goal for the grant was to increase water access by extending existing water supply services, giving water access to 25,050 people living in the urban area and periphery. The Project also sought to strengthen the sustainable management of water services within the area by creating and improving the management capacity and business skills of 12 water management bodies or water service operators. Mouvman Fanm Soléy No (MESN) is a women’s organization that was created in 2005. MFSN received a subcontractor from LWI as part of their EAF grant. The MFSN grant covered the cost of constructing two boreholes to supply the kiosks already built under a grant agreement with the Living Water Foundation as well as awareness and training activities for the kiosk management committees. The LWI site visit/evaluation completed in July 2020 indicates that there were some changes in the activities during project implementation, but according to LW/I, all activities of the two main components were carried out. Rehabilitation of the handpumps and construction kiosks was completed, but there was no specific monitoring and evaluation plan for this Project. LWI was not able to generate detailed information on beneficiaries. Additional follow-up was needed to understand the impact six months after completion as well as a lessons learned workshop planned for August 2020. These additional reports were not shared with the ET, so it is unclear if they were produced.30 The KiIls discussed that COVID-19 made the implementation process long, and there was also an increase in prices, especially fuel, that affected the Project overall. À renewable energy source and a complete treatment system would have been a better option. The majority of the kiosks started operating at the end of 2020. The evaluation of the MFSN Project stated that the objective to provide water to 3,000 people was not reached because of the near non-functionality of the kiosks; this was confirmed during the Kils. Due to these non-functioning kiosks, the management committees also never functioned. One kiosk never functioned because the borehole could never supply it, and no investigation was done to determine why (at the time of this evaluation). The MFSN evaluation states the second kiosk is only 10-20 percent functional due to issues with the batteries. According to respondents, some members of the kiosk try to run it with a 400-watt generator. The batteries were described as being overloaded by the population by the MFSN evaluation; the Kils described it as more of an issue with quality. The evaluation stated that one of the biggest problems that led to the poor results of this grant is a lack of communication between DAI, Living Water, and MESN, and this was also reiterated during the Kils. MFSN was brought 30 Evaluation of Living Water International July 2020 31 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 47] in to complete the Project as a sub to Living Water International (LW/PS but was not provided with all the information upfront.3! EAF and DAI KII respondents shared that the LWIS and MFSN water kiosks did provide access to 24,000 people in areas that had previously struggled with access to clean water. A total of 14 kiosks were constructed, but due to equipment and water quality issues many of these kiosks are not currently functioning. During the fuel crises, many of the systems were not working; only those with solar panels worked. It is clear that the two kiosks that MFSN and LWIS worked on together are not functional, but to get a clear understanding of the issue a thorough investigation would be needed. There have been complaints regarding water quality, either in relation to issues with turbidity and/or salinity. There were also training workshops for 14 operators to help in small business management. However, the kiosks” management committees/businesses currently have little savings for future maintenance and expenses, which is concern for the kiosks’ long-term sustainability. Their sustainably will depend on their functionality, and at the time of the ET some were not functioning and were unable to make repairs. UNIVERSITY OF LIMONADE The University of Limonade worked with Auburn University to 1) develop a water testing laboratory; 2) train university students in water quality testing techniques; and 3) develop a database containing water tests from providers in the region. The proposal for the University of Limonade was not shared with the ET, but KII respondents confirmed that the objective of the grant was to continue the support of the university’s technical lab. A previous grant by the USAID AVANSE (Appui a la Valorisation du potentiel Agricole du Nord, pour la Securite Economique et Environnementale) Project funded the development of a soil testing lab. This grant was used to develop a water quality lab (testing equipment purchase and complete some training). This additional support allows the University of Limonade to conduct both water and soil quality testing for the North and Northeast departments, something that was not available in the area beforehand. During the Kils with the EAF, it was shared that the marketing of these new services was not very effective but the lab was able to do some testing for kiosks. At the time of writing this evaluation, the laboratory is unable to pay the technicians since they do not yet have very many clients but they agree nonetheless that there was a positive impact on academic improvements for the university. Auburn University developed the laboratory procedures for drinking water analysis. In addition, a study is being prepared to ensure that water quality in Ouanaminthe meets drinking water standards.3 The KIIs also discussed if requirements regarding water quality testing of kiosks would help the overall business of the lab, but for now there is limited demand for the services. This is likely to improve once more people are aware of the services offered. LE LEVIER Le Levier is a federation that has expertise in housing finance. The LEVIER Federation planned to use the grant to pool non-financial services (awareness and training) and financial services (product 31 Evaluation Mouvman Fanm Soléy N6 August 2021 32 WATSAN Quarterly Report Jan-March 2022 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 32 [page 48] development, savings advice, and credit financing) to stimulate the populations of the five zones the Project is working in to create a healthier environment for both individual and collective well-being by constructing household toilets and latrines.33 Le Levier planned to make sure that the credits disbursed will be effectively allocated to the construction of toilets and/or bathrooms. The granted credits will be disbursed in installments; the first disbursement will be made according to the estimate or arrangements made between the borrower and the contractors hired, and subsequent disbursements will be made according to the level of progress of the work validated by the credit officers of the various funds concerned.34 The EAF Gls revealed that the micro loan program for toilets and latrines was able to provide 120 credits and trained four people, but the target number 500 of beneficiaries was not reached. The Kils with the DAI staff and the EAFSs grantees noted that the fund will be able to continue even after the Project ends to provide access to credit for people who want to build a toilet. Le Levier feels this Project was successful and was surprised with the demand for modern toilets instead of latrines. This demand shows the financial products that have been designed can be adapted. This increase in demand for modern toilets required more money to be invested per household credit than planned. This additional information will help them plan and adjust for future projects. Each fund in the network had a different experience in the implementation of the Project. In Plateau Central, the funds had a liquidity problem that had hindered the implementation of the project, while Cap Haitien and Croix des Bouquets did not have the same difficulties. Le Levier hopes to continue the Project’s activities as the funds are still available; the repayment of the micro credit keeps the funds available to be used with new beneficiaries. SOIL SOIL is an NGO that provides household sanitation services where revenues are collected from toilet users and through the sales of compost. For now, household service is only available because it is subsidized by SOIL to the general public. The objective of the grant was to help with the diversification of waste to value end products. The grant funded pilot trials on black solder fly larvae (BSFL) production, which is used to break down waste and in turn transforms the larvae into protein rich chicken feed. There were four objectives for this grant: 1) cultivate BSFL colony for continuous supply of five-day old larvae (when the larvae are ready to be put on the biowaste); 2) achieve consistent and optimized waste transformation into larval biomass; 3) ensure product safety and nutritional value; and 4) understand market demand and sales pricing for BSFL animal feed in Haiti.35 During the Kils with EAF, SOIL was able to confirm that it was viable to produce the black fly larva and that there was a market for this product as a feed for chicken and fish. SOIL is hopeful that they will be able to start production of the product in the near future, which will help them make more money from the composting toilet. This will in turn help the overall sustainability of SOIL and the composting toilet service they offer. 33 Le Levier Proposal 3 Le Levier Proposal 35 SOIL Proposal 33 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 49] UNIVERSITY QUISQUEYA This Project was carried out by the University Quisqueya (UniQ, Haiti) with other universities.36 The grant’s main objective was to evaluate, on the basis of a full-scale or real demonstration trial, the feasibility of implementing the Projet de Lits de Séchage Plantés de Végétaux (LSPV) technology for the treatment and reclamation of latrine pit sludge in Haïti. KII respondents discussed that due to the delays of materials, only one deliverable was produced at the time of this evaluation. The materials purchased as part of the grant arrived in March 2020, meaning the acquisition of this equipment took more than 12 months. Extensions on the other deliverables have been given to UniQ, on which it will work once the equipment has been installed. This will allow UniQ to continue the work and hopefully address some of the sustainably issues. SANITATION 509 Sanitation 509 is a business that has 20 years’ experience working in waste management (including collection and treatment of waste). This Project has already received funding from the USAID — AVANSE project for the acquisition of equipment and materials. This EAF grant was to be used to develop the physical and electrical infrastructure and will be used to purchase vehicles to start the collection and treatment of the hazardous waste, including a waste storage unit, a processing unit, a sterilization unit, an incineration unit, and a depot to store equipment and associated amenities. GI participants said that due to delays the Project was not completed by the time of the evaluation. Some of the work had finished, such as part of the perimeter fence and work on the electrical supply, but the majority of the equipment had yet to arrive on site, soit is difficult to determine the success and impact of the grant or look at the overall sustainability of the grant or business activities. OVERALL EAF CONCLUSIONS The grants were successful in terms of increasing the number of clients for the organizations both in terms of access to markets and in terms of the range of services offered. For example, the kiosks provided access to water to 24,000 people, the micro grants for latrines and toilets provided access to credit for 100 people, and Le Levier will continue this service. In addition, the lab at the University of Limonade can now provide water quality and soil testing. The research activities for SOIL and UniQ proved to be a success as activities have been able to get off the ground. Due to delays and some challenges, it is difficult to determine if the sustainability of the organizations was improved by the grants. In some instances, there have been issues with equipment and possible maintenance issues that affect their overall sustainability. Especially due to some of the delays and how some of the grants were not finished at the time of this evaluation, the question regarding sustainability is difficult to answer at the moment; this will be clearer as the grantees finalize the Projects and carry the activities into day-to-day production or operation— 36 UniQ worked in collaboration with the State University of Haiti (UEH) and the Ecole de Technologie Superieure (ETS) in Montreal and the Institute national de recherche en sciences et technologies pour environment et agriculture (IRSTEA, France). UEH, UniQ, and ETS have an MOU that was signed in January 2016. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 34 [page 50] and if evaluations from the Project were planned and carried out. In addition to the delays, there were communication issues that affected the overall implementation of the grants, and the overall application process was not always clear and straight forward. EVALUATION QUESTION 3. How effective was WATSAN’s approach of targeting the end of the sanitation value chain in improving the overall sanitation value chain? The ET looked at a number of sub questions and interviewed many people who supported and worked with the FSMs. The FSM management teams and users for Fonfred and Morne a Cabri were interviewed, as well as key personnel from DAI that worked with the FSM teams. Before the Project, the Fonfred site was not operational and there was no place in the area to safely dispose of fecal waste. The Morne a Cabri site was open and accepting waste, but the treatment basins were full and the site was not properly managed. During the Project, the overall management of the sites was formalized, now under the supervision of OREPA and the Waste Water Treatment Plant Coordinator.37 The Project assisted in the coordination and involvement of different actors including the FSM sites, SMES, LEAs, and MTFs. The interviews with the sanitation stakeholders indicated that these activities encouraged improvement to the sanitation situation for the towns; there was knowledge-sharing about the importance of latrines and the services available and information on the service providers. Some of this work also dealt with the creation of sanitation zoning, but there are limited regulations being implemented for toilet construction and emptying of toilets. According to the FGDs with the LEAs, the Project increased their visibility due to the marketing support and coordination with the MTFSs, so the LEAs were able to build their businesses and increase their customer base. In FGDs, LEAs in the area of the FSM sites shared that access to these sites means they no longer have to dump the sludge in a ravine or dig a hole on the property of the client to relocate waste. Also, with the additional training, the LEAs discussed that they are less likely to take on sludge removal work if there is nowhere to dispose of the sludge. Users for both sites and other sanitation stakeholders all mentioned the improvement to the general cleanliness, smell, and environment in the surrounding areas as well as the MTF in Les Cayes. During the interviews, there was a discussion of the perceived improvement to the surrounding areas in terms of the smell and improvements to the health of the surrounding area. The users of the sites shared during the Gls that they recommend the use of this site to other businesses, but it was discussed that more informal businesses are not likely to use the site due to the current level of tariffs. Some of the companies that use the sites to provide sludge removal and disposal services shared they have a number of clients, ranging from 10 to 300, including households, NGOs, state institutions, and private companies. Having an operational FSM allows the users to provide receipts to clients and advertise that they properly dispose of waste as part of the services they provide. This is particularly important to some clients such as international NGOs and the U.S. Embassy. 37 Draft Operations Manual pg. 17 35 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 51] This Project was able to focus management support for the FSM that addressed the end of the sanitation value chain. MORNE A CABRI The Project was able to support the FSM both with infrastructure inputs and building the capacity of the staff. The Kils with DAI and the FSM staff noted that this site was able to implement Office National d’Assurance-Vieillesse (ONA) and Office d'Assurance Accidents du Travail, Maladie et Maternité (OFTAMA) insurance for their staff, which is a requirement for businesses in Haïti. Also, currently the site is running on a cost recovery level, so it is making enough to cover salaries and general costs. The capacity building activities included development of a business plan, help with accounting, software to manage invoices, and training on conflict management, which the staff felt have really helped with human resources management and negotiating with clients to pay user fees. The FSM staff shared that the site currently had 15 companies/users that dispose of waste on a consistent basis. The ET was able to conduct a GI with some of the users and found that they are generally satisfied with the FSM site and worry about if the site will close after the Project support ends. The FSM users shared that there are some complaints regarding wait times, disposing of waste, receiving and processing paperwork, and the tariff level. The site being open officially allows clients to advertise and inform their clients of the proper disposal of the waste. This is particularly important for NGOs and other organizations such as the U.S. Embassy that request documentation of proper disposal. FONFRED During the Gls with different sanitation actors in the area of Fonfred, it was noted that there is better environmental management of the site and the surrounding area now than before the Project. The sanitation actors said there are issues regarding sustainability since the amount of sludge received each month is not enough to support the site and cover all costs. Right now, the site is paying salaries and had an accountant but due to issues with transportation, it is not able to meet the demand in requests for help with transport to dispose of waste, meaning they are missing out on revenue due to a lack of transportation. À World Bank project was supposed to purchase ten motorcycles with trailers for the site, but this was delayed, and it is unclear when this will be implemented. The KIls with sanitation stakeholders noted that the site currently had seven companies/users that dispose of the waste on a consistent basis. The users are generally satisfied with the FSM site and discussed how they have to explain the importance of the disposal of sludge at the site to their clients. EMPOWERMENT OF SANITATION STAKEHOLDERS The FSM teams really felt supported by the Project, and now there are two FSM sites operating in Haiti. The LEAs feel empowered with their work and have been able to improve their businesses and the services they provide, and also increase their community profile, acceptance, and marketing, which in turn gave them access to new clients such as churches, schools, and NGOs. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 36 [page 52] CHALLENGES AT FSM SITES Some of the challenges in the sanitation sector come from the fact that the FSM sites in Morne a Cabri and Fonfred are the only two functioning sites for all of Haïti, so some of the LEAs, SMEs, and MTFs that worked with the Project do not have access to an FSM site (such as in Cap Haitien and Mirebalais). The sanitation stakeholders said that there were a number of challenges that the Project activities faced, especially the coordination and delay in Project activities from the other donors that created issues for the sanitation Project activities, such as the motorcycles with trailers from the World Bank and the work with IDB for the FSM site at Mouchinette. The delay in Fonfred exacerbated the transportation issues, which in turn may affect long term sustainability of the sites. The challenge for companies is their inability to convince people to seek their toilet construction services. There is a lack of knowledge at that level. It was difficult to change people’s behavior regarding building toilets in the traditional way. The GOH should encourage people to build toilets in their homes, which would increase the demand for businesses. CONCLUSIONS With the help of the Project, both sites are operational and are on the path to operating like a business. They operate under a documented business model but were not yet profitable at the time of the evaluation. These sites provide the communities around them in Port-au-Prince and Les Cayes a safe place to dispose of waste. The opening of the FSM sites allows business to demonstrate proper disposal to clients and this helped their businesses. It is important to note that while the infrastructure and management capacity of the FSM sites improved due to the Project activities, sustainability is still an issue as the sites having problems with transportation and those not receiving enough customers/sludge each month to be profitable. There have been perceived health and environmental improvements in the communities surrounding the two FSM sites, as the waste is now disposed of in a controlled area and there is less disposal in the surrounding ravines. The coordination and involvement of different sanitation stakeholders encouraged improvement to the overall sanitation situation in the Project areas. The coordination included knowledge sharing about the importance of latrines and what services are available in the area as well as information on service providers. By targeting the FSM sites, the Project was able to focus on the end of the sanitation value chain, and these improvements increased access and use of a safe disposal system for the sanitation stakeholders. This also allowed the users of the sites to expand and market their businesses. The long-term impact Will depend on how the FSM sites can function without the support of the Project, as there are still ongoing issues at the site including transportation and receiving enough sludge to cover costs. 37 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 53] CROSS-CUTTING ISSUES The ET considered some of the cross-cutting issues that affected all aspects of the Project. The main events the Project had to navigate are summarized in Figure 15. Except for the first year, the Project faced a constant series of crises through implementation. 2018 2019 2020 2021 202 tr4 Qtr1 Qtr2 Qtr3 Qtr4 Qtri Qtr2 Qtr3 Qtr4 Qtr1 Qtr2 Qtr3 Qtr4 Qtri Qtr2 Qtr3 Qtr4 Qtr1 Qtr2 | Qtr3 ; Evacuation of USAID personnel : Î É Lockdown over COVID : Ë COVID Restrictions É 38FIGURE 15: PROJECT ISSUES TIMELINE GENDER Although addressing gender-related issues was not the focus of the Project, as stated by many of the participants, the Project did have a gender specialist present, and some trainings were offered to help address sexual harassment and gender roles for the CTE staff, kiosks managers, and SME and LEA workers. As part of the research on the Project’s approach to gender integration and gender equality outcomes of activities, the ET asked the CTE staff a number of questions regarding gender issues, integration, and related training activities. Each figure below shows the percentage of respondents for each CTE. 38 Applying Adaptive Management for Results in a Fragile Environment, USAID Water and Sanitation Project Learning Note, 27 April 2022 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 38 [page 54] 71% 77% . 78% 67% " 71% 25% * 2 50% 29% | Cap Haitien Mirebalis Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets M Participated in a sexual harrassment training mDid not participate in a sexual harrassment training + FIGURE 16: PARTICIPATION IN SEXUAL HARRASSMENT TRAINING Figure 16 demonstrates that for the majority of CTES, 50 percent or more of the staff were able to participate in sexual harassment training as part of the Project. 100% TE 1% 13% 80% En 60% LL U © FU © : 40% 25% F 20% Es 0% Male Female} Male Female! Male Female! Male Female! Male Female! Male Female} Male Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience m Negative experience Experience rating N/A FIGURE 17: CTE'S MANAGEMENT OF CTE'S GENDER ISSUES BY SEX Figure 17 shows while 93 percent of CTE staff noted a positive experience regarding the CTE responsiveness to gender issues responses from male and female staff differed. Ninety-seven percent of male staff noted a positive experience, while 77 percent of females expressed the same opinion. This statistically significant difference between the two groups is due to the mixed responses from the Cap Haitien CTE staff. Eighty nine percent of male staff indicated a positive experience regarding the CTE’s responsiveness to gender issues, with only 25 percent of female staff responding that they have had a positive experience. It is notable that half of the female staff from Cap Haitien chose to not rate their experience. 39 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 55] 100% 8% 14% [10% | 11% 75% 23% 40% 10% 11% 29% 25% 0% Cap Haitien Mirebakis Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets M Positive experience Neutral experience m Negative experience Experience rating N/A FIGURE 18: CTE'S MANAGEMENT OF COMMUNITY GENDER ISSUES When discussing community gender issues, the CTE staff still rated the CTE’s overall management experience as positive but there is more variation than when discussing the management of gender issues internally (See Figure 18). Overall, the data indicate that the CTE staff have positive experiences regarding gender issues. The question in the survey was very general, so it is difficult to define or talk more in depth about the gender issues faced by the CTE staff. The CTE management staff also shared during the FGDs that the “training sessions raised awareness on the participation and respect of women in the workplace and in the management of kiosks. The only obstacle to the participation of women in the execution of activities and jobs remains gender stereotyping of gender roles. However, as clients, women receive the same service as men because the focus is on households.” The CTE HR manual notes a 30 percent quota for the minimum proportion of women working for the CTEs. Some of the CTEs have reached this goal. During the FGDs with the kiosk managers, LEAs, and SMEs, the participants said they enjoyed the trainings, which reenforced that woman are capable of doing the same jobs and activities as men. Some of the focus groups with kiosk managers discussed how the increase in water access positively affects women more since they collect the water and they also benefit from the better-quality kiosk water in terms of health (e.g., a reduced incidence of skin diseases). HAITI POLITICAL/ECONOMIC SITUATION AI stakeholders discussed how the political situation in Haïti is very complicated and affects everyone’s daily life. It affects all aspects of the economic system: lack of suppliers and less competition, price increases (especially fuel), associated delays in activities and Project implementation, difficulties with monitoring activities in the field, and difficulties with access and travel due to à fluid security environment, all of which affect WASH service delivery. There are demonstrations that affect how or if people can get to work or to a planned activity. In particular, the CTES said that fuel shortages have affected transportation and the availability and cost of fuel for CTESs to pump and treat water. To help mitigate fuel price fluctuations, the Project in some cases provided fuel subsidies to help with the cost of purchasing fuel and funded construction of fuel storage facilities to ensure fuel availability and continuous water supply. The CTEs and OREPA South discussed how there are also increased security issues in the area of Martissant, which affects the transport of people and equipment to this region and can cause delays and increase the price of materials. The CTE in Croix de Bouquets and some of the implementing partners USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 40 [page 56] in that area discussed the impact of security and gang Warfare in the Croix de Bouquets area, which affected the construction work on the water system and monitoring and follow up activities. CTE staff in Croix de Bouquets are now working out of Delmas due to these security issues. The insecurities in the Port-au-Prince area also affect the usage and access to the Morne a Cabri FSM site. For some implementing partners, the biggest challenge was security. For example, during construction work in Criox de Bouquets, workers there had to talk with and manage gang members on a daily basis to negotiate access to Work sites. COVID-19 The COVID-19 outbreak hit in March 2020. At first Haiti was closed to international travel, and in country there were many requirements regarding the organization of events. During the Kils with DAI and FGDs with the CTE staff, it was shared that because the Project already had support staff based in the CTES, this did not have a huge impact. The Project was able to be flexible and still move forward with its objectives, especially the capacity building at the CTE and FSM level. Many of the participants in the interviews and focus groups mentioned that COVID-19 did not really affects the Project, except with some adjustment in how activities were planned. EARTHQUAKE The earthquake that hit the South of Haiti on August 14, 2021, was devastating, and had impacts on the water and sanitation services in the area. The FGDs and KIls with Les Cayes Project participants shared that there were damages to some of the pipes of the Les Cayes water system, which triggered a deterioration in the quality of water. Meanwhile, some of the LEAs in the South also had to change office locations, which temporarily affected their visibility. ONEPE/OREPA shared that DINEPA and OREPA were able to help organize and coordinate the NGO-supported earthquake response, which was done by creating and using a survey on mWater to evaluate the current status of different water sources, including kiosks. This helped determine where water trucking was needed and the best locations for temporary water storage facilities. LAND TENURE Land tenure seemed to be an overarching theme, which was brought up during some of the higher level Kils, especially in relation to obtaining permission for household toilet construction. In Haïti, land tenure issues affect all projects, including where kiosks could be constructed. Most people rent or do not have titles for their land, so investing in a toilet or latrine is difficult or not a priority, or simply not viewed as their responsibility. CROSS-CUTTING CONCLUSIONS Some Project stakeholders perceive that due to increased access to water, and the fact that women tend to be the primary collectors of water, they have benefited more proportionally from the Project than men. The Project was able to help with some of the impacts from insecurity (such as fuel shortage and price increases), but it is impossible to mitigate all impacts of this issue. The Project was also able to adapt to the COVID-19 situation. The earthquake had a negative impact on the Les Cayes water system, 41 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 57] but DINEPA and the Project were able to respond quickly and effectively, and USAID has extended the Project by six months. RECOMMENDATIONS USAID/HAITI 1. Ensure future programming supports CTESs to establish milestones for reaching defined service delivery standards that are linked to the SDGs and tracked consistently in mWater. 2. Engage DINEPA/GOH to enable SIEPA to have better data connectivity between the various software packages for the CTESs. 3. Continue the collaboration between DINEPA/GOH and mWater to develop checklists, surveys and tools to be used by the CTESs to respond to climate-related weather emergencies. 4. Continue to support the FSMs in their efforts to reach profitability/sustainability by expanding infrastructure to help them increase the amount of sludge they can process and, in particular, enable greater fecal sludge transportation capacity. A lot of progress was made with the FSMs during the Project, but this support will need to continue to ensure the progress is not lost. 5. Continue to support coordination among sanitation sector stakeholders to build GOH capacity in the development and enforcement of FSM regulations. To help ensure households have toilets and latrines and use FSM sites for disposal of waste, there will need to be national regulations that are enforceable at the local level. 6. Engage DINEPA/GOH to make requirements for water testing for water suppliers to ensure clean water is available to the communities (these suppliers could be clients of the water quality lab at the University of Limonade). 7. Encourage the use of mWater for sanitation management as well. Take the successes from the CTESs and apply them to the FSMs. Ensure via stakeholder consultation and engagement that relevant data is reliably collected and visualized to help with decision making and future sanitation business development. Ensure training regarding use of mWater and other software are provided to the FSM management teams. USAID IMPLEMENTERS I. Undertake a needs assessment with CTEs on how they can grow their financial and asset management capacity to support accountable capital expenditure on infrastructure. 2. Address data continuity/quality issues for data reported on mWater. 3. Support mWater’s capacity to map all water supply infrastructure to improve CTE technical operations. 4. Develop a standardized approach for all CTE-managed kiosks that includes an indicator(s) for kiosks to supplement the current single (technical) indicator of percentage functional kiosks (e.g., the hours per day of operation metric used by CTESs for household connections). USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 42 [page 58] 5. Use EAF-type grants to support female-lead/owned organizations and projects. These grants are a great opportunity to support organizations that struggle with access to funding. With more attention given to gender issues and by providing the added support these organizations might need, there could be a more focused impact on women and girls for local water and sanitation services. 6. Ensure the activities supported by EAF grants are more strategically aligned with WATSAN activities and goals. Both FSMs struggle with transportation of waste to the sites. As the goal of the EAF is to support innovation, it would be advantageous to try to support organizations and businesses that directly influence areas that are challenging for the Project. 7. Improve the organization of the EAF grants process to better manage grantee expectations around the proposal steps and communication arrangements. The overall organization of the grants could be improved, especially regarding communication with grantees and other stakeholders on the application and implementation process (such as the status of equipment purchases). 8. Ensure monitoring and evaluations are completed for the EAF grants. To ensure the grants have the intended impact, it is important to discuss and agree upon indicators with the grantees, so they are able to monitor and share with the implementing partner. It is also important that evaluations of the different stages of the process are undertaken by the implementor so improvements can be made. The extension of the Project provides DAI with the opportunity to complete the evaluations for these EAF grants as they are finishing up. 43 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 59] ANNEX A. SCOPE OF WORK STATEMENT OF WORK FINAL PERFORMANCE EVALUATION OF USAID/HAITI'S WATER AND SANITATION (WATSAN) PROJECT INTRODUCTION SUMMARY INFORMATION BACKGROUND 3.1 Description of the Problem and Context 3.2 Description of the Intervention to be Evaluated 3.2.1 Goal and Strategic Approach 3.2.2 Development Hypothesis and Theory of Change 3.2.3 Performance Information EVALUATION PURPOSE 4.1 Evaluation Use and Users EVALUATION QUESTIONS DESIGN AND METHODOLOGY DELIVERABLES AND REPORTING REQUIREMENTS Z.I Report quality criteria TEAM COMPOSITION SCHEDULE FINAL REPORT FORMAT OTHER REQUIREMENTS 44 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 60] STATEMENT OF WORK (SOW) FINAL FINAL PERFORMANCE EVALUATION OF USAID/HAITIS WATER AND SANITATION (WATSAN) PROJECT LL INTRODUCTION USAID/Haitis Water and Sanitation (WATSAN) Project is a four-and-a-half-year Task Order awarded to DAI Global, LLC, with V3 Engineering, Ayiti Nexus, mWater, Centre et Formation et d’Encadrement, and Zanmi Lasante as sub-awardees. The activity started in December 2017 and will be completed in June 2022. WATSAN interventions aim to support access to sustainable water supply and sanitation services, and strengthen the enabling environment for sustainable delivery, operation, and maintenance of water and sanitation services. The project started by targeting the urban areas of five communes: Cap Haïitien, Mirebalais, Croix des Bouquets (Canaan), Les Cayes, and Jeremie. Later, WATSAN added five more communes: Belladere, Hinche, Lascahobas, Ouanaminthe, and Pignon. The key objectives of the project are to build the governance and financial capability of targeted Haitian public utilities and private operators while enabling 250,000 people to gain access to new or improved water services and 75,000 people to gain access to basic or safely managed sanitation services. The WATSAN project supports sector institutions by implementing a combination of targeted infrastructure improvements and technical assistance prescribed by the “Utility Turnaround Framework”. WATSAN also supports sector MSMEs and promotes innovations through its Enterprise Acceleration Fund. IL SUMMARY INFORMATION Total Estimated Ceiling of the Evaluated $41,812,295 Project/Activity (TEC) Active Geographic Regions Belladere, Cap-Haitien, Croix des Bouquets, Hinche, Jeremie, Lascahobas, Les Cayes, Mirebalais, Ouanaminthe, Pignon Office of Infrastructure, Energy and Engineering (OIEE) 45 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 61] III. BACKGROUND 3.1 DESCRIPTION OF THE PROBLEM AND CONTEXT Access to water and sanitation in Haiti is the lowest in the Western Hemisphere. Only 65 percent of the population has access to improved water sources, a slight increase and slow progress since 1990, when the access rate Was 62 percent, as increases in services were not able to keep pace with population growth. However, actual access rates may be lower, as many water systems’ functionality is poor. Only 28 percent of the Haitian population has access to improved sanitation, a rate that has been relatively constant since 1990. Safe collection, transport, and treatment of human excreta are practically non- existent throughout Haiti. The Government has focused on awareness and promotion campaigns to encourage households to build their own latrines while also addressing the need for sanitation facilities in public schools, health institutions, and other public spaces. These campaigns to reduce open defecation have been somewhat successful; open defecation rates have dropped from almost 50 percent in 1990 to about 20 percent in 2015. However, surveys indicate that the use of improved sanitation facilities has only increased from 18 to 28 percent, meaning that much of the population still relies on shared or unimproved sanitation facilities. While sanitation access rates in urban areas are higher than the national average of 28 percent, they are still below 50 percent. In informal or unplanned settlements, where the poorest and most vulnerable urban populations generally live, urbanization and high localized population density may be accompanied by an increased risk of infectious disease transmission, primarily impacting the poor. The combination of a high fecal-related disease burden and inadequate infrastructure suggests that investment in expanding sanitation access in densely populated urban slums can yield important public health gains. In addition, climate change and other natural and manmade disasters as well as rapid urbanization will threaten water resources and gains made in the water supply and sanitation sector. Building the capacity of utilities and private operators to anticipate needs, plan for and finance improvements and to expand access to underserved communities on a sustainable basis is the strategy promoted by the USAID Water and Development Plan. 3.2 DESCRIPTION OF THE INTERVENTION TO BE EVALUATED The WATSAN project is in its last year of implementation covering 10 communes (see Figure |). The project is working towards building a foundation for long-term, sustainable growth in access to safe drinking water and sanitation in Haïti where many communities suffer from high incidences of cholera and diarrheal disease. The Project is implemented in collaboration with Haitis National Directorate of Potable Water and Sanitation (DINEPA), the branch of the Haitian government tasked with ensuring water and sanitation services for its citizens with the overall goal of improving sanitation and water for all Haitians. WATSAN initially focused on five areas: three areas hard hit by the cholera outbreak (Cap-Haitien, Mirebalais, and Canaan) and two areas hard hit by Hurricane Matthew (Les Cayes USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 46 [page 62] and Jeremie). During the last two years, it added five additional areas: Ouanaminthe, Pignon, Lascahobas, Belladere, and Hinche. The WATSAN project’s three primary goals are to: e Help 250,000 people get access to basic or improved water e Help 75,000 people get access to basic or improved sanitation e Lay the foundation for sustainable increases in access to water and sanitation across Haiti The project is reaching these goals by focusing on its core values of self-reliance, resilience, and the private sector approach. nd - nn O=—— Figure l: Map of WATSAN intervention areas 3.2.1 GOAL AND STRATEGIC APPROACH Component l: increasing Access to Sustainable Water Services The Project activities under Component | are focused on providing basic water or improved water services to 250,000 people by building the capacity of the Technical Centres of Exploitation (CTESs) to provide water on a sustainable basis. While the main focus of the project is providing technical assistance to help the CTEs manage themselves on a cost recovery basis, WATSAN also funds infrastructure improvements that serve to make the systems more reliable and sustainable. Task 1.1: Water Infrastructure Engineering Services 47 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 63] Deliverables under Task 1.1: e Initial assessment report for all water systems to be supported, with prioritized list of proposed construction activities e EMMP for all construction activities e 100% design specifications for all construction activities e Signed firm fixed price contract(s) with construction firm(s) Task 1.2: Water Infrastructure Construction Deliverables under Task 1.2 e Certificate of completion for all newly constructed infrastructure, signed by the QA firm e Water quality test results from newly constructed water systems meet JMP standards. Task 1.3: Technical Assistance for Water Service Providers Deliverables under Task 1.3: e 5 service providers demonstrating at least 10% increase in cost recovery over a baseline established in Year | e 5 service providers demonstrating improved management capacity, according to a Contractor proposed index e 5 CTEs have plans to ensure water safety Component 2: increasing Access to Sustainable Sanitation Services The project activities under Component 2 are focused on providing basic or improved sanitation to 75,000 people by strengthening the capacity of small and medium enterprises (SMESs) to market sanitation products and by assisting DINEPA to reopen the fecal sludge management facilities in Les Cayes and near Canaan. Task 2.1: Support to Sanitation Enterprises Deliverables under Task 2. l: e 10 FSMs entrepreneurs and/or SMEs supported with capacity and business model assistance e 15 SMEs demonstrating increased sale of latrine products Task 2.2: Waste Treatment & Fecal Sludge Management Engineering Services Deliverables under Task 2.2: USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 48 [page 64] e Initial assessment report for all proposed wastewater treatment systems to be constructed, with prioritized list of proposed construction activities e 611(e) analysis for all construction activities e EMMP or EA for all construction activities e 100% design specifications for all construction activities e Signed firm fixed price contract(s) with construction firm Task 2.3: Wastewater Treatment & Fecal Sludge Management Construction Minimum Deliverables under Sub-task 2.3: e Certificate of completion for all newly constructed infrastructure, signed by the QA firm e Wastewater and/or byproduct quality test results from newly constructed treatment systems meet WHO/GOH standards. e 3 communes have waste treatment capacity Component 3: Improving the Enabling Environment for Sustainable Implementation, Operation, and Maintenance of Water and Sanitation Services The project’s third goal is to lay the foundation for sustainable increases in access to safe water and sanitation. It is meeting this goal by building successes at the local level and then working through DINEPA (and other institutions) to share these on a larger scale, for instance by holding national events and publishing the lessons learned. Task 3.1: Technical Assistance to National and Sub-National Governments Deliverables under Task 3.1 e 5 national and/or sub-national institutions successfully received training e Strategies/plans for improved water and sanitation service delivery in place for all supported geographies Task 3.2: Knowledge Dissemination & Learning Deliverables under Task 3.2 e Three national or regional events on activity learning involving government, civil society, private sector and donor partners e Five published knowledge products spanning all three activity components 3.2.2 DEVELOPMENT HYPOTHESIS AND THEORY OF CHANGE The WATSAN development hypothesis is that reducing the prevalence of waterborne diseases in priority geographic areas is achievable - if the project: 49 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 65] e Strengthens the ability of the Technical Centres of Exploitation (CTESs), Regional Offices for Potable Water and Sanitation (OREPAs), and the National Directorate for Potable Water and Sanitation (DINEPA) to provide water on a cost recovery basis; e Develops Fecal Sludge Management (FSM) facilities that can operate on a cost recovery basis; and e Strengthens the ability of the private sector to offer water and sanitation services. Effective implementation of WATSAN programming leading to reduced waterborne diseases can be achieved if the following conditions are met: e Self-reliance: The institutions are able to generate enough revenue to fully cover their operations without any subsidies to cover their operating costs. e Resilience: Given the volatile climate in Haïti, the institutions can design, build, operate, maintain, and upgrade systems that can withstand disruptions and economic uncertainty. e Private Sector Approach: The CTEs and Fecal Sludge managers are adopting a private sector, customer-driven approach. ennnans 000 oo © Then we will Ce) Then we will achieve these achieve this project objectives goal + Improvingthe operational and managerial capacities and reduce + Helping 250,000 people gain non-revenue losses enable the access to basic or improved CTEs, OREPAs, and DINEPA to access to water Reduce the prevalence of provide water on a cost recovery + Helping 75,000 people get cholera and other basis access to basic or improved waterborne diseases and + Promoting viable business models sanitation services thereby improve the for SMEs and FSM and closing + Layingthe foundationfor health and prosperity of financing gaps for market entrants sustainable increases in the Haitian People by increasing access to credit to access to water and achieve broader access to sanitation throughout Haiti sustainable sanitation services Theory of Change 3.2.3 PERFORMANCE INFORMATION USAID/Haiti expects that the evaluation team will review key strategy and project documents while carrying out this evaluation, which contain valuable performance information to involve in and help inform the evaluation design and analysis plan. Key internal sources of data and information include the following: e _mWater data: mWater is a digital platform that serves as a tool for national/local water utility/operator uploads, monitoring and reporting e __WATSAN baseline survey data and reports (completed in 2020 by SI/ESS) USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 50 [page 66] e __WATSAN project design documents, Theory of Change, and Monitoring, Evaluation and Learning (MEL) plans including all available performance indicator data and information ° Annual work plans e All available progress and quarterly reports e USAID/Haiti 2020-2022 country strategy (public version), Water and Development Plan, and other strategy documents e Other documents and datasets, such as project success stories, case studies, presentations, snapshots, and other communication materials such as videos e Other MEL data, including GIS information, cartography, and knowledge products IV. EVALUATION PURPOSE The primary purpose of the WATSAN final performance evaluation is to determine the effectiveness of the project in achieving its objectives (USAID Water and Development progress indicators). The evaluation will assess the project’s design and approaches used to achieve its objectives in order to inform future USAID/Haiti programming similar in scope or approach. Considering the Theory of Change components, this analysis will consider factors related to partnerships, capacity building, and sustainability. For instance, as a key component of the project, the evaluation will measure whether WATSAN interventions in the 10 communes of implementation have resulted in improved capacity of local utilities and local actors to provide households and communities with sustainable, improved access to water and sanitation services and to what extent the interventions have contributed to improved institutional capacity in the sector. A baseline study conducted at the start of the project assessed access to basic water and sanitation, and found that there are considerable opportunities to expand access to potable water and improve service delivery. The study also concluded that there was potential opportunity to reduce open defecation by willingness to pay for sanitation services and making improvements to sanitation facilities. The baseline study helped inform pivots in the WATSAN activity, and it is notable that the focus of this final performance evaluation extends beyond the initial objectives outlined in the baseline study. For a few reasons associated with WATSAN activity adaptations and achievements, this final performance evaluation will refer to but not build on this baseline data, largely because those baseline indicator measurements have changed. For this final evaluation, data and information to measure changes in outcomes from before and after the project implementation will be derived from other sources, such as the mWater data platform and key stakeholders. Hence, this evaluation will assess the final WATSAN activity indicators and will include a significant focus on outcomes related to capacity building of local actors and service providers, as the WATSAN actions aim to promote sustainable water and sanitation access in the target areas. 51 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 67] 4.1 EVALUATION USE AND USERS The primary stakeholders for this evaluation include USAID/Haiti, WATSAN implementing partners including DAI and its partners Ayiti Nexus (communications/BCC), Centre de Formation et d’Encadrement Technique (CFET) (capacity development), Zanmi Lasante (health and advocacy), V3 (engineering) and mWater (digital monitoring and reporting), and the National Directorate of Potable Water and Sanitation (DINEPA). Other key stakeholders who will benefit from the evaluation include other donors and entities active in Hait}”s WASH sector (IDB, Spanish Aid, World Bank, etc.) as well as a variety of private entrepreneurs, NGOs, and Faith-based organizations devoted to improving the status of Haiti’s water and sanitation service provision. Furthermore, this evaluation will help inform USAID/Haitis broader learning questions regarding its strategic approach to engaging both central and local government as well as local actors as key avenues for bolstering capacity to deliver quality services. V. EVALUATION QUESTIONS In line with the stated purpose of this final evaluation, the following evaluation questions have been designed to help USAID/Haiti and partners better understand the WATSAN project’s approach and Theory of Change: |. To what extent has WATSAN met its Task Order objectives to build governance capacity at multiple levels (national, regional, local) to improve sustainable water supply and sanitation service delivery? Criteria that inform sustainability in this context are the ability to consistently generate revenue and grow the customer base, and the ability to retain and develop technically competent staff (of water and sanitation utilities). This question should investigate the resiliency of WASH institutions (e.g., DINEPA, CTEs, OREPAs, and water and sanitation related MSMESs), and how the activity supported water and sanitation utilities at multiple levels to effectively manage crises in different geographical areas, such as in the South department after the recent August 2021 earthquake and tropical storm Grace, and in the North when handling droughts. Answering this question should also look at how using the Utility Turnaround Framework at multiple levels supported sustainable service delivery by water and sanitation utilities. 2. How was the Enterprise Acceleration Fund utilized and to what extent did those grants support WASH enterprises (such as MSMEs, NGOs, and Bayakous) to move towards sustainable service delivery? In answering this question, USAID/Haiti would like to know to what extent this Fund should be utilized or adapted in future programming similar in scope. 3. How effective was WATSAN’s approach of targeting the end of the sanitation value chain in improving the overall sanitation value chain? The end of the value chain includes components like excrement pick up, delivery, and treatment. Answering this question should investigate to what extent this approach has supported progress towards Haitis 2030 goal of ending open defecation. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 52 [page 68] In answering the above evaluation questions, the evaluation team should provide ample evidence, identifying areas to improve as well as lessons learned and best practices. The recommendations should guide future USAID/Haiti WATSAN programming, and the results will also be used by the wider WASH sector in Haiti for learning and adaptations. The evaluators will review and confirm a shared understanding of the evaluation questions in collaboration with USAID/Haïti prior to finalizing the evaluation design. VI. DESIGN AND METHODOLOGY The evaluation team is expected to propose an evaluation methodology and analysis plan that guarantees the highest degree of rigor so as to ensure credible findings and recommendations. A mixed method approach is possible, including but not limited to the following methods: e Utilization of the mWater performance data to determine how the activity has reached its beneficiary targets for access to water and sanitation (using the final WATSAN indicator definitions) e Utilization of the mWater performance data on revenue collection, operations and maintenance, financial performance, and asset management to assess and quantify WATSAN's capacity building and systems strengthening efforts e Detailed desk review of relevant project documents and reports e Detailed desk review of external documents, such as GOH documents, DINEPA operating agreements, other donor-funded programs/reports, and published WASH literature e Interviews and/or assessments and/or group discussions with key players and stakeholders, including USAID/Haiti staff, relevant GOH staff at national and regional levels, WATSAN staff, private sector actors, civil society organizations, and knowledgeable beneficiaries. A list of proposed interviewees will be submitted with the inception / design report. e Other qualitative and/or quantitative data collection methods, as appropriate to the evaluation questions The evaluation team is expected to describe which methods can best achieve which evaluation questions, and why, providing analysis plans and disaggregation methods in the evaluation design document(s). The methodology proposed should have a suitable representation and analysis of the geographic areas targeted (including considering contextual challenges faced in the different communes), population groups targeted (e.g., women, youth), and partners engaged (such as the private sector, local authorities, service providers, etc.). The data collection, analysis and reporting should provide data disaggregation by these (and other) markers where useful and necessary (such as by age, gender, location, activity approach, contextual changes, type of utility, etc.). Finally, the methodology will include triangulation of primary data with secondary data sources to produce robust and verifiable evaluation results. 53 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 69] VIL._ DELIVERABLES AND REPORTING REQUIREMENTS This section lists the expected deliverables and reporting requirements for this evaluation. This information complies with the reporting requirements already agreed upon with Social Impact (SI) through their Evaluation and Survey Services (ESS) contract with USAID/Haiti as well as the revised September 2021 Process Map (v5) that outlines the entire evaluation process in greater detail. Key deliverables and their schedule include: 1. Work plan: Upon receipt of the final SOW, ESS shall submit, within twenty (20) business days, a draft work plan to the ESS Contracting Officer’s Representative (COR) at USAID/Haiti. The work plan will include: (1) the anticipated schedule and logistical arrangements; (2) a list of the members of the evaluation team, delineated by roles and responsibilities with their level of effort (LOE); (3) the identification of other required personnel and relevant local subcontractors, their LOE, roles and responsibilities, and qualifications; and (4) the deliverable schedule. 2. Evaluation Inception / Design combined report: Within five weeks of approval of all evaluation team members, ESS must submit to the COR a combined Inception / Design Report, which will document: (1) a summary of what the team learned from its review of Activity background documents and other relevant literature; (2) a brief summary of the information gaps to be filled through the primary data collection; (3) a detailed evaluation design, including key stakeholder groups to be participants, sampling approaches, data collection and analysis methods, and an evaluation matrix linking each evaluation question in the SOW to the data sources, methods, and data analysis plan that will facilitate completely addressing the questions; (4) draft questionnaires and other data collection instruments or their main features; (4) the list of potential interviewees and sites to be visited; (5) known limitations to the design; and (6) a dissemination plan. Throughout this process, alongside the document review, the evaluation team (ET) will use information from a Kickoff meeting as well as an In-briefing presentation (see below), both held virtually. USAID/Haiti will take up to 10 business days to review and consolidate comments on the report through the COR or other designated evaluation representative in the USAID/Haiti MEL team. Once the ET receives the consolidated comments on the combined Inception / Design Report, they are expected to return a revised report within five (5) business days. 3. In-briefing/Presentation: While developing the detailed evaluation design, the evaluation team is expected to hold an In-briefing presentation with key USAID/Haiti staff and the (main) Implementing Partner (IP) to discuss having a shared understanding of the design approach and proposed methodology, etc. This is an opportunity to clarify assumptions and make sure that data collection tools, sampling, and analysis plans will capture the information needed to answer the evaluation questions thoroughly and with the highest quality. Also serving to provide quality control of the evaluation, ESS, the ET, and USAID/Haiti key staff will meet bi-weekly, or as possible at key moments, starting after submission of the USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 54 [page 70] combined Inception / Design Report. Hence, the first bi-weekly call will offer an opportunity to discuss USAID’s feedback on the draft report. 4. Out-briefing/Presentation: Within 20 business days after the fieldwork concludes, the evaluation team is expected to hold at least the first of two Out-briefing presentations to discuss the evaluation findings, conclusions and recommendations. The first Out-briefing will include key staff from the USAID/Haiti Program and Technical Offices (including the activity staff); this session will be prioritized, while the second Out-briefing will be open to the entire Mission and can occur at a later date. These sessions will spark deeper discussions within USAID/Haiti, and the Program Office expects to use the Out-briefings as launching points for its post-evaluation management plan. 5. Draft Report: The draft report should be consistent with the guidance provided in Section X: Final Report Format. The report will address each of the questions identified in the SOVW and any other issues the team considers bearing on the evaluation objectives. Any such issues can be included in the report only after consultation with USAID. The draft report will be submitted fifteen (15) business days after the first Out-brief presentation. Once the draft evaluation report is submitted, USAID will have 10 business days in which to review, comment on the draft, and submit the consolidated comments to the evaluation team. The ET will then be asked to submit a revised final report 10 business days hence, and again USAID/Haiti will review and send additional comments or approve the final evaluation report Within 10 business days of its submission. 6. Final Report: The evaluation team will be asked to take no more than 10 business days to respond to and incorporate USAID/Haitÿs comments on the draft report. ESS will then submit the final report to its COR. All activity data and records shall be submitted in full and should be in electronic form in easily readable format, organized, and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. 7. Collected data: ESS will then upload the final report and all evaluation data, respectively, to the USAID Development Experience Clearinghouse (DEC) and Development Data Library (DDL). ESS will also provide USAID/Haiti a full Close-out file of the evaluation by May 2022. 7.1 REPORT QUALITY CRITERIA To help ensure a high-quality evaluation report, these quality criteria (as described in ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report) must underline all work under this SOW: e Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project or activity. e Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. e The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. 55 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 71] e Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. e Evaluation methodology should be explained in detail and sources of information properly identified. e Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). e Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. e Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. e lfevaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. e lfrecommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. VIII TEAM COMPOSITION Selection of the evaluation team will be done in close collaboration with USAID/Haiti and according to the minimum standards described in this section. The evaluation team must have an appropriate mix of technical skills to successfully conduct this evaluation. The evaluation team shall include a Team Leader (TL) and a Sector Specialist/Assistant Team Leader (ATL). The two positions are considered key personnel and essential to the work being performed. The TL is ultimately responsible for the overall management of the evaluation team and final products, in conformity with the added layer of quality control provided by SI. Given this SOW, we anticipate that the TL should meet the following minimum standards: e The TL must be an experienced evaluation expert, with a documented track record of at least 10 years of experience conducting evaluations. e The TL shall have demonstrated experience in evaluating WASH activities. Familiarity with urban utility reform approaches and market-based approaches to service delivery is highly preferred. À demonstrated background conducting analyses of WASH infrastructure design, functionality, and sustainability will be an asset. e The TL shall have at least a master’s degree in social science (population and demography), public health or other related WASH field, and will possess excellent writing and interpersonal skills. e S/he must be fluent in English and French. e Work experience with USAID will be an asset. e Deep familiarity with the local context will also be an asset. The Sector Specialist/ATL shall complement the role of the TL; we anticipate that the ATL should meet the following minimum standards: USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 56 [page 72] e The ATL must be an experienced evaluation or sector expert, with a documented track record of at least 7 years of experience coordinating and designing performance evaluations. e The ATL shall have demonstrated experience in evaluating or working in WASH activities. Familiarity with urban utility reform approaches and market-based approaches to service delivery will be an asset. e The ATL shall have at least a master’s degree in social science (population and demography), public health or other related field, and will possess excellent writing and interpersonal skills. e S/he must be fluent in English and French. e Work experience with USAID will be an asset. e Deep familiarity with the local context will also be an asset. ESS is strongly encouraged to sub-partner with one local Haitian firm for data collection purposes or to include additional researchers as part of the evaluation team, as needed. If planning to partner with a local data collection firm, the selected Haïitian firm should demonstrate proven capacity in collecting performance evaluation-related data and should have data analysis capabilities to perform all related data management functions. This approach is encouraged to build the local firm capacity and will also provide a Haitian perspective for the data collection and analysis. IX. SCHEDULE The estimated period of performance for this evaluation is September 2021 to April/May 2022. Following the information provided in Section VII: Deliverables and Reporting Requirements, USAID/Haiti has outlined the illustrative evaluation timeline below in Table |: Tentative Schedule with Key Tasks. SI will work closely with USAID/Haiti to expedite the timeline when possible, and to avoid and overcome delays and challenges while maintaining the integrity of the evaluation data, results, and use. As the below schedule is illustrative, some dates may change (e.g., tasks completed earlier or pushed back). It is expected that SI will keep the schedule updated regularly, and SI or the ET may add additional tasks or more details as needed. Table l: Tentative Schedule with Key Tasks TASKS TENTATIVE TIMELINE Final evaluation SOW Sept. 22, 2021 Evaluation Work Plan draft submitted October 18 USAID Work Plan and Team approval Oct. 25 ET onboarding and training Oct. 25-29 Kickoff with USAID and IP November | In-briefing presentation with USAID and IP Nov. 15 57 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 73] TASKS TENTATIVE TIMELINE Inception/Design combined Report draft submitted, with Budget and Narrative, and Nov. 29 IRB processes/other approvals commenced USAID provides feedback on Inception/Design Report and Budget/Narrative December 13 Incorporate USAID comments and submit final Inception/Design combined Report Dec. 20 USAID approval of combined Inception/Design Report and Budget/Narrative Dec. 31 Team planning meeting / Field work Planning Likely early January 2022, or starting in December if/as possible Evaluation Data collection and Fieldwork January (or December if/as possible) Data Analysis and Report writing Jan-Feb. Prepare and conduct evaluation debriefing presentations February 2022 Draft report submitted to USAID April | Incorporate USAID comments and submit Evaluation Final Report May 2 Upload collected data to DDL, upload the evaluation report to the DEC, and share a May 2022 Closeout Folder with USAID/Haiti X. FINAL REPORT FORMAT The evaluation final report should include an abstract of no more than 250 words; executive summary of no more than 35 pages; background on the local context and strategies/projects/activities being evaluated; the evaluation purpose and questions; the methodology or methodologies; study limitations; findings, conclusions, and recommendations. The executive summary should be 2-5 pages in length and summarize the purpose, background of the activity being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in detail in the report. Limitations to the evaluation shall be disclosed in the report, with attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). The final report format should be as follows: -_ Abstract - Executive Summary -__ Evaluation Purpose - Background on the Context and the Strategies/Projects/Activities being Evaluated - Evaluation Questions - Methodology USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 58 [page 74] -__ Limitations to the Evaluation -_ Findings, Conclusions, and Recommendations - Annexes The annexes to the report shall include: - The Evaluation SOW - All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides -__ All sources of information properly identified and listed -__ Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest - Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team XI. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non- proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the activity or the evaluation. USAID will retain ownership of the survey and all datasets developed. AI modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the final report. 59 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 75] ANNEX B. DOCUMENTS REVIEWED ANNUAL REPORTS USAID Water and Sanitation Project FY 18 Annual Report USAID WATSAN FY 19 Annual Report USAID WATSAN FY 20 Annual Report USAID WATSAN FY 21 Annual Report EMMR -— FY2020 Water and Sanitation Project WORK PLANS USAID Water and Sanitation Project Y3 Revised Work Plan USAID Water and Sanitation Project Y4 Work Plan USAID Water and Sanitation Project Y5 Work Plan QUARTERLY REPORTS QI and Q2 FY 18 Quarterly Report Q3 FY 18 Quarterly Report QI FY 19 Quarterly Report Q2 FY 19 Quarterly Report Q3 FY 19 Quarterly Report QI FY 20 Quarterly Report Q2 FY 20 Quarterly Report Q3 FY 20 Quarterly Report QI FY 21 Quarterly Report Q2 FY 21 Quarterly Report Q3 FY 21 Quarterly Report QI FY 22 Quarterly Report Q2 FY 22 Quarterly Report PUBLICATIONS Catalogue de toilette USAID WB Water Utility Turnaround Framework USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 60 [page 76] USAID-Haiti-Strategic-Framework-Gender-Analysis EAF PROPOSALS AND DOCUMENTS LWI Proposal MFSN Proposal Proposal 509 Sanitation Proposal UniQ FECAH Le LEVIER Proposal SOIL USAID BSF Application MFSN Proposal Proposal 509 Sanitation Proposal UniQ LWI Proposal USAID BSFL Concept Note — from SOIL Rapport d’evaluation de LWI Rapport preliminaire d’evaluation MFSN OTHER RELEVANT DOCUMENTS Haiti WATSAN learning note - data driven management for water service providers Haiti WATSAN Learning Note — Adaptive Management for Results in a Fragile Environment Lessons Learned in Urban Sanitation CTE Evaluation under USAID Water and Sanitation Project CTE Performance Evaluation under USAID Water and Sanitation Project MEL Plan FY 21 May Revision MEL Plan FY 22 Final October 2021 Manual Pit Emptier visit Morne-4-Cabri Opening the Fonfred FSM Public Kiosks in Jeremie Presentation Result CTE Survey Monthly Reports for the CTEs from mWater Zonage de l’assainissement_Formatté Annexe A - Guide d'Utilisation de QuickBooks DINEPA - CTE V2 61 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 77] HR Manual DRAFT-3 Les expériences en assainissement urbain (Twelve years of Sanitation Learning in Haiti) USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 62 [page 78] ANNEX C. KIIS/GIS/FGD EVENT LIST 3/10/2022 KI FGD CTE Pignon Manager | (l 1 0 3/28/2022 FGD FGD CTE CTE Cap Haitien 3 (l 0 3 3/29/2022 FGD FGD CTE CTE Ouanaminthe 2 (l 2 0 3/31/2022 GI FGD CTE CTE Mirebalais 3 (l 2 1 4/1/2022 GI FGD CTE CTE Croix des Bouquets 2 ll 1 1 4/5/2022 GI FGD CTE CTE Cayes 2 (l 0 2 4/16/2022 FGD FGD CTE CTE Hinche 3 (l 1 2 4/16/2022 GI FGD CTE CTE Jeremie 4 (l 3 1 4/17/2022 FGD FGD Kiosk Kiosk Managers Cap Haitien 7 (l 5 2 4/1/2022 FGD FGD Kiosk Kiosks Mangers Cap Haitien 6 ll 5 1 4/1/2022 FGD FGD Kiosk Kiosk Managers Ouanaminthe 6 ll 5 1 4/1/2022 FGD FGD Kiosk Kiosk Managers Ouanaminthe 6 ll 5 1 4/2/2022 FGD FGD Kiosk Kiosk Managers Ouanaminthe 6 ll 5 1 41612022 GI FGD Kiosk Kiosk Managers Cayes 2 ll 1 1 4/16/2022 GI FGD Kiosk Kiosk Managers Mirebalais | (l 0 1 4/8/2022 GI FGD Kiosk Kiosk Managers Croix des bouquets | (l 0 1 4/15/2022 KI FGD Kiosk Kiosk Managers Jeremie 3 (l 2 1 3/30/2022 FGD FGD LEA LEA Nord 5 3 5 0 63 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 79] 3/31/2022 [el FGD LEA LEA 6 - 6 0 41/5/2022 [el FGD MTF MTF Sud 3 ll 2 1 4/17/2022 [el FGD MTF MTF Mirebalais 3 ll 3 0 41/8/2022 [el FGD OREPA OREPA Nord I ll 1 0 41/8/2022 [el FGD OREPA OREPA Centre I ll 1 0 4/15/2022 KII FGD OREPA OREPA Sud I ll 1 0 4/15/2022 KII FGD OREPA OREPA Ouest I ll 1 0 3/29/2022 [el FGD SME SME 5 ll 5 0 4/4/2022 GI FGD SME SME Croix des Bouquets 5 2 5 0 41/5/2022 [el FGD SME SME Cap Haitien 4 ll 4 0 4/28/2022 [el FGD SME SME 5 ll 4 1 3/25/2022 [el GI CTE Liason CTE Liaison 2 2 0 2 3/25/2022 [el GI CTE Liason CTE Liaison 2 2 1 1 3/25/2022 KII GI CTE Liason CTE Liaison I ll 1 0 3/28/2022 GI GI EAF EAF Limonade Henry Campus 3 (l 3 0 4/1/2022 KII GI EAF EAF Le Levier I 4 1 0 4/1/2022 [el GI EAF EAF Living Water 2 ll 2 0 41612022 KII GI EAF EAF UniQ I ll 1 0 4/17/2022 [el GI EAF EAF Incinerator 509 2 ll 1 1 4/17/2022 [el GI EAF EAF MFSN 3 ll 0 3 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 64 [page 80] 4/12/2022 KII GI EAF EAF SOIL I ll 0 1 3/30/2022 GI GI FSM Staff FSM Morne a Cabri 2 ll 1 1 3/30/2022 GI GI FSM Staff FSM Fon Fred 2 ll 2 0 3/29/2022 GI GI FSM Users FSM Users Morne Cabris 2 (l 0 2 3/30/2022 GI GI FSM Users FSM Users Fon Fred 2 (l 1 1 2/28/2022 KII GI IP Ayiti Nexus I A7 0 1 3/2/2022 GI GI IP mWater 3 A7 2 1 3/7/2022 KII GI IP V3 I A7 1 0 3/30/2022 KII GI IP ZL I - 0 1 3/30/2022 GI GI IP CFET 2 A7 0 2 2/21/2022 KII KII DAI DAI I A7 1 0 2/24/2022 KII KII DAI DAI I A7 1 0 2/24/2022 KII KII DAI DAI I A7 0 1 2/25/2022 KII KII DAI DAI I A7 1 0 3/2/2022 KII KII DAI DAI I A7 0 1 3/10/2022 KII KII DAI DAI I A7 1 0 3/14/2022 KII KII DAI DAI I A7 1 0 3/24/2022 KII KII DAI DAI I A7 0 1 2/22/2022 KII KII ONEPA ONEPA I A7 0 1 4/11/2022 KII KII ONEPA ONEPA I A7 1 0 65 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 81] 2/21/2022 Kil KI! USAID USAID l A7 (e) 1 3/3/2022 GI KI! USAID USAID 2 A7 1 1 3/7/2022 Kil KI! USAID USAID l A7 (e) 1 Total 146 100 46 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 66 [page 82] ANNEX D. MWATER SECONDARY DATA REVIEW SUMMARY OF DATA CLEANING PROCESS For the monthly report data on mWater, the ET downloaded the data from the mWater platform’s Suivi-SAEP link on May 4, 2022. The data was then imported to Stata and adjusted to be suitable for Stata analysis (variable labels, string variables). Data cleaning process mainly included two steps: |) identifying and addressing duplicates in Monthly Reports; and 2) identifying and addressing outliers of variables used for the 11 key indicators. e _Identifying and addressing duplicates in Monthly Reports: Two Monthly Reports were available for the same month in three instances. The ET reviewed the data and kept the report with more data points and removed the other duplicate (removed the CTE Belladere report submitted on “2020-10-20 22:42:48,” CTE Lascahobas report submitted on “2021-07-15 18:53:23,” and Pignon report submitted on “2020-06-01 15:57:26”). e _Identifying and addressing outliers of variables used for the 11 key indicators: Outliers are identified for each variable contributing to the 11 key indicators using IQR method3? at each SAEP level, which are addressed by: |) keeping the data as is; 2) replacing the data as missing value; and 3) adjusting the data to correct apparent errors. All edits made to the dataset are listed in Table 6. TABLE 6: DATA OUTLIERS MONTHLY ORIGINAL REVISED SAEP REPORT INDICATOR VARIABLE DATA DATA RATIONALE Active Active subscribers Likely a typo because | CTE Belladere July 2020 : (Total abonnes actifs 15,000 1,500 other months are around subscribers : debut du mois) 1,500 Average hours of Likely an entry error 2 CTE ., December Service service/week (Moyen 25 : because other data are Jeremie 2018 continuity heures / semaine de (dropped) all less than 10 and this is service) the first Monthly Report : Arrears at the Likely an entry error 3 CTE : 2021 Collection beginning of the month 1031,962,425 ‘ because average of other Jeremie January eficlency (RECETTES: Arriérés PT (dropped) months was around arrears début mois) 10,000,000 : Active subscribers Likely a typo and 4 CTE July 2020 Active (Total abonnes actifs 9,058 908 previous two months Lascahobas subscribers : “ong’ debut du mois) were ‘908 CTE Collection : i 5 March 2021 ‘ Total monthly recovery 762,120 Likely an entry error Lascahobas efficiency — total (RECETTES: (dropped) because average other 3 Interquartile range (IQR) method refers to identifying values that fall outside of a range of minimum and maximum value determined by IQR, which is Quartile | (QT) value subtracted from Quartile 3 (Q3) value. The minimum value range is determined by QI — 1.5IQR and the maximum value range is determined by Q3 + 1.5IQR. Any observations that are more than 1.5 IQR below QI or more than 1.5 IQR above Q3 are considered outliers. 67 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 83] MONTHLY, ORIGINAL REVISED SAEP REPORT MNPICATOR VARIABLE DATA DATA RATIONALE Recouvrement total mois months were below - Montant) ‘150,000° : Arrears at the Likely an entry error CTE F Collection nn . 6 ebruary efficiency — beginning of the month 259.486.7 because second lowest Mirebalais 2021 arrears Y (RECETTES: Arriérés Fm (dropped) value other than this was début mois) ‘1,689,456" Total monthly recovery Likely an entry error 7 CTE December Collection (RECETTES: 1,542,552.4 : because average other Mirebalais 2019 efficiency — total Recouvrement total mois "7" (dropped) months were around - Montant) ‘150,000° Likely an entry error CTE Quantity of water . because other months 8 Mirebalai July 2019 Total production produced (Quantité 285,120 were all below 100,000, ITEDaraIs d’eau produite) (dropped) and this was the first Monthly Report CTE Quantity of water Likely a typo, and other 9 Mirebalais June 2020 Total production produced (Quantité 336,000 33,600 months were all below d’eau produite) 100,000 : Billing during months . Likely an entry error 10 GE inne Juy2020 Collection | (RECETTES: Facturation 1,300 because other months Y durant mois) (dropped) were all above 14,550 Le Kiosk - Total Qty . Likely an entry error 11 CTE de Cayes Sepromber Functionin£ (PATRIMOINE: Kiosque - O because other months 1oS's Total Qté) (dropped) were all above 13 Quantity of water . Likely an entry error 12 Pignon June 2020 Total production produced (Quantité 237,300 because other months d'eau produite) (dropped) were all below 3,000 SUMMARY OF AVAILABLE DATA In total, 294 reports were reviewed by the ET for the 10 WATSAN SAEPSs (CTE Cap Haitien, CTE des Cayes, CTE Croix des Bouquets, CTE Jeremie, CTE Mirebalais, CTE Hinche, CTE Ouanaminthe, CTE Belladere, CTE Lascahobas, Pignon Centre Ville). Table 7 summarizes the reports that were available on mWater as of May 4, 2022 for the | | SAEPs. TABLE 7: AVAILABLE DATA FIRST AVAILABLE LAST AVAILABLE NOMEER ON SAEP REPORT REPORT AVAILABLE MONTHLY REPORTS CTE Cap Haitien 12/1/2018 3/1/2022 40 CTE des Cayes 12/1/2018 3/1/2022 40 CTE Croix des Bouquets 5/1/2019 3/1/2022 35 CTE Jeremie 12/1/2018 2/1/2022 39 CTE Mirebalais 7/1/2019 1/1/2022 31 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 68 [page 84] MONTHLY REPORTS CTE Hinche 6/1/2019 1/1/2022 32 CTE Ouanaminthe 5/1/2020 3/1/2022 23 CTE Belladere 5/1/2020 9/17/2021 17 CTE Lascahobas 1/1/2020 9/17/2021 21 Pignon Centre Ville 5/1/2020 8/1/2021 16 Total 294 69 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 85] INDICATOR 1. ACTIVE SUBSCRIBERS Definition: Subscribers that are not: 1) passive subscribers (subscribers disconnected for debt; 2) subscribers that are connected but do not have water because of a problem in the network; and 3) subscribers that paid their connection fee but are not yet connected. SAEP Active Subscribers 4500 ; : ? Hinche & À Lascahobas, Belladere, & 4000 +Oumnrintte ; Pignon Ê 3,500 ] = \/ — Cap Haitien Ë 300 É [TA Les Cayes Ê 250 F ra es Bouger $ ) | —jeremie ë 2,000 L Î — Mirebalais Ë 1,500 = —Hinche À 100 €} H — Ouaraminthe 5 0 Ti — — —Belladere ë [7 : Î — Lascahobas ° | ‘ —Pgron PP EE ES NP D PS SP PS PS D D D SN À NW GE RE 0 à RE OÙ EE RS Monthly Report SAEP Active Subscribers Six-month Moving Average <0® EHinche & Ÿ Lascahobas, Belladere, & À Ouanminthe À Pignon a 3,500 i H & Î 5 — Cap Haitien 5, 2,500 l (Croix des Bouquets È Î H —Jeremie & 200 É î —Mirebalais £ 1500 i É —Hinche Ê | Z 1] î — Ouanaminthe $ 190 7] - —Belladere ä ns F 500 7 —Lascahobas 4, Î —Pgnon Q Q Q Q © © P © © © SJ SJ S d SI Ÿ 2 2 # SA S a S SP ES PP & PE Monthly Report SAEP Active Subscribers 4000 2 3500 É Ê 3000 $ 2500 È E El £ 2000 £ $ 1.500 E a Ë no | Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Ouaraminthe Belladere Lascahobas Pignon Bouquets USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 70 [page 86] INDICATOR 2. COLLECTION EFFICIENCY (CURRENT) Definition: Percentage of the amount collected for charges for the month over the amount billed for the current month. SAEP Collection Efficiency (Current) 1e ?Hinche & ? Lascahobas, Belladere, 140 ?Ouanaminthe JR PInon È Ê i —Les Cayes à Î ; —— (Croix des Bouquets 5 © H H : —Mirebalais £ : H —Hinche ÿ 4 = É 5 à + FAP — Ouanaminthe Lo] À 20 A / \\ /| —Belladere 2 XX) TD LT ” S SSI LR SUR 7 SZ SES VA em tascahobas ON ON 9 S S SP P P P PS SP D D DS À À À P —Pigron D RE RE OÙ DE Rp OÙ EE SRE OÙ D Monthly Report SAEP Collection Efficiency (Current) Six-month Moving Average ‘ ? Hinche & ? Lascahobas, Belladere, & ? Ouanaminthe À Pignon & 50 | Î — Cap Haitien È Î —Les Cayes 5 4 t — Croix des Bouquets > H ——Jeremie 5 30 î , 8 : — Mirebalais 8 20 TLAD inche nd! XP — Suns 8 7 ] LE RE ir = — —Lascahobs 0 à —Pignon PO A PP PP PS PO D À D ED À À À Re ES ge 7 ge a 7 St 7 7 g — Monthly Report SAEP Collection Efficiency (Current) 160 g 140 $ 120 ë * 100 à & ü 80 $ 5 6 H É “5 4 Ü ant Hill i k | | | o ill NN TR | 1 il... M 1 | ll [M bat Cap Haitien Les Cayes Croixdes Jeremie Mirebalais Hinche Ouamaminthe Belladere Lascahobas Pignon Bouquets 71 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 87] INDICATOR 3. COLLECTION EFFICIENCY (ARREARS) Definition: Percentage of the arrears recovered over the arrears at the beginning of the month. SAEP Collection Efficiency (Arrears) 45 5 : ; Hnchea ? Lascahobas, Belladere, & 40 À Ouanaminthe À Pignon % 3 Î Î — Cap Haitien Ë 30 É î —Les Cayes 7 Î 4 2 : : — Croix des Bouquets Ë 20 i i — jeremie 5 Ë H — Mirebalais Ê —Hinche FA & 10 + A À , Ki [A [> — Ouanaminthe Ô » Sos 5 c — la WA = / XI LES LA — Beladere 0 ° : —Lascahobas NON OS ON PS SP SP PS PS PP ES SD D |» } à NO À 9 A SO P à 9 D Où a —?" DE RE SE D D RE RE SU RE D D EE ST RE D D ignon Monthly Report SAEP Collection Efficiency (Arrears) Six-month Moving Average 25 ; ; Hinche& À Lascahobas, Belladere, : À Quanaminthe | & Pignon € 2 ; ; — Cap Haitien 8 É i — Les Cayes L 5 ï H — Croix des Bouquets Ë Î Î —jeremie "4 Î H $ 10 i Î —Mirebalais ë = —Hinche 5, en TRE — — Ouanaminehe ë D Re — EE — << —— © —Belladere 0 H i — Lascahobas PE PP SE PS PO À D SOS OÙ D D —)Pignon et 7 7 a 7 7 ge 7 Monthly Report SAEP Collection Efficiency (Arrears) 45 40 35 € 30 8 E 25 ë Z 20 8 É 15 d] D | | | Ë Ki CES : il il | fl. (is ” Usb ds | | di Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Ouanaminthe Belladere Lascahobas Pignon Bouquets USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 72 [page 88] INDICATOR 4. COLLECTION EFFICIENCY (OVERALL) Definition: Percentage of total amount collected over the total amount billed. SAEP Collection Efficiency (Overall) ? Hinche& À Lascahobas, Belaadere, À Ouanaminthe } & Pignon — Cap Haïtien € Î É —Les Cayes Ë 400 : à —— Croix des Bouquets > Î Î — Jeremie ë 300 s L — Mirebalais $ H É —Hinche £ H i $ 20 ]\ 4 É /\ \ À — Ouanaminthe S106 . AN ”) X PATES A XS N. PS —Belladere Y ; {) VS. À XS4 PE — Lascahobas SN RPM - M 0 = Ù — Pignon ee SN SN S 9 0 PP D À? D PP D PE À + D À D ER RE OÙ D ER NS RE OÙ D RÉ NS RÉ OÙ D Monthly Report SAEP Collection Efficiency (Overall) Six-month Moving Average 300 : : ? Hinche & ÎLascahobas, Belladere, & } Ouanaminthe {Pignon g 250 . Î — Cap Haitien 6 200 ; : — Croix des Bouquets 4 150 : À —Jeremie È s = D — Mirebalais 100 CP I _ A? IT ZX — Hinch, É IS ST 7 NS . ê p— NN D PT TN ON Ouanaminthe o - É — Lascahobas PO RO A SP PP PS SP PS D D OS ES DS ES D D Penn a NS S SR at ef oo SR at ef KA S Monthly Report SAEP Collection Efficiency (Overall) 600 € = 500 S 8 à 400 $ 1 $ 300 £ È 3 200 Ô , MALE MALO 1 dll UM du di Cap Haitien Les Cayes Croixdes Jeremie Mirebalais Hinche Ouanaminthe Belladere Lascahobas Pignon Bouquets 73 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 89] INDICATOR 5. REVENUE Definition: Amount of subscriber payments this month + arrears recovered + subscribers paid in advance + reconnection fees + quotes/connection fees + meter installation fees + penalties + kiosk water sales + used materials sales + TCA collected. (Note: Does not include subsidies) SAEP Revenue 3,500 À Hinche& ? Lascahobas, Belladere, À Ouanaminthe À & Pignon 3,000 Î : — Cap Haitien 8 É i — Les Ca £ 2500 j i s Cayes à H H = Croix des Bouquets 5 2000 Î : — Jeremie Ê 1,500 H Mirebalais $ —Hinche 8 $ 1,000 \\ — Ouanaminthe Le + ON N À —Bclladere RSS > DA AI - PNG TT NE ——tascahobas 0 su - — Pignon D NON NON ON ON PO PS PS PP P SP À» À À» À À À» À DE EE NE OÙ D RE NS RÉ OÙ D ES RÉ OO Monthly Report SAEP Revenue Six-month Moving Average 5% Ep Ï Lascahobas, Belladere, À Hinche& | À Ouanaminthe : & Pignon Z 2.000 H i — Cap Haitien È Î i 5 | i ——Les Cayes £ 1.500 À Ÿ — Croix des Bouquets og : : — Jeremie z 1,000 H i — Mirebalais > H — 5 — Bcladere 0 EE Lascahobas PO PP PP PP DS SOS SOS OS POP 7" Re ge NS 7 ge el 7 7 Paron Monthly Report SAEP Revenue 3,500 3,000 2 2500 H 3 2000 ê = L 1,500 TE 8 2 1000 £ a L À \ | | | di | | , M db ARR Al dla vu: Cap Haitien Les Cayes Croix des Bouquets Jeremie Mirebalais Hinche Ouaraminthe Beladere Lascahobas Pignon USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 74 [page 90] INDICATOR 6. EXPENSES Definition: Bank account balance of expenses. SAEP Expenses 50 ? Hinche & ? Lascahobas, Belladere, & 4,000 ? Ounaminthe À Pignon & 3500 É | — Cap Haitien à 3000 i i ——Les Cayes Ë 3500 i : Croix des Bouquets 0 È H ——Jeremie = : V/aN A —Mirebalais 8 1,500 i $ 3 —Hinche À 1000 Ÿ FR DA CR IS ( — Ouanaminthe 50 A AS NL = SU RS NN PRET NT LINE Belladere ——Lascahobas NON NON OX ON DS PS PS PS ES PE TT DS ES » À à ù Ÿ ON 0 2 POP à à D À —? D RÉ RE OÙ D LE RE OÙ DE LE RE OÙ D se Monthly Report SAEP Expenses Six-month Moving Average 5 ? Hinche& } Lascahobas, Belladere, & ? Ouanaminthe À Pignon & 200 t Î — Cap Haitien É ; Î —Les Cayes 6 1,500 È È — Croix des Bouquets £ È ; j — jeremie Z 1,000 ; É — Mirebalais Ë Î É —Hinche ë 50 eZ TT — Ouanaminthe = — = —Bcladere 0 Em ai hp NO D PP PS PP PSP ES SE OS Op À » Pi RON et ge get et Gt ge te Monthly Report SAEP Expenses 4,500 4,000 3,500 8 £ 3000 5 3 #5 2500 = © E 200 3 1500 & ä 1,000 L Li il al à il 0 ul | | th ll il abdl | | al 1 TT ae I Cap Haitien Les Cayes Croix des Bouquets Jeremie Mirebalais Hinche Ouanaminthe Belladere Lascahobas Pignon 75 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 91] INDICATOR 7. OPERATING RATIO Definition: Total expenses over total revenue. SAEP Operating Ratio 600 ? Hinche& À Lascahobas, Belladere, & ? Ouanaminthe À Pignon _ | : —Les Cayes £ Î i — Croix des Bouquets © 300 Î | —jeremie FI H É —Mirebalais "AA AA AMUWAN AAFA | Ë 0 VAN DÉPENS ol A, S'SN XEROX rm Aie UG \ Ze, LORS UE —Belhdere 0 V a —Lascahobas ER DS ER SN RS SOS S © D D RD D D DE ES SN —Pignon Of EE NS RO OÙ EE NS RÉ OÙ OÙ ES NÉ D D Monthly Report SAEP Operating Ratio Six-month Moving Average 250 5 ; ? Hinche & Lascahobas, Belladere, 8 À Ouanaminthe Pignon 200 —û— ; — Cap Haitien & 150 î ST | É : —— Croix des Bouquets 5 i A <" o NI ——Jeremie ® 100 CE ÈS. OS, > D ON — Mirebalais 5 Pa r. So — LE A & HA ° ESP SEEN re © so : : — Ouanaminche | | —Beladere 0 Ë H —Lascahobas a Q QI Q © © S © © © NS Sd S NS S Sd 42 #2 ST S Se 7 cu Se gt SP os ST nee Monthly Report SAEP Operating Ratio 600 500 Æ 40 É Ë 300 FI S 200 : | i ï "M h | il | j | il | | Cap Haitien Les Cayes Croix des Bouquets Jeremie Mirebalais Hinche Ouamaminthe Belladere Lascahobas Pignon USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 76 [page 92] INDICATOR 8. TOTAL WATER PRODUCTION Definition: Amount of water that was produced from the springs and wells to be delivered to the reservoir. SAEP Total Water Production 250,000 , : = À Hinche & } Lascahobas, Belladere, & Pignon & ? Ouanaminthe $ 200,000 Î — Cap Haitien E Î è Î ——Les Cayes à 150,000 i —— Croix des Bouquets Ë | —jeremie Ë Mira 2 100,000 nr] — Mirebalais ë , Sax —Hinche Es — Ouanaminthe à RARIT NET ZE V PILE — ; DA A VA ee Ninon —tascnobas SN —— Paron DFE NS RO Of EE NS OÙ Of RE NS RE OÙ D Monthly Report SAEP Total Water Production Six-month Moving Average . 180/000 Hinche & Lascahobas, Belladere, 8 Æ 160,000 fOuanaminthe fPignon À 140,000 | Î — Cap Haitien Ÿ 120000 —— Les Gares $ 100,00 s Croix des Bouquets 5 H H — jeremie TZ 80,000 H L Mirebalat &. Ï Ë S — Mirebalais 5 60,000 i inch s ; i 7] QC —Hinche É 40,000 y LS > —— Ouanaminthe & 20000 cé Pr H << —selladere 0 —Lascahobas POP SOS POP PO E PS PE ED OP EE OO EP À À j ON CO CC CS —Pignon Monthly Report SAEP Total Water Production 250,000 H à 200,000 3 £ € $ 150,000 ê Ë à 8 & ÿ 100.000 Ë 2 U | | l l l | . mul | à la : | ll NM! | m | Cap Haitien Les Cayes Croixdes Jeremie Mirebalais Hinche Ouamaminthe Beladere Lascahobas Pignon Bouquets 77 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 93] INDICATOR 9. SERVICE CONTINUITY Definition: Average number of hours per day of service multiplied by the average number of days per week of service (average for all sectors in the system). SAEP Service Continuity u 180 Hinche & Flascahobas, Belladere, & Pignon Ë 160 \Ouanamin à 140 H i — Cap Haitien g H H È # 120 i { ——Les Cayes HIS Î É —— Croix des Bouquets È à 100 i : Eu H H —— Jeremie 80 + : 33 H Ë / — Mirebalais 55 4 . i 1] —— Ouanaminthe 82 2 H i —Belladere d 0 - —Lascahobas 8 œ Q Q a Q Q Q © © © © © © SJ d D d d d 42 $ PO NC ES PS NP PP PS POP D D D D OP À À — Pignon Fo RE RE OÙ D LE RE OÙ D EE RÉ OÙ D Monthly Report SAEP Service Continuity Six-month Moving Average & 180 EHinche & Lascahobas, Belladere, & Pignon £ 160 Ouanaminthe ; & 140 i H — Cap Haitien g î 2% H 7 — T8 120 t É Les Cayes Hs i É — Croix des Bouquets 5 5 100 4 Î se ; H — Jeremie 3 © : Î — Mirebalais LE © ! ! FE Î Î —Hinche gp 4 ; H =— Ouanaminthe &S 20 i ï —Belladere ÿ 0 = —— Lascahobas £ H A PP PS PS PO D SO D SO D ES PO» LORS St D D 7 gg 7 ST 7 7 a Paron Monthly Report SAEP Service Continuity 180 " ÿ 160 3 & 140 £z ë ë 120 £ & 100 + 3 æ ge ST 60 £ 5 £ 3 4 EE 5 i * mit | | 3 0 amuliitth ï ulhall a 1 1 [dt [en 1 là ( ll Cap Haitien Les Cayes Croix des Bouquets Jeremie Mirebalais Hinche Ouanaminthe Beladere Lascahobas Pignon USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 78 [page 94] INDICATOR 10. RESIDUAL CHLORINE CONFORMITY Definition: Proportion of residual chlorine tests that conform to norms calculated by the number of residual chlorine tests that conform to norms over the number of residual chlorine tests conducted. SAEP Residual Chlorine Conformity £ Hinche & Ouanaminthe Lascahobas, Belladere, & Pignon Ë 00 _ ë Î £ f — Cap Haitien $ : l ss res EP © : E — Croix des Bouquets sE : : —Jeremie 5 o . | . — Mirebalais Æ 5 i —Hinche É > : Î —— Ouanaminthe $ Î N Î — Belladere 5 o Î Î —— Lascahobas $ ÿ X OX OX ON SON SP PSP P PP PE PSP ES D D D D À — Pignon QE RE EE D D RÉ SU RE OÙ GE RÉ Sp D D Monthly Report SAEP Residual Chlorine Conformity Six-month Moving Average Hinche & Ouanaminthe Lascahobas, Belladere, & Pignon 100 £ “ — 5 s Cap Haitien Î Î —Les Cayes Ë | | — Croix des Bouquets EDS Î | —jeremie SE 4 i : — Mirebalais 38 20 H Î — Ouanaminthe H Î | —Belladere È o i l —Lascahobas LRO PS PS PS PS PS PS D À D D D D D D )"Pignon Et at ee 7 a a a 7 7 a a 7 BON SONO OS OR ON SO RON OO ON SON € Monthly Report Residual Chlorine Conformity 100 2 5 5 $ & È 5 ÿ __ 60 2€ £e SË ÿ 3 È 2 & ÿ Ë 0 Cap Haitien Les Cayes Croix des Bouquets Jeremie Mirebalais Hinche Ouaraminthe Belladere Lascahobas Pignon 79 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 95] INDICATOR 11. FUNCTIONING KIOSKS Definition: Number of functional kiosks over the number of total kiosks. SAEP Functioning Kiosks Hinche & Ouanaminthe Lascahobas, Belladere, & Pignon Z 100 7 5 3 \/ H \/ 4 AA I/ — Cap Haitien 8 80 i \? £ 4 1 ——Les Cayes È u Î — Croix des Bouquets F5 E Ê Î ——Jeremie 5 4 É —Mirebalais & | —Hinche 5 — ê î \] t —Belladere 0 + : . —Lascahobas SONORE ON PS PS SP PS PS PS DO» SO SO» À À EE pt OP QE NT RS OÙ D En ST do € re Monthly Report SAEP Functioning Kiosks Six-month Moving Average Hinche & Ouanaminthe Lascahobas, Belladere, & Pignon 100 € _ NN ZT SL comen E 80 i H É —— Les Cayes El Î H 2 — Croix des Bouquets E 60 ! î — Jeremie E Î H — Mirebalais È 40 : ; —Hinche 5 Î </ — Ouanaminthe ÿ 20 ; : & H i —Belladere H Î —— Lascahobas 0 | 1 SR ER EE TT TS St 7 gt 7 7 a el 7 _ Monthly Report SAEP Functioning Kiosks 100 E £ 8 & ë £ 60 ë È ë 5 40 ; 8 & 2 MN | Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Ouanaminthe Belladere Lascahobas Pignon Bouquets USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 80 [page 96] HOUSEHOLD WATER ACCESS COVERAGE BY CTE Household Water Access Coverage by CTE 40,000 38.012 35,000 8 9 30.000 27351 SS Households in the CTE $ à 25.000 coverage area $ Fi 20.000 M Functional kiosk 6 5 5 Residential 3 : 15,000 ar TDR mResidential connection 5 9,229 | 1505 . u Z 2 10.000 7,957 | 6257 Commercial connection T " 4,794 500 Le 3.205 2387 M Institutional connection ° 3.454 FE NA D mx … s é $ e Û € e se s s eo . 7 e “ # ra ri # # & & ÿ “ _ ÿ «° Source: e Residential connection: mWater Monthly reports for Mirebalais (Jan 2022), Belladere (Sep 2021), Lascahobas (Sep 2021), and Pignon (May 2020), all other data from Cadastre (date unspecified) e Commercial connections: Cadastre (date unspecified) e Institutional connections: Cadastre (date unspecified) e Functional kiosk: Cadastre (date unspecified) e _ Household in the CTE coverage area: Cadastre (date unspecified) *Data for Croix des Bouquets are not presented other than the households served because available data on their functional kiosks are inaccurate. 81 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 97] ANNEX E. CTE STAFF SURVEY RESULTS SUMMARY OF DATA CLEANING PROCESS CTE staff survey data were reviewed for consistency and checked for any duplicates and outliers. No data issues were found during the data quality review. Data presented below, therefore, is the analysis of the original data received from the Haïitian data collection firm. STAFF MANAGEMENT Formal job descriptions 100% 100% 100% 100% 100% 100% 100% o 92% 75% 50% 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets Has à formal job description # Doesn't have a formal job description Job expectation 100% 100% 100% 100% 100% 100% 100% 75% ELA 50% 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets # Clear job expectations M Unclear job expectations USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 82 [page 98] Number of years working at CTE 100% 29% Q 38% 2 % 75% 50% 43% a 56% 80% 50% 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets Less than a year 1-2 years m2-3ÿears m3-4years Mm4-5 years 6 or more years Performance evaluation . [00 29 29 4 Je € o % 75% ue = QUE . 69% 71% 73% _ 71% 50% 44% 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets mHad a performance evaluation # Didn't have a performance evaluation Don't know 83 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 99] EXPERIENCE USING TOOLS Experience using SIGA 100% m2 6% n ” 75% _ 43% 50% 54% 86% 73% 100% 67% e Q . _ 0% | Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience Experience rating N/A # Don't use SIGA Don't know if CTE uses SIGA Experience using mWater 100% 20% 14% 75% 52 s 43% = 50% 17% Ê0e 89% 57% =s El 25% Q 42% 40% © 0% 14% FE ET Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience Experience rating N/A # Don't use mWater Don't know if CTE uses mWater Experience using SISKLOR 100% 14% 13% 14% 75% 31% 14% 67% x 29% 50% 100% 83% 86% 87% 25% 54% % 33% 43% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience ® Neutral experience # Experience rating N/A 8 Don't know if CTE uses SISKLOR USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 84 [page 100] Experience using QuickBooks 100% 75% 50% SF 62% 67% 71% e 90% 89% po == 43% 35% LL 17% | 6 14% 31% 25% = 0% _ 14% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience x Negative experience Experience rating N/A # Don't use Quickbooks Don't know if CTE uses Quickbooks MANAGEMENT OF GENDER ISSUES AND TRAINING Sexual harrassment training 100% 23% 9 22% 29% 33% o e 75% a 50% 71% 50% 77% 9 78% 71% 67% , le 25% Œe 50% 29% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets M Participated in a sexual harrassment training Did not participate in a sexual harrassment training CTE’s management of CTE’s gender issues 100% NE 11% 75% _ 50% 100% 100% 100% 100% 89% 100% 69% 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience x Negative experience Experience rating N/A 85 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 101] CTE’s management of CTE’s gender issues by sex 100% . 11% 13% 80% 50% 60% : 4 4 100% 100% 4 , 40% 89% mn 88% 20% | 25% 0% Male Female! Male Female! Male Female) Male Female, Male Female! Male Female, Male Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience Negative experience Experience rating N/A CTE’s management of community gender issues EZ 14% 11% > 10% 11% 75% 23% 40% 50% Q 0 . 72 23 80% 78% o 69% 60% 71% 25% ? 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience Negative experience Experience rating N/A DAs management of gender issues 100% g 15% 14% 13% 10% 11% 14% 75% 15% 50% % o D 100% 86% 87% 90% 89% 86% 69% 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience Negative experience Experience rating N/A USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 86 [page 102] VIEWS ON SERVICE QUALITY View on CTE's service for the community 100% 14% 20% 11% 75% 50% 92% 92% 86% 93% __. 89% JE 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets m Positive experience Neutral experience f Negative experience Service improvement in the last two years 100% 75% 50% 100% 100% 100% 93% 100% 100% 100% 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets a CTE services improved CTE services did not improve VIEWS ON WILLINGNESS TO PAY Tariff level 100% 8% re 10% 11% 75% 56% 50% 92% Ex 100% 100% 90% 100% 25% 33% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets nm Tarif is appropriate for the services Tariff is not appropriate Don't know 87 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 103] Clients’ willingness to pay 100% 22% 75% 50% 100% 100% 100% 100% 100% 100% 78% 25% 0% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets # Clients are willing to pay for the services Clients are not willing to pay VIEWS ON THE FUTURE FOCUS OF CTES CTE’s Future Focus Area —e— Cap Haitien e—Mirebalais ——e— Croix des Bouquets =@=» Les Cayes Jeremie —e—Hinche —e— Ouanaminthe Capacity building 12 Other 10 Î Staff gender equality 8 Developing pricing structure ‘ Better use of mWater data [| Î Better prioritization of << Compliant management & priori : Ce pl 8 pairs { response N! Reducing downtime Customer communications Expansion of water supply Promoting water supply services to new areas services USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 88 [page 104] CTE's Future Focus Area 100% 4% el 7% er 8% 8% 90% 19% 11% LE 13% 8% 8% Q 7% 2 . 11% 80% 7% 3% 12% 16% Other 6 % 11% ing pricil 70% x 24% # Developing pricing structure : 20% Better prioritization of repairs 60% 13% 16% : ; 43% Ô # Reducing downtime 3% 50% 27% 12% m Expansion of water supply services to new areas m Promoting water supply services 20% 14% _—. 8 PPIY 40% 28% 12% Customer communications 0 7. 11% 10% # Compliant management & response e 2 14% 6% 8% m Better use of mWater data 20% 14% : 13% m Staff gender equality 7e xx # Capacity building % 25% n % 10% 24% 24% 14% 14% 14% 0% 4% Cap Haitien Mirebalais Croix des Les Cayes Jeremie Hinche Ouanaminthe Bouquets 89 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 105] ANNEX F. 2021 CTE CUSTOMER SURVEY SECONDARY DATA REVIEW SUMMARY OF DATA CLEANING PROCESS For the CTE customer survey data, the ET imported the data into Stata and adjusted it to be suitable for Stata analysis (variable labels, string variables). Data cleaning process mainly included three steps: 1) identifying and addressing duplicates; 2) identifying and addressing outliers; and 3) re-categorizing “other” options. ° _Identifying and addressing duplicates: The ET reviewed the data for duplicates, and each entry was assigned a unique ID for the analysis. ° _Identifying and addressing outliers: Outliers are identified for each numeric variable using IQR method, and they are addressed by: 1) keeping the data as is where data was plausible: 2) replacing the data as missing value if deemed unplausible: and 3) adjusting the data to correct apparent errors. Two variables (how long have you been a customer and birthdate) were reviewed and corrected. ° _Re-categorizing “other” options: Upon reviewing the categorical responses, a large portion of the respondents chose the “other” options where available and described responses that could have been classified as one of the response options already provided. These were reviewed and re-categorized to correct the large portion of “other” responses. CUSTOMER CHARACTERISTICS Number of surveys completed (active & passive customers) 5,000 4,264 4,000 2,911 3,666 3,000 2,035 2,056 2,000 2,935 942 0 D | Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets # Survey completed Number of clients (SIGA) 90 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 106] Active vs. passive customer 100% o 98% % 0 % 75% 95% 90% ox 86% 50% 56% 25% 0% Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets mActive client M Passive client Customer gender (active & passive customers) 100% jo o) ro, 29% 33% 29% 75% 39% 38% 37% 50% 61% 62% 63% 10° ÊTRE de 25% 0% Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets mFemale mMale 91 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 107] Average customer age (active & passive customers) 60 50 57 57 51 52 4 53 40 30 20 10 0 Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets Average time as a CTE customer (active & passive customers) 140 120 124 100 80 92 60 40 48 c © 0 Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 92 [page 108] PASSIVE CUSTOMERS Reasons for being a passive customer (passive customers) 100% | + —— — — 14 20 75% 196 il 192 50% 47 25% 315 ” 13 31 100 16 0% Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets ml haven't had water in over a year i Water is not distributed on time m1 have too much debt # | buy water from water truck l'have a well m Water is too expensive ml get water from kiosks # Water is still dirty Willingness to become an active customer again (passive customers) 100% 75% 50% 88% 100% 100% ce 25% 57% 50% : 0% Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets mYes BNo 93 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 109] CTE SERVICE Water distribution frequency (active customers) 100% 16% 0 16% © 80% 97% 93% 96% 83% 62% 79% 60% 40% 20% 5 0% Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets mAt least once a day MEveryday Less than 3 times a week More than 3 times a week Water bill received in the last 2 months (active customers) 100% 75% 50% 97% 0, D 92% 91% _—. : 78% 25% SE 0% Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets mYes BNo USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 94 [page 110] Payment mode (active customer) m Bureau CTE # Bureau CTE/MonCash m Other # Collection agent MonCash OX m Check # New subscriber m Bureau CTE/Bank # Bureau CTE/Other # Bureau CTE/MonCash/Credit Card m Bank # Bureau CTE/Credit Card SATISFACTION LEVEL Satisfaction with CTE service (active customers) 100% 75% 59% 54% Lu 2 50% 58% 83% 89% 25% 41% 46% 40% 17% 0% ” Cap Haitien Les Cayes Croix des Jeremie Mirebalais Hinche Bouquets mYes &No 95 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 111] Most satisfied services (active customers) 1,800 1,600 = 1,400 1,200 1,000 _ 800 600 400 200 EE e a e $ .& & N N S a de a & S S # A & & S o S EN Es S Aù À S & Q a Eù $ x RO) < © Ÿ E S s oŸ m Cap Haitien mLes Cayes m Croix des Bouquets mJeremie Mirebalais m Hinche Most dissatisfied services (active customers) 1,800 1,600 1,400 || 1,200 1,000 800 600 = E # EE ES 200 ES nm PSS ES NS à Ÿ à 5 Ka C O < Q S G PS Ô S NS Va S # & S Cà CS < <Ÿ © F2 $ À S° ° S ? [où ro # Cap Haitien MLes Cayes M Croix des Bouquets Jeremie Mirebalais m Hinche USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 96 [page 112] ANNEX G. LIST OF STAKEHOLDERS CONSULTED EAF SOIL Mouvman Fanm Solèy No FECAPH — Le Levier Henry Christophe Campus in Limonade Water Quality Testing Quisqueya University (plant-based FS drying) Living Water Incinerator 509 Sanitation WORKING PARTNERS V3 Engineering mWater (Brain Jensen and John and Annie Feighery) SIGA Ayiti Nexus SISKLOR Centre et Formation et d’Encadrement Zanmi Lasante Management SMALL BUSINESSES AND ASSOCIATIONS SMEs e OJPMBTPA e_ OMEKVIGS e__ ETCNH/FISAJ Construction e _ATTA-EL ROI Construction e ASDA e ATA e RTMC e RETCOM e REVDO e _ASTIAS 97 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 113] e RECCHAC e ATESCA e APGA e BEAUGAZ e SANI-PREFAB e ACM e ESSAM-Plus e AVIE LEAs e _ Boucard Pest Control e SANCO e ADVNN e SANI BON SEVIS e LAVABLECO e ECLAT SERVICE TOTAL e DENSCO e TOP VIDANGE e PAFEFO e Tet Ansanm e LAVI PA FASIL Mayoral/municipal task forces on sanitation e _ CCAM - Mirebalais e CERAAC - Les Cayes FSM Fonfred FSM Management Morne-à-Cabri FSM Management Users of Fonfred FSM Users of Morne-à-Cabri FSM USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 98 [page 114] CTE STAFF FOR FGD Cap Haitien Mirebalais Croix des Bouquets Les Cayes Jeremie Ouanaminthe Hinche CTE STAFF FOR SURVEYS Cap Haitien Mirebalais Croix des Bouquets Les Cayes Jeremie Ouanaminthe Hinche KIOSKS CTE kiosk managers (Ouanaminthe, Les Cayes, Jeremie, Croix des Bouquets) Living Water kiosk managers (Cap Haitien) ONEPA/OREPA Myriame Dorfeuille, Director, ONEPA Neud Pharo Joseph, Systems Specialist, ONEPA Gabriel Tondreau, ONEPA-SISKLOR OREPA directors (Sud, Centre, Ouest, Nord) DAI COP Daniel O’Neil DCOP Samuel Diery Mondestin MEL Officer Nina Bernard Jean Fenzie 99 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 115] Celestin Valdes Milner St Fluer Ana-Flore Leroy Papa Diop CTE Liasons e Lovitha Baptiste - Cap Haitien e Nadine Rene - Hinche e _Zacharie Popote - Mirebalais e _ Ruffine Astremon - Cayes e _Karl-Henry Olias - Ouanaminthe e Camelot Junior - Croix des Bouquets USAID WASH Program Officer/USAID WATSAN Contracting Officer’s Representative Marcia Urquhart Glenn Environmental Officer Abdel Abellard Assistant Environmental Officer Cynthia Figaro WASH Advisor Carmelita Francois OTHER Neil Van Dine - Pignon USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 100 [page 116] ANNEX H. WATSAN PROJECT METHODOLOGY USAID WATSAN THEORY OF CHANGE snnanns oo 000 00 © Then we will (es) Then we will achieve these achieve this project objectives goal + Improving the operational and managerial capacities and reduce + Helping 250.000 people gain non-revenue losses enable the access to basic or improved CTESs. OREPAs, and DINEPA to access to water Reduce the prevalencé of provide water on à cost recovery + Helping 75,000 people get cholera and other basis access to basic or improved waterborne diseases and + Promoting viable business models sanitation services thereby improve the for $MEs and F$SM and closing + Laying the foundation for health and prosperity of financing gaps for market entrants sustainable increases in the Haitian People by increasing access to credit to access to water and achieve broader access to sanitation throughout Haiti sustainable sanitation services Theory of Change FIGURE |: THEORY OF CHANGE USAID WATSAN ACTIVITY INTERVENTIONS AND PHASES The Project’s activities are organized around three closely linked components with specific tasks for each. TT + Finalize operations and maintenance manuals + Implement MIS for each for CTE CTE + Expand non-revenue + Complete EAF grants * parer rater water loss to all CTES + Finalize training cycle . i . ” + Provide training on +_ Finalize operations and . Deveio® MIRE Barton . avrard fr te grantsto customer management + Complete EAF grants + Publish and distribute and MIS for CTES promote private sector *_ Monitor FAF grantees + Finalize training cycle CTE manuals bb)" )") + _identify priority projects. + Finalize designs and put + Complete first projects + Complete all + Final Completion for all + Begin conceptual out for bid first projects + complete design and construction activities projects designs + Begin conceptual procurement of all + Monitor projects during designs of remaining construction projects guarantee period projects FIGURE 2: WATER MILESTONES 101 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 117] + Provide TA to private sector sanitation + Develop sanitation task . force and private sector network and task force _ Continue support to sanitation network in + Award remaining EAF sanitation task force and each area grants and monitor EAF network + Promote the construction grantees + Provide management of toilets for those + Complete EA for FSM training at FSM sites + Develop understanding without sites + Develop standard : with DINEPA and other + Award first EAF grants to + Develop management operating manual for + Publish and distribute projects. promote private sector plan for FSM sites FSM in Haiti FSM manual _ = nn ) ” ) + _Identify priority projects. + Develop scope of work *_ Complete EA for + Complete all FSM +_ Final Completion for all for environmental sanitation activities construction activities projects assessment + Complete design and + Monitor projects during procurement of all FSM guarantee period construction FIGURE 3: SANITATION MILESTONES The USAID Water and Sanitation Project 1 2 Develop MIS Implement an Implement à 5 V4 " (mWater) Annual Plan multi-year plan Li Organization Performance Strategy goals Hi Staffing plan evaluation & based V4 cs training compensation Management AS Manage annual Develop capital ri budget and budget Financial accountin, g 2 management system E Implement Model of water Fa a ani mel distribution Ensure water AE V2 Technical reduce NRW and monitor quality slots operations losses performance ane ( /\ Effective billing Tariff based on Eie V2 & collections consumption CUROTRE Commercial strategy Operations FIGURE 4: USAID WATER AND SANITATION PROJECT USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 102 [page 118] ANNEX I. DATA COLLECTION INSTRUMENTS INFORMED CONSENT AGREEMENT A - DAI, USAID AND DAI IMPLEMENTING PARTNERS Introduction and Purpose: Thank you for taking the time to speak with us today. My name is [NAME]. | am a researcher from Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haiti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to know what were the most important challenges it faced, and to identify lessons learned and good practices to help future water and sanitation initiatives in Haïti or elsewhere. We are speaking to you today in your capacity as the donor/prime/implementing partner (DELETE AS APPROPRIATE) with/for the USAID WATSAN Project. Request: We would like your honest impressions, opinions and thoughts about various issues related to this project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are we from or represent the Government of Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this. If you choose to participate you will be 1 of 35 interviews that are being conducted. You can opt out at a later time if you wish by informing by speaking or private message the facilitator at any time during the course of the discussion. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future programs to increase access to water and sanitation services in Haïti. Procedures: If you agree to participate, we will ask you to discuss your experience and opinion of the activities and services implemented under the USAID WATSAN Project. The interview will take about one and half hours of your time. As part of the evaluation, we may share our notes from this interview With USAID for posting to a public database. However, the notes we may share will not contain your name, title, organization or any details that could identify you as a source of information. Contacts: If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 or at kfelizor@socialimpact.com or +509 3198 6328. | will leave a copy of this form with you. 103 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 119] If you accept, we would like to record our conversation. The purpose of the recording is to help us with our notetaking. Only the evaluation team will have access to the recording, and we will destroy it once we have completed our notes. Do you have any questions? [interviewer should answer any questions] Permission to Proceed l'understand the purpose of the interview as outlined above and understand that | can withdraw from the interview at any time and for any reason. | agree to participate in the interview (Evaluator records). Yes / No Are you willing to be recorded? Yes / No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: Date B -ONEPA AND IPS Introduction and Purpose: Thank you for taking the time to speak with us today. My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haiti] and Social Impact, from Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haiti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to know what were the most important challenges it faced, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haiti or elsewhere. We are speaking to you today because you collaborated directly with USAID WATSAN as a key stakeholder and/or with USAID WATSAN partners that are involved in the water/sanitation sector. Request: We would like your honest impressions, opinions and thoughts about various issues related to this project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are we from or represent the Government of Haiti. Voluntary Participation: Your participation in this interview is voluntary and will not impact the services you have received or may receive in the future. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this. You can opt out at a later time if you wish by informing by speaking or private message the facilitator at any time USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 104 [page 120] during the course of the discussion. If you choose to participate you will be 1 of 35 interviews that are being conducted. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future programs to increase access to water and sanitation services in Haïti. Procedures: lf you agree to participate, we will ask you to discuss your experience and opinion of the activities and services implemented under the USAID WATSAN Project. The interview will take about one and half hours of your time. As part of the evaluation, we may share our notes from this interview with USAID for posting to a public database. However, the notes we may share will not contain your name, title, organization or any details that could identify you as a source of information. Contacts: If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 or at kfelizor@socialimpact.com or +509 3198 6328. | will leave a copy of this form with you. If you accept, we would like to record our conversation. The purpose of the recording is to help us with our notetaking. Only the evaluation team will have access to the recording, and we will destroy it once we have completed our notes. Do you have any questions? [interviewer should answer any questions] Permission to Proceed l'understand the purpose of the interview as outlined above and understand that | can withdraw from the interview at any time and for any reason. | agree to participate in the interview (Evaluator records). Yes / No Are you willing to be recorded? Yes / No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: Date 105 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 121] C - EAF GRANTEES Introduction and Purpose: Thank you for taking the time to speak with us today. My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haiti] and Social Impact, from Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haiti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to know what were the most important challenges it faced, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haiti or elsewhere. We are speaking to you today because you as a USAID WATSAN grantee and collaborator involved in the water/sanitation sector. Request: We would like your honest impressions, opinions and thoughts about various issues related to this project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are we from or represent the Government of Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this. You can opt out at a later time if you wish by informing the facilitator by speaking or private message at any time during the course of the discussion. If you choose to participate you will be | of 35 interviews that are being conducted. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future programs to increase access to water and sanitation services in Haïti. Procedures: If you agree to participate, we will ask you to discuss your experience and opinion of the activities and services implemented under the USAID WATSAN Project. The interview will take about one and half hours of your time. As part of the evaluation, we may share our notes from this interview with USAID for posting to a public database. However, the notes we may share will not contain your name, title, organization or any details that could identify you as a source of information Contacts: If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 or at kfelizor@socialimpact.com or +509 3198 6328. | will leave a copy of this form with you. If you accept, we would like to record our conversation. The purpose of the recording is to help us with our notetaking. Only the evaluation team will have access to the recording, and we will destroy it once we have completed our notes. Do you have any questions? [interviewer should answer any questions] Permission to Proceed l'understand the purpose of the interview as outlined above and understand that | can withdraw from the interview at any time and for any reason. | agree to participate in the interview (Evaluator records). Yes / No USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 106 [page 122] Are you willing to be recorded? Yes / No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: Date D - FSM MANAGEMENT AND FSM USERS Introduction and Purpose: Thank you for taking the time to speak with us today. My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haiti] and Social Impact, from Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haiti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to know what were the most important challenges it faced, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haiti or elsewhere. We are speaking to you today as a key stakeholder and implementing partner with the USAID WATSAN project at this FSM site OR because you have been directly affected by the USAID WATSAN project intervention at this FSM site (delete as applicable). Request: We would like your honest impressions, opinions and thoughts about various issues related to this project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are we from or represent the Government of Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this. You can opt out at a later time if you wish by informing the facilitator by speaking or private message at any time during the course of the discussion. If you choose to participate you will be | of 35 interviews that are being conducted. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future programs to increase access to water and sanitation services in Haïti. Procedures: If you agree to participate, we will ask you to discuss your experience and opinion of the activities and services implemented under the USAID WATSAN Project. The interview will take about one and half hours of your time. As part of the evaluation, we may share our notes from this interview with USAID for posting to a public database. However, the notes we may share will not contain your name, title, organization or any details that could identify you as a source of information. 107 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 123] Contacts: If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 or at kfelizor@socialimpact.com or +509 3198 6328. | will leave a copy of this form with you. If you accept, we would like to record our conversation. The purpose of the recording is to help us with our notetaking. Only the evaluation team will have access to the recording, and we will destroy it once we have completed our notes. Do you have any questions? [interviewer should answer any questions] Permission to Proceed | understand the purpose of the interview as outlined above and understand that | can withdraw from the interview at any time and for any reason. | agree to participate in the interview (Evaluator records). Yes / No Are you willing to be recorded? Yes / No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: Date E- SMES, LEAS, MFTS, KIOSK MANAGERS Date: Facilitator: Primary Notetaker Name: Respondent Summary: FOCUS GROUP CONTACT SCRIPT Introduction and Purpose: My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haïti] and Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haïti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to assess what were the most important challenges it faces, and to identify lessons learned USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 108 [page 124] and good practices that might help future water and sanitation initiatives in Haiti or elsewhere | was given your contact information by DAI. We are conducting focus group discussions with small businesses and organizations that are USAID WATSAN stakeholders to learn about their participation and perception of the project. These organizations will include but are not limited to Small and Medium Sized Enterprises, Latrine Emptying Associations, Mayoral Task Forces on Sanitation and water kiosk managers. Request Your organization received technical assistance from USAID/Haiti project to improve their water and sanitation services. Your experiences with DAI and the USAID WATSAN Project will help us to assess whether the assistance was effective or not. We would like, with the help of this Focus Group, to learn about your impressions, opinions and thoughts about various issues related to the USAID WATSAN project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are We from or represent the Government of Haiti. Your participation in this focus group discussion is voluntary and will not impact the services you have received or may receive in the future. You will be taking part in | of 24 focus groups that are being organized. Procedures: If you agree to participate in the Focus Group Discussion, we will ask the group members to talk about their experience and opinion of the activities and services implemented under the USAID WATSAN Project. You can opt out at a later time if you wish by informing the facilitator at any time by speaking or private message. The discussion will take about 90 minutes of your time. Because of the current public health situation, the focus group discussion will be conducted online using WhatsApp and/or Zoom. If you agree to participate, we will provide you the details about the date and time of the call and provide you with credit to allow you to access the Internet for the call. You are under no obligation to participate. Do you agree to participate? Yes No Initials of evaluator to indicate receipt of verbal consent to be interviewed': Initials of evaluator to indicate receipt of verbal consent to be recorded: Date If they agree to participate: Thank you. We will share more information about the date and time of the call and provide you with credit to allow you to access the Internet for the call. Prior to the call, we have just a few questions about you and your household: 1. Which best describes your situation? ° __Married, living with spouse 109 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 125] e__ Married, not living with spouse e _Female-headed household e Single °__Widow e Other (please describe) 2. What organization do you work for and what is your job title? INFORMED CONSENT AGREEMENT - FGD Thank you all for agreeing to participate in todays’ call. Introduction and Purpose: My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haïti] and Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haïti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to assess what were the most important challenges it faces, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haiti or elsewhere | was given your contact | information by DAI. We are conducting focus group discussions with small businesses and organization that are USAID WATSAN stakeholders to learn about their participation and perception of the project. Request Your organization received technical assistance from USAID/Haiti project to improve their water and sanitation services. Your experiences with DAI and the USAID WATSAN project will help us to assess whether the assistance was effective or not. We would like, with the help of this Focus Group, to learn about your impressions, opinions and thoughts about various issues related to the USAID WATSAN project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are We from or represent the Government of Haiti. You will be taking part in 1 of 24 focus groups that are being organized. Procedures: Today we are going to discuss your experience and opinion of the support received and the activities you participated in as part of the USAID WATSAN project. The discussion with last about an hour and a half. Your answers will be kept confidential. DAI has shared your information with us but will not know whether you chose to participate or any details of your response. As part of the evaluation, we may share our notes from this interview with USAID for posting to a public database. However, the notes we may share will not contain your name, title, organization or any details that could identify you as a source of information. Due to the private nature of this research, we ask that all focus group participants agree not to share anything that is discussed with anyone outside of this group once this conversation ends. Nonetheless, there is a risk that other discussion participants will repeat what is shared here today. Remember that you are free to refuse to answer any question. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 110 [page 126] Risks/Benefits: You will not receive any direct benefit or compensation for participating in this discussion group, either from Social Impact or from [XXX]. If you have not already, you will be provided with a credit stipend of [amount] to cover the data costs of participating. Voluntary Participation: Participation in this discussion group is completely voluntary. Your participation in this focus group discussion is voluntary and will not impact the services you have received or may receive in the future. | You can opt out at a later time if you wish by informing the facilitator by speaking or private message at any time during the course of the discussion. If you choose to participate you will be | of 24 focus groups that are being conducted. Do you have any questions? [interviewer should answer any questions] Contacts: If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 or at kfelizor@socialimpact.com or +509 3198 6328. | will leave a copy of this form with you. Permission to Proceed If you accept, we would like to record our conversation. The purpose of the recording is to help us with our notetaking. Only the evaluation team will have access to the recording, and we will destroy it once we have completed our notes l'understand the purpose of the interview as outlined above and understand that | can withdraw from the interview at any time and for any reason. | agree to participate in the interview (Evaluator records). Yes No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: Date Brief Description of FGD Process This will be conducted as a focus group. | will ask a series of questions to the group and facilitate some discussion. We want everyone to feel free to express themselves and participate, especially if you have a different opinion than what is being shared by others. Not everybody needs to respond to each question, though we hope you to speak up if you have something to share. Because we are having this discussion over Zoom/WhatsApp instead of in person, | may call on individuals to make sure everyone has an opportunity to contribute. If at any point you do not wish to respond you can say so. F - CTE MANAGEMENT Date: 111 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 127] Facilitator: Primary Notetaker Name: Respondent Summary: FOCUS GROUP CONTACT SCRIPT Introduction and Purpose: My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haïti] and Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haïti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to assess what were the most important challenges it faces, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haiti or elsewhere | was given your contact information by DAI. We are conducting focus group discussions with USAID WATSAN stakeholders including CTE Management staff to learn about their participation and perception of the project. Request Your organization received technical assistance from USAID/Haiti project to improve their water and sanitation services. Your experiences with DAI and the USAID WATSAN Project will help us to assess whether the assistance was effective or not. We would like, with the help of this Focus Group, to learn about your impressions, opinions and thoughts about various issues related to the USAID WATSAN project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are We from or represent the Government of Haiti. You will be taking part in 1 of 24 focus groups that are being organized. Procedures: |f you agree to participate in the Focus Group Discussion, we will ask the group members to talk about their experience and opinion of the activities and services implemented under the USAID WATSAN Project. You can opt out at a later time if you wish by informing the facilitator at any time. The discussion will take about 90 minutes of your time. Because of the current public health situation, the focus group discussion will be conducted online using WhatsApp and/or Zoom. lf you agree to participate, we will provide you the details about the date and time of the call and provide you with credit to allow you to access the Internet for the call. You are under no obligation to participate. Do you agree to participate? Yes No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 112 [page 128] Date INFORMED CONSENT AGREEMENT - FGD Thank you all for agreeing to participate in todays’ call. Introduction and Purpose: My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haïti] and Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Activity. USAID WATSAN is a project to help improve access to water and sanitation services in Haïti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to assess what were the most important challenges it faces, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haiti or elsewhere | was given your contact information by DAI. We are conducting focus group discussions with CTE Management Staff to learn about their participation and perception of the project. Request Your organization received technical assistance from USAID/Haiti project to improve their water and sanitation services. Your experiences with DAI and the USAID WATSAN project will help us to assess whether the assistance was effective or not. We would like, with the help of this Focus Group, to learn about your impressions, opinions and thoughts about various issues related to the USAID WATSAN project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are we from or represent the Government of Haiti. You will be taking part in 1 of 24 focus groups that are being organized. Procedures: Today we are going to discuss your experience and opinion of the support received and the activities you participated in as part of the USAID WATSAN project. The discussion with last about an hour and a half. Your answers will be kept confidential. DAI has shared your information with us but Will not know whether you chose to participate or any details of your response. As part of the evaluation, we may share our notes from this interview with USAID for posting to a public database. However, the notes we may share will not contain your name, title, organization or any details that could identify you as a source of information Due to the private nature of this research, we ask that all focus group participants agree not to share anything that is discussed with anyone outside of this group once this conversation ends. Nonetheless, there is a risk that other discussion participants will repeat what is shared here today. Remember that you are free to refuse to answer any question. Risks/Benefits: You will not receive any direct benefit or compensation for participating in this discussion group, either from Social Impact or from [XXX]. If you have not already, you will be provided with a credit stipend of [amount] to cover the data costs of participating. 113 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 129] Voluntary Participation: Participation in this discussion group is completely voluntary. Your participation in this focus group discussion is voluntary and will not impact the services you have received or may receive in the future. If at any point, any of you don’t want to continue, please let me know. You can opt out at a later time if you wish by informing by speaking or private message the facilitator at any time during the course of the discussion. Do you have any questions? [interviewer should answer any questions] Contacts: If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 or at kfelizor@socialimpact.com or +509 3198 6328. | will leave a copy of this form with you. Permission to Proceed If you accept, we would like to record our conversation. The purpose of the recording is to help us with our notetaking. Only the evaluation team will have access to the recording, and we will destroy it once we have completed our notes l'understand the purpose of the interview as outlined above and understand that | can withdraw from the interview at any time and for any reason. | agree to participate in the interview (Evaluator records). Yes No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: Date Brief Description of FGD Process This will be conducted as a focus group. | will ask a series of questions to the group and facilitate some discussion. We want everyone to feel free to express themselves and participate, especially if you have a different opinion than what is being shared by others. Not everybody needs to respond to each question, though we hope you to speak up if you have something to share. Because we are having this discussion over Zoom/WhatsApp instead of in person, | may call on individuals to make sure everyone has an opportunity to contribute. If at any point you do not wish to respond you can say so. G - OREPA Date: Facilitator: Primary Notetaker Name: USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 114 [page 130] Respondent Summary: FOCUS GROUP CONTACT SCRIPT Introduction and Purpose: My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haïti] and Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haïti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to assess what were the most important challenges it faces, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haiti or elsewhere | was given your contact information by DAI. We are conducting focus group discussions with OREPA Reginal Directors to learn about their participation and perception of the project. Request Your organization received technical assistance from USAID/Haiti project to improve their water and sanitation services. Your experiences with DAI and the USAID WATSAN Project will help us to assess whether the assistance was effective or not. We would like, with the help of this Focus Group, to learn about your impressions, opinions and thoughts about various issues related to the USAID WATSAN project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are We from or represent the Government of Haiti. You will be taking part in 1 of 24 focus groups that are being organized. Procedures: If you agree to participate in the Focus Group Discussion, we will ask the group members to talk about their experience and opinion of the activities and services implemented under the USAID WATSAN Project. The discussion will take about 90 minutes of your time. Because of the current public health situation, the focus group discussion will be conducted online using WhatsApp and/or Zoom. If you agree to participate, we will provide you the details about the date and time of the call and provide you with credit to allow you to access the Internet for the call. You are under no obligation to participate. Do you agree to participate? Yes No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: Date INFORMED CONSENT AGREEMENT - FGD 115 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 131] Thank you all for agreeing to participate in todays’ call. Introduction and Purpose: My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haïti] and Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Activity. USAID WATSAN is a project to help improve access to water and sanitation services in Haiti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to assess what were the most important challenges it faces, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haïti or elsewhere | was given your contact information by DAI. We are conducting focus group discussions USAID WATSAN Stakeholders including OREPA directors to learn about their participation and perception of the project. Request Your organization received technical assistance from USAID/Haiti project to improve their water and sanitation services. Your experiences with DAI and the USAID WATSAN project will help us to assess whether the assistance was effective or not. We would like, with the help of this Focus Group, to learn about your impressions, opinions and thoughts about various issues related to the USAID WATSAN project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are We from or represent the Government of Haiti. You will be taking part in 1 of 24 focus groups that are being organized. Procedures: Today we are going to discuss your experience and opinion of the support received and the activities you participated in as part of the USAID WATSAN project. The discussion with last about an hour and a half. Your answers will be kept confidential. DAI has shared your information with us but will not know whether you chose to participate or any details of your response. As part of the evaluation, we may share our notes from this interview with USAID for posting to a public database. However, the notes we may share will not contain your name, title, organization or any details that could identify you as a source of information. Due to the private nature of this research, we ask that all focus group participants agree not to share anything that is discussed with anyone outside of this group once this conversation ends. Nonetheless, there is a risk that other discussion participants will repeat what is shared here today. Remember that you are free to refuse to answer any question. Risks/Benefits: You will not receive any direct benefit or compensation for participating in this discussion group, either from Social Impact or from [XXX]. If you have not already, you will be provided with a credit stipend of [amount] to cover the data costs of participating. Voluntary Participation: Participation in this discussion group is completely voluntary. Your participation in this focus group discussion is voluntary and will not impact the services you have received or may receive in the future. If at any point, any of you don’t want to continue, please let me know. _ If you wish to stop the survey at any point please tell the data enumerator by speaking or private message. Do you have any questions? [interviewer should answer any questions] Contacts: If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 or at kfelizor@socialimpact.com or +509 3198 6328. | will leave a copy of this form with you. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 116 [page 132] Permission to Proceed If you accept, we would like to record our conversation. The purpose of the recording is to help us with our notetaking. Only the evaluation team will have access to the recording, and we will destroy it once we have completed our notes l'understand the purpose of the interview as outlined above and understand that | can withdraw from the interview at any time and for any reason. | agree to participate in the interview (Evaluator records). Yes No Initials of evaluator to indicate receipt of verbal consent to be interviewed: Initials of evaluator to indicate receipt of verbal consent to be recorded: Date Brief Description of FGD Process This will be conducted as a focus group. | will ask a series of questions to the group and facilitate some discussion. We want everyone to feel free to express themselves and participate, especially if you have a different opinion than what is being shared by others. Not everybody needs to respond to each question, though we hope you to speak up if you have something to share. Because we are having this discussion over Zoom/WhatsApp instead of in person, | may call on individuals to make sure everyone has an opportunity to contribute. If at any point you do not wish to respond you can say so. H- CTE STAFF SURVEY INFORMED CONSENT AGREEMENT Introduction and Purpose: Thank you for taking the time to speak with us today. My name is [NAME]. | am a researcher representing s [Name of TBD local data collection firm based in Haiti] | and Social Impact, a company that is based in the United States. Our team is conducting an evaluation study of the USAID/HAITI WATSAN Project. USAID WATSAN is a project to help improve access to water and sanitation services in Haiti. The purpose of the study is to understand the extent to which USAID WATSAN was successful meeting its objectives, to assess what were the most important challenges it faces, and to identify lessons learned and good practices that might help future water and sanitation initiatives in Haiti or elsewhere. We are speaking to you today because you collaborated directly with USAID WATSAN or with USAID WATSAN partners, are involved in the water/sanitation sector, or because you may have been directly or indirectly affected by the USAID WATSAN Project interventions. The objective of this CTE staff survey is to see how the WATSAN CTE capacity building and support affected CTE staff capacity and how the changes in management affected CTE staff in their day-to-day 117 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 133] work with the CTE. Request: We would like your opinions about various issues related to this project’s implementation and outcomes. We are independent consultants who have no affiliation with the firms and institutions that implemented USAID WATSAN, nor do we represent any other private sector firm in the water/sanitation sector, nor are we from or represent the Government of Haiti. Approximately 70 people will be surveyed. Voluntary Participation: Your participation in this survey is voluntary and will not impact the services you have received or may receive in the future. You do not have to agree to participate in this study. You are free to end the survey at any time or to decline to answer any question which you do not wish to answer. lf you decline to participate in the survey, no one will be informed of this. If you wish to stop the survey at any point please tell the data enumerator by voice or private message. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future programs to increase access to water and sanitation services in Haïti. Procedures: If you agree to participate, we will ask you to ask you a series of questions about your experience and opinion of the activities and services implemented under the USAID WATSAN Project and your role with the CTE. The survey will take about 30 minutes of your time. As part of the evaluation, we may share our notes from this interview with USAID for posting to a public database. However, the notes we may share will not contain your name, title, organization or any details that could identify you as a source of information. Contacts: If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 or at kfelizor@socialimpact.com or +509 3198 6328. | will leave a copy of this form with you. Do you have any questions? [interviewer should answer any questions] Permission to Proceed lunderstand the purpose of the survey as outlined above and understand that | can withdraw from the survey at any time and for any reason. | agree to participate in the survey (evaluator records). Yes / No Initials of evaluator to indicate receipt of verbal consent: Date USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 118 [page 134] QUALITATIVE AND QUANTITATIVE QUESTIONNAIRES USAID KII 1. Date: 2. Start time: 3. End time: 4. Modälity: in person/remote 5. interviewer Name: 6. Primary Notetaker Name: 7. Respondent(s)Name(s) 8. Respondent(s) Title(s): 9. Respondent Organization: 10. [Separate: Respondent contact information] 11. Anyone else present: 12. # Months Respondent has worked with the organization: 13. Sex of respondent: Questions 1... Whatis your role in your organization and how does it relate to the WATSAN activity? 2. How long have you been in this role? 3. What was your experience of working with DAI and other Implementing Partners? a Probe for details, successes, challenges. 4... What was your experience working with and including the GOH in the WATSAN activities? 5. How was your experience/understanding of implementation, regarding star- up ofthe project, mid-point and close out? 6. Please describe your experience of using the Water Utility Turnaround Framework? 7. Probe for successes and probe for challenges. 8. What do you think were the biggest successes of the WATSAN project? a Probe Water, HH sanitation, FSM, governance, sustainability 9. What do you think were the biggest challenges of the WATSAN project? a Probe Water, HH sanitation, FSM, governance, sustainability 10. Do you think there were any gaps in the support by the WATSAN project? a Probe regarding water supply services, support to CTESs, sanitation services, support to SMEs?. 11. In terms of project outcomes were these what you expected? a lfyes - probe details If no - probe details 12. Were there any project outcomes you did not expect? a lfyes - probe details 119 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 135] 13. Please describe any impact on project implementation due to Natural Disasters, COVID, Political unrest, if so what and how have they impacted WATSAN? 14. What was your experience of monitoring the WATSAN activity? a. Were data quality assessments completed b. Probe details regarding specific indicator tracking, reporting issues? 15. Can you share any lessons learned, pros or cons of the monitoring process? 16. Did DAI and/or the IP bring Gender/GESl into the implementation, if so how? 17. Do you feel the WATSAN project addressed sustainability? if so how? 18. What are your thoughts on EQI 19. What are your thoughts on EQ2 20. What are your thoughts on EQ3 21. Is there anything else you would like to mention or discuss about the WATSAN activity? ENTERPRISE ACCELERATION FUND (EAF) GI | ! HOUR 30 MINUTES DURATION EAF Project Name: 1. Date: 2. Start time: 3. End time: 4. Modality: in person/remote 5. interviewer name: 6. Primary notetaker name: 7. Respondent(s) title(s): 8. Respondent organization: 9. # months respondent has worked with the organization: 10. Sex of respondent: [Separate: Respondent contact information] 11. Respondent(s) name(s) 12. Anyone else present: QUESTIONS 1. Whatis your role at your organization and how long have you been in this role? 2. How does your role relate to the USAID WATSAN Project? What are your thoughts about the EAF funding application process? a. Probe: what went well? b. Probe: what could have gone better? 3. How long did the EAF funding application process take? [please tick one] "Less than a month "Between | month and 2 months "Between 2 months and 4 months "Between 4 and 6 months USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 120 [page 136] "More than 6 months " DK 4. What support did USAID WATSAN project provide during the EAF funding application process? 5. Would you recommend any modifications to the EAF funding application process? Probe: if yes, what and why? 6. Please describe how EAF funds were used on the project? 7. Where the funds sufficient or not? Probe: if not, please explain 8. Did your use of the EAF funding change over time? Probe: if yes, please explain. 9. What was your experience of the EAF project implementation process? Probe: positive experiences Probe: challenges, including the Payment By Results modality 10. Apart from funding, what support did USAID WATSAN project provide during the EAF project implementation process? Probe: technical support/training for your particular field of work? Probe: non-technical support/training relevant to your particular business activity? Probe: how was the support/training delivered (face to face, online, other) and by who? 11. Can you describe the EAF project outputs and their current status? Probe: details of achievements Probe: details of challenges/difficulties with delivering outputs 12. What were the overall results from using the EAF funding? Probe: unexpected project impacts? Details? 13. Please describe how your business and client base has or has not developed/increased compared to before the implementation of the EAF grant. Probe: if yes, in what ways and how? Success factors? Probe: if no, why not? Factors negatively affecting your business? 14. On reflection, what are the strengths of your business and client base now? 15. On reflection, what are the weaknesses of your business and client base now? 16. Looking forward into the future, what are the opportunities for your business and client base? 17. Looking forward into the future, what threats are there to your business and client base? 18. Where do you see your business and client base in six months’ time? 121 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 137] 19. How was the progress of you EAF project monitored? Probe: positive and negative aspects of monitoring process Probe: from the monitoring process could you please share any lessons learned about the project? Do you take sustainability into account for your businesses and services? If so, how? Probe if so, what if any impacts has/does the EAF project have on sustainability? Have there been impacts on the project due to natural disasters, COVID-19 or political unrest (including fuel issues)? Probe: If yes, please describe them 20. Please describe the communication between you and your team with DAI during the application process and implement? Probe: issues, recommendations for improvement 21. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN project provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how!?Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 22. Is there anything else you would like to mention or discuss about your use of and experience with the EAF grant for your business? ONEPA KII | I HOUR 30 MINUTES DURATION 1. Date: 2. Start time: 3. End time: 4. Modality: in person/remote 5. interviewer name: 6. Primary notetaker name: 7. Respondent(s) name(s): 8. Respondent(s) title(s): 9. Respondent organization: [Separate: Respondent contact information] 10. Anyone else present: USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 122 [page 138] 11. # months respondent has worked with the organization: 12. Sex of respondent: QUESTIONS 1. Whatis your role in ONEPA, and how long have you been in this role? 2. How does it relate to the USAID WATSAN Project? 3. What activities have you individually taken part in as part of the USAID WATSAN Project? 4. How do you or others at ONEPA use mWater data? 5. Please discuss your experience/understanding regarding how the CTEs manage and upload the mWater data? 6. VWVhat decisions does mWVater data enable and how? 7. How do you or others at ONEPA use SIGA data? 8. Please discuss your experience/understanding regarding how the CTEs manage and use SIGA data. What decisions does SIGA data enable and how? 9. How do you or others at ONEPA use SISKLOR data? What decisions does SIGA data enable and how? 10. Please discuss your experience/understanding regarding how the CTEs manage and use SISKLOR data. What decisions does the SISKLOR data enable? 11. How do you or others at ONEPA use QuickBooks information? What decisions does the QuickBooks information enable? 12. Please discuss your experience/understanding regarding how the CTEs manage and use QuickBooks. 13. What is ONEPA's experience of managing all this data? Probe: successes Probe: challenges Probe: gaps in data collection/analysis management capacity Probe: data usage opportunities Probe: data usage constraints 14. How is all this data influencing decision-making and planning on a day-to-day and month-to- month basis? 15. How do you think this data/information management does or does not affect the capacity of the CTESs to grow their businesses? 16. Please describe your experience of using the Water Utility Turnaround Framework. Probe: successes and challenges 17. Do you feel there is a difference between the CTESs that had direct USAID WATSAN support and the other CTESs in the country who were not directly supported by USAID WATSAN? 18. In terms of project outcomes, were these what you expected? Probe: details for all responses 123 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 139] 19. Were there any project impacts you did not expect? Probe: for details 20. Have you had any opportunities to share your knowledge and skills with colleagues/CTEs? Probe: for details on how knowledge sharing is managed/encouraged 21. What is your experience regarding the USAID WATSAN activities in the sanitation sector? Probe: for details, success and challenges for different project activities 22. Has the USAID WATSAN Project addressed the sustainability of the CTEs and the water supply service delivery and sanitation service delivery? 23. How, if at all, does this relate to ONEPAs longer-termsustainability? 24. Please share your thoughts on the success and challenges of the USAID WATSAN project. In the sanitation and water sector 25. What do you think should be the ONEPA'’s main area of focus in 2022? 26. Looking into the future, what you do you feel ONEPA is doing regarding increasing access to water and sanitation services in Haïti? 27. Have there been impacts on the project due to natural disasters, COVID-19, or political unrest? If so, what and how? 28. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how? Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 29. Did DA or the CTESs take the needs of marginalized populations and how best to support them into account when implementing project activities? Probe: If yes, how Probe: if no, why not? 30. Is there anything else you would like to mention or discuss about the USAID WATSAN Project? USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 124 [page 140] IMPLEMENTING PARTNER GI | ! HOUR 30 MINUTES DURATION Centre et Formation d'Encadrement, Zanmi Lasante . Date: 2. Start time: 3. End time: 4. Modality: in person/remote 5. interviewer name: 6. Primary notetaker name: 7. Respondent(s) name(s): 8. Respondent(s) title(s): 9. Respondent organization: [Separate: Respondent contact information] 10. Anyone else present: LL. # months respondent has worked with the organization: 12. Sex of respondent: QUESTIONS 1. Whatis your role in [NAME OF IP], and how long have you been in this role? 2. How does your role relate to the USAID WATSAN activity? 3. What activities have you individually taken part in as part of the USAID WATSAN project? 4. What, if any, support did USAID WATSAN provide during the USAID WATSAN contracting process? 5. Would you recommend any modifications to the USAID WATSAN contracting process? Probe: if yes, what and why 6. How were USAID WATSAN funds used? Probe: sufficient or not? Please explain. Did your use of the USAID WATSAN funding change over time? Probe: if yes, record details 7. What was your experience of the USAID WATSAN Project implementation process? Probe: positive experiences and challenges. 8. Apart from funding, what support did USAID WATSAN provide during the project implementation process? Probe: technical support/training for your particular field of work? Probe: non-technical support/training relevant to your particular business activity? Probe: how was the support/training delivered (face to face, online, other) and by who? 9. Can you describe the USAID WATSAN Project outputs and their current status? Probe: achievements, details; if no, probe for details. 125 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 141] 10. Overall, what were the results from your organization using the USAID WATSAN funding? Probe: unexpected project impacts? Did you or DAI monitor the USAID WATSAN Project? Probe: positive and negative aspects of monitoring process Probe: from the monitoring process could you please share any lessons learned about the project? 23. Do you take sustainability into account for your businesses and services? If so, how? Probe if so, what if any impacts has/does the project have on sustainability? 24. Have there been impacts on your project due to natural disasters, COVID-19 or political unrest (including fuel issues)? Probe: if yes, please describe them 25. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how? Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 26. Did DAI or the [name of IP] take needs of marginalized populations and how best to support them to them into account when implementing project activities? Probe: If yes, how Probe: if no, why not? 27. Is there anything else you would like to mention or discuss about your use of and experience with the USAID WATSAN project and/or DAI? USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 126 [page 142] DAI GI CTE LIAISON OFFICERS | I! HOUR 30 MINUTES DURATION 1. Date: 2. Start time: 3. End time: 4. Modality: in person/remote 5. interviewer name: 6. Primary notetaker name: 7. Respondent(s) name(s): 8. Respondent(s) title(s): 9. Respondent organization: [Separate: Respondent contact information] 10. Anyone else present: 11. # months respondent has worked with the organization: 12. Sex of respondent: QUESTIONS 1. For which CTE(s) are you the Liaison Officer for and how long have you been in this role? 2. Whatis your role as CTE Liaison Officer and how does it relate to supporting the work of the CTE? Probe: for successes and challenges. 3. What support were you involved in providing for CTE staff? Probe: technical support/training for your CTE water supply functions? Probe: non-technical support/training relevant to CTE business development? Probe: how was the support/training delivered (face to face, online, other) and by who? We are now going to talk with you about the various water utility management software packages used by the CTEICTES in their day-to-day operations. 4. Please describe your experience with the CTESs of using the Water Utility Turnaround Framework? Probe: for successes and challenges. 5. How is the Operational Manual useful to the CTEs? Probe: for successes and challenges 127 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 143] 6. What has been the impact/experience of using mWater for the CTEs and DINEPA/ONEPA? Probe: for successes and challenges Probe: what decisions has mWater data enabled for your CTE? 7. What has been the impact/experience of QuickBooks for the CTEs and DINEPA/ONEPA? Probe: for successes and challenges Probe: what decisions has QuickBooks enabled for your CTE? 8. What has been the experience of SIGA for the CTEs and DINEPA/ONEPA? Probe: for successes and challenges Probe: what decisions has SIGA enabled for your CTE? 9. What has been the experience of SISKLOR for the CTEs and DINEPA/ONEPA? Probe: for successes and challenges Probe: what decisions has SISKLOR enabled for your CTE? 10. How did DAI and the CTE use the budget for operational and planning, and has this changed over the course of the project! If so, how? Please describe the impacts. 11. How do the CTEs manage collection of tariffs, and has this changed over the course of the project} If so, how? Please describe the impacts. 12. Do you feel the tariff is appropriate for the services provided? Probe: Are clients willing to pay for the services? 13. How do you think the CTESs can continue to grow and increase their revenue? 14. What is the approach of the CTE to non-revenue water losses? 15. How does the CTE retain current subscribers? 16. How does the CTE recruit and manage new subscribers? Probe: subscribers’ goal, how does the CTE meet those goals? 17. What has been the impact/experience of the kiosks (if any) in your system, and how has this effected the client base? 18. What do you think attracts people to work at the CTE? 19. What do you think are the reasons why staff leave the CTE? Pause here, if necessary. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 128 [page 144] 20. In terms of project impacts for CTEs, were these what you expected? Probe: if yes for details Probe: if no for details 21. Were there any project impacts for CTEs you did not expect? If yes, probe for details. 22. 23. Have there been impacts on the project due to natural disasters, COVID-19 or political unrest (including fuel issues)? Probe: If yes, please describe them 24. What are your thoughts on how USAID WATSAN activities will continue after the USAID WATSAN Project ends? 25. What are or should be the future business development priorities for the CTESs regarding water supply services? 26. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how? Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 27. Did DA and your CTESs take the needs of marginalized populations and how best to support them into account when implementing project activities? Probe: If yes, how, if no, why not? 28. Is there anything else you would like to discuss about your experience of working with the CTESs? 129 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 145] OREPA FGD | I HOUR 30 MINUTES DURATION 1. Date: 2. Start time 3. Endtime 4. Interviewer name: 5. Primary notetaker name: QUESTIONS 1. What was the experience of OREPA working with USAID WATSAN? Probe: for details. 2. Please describe OREPA's experience of using the Water Utility Turnaround Framework? Probe: for successes and challenges. 3. What is your opinion of the capacity of the CTEs to manage data (mWater, SIGA, SISKLOR, Quickbooks) and how has this changed over the course of the project?How is this data enabling decision-making and planning on a day-to-day and month-to-month basis? 4. If so, how and please describe the impacts. Probe: mWater. Probe: SIGA. Probe: SISKLOR. Probe: Quickbooks. 5. How do you all think this data/information management is affecting the capacity of the CTESs to grow their businesses? 6. Does OREPA have a knowledge-sharing program? If so, how is knowledge sharing encouraged and managed? Probe: is there knowledge sharing between CTEs? 7. Has the USAID WATSAN Project addressed the sustainability of the CTE to deliver water supply services and sanitation services? USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 130 [page 146] Probe: to new customers? Probe: to existing customers in terms of service quality? 8. Have there been impacts on the project due to natural disasters, COVID-19, or political unrest? If so, what and how have they affected the project? 9. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how? 10. Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 11. What activities did you support or were in your region regarding sanitation activities for the WATSAN USAID project Probe: success, challenges, details on activities undertaken. 12. What do you think should be OREPA’s main area of focus this year? 13. Is there anything else you would like to mention or discuss about the USAID WATSAN activity and its support to CTEs? 131 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 147] SMALL MEDIUM ENTERPRISE FGD | I! HOUR 30 MINUTES DURATION 1. Date: 2. Start time 3. Endtime 4. Interviewer name: 5. Primary notetaker name: QUESTIONS 1. What did your SME accomplish with support from USAID WATSAN? 2. What activities did you participate in as part of the USAID WATSAN project? Probe: details, thoughts on trainings, information/materials provided, success, challenges 3. What are some success and/or challenges you faced when working with USAID WATSAN? 4. Please explain/describe how or how not your organization’s capacity changed during your participation in the USAID WATSAN project? a. Probe for capacity building. 5. Asa group, what do you think about the sustainability of the services you provide? 6. Has there been a trend in the change in the number of customers? If so, please explain? 7. Has there been a trend in the change of your SME’s revenue? [f so, please explain. 8. Has there been a trend in the change in the number of services you provide? If so, please explain. 9. Will you or any of your customers use an FSM site for fecal sludge disposal? a. Probe why or why not. 10. How do clients transport material to the FSM, what are the challenges with transportation for the FSMs and/or for the users/clients? 11. What FSM activities are being funded by other organizations (WB, IDB etc) and what have been the successes and challenges of this collaboration? 12. What are the strengths of your business and client base now? 13. What are the weaknesses of your business and client base now? 14. What are the opportunities for your business and client base now/in the future? 15. What are the threats to your business and client base now/in the future? 16. Where do you see your business and client base in six months’ time? USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 132 [page 148] 17. Have there been impacts on your activities/business due to natural disasters, COVID-19, or political unrest? If so, what and how have they affected the project? 18. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how? Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 19. Did DAI or the SME take the needs of marginalized populations and how to support them into account regarding the project activities? If so, how and why? 20. Is there anything else you would like to mention or discuss about the USAID WATSAN activity? 133 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 149] LATRINE EMPTIER ASSOCIATION FGD | | HOUR 30 MINUTES DURATION I. Date: 2. Start time 3. End time 4. Interviewer name: 5. Primary notetaker name: QUESTIONS 1. What activities did you participate in as LEA members, and what are your thoughts about USAID WATSAN implementation? 2. What were you able to do with support from USAID WATSAN with your LEA? 3. Can you please share the advantages and disadvantages of these activities? 4. Will you use an FSM site? Probe: why/why not. 5. What are some successes and/or challenges you faced when working with USAID WATSAN? 6. Please explain/describe how or how not your organization’s capacity changed during your participation in the USAID WATSAN Project? Probe: for capacity building 7. How do clients transport material to the FSM, what are the challenges with transportation for the FSMs and/or for the users/clients? 8. What FSM activities are being funded by other organizations (WB, IDB etc) and what have been the successes and challenges of this collaboration? 9. Do you consider sustainability regarding the services you provide? If so, how? Please explain the importance. 10. Has there been a trend in the change in the number of customers? If so, please explain. 11. Has there been a trend in the change of revenue? If so, please explain. 12. Has there been a trend in the change in the number of services you provide? If so, please explain. 13. What are the strengths of your business and client base now? 14. What are the weaknesses of your business and client base now? USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 134 [page 150] 15. What are the opportunities for your business and client base now? 16. What are the threats to your business and client base now? 17. Where do you see your business and client base in six months’ time? 18. Have there been impacts on your activities/business due to natural disasters, COVID-19, or political unrest? If so, what and how have they affected the project? 19. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how? Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 20. How do you take the needs of marginalized populations and how to support them into account regarding the work of the LEA? If so, how and why? 21. Is there anything else you would like to mention or discuss about the USAID WATSAN Project and its work with you? 135 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 151] MAYORAL TASK FORCE FGD | |! HOUR 30 MINUTES DURATION e Date: e Start time e Endtime e interviewer name: e Primary notetaker name: QUESTIONS 1. What activities does the MTFparticipate in, and what are your thoughts about USAID WATSAN Project ? 2. What did you accomplish with support from USAID WATSAN? 3. Can you please share the advantages and disadvantages of these activities? 4. What are some of the successes you encountered when working with USAID WATSAN Project? 5. What are some of the challenges you encountered when working with the USAID WATSAN project? 6. Please explain/describe how or how not your organization’s capacity changed during your participation in the USAID WATSAN Project? Probe: for capacity building. 7. Do you consider sustainability regarding the services you provide? If so, how? Please explain the importance. 8. How do clients transport material to the FSM, what are the challenges with transportation for the FSMs and/or for the users/clients? 9. What FSM activities are being funded by other organizations (WB, IDB etc) and what have been the successes and challenges of this collaboration? 10. Has there been a trend in the change in the number of customers? If so, please explain. 11. Has there been a trend in the change of revenue? If so, please explain. 12. Has there been a trend in the change in the number of services you provide? If so, please explain. 13. Have there been impacts on your activities due to natural disasters, COVID-19, or political unrest? If so, what and how have they affected the project? USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 136 [page 152] 14. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how? Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 15. Did the MTF take the needs of marginalized populations and how to support them into account When planning and implementing their work? Probe: examples If so, how and why? 16. Is there anything else you would like to mention or discuss about the USAID WATSAN Project and the work you are doing with the MTF? 137 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 153] CTE STAFF FGD | 2 HOURS DURATION I. Date: 2. Start time 3. End time 4... How many staff work at this CTE? 5. interviewer name: 6. Primary notetaker name: QUESTIONS 1. When did your CTE start working with the USAID WATSAN Project? 2. What was your experience working with USAID WATSAN? a. Probe for details. 3. Please describe your experience of using the Water Utility Turnaround Framework? a. Probe for successes and challenges. 4. How does the CTE manage data (mWater, SIGA, SISKLOR, Quick Books), and has this changed over the course of the project? If so, how? Please describe the impacts. 5. Howis this data influencing decision-making, planning, and business development on a day-to- day and month-to-month basis? 6. How do your staff manage collection of tariffs? Has this change over the course of the project? If so, can you describe how and the impacts? 7. How can the CTE continue to grow and increase revenue? 8. What is the approach towards minimising non-revenue water financial losses? 9. Do you feel the tariff is appropriate for the services provided? Please provide details, if clients are willing to pay and/or if complaints are received regarding costs. 10. How does the CTE recruit and manage new subscribers? Probe for subscribers’ goals and how the CTE meets those goals. 11. How does the CTE retain current subscribers? 12. Has the USAID WATSAN Project addressed the sustainability of the CTE and the water services? If so, please explain. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 138 [page 154] 13. How do you feel this CTE and ONEPA/OREPA is moving toward the future regarding managing water and increasing access in Haiti? Pause if needed 14. What has been the impact/experience of the kiosks (if any) in your system, and how has this effected the client base? 15. What attracts people to work at the CTE? 16. What are the reasons why staff leave the CTE? 17. Have you received any training? Probe: for details of the training, opinion about the utility of the training, advantages and disadvantages of trainings. 18. Does this CTE/OREPA/ONEPA have a knowledge sharing program? If so, how is knowledge sharing encouraged and managed? Probe: is there knowledge sharing between CTEs? 19. Have there been impacts on the project due to natural disasters, COVID-19, or political unrest? If so, what and how have they affected the project? 20. What are the business development priorities for your CTE? 21. What do you think should be the main area of focus for your CTE this year? 22. In what ways, if any, were gender (both women and men) considered as a part of activity implementation? Probe: Did USAID WATSAN provide you (or your organization) any training, guidance, or other resources related to gender? If yes — please describe. Probe: Were there any gender differences in how activities were implemented? If yes, please share any observations related to enabling factors, or barriers, to reaching both women and men. Probe: Do you take gender into account regarding the services you provide, if yes, how? Probe: are there any differences in the number of men and women using your services and/or the way they access your services? 23. Did your CTE take the needs of marginalized populations and how to support them into account When planning and implementing your work? Probe: examples If so, how and why? 24. Is this CTE involved in any sanitation activities, please explain, do you feel this is an area of growth? 25. Is there anything else you would like to mention or discuss about the USAID WATSAN Project and its support to your CTE? 139 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 155] KIOSK MANAGERS FGD | I! HOUR 30 MINUTES DURATION I. Date: 2. Start time 3. End time 4. Interviewer name: 5. Primary notetaker name: QUESTIONS 1. Can you please explain the kiosk services you provide to the community? 2. How many users visit the kiosk each day? 3. What are the hours of operation, and what are the peak hours? (Use the table below to provide answers.) KIOSK NUMBER OF USERS HOURS OF PEAK HOURS 4... What support do you receive from the CTE/Living Water (delete as applicable)? USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 140 [page 156] 5. Has there been a trend in the change in the number of customers? If so, please explain. 6. Are users Willing to pay for water? If not, please explain the issues clients face. 7. How do you manage the money for the kiosk and ensure money is available for repairs and bills (from CTE)? 8. Are you making a profit? If not, can you please provide details on the issues affecting this? 9. Has there been a trend in the change of revenue? If so, please explain. 10. Has there been a trend in the change in the number of services you provide? If so, please explain. (e.g., secondary distributes for household delivery) 11. What are the strengths of your business and client base now? 12. What are the weaknesses of your business and client base now? 13. What are the opportunities for your business and client base now? 14. What are the threats to your business and client base now? 15. Where do you see your business and client base in six months’ time? 16. Do you consider sustainability regarding the services you provide? If so, how? Please explain the importance. 17. Have there been impacts on the project due to natural disasters, COVID-19, or political unrest? If so, what and how have they affected the project? 18. Have you observed any changes in how women, men or children are using your services? If so, how, and why? 19. Is there anything else you would like to mention or discuss? 141 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 157] FECAL SLUDGE MANAGEMENT USER GI | I HOUR 30 MINUTES DURATION I. Date: 2. Start time: 3. End time: 4. Modality: in person/remote 5. Interviewer name: 6. Primary notetaker name: 7. Respondent(s) name(s): 8. Respondent(s) title(s): 9. Respondent organization: [Separate: Respondent contact information] 10. Anyone else present: 11. # months respondent has worked with the organization: 12. Sex of respondent: QUESTIONS 1. How long have you been moving Fecal Sludge? 2. How much Fecal Sludge do you move per week? 3. Before the FSM site opened, where did you dispose of your fecal sludge? 4. How many clients do you have? 5: Do you only use the FSM site now for disposal? if not where else do you empty your Fecal Sludge? 6. Why do you bring your FS to the FSM site? 7. What kind of clients do you serve? Probe regarding private household, business, institutions (schools, hospitals, etc.) 8. How do clients transport material to the FSM, what are the challenges with transportation for the FSMs and/or for the users/clients? 9. What FSM activities are being funded by other organizations (WB, IDB etc) and what have been the successes and challenges of this collaboration? 10. Has there been a change in your business since bring the FS to this FSM? If so, please explain? Probe regarding costs, ease of disposal, attractive to clients? 11. Do you recommend this site to other FS transporters, if so why? 12. What is your experience and opinion with the charges you levy to your customers for collecting their FS? What is your experience and opinion with the charges you pay the FSM for disposal? 13. What is your experience and opinion with the management of the FSM? 14. What is your experience and opinion with the FSM customer service? 15. What is your experience and opinion with the ease of use of the FSM site? 16. What is your experience and opinion with the wait times for disposal? 17. What is your experience and opinion with the treatment of the FS? 18. Looking back, what are the strengths of your business/bringing your FS to the FSM site? USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 142 [page 158] 19. Looking back, what are some of the weaknesses of your business/bringing your FS to the FSM site? 20. Looking forward, what are some of the opportunities you have bringing your FS to the FSM site? 21. Looking forward what are some of the threats your work faces bringing your FS to the FSM site? 22. What personal protection equipment do you use? And why? 23. ls there anything else you would like to mention or discuss about your use of the FSM site? 143 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 159] CTE STAFF SURVEY I. Date: 2. Surveyor name: 3. Respondent name: 4. Respondent job title: 5. Respondent’s CTE: 6. Commune: 7. Sex of respondent: a. Male b. Female c. Other 8. Do you have a formal written job description? a Yes b. No c. Don't know 9. Are the expectations clear in your job description? a Yes b. No c. Notsure 10. How long have you worked at the CTE? a. 0-1 year b. 1-2 years c. 2-3 years d. 3-4 years e. 4-5 years f. 6 or more years 11. Have you had a performance evaluation in the past year? a Yes b. No c. Don't know 12. Is SIGA used at this CTE? a Yes b. No c. Don't know 13. How would you rate the experience of using SIGA? a. Positive b. Neutral c. Negative d. NA 14. Is mWater used at this CTE? a Yes b. No c. Don't know USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 144 [page 160] 15. How would you rate the experience of using mWater? a. Positive b. Neutral c. Negative d. NA 16. Is SISKLOR used at this CTE? a Yes b. No c. Don't know 17. How would you rate the experience using Sisklor? a. Positive b. Neutral c. Negative d. NA 18. Is QuickBooks used at this CTE? a Yes b. No c. Don't know 19. How would you rate the experience using Quick Books? a. Positive b. Neutral c. Negative d. NA 20. How do you feel about the services your organization provides to the community? a. Positive b. Neutral c. Negative d. NA 21. How do you feel about how the CTE manages gender issues within the CTE? a. Positive b. Neutral c. Negative d. NA 22. Have you participated in a sexual harassment training in the past 2 years? a Yes b. No c. Don't know 23. How do you feel about how DAI manages gender issues in its work with the CTEs? a. Positive b. Neutral c. Negative d. NA 24. How do you feel about how the CTE manages gender issues in the community? a. Positive 145 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV [page 161] b. Neutral c. Negative d. NA 25. Would you agree that the capacity of the CTE has improved over the past two years? a Yes b. No c. Don't know 26. Do you agree that the services that the CTE provides have been improved in the past two years? a Yes b. No c. Don't know 27. Are you familiar with the operations manual? a Yes b. No c. Don't know 28. What improvements do you think the CTE should focus on next? (rate your top three) a. Training/capacity building of CTE staff (water engineering,plumbing, repairs) b. Training/capacity building of CTE staff (information management, e.g. use of mWater c. Gender equality among staff d. Promoting CTE water supply services e. Reducing downtime f. Customer communications g. _ Compliant management and response h. Better prioritization of repairs i. Developing pricing structure j. Making better use of mWater data k._ Making better use of SIGA data? 1. Making better use of SISKLOR data? m. Making better use of QuickBooks data? n. Expansion of water supply services to new areas o. Getting involved in providing sanitation services p. Other (please state) 29. Do you feel the tariff is appropriate for the water supply services provided? a Yes b. No c. Neutral 30. Are clients willing to pay for the services? a Yes b. No c. Neutral 31. Is the service valued in the community a Yes b. No USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 146 [page 162] c. Neutral 147 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV