Évaluation finale de la performance du projet USAID Eau et Assainissement

PHOTO CREDIT: DAN O’NEILL, DAI HAITI USAID/HAITI FINAL PERFOMANCE EVALUATION OF USAID WATER AND SANITATION PROJECT FINAL REPORT Haiti Evaluation and Survey Services Project (ESS) July 21, 2022 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. USAID/HAITI FINAL PERFORMANCE EVALUATION OF USAID WATER AND SANITATION (WATSAN) PROJECT FINAL REPORT Date: July 21, 2022 Submitted to USAID/Haiti Contract Number: AID-521-C-17-00002 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. Contact: Salima Mutima, Chief of Party 2300 Clarendon Blvd, Suite 1000 Arlington, VA 22201, USA DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS The evaluation team (ET) would like to thank the current and former staff at the prime implementer Development Alternatives International (DAI) and select implementing partners mWater, V3 Engineering, Ayiti Nexus, Centre et Formation et d’Encadrement, and Zanmi Lasante for their participation and provision of documentation and valuable feedback. The ET also appreciates Chase’s provision of local data collection services. The ET also thanks the staff at USAID, including Raina Zantout and Marcia Urquhart Glenn, for their support and feedback throughout the entire evaluation process. Finally, the ET would like to thank the many stakeholders who spoke with us about the WATSAN/USAID activity. The support and engagement of these organizations and stakeholders proved invaluable during the evaluation process. TABLE OF CONTENTS ABBREVIATIONS AND ACRONYMS III ABSTRACT VII EXECUTIVE SUMMARY VIII INTRODUCTION 1 BACKGROUND 1 HAITI’S WATER AND SANITATION STATUS 1 USAID WATSAN ACTIVITY GOAL AND OBJECTIVES 2 USAID WATSAN ACTIVITY INTERVENTIONS AND PHASES 3 PROJECT PERFORMANCE SUMMARY 4 EVALUATION QUESTIONS 5 EVALUATION QUESTION 1 5 EVALUATION QUESTION 2 5 EVALUATION QUESTION 3. 5 METHODOLOGY 5 DATA COLLECTION 6 DATA ANALYSIS 9 FINDINGS AND CONCLUSIONS 12 EVALUATION QUESTION 1 (WATER SUPPLY). 12 EVALUATION QUESTION 1 (SANITATION). 24 EVALUATION QUESTION 2 29 EVALUATION QUESTION 3 35 CROSS-CUTTING ISSUES 38 RECOMMENDATIONS 42 USAID/HAITI 42 USAID IMPLEMENTERS 42 ANNEX A. SCOPE OF WORK 44 ANNEX B. DOCUMENTS REVIEWED 60 ANNEX C. KIIS/GIS/FGD EVENT LIST 63 ANNEX D. MWATER SECONDARY DATA REVIEW 67 i | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV ANNEX E. CTE STAFF SURVEY RESULTS 82 ANNEX F. 2021 CTE CUSTOMER SURVEY SECONDARY DATA REVIEW 90 ANNEX G. LIST OF STAKEHOLDERS CONSULTED 97 ANNEX H. WATSAN PROJECT METHODOLOGY 101 ANNEX I. DATA COLLECTION INSTRUMENTS 103 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | ii TABLES AND FIGURES TABLE 1: SUMMARY OF PERFORMANCE INDICATORS FROM THE PROJECT RESULTS FRAMEWORK 4 TABLE 2: KIIS, GIS, AND FGDS 7 TABLE 3: CTE STAFF SURVEY DATA COLLECTION 8 TABLE 4: MWATER INDICATORS AND METRIC CATEGORIES 12 FIGURE 1: THE “SPIRAL OF DECLINE” LEADING TO POOR PERFORMANCE (AFTER GALAITSI 2016) 13 FIGURE 2. THE DATA VALUE CHAIN MODEL FOR TURNING DATA INTO USEFUL INFORMATION 14 FIGURE 3: CTE STAFF EXPERIENCE OF USING MWATER 15 FIGURE 4: CTE EXPENSES DATA FROM MWATER (6 MONTH MOVING AVERAGE) 16 FIGURE 5: CTE REVENUE DATA FROM MWATER (6 MONTH MOVING AVERAGE) 16 FIGURE 6: CTE OPERATING RATIO DATA FROM MWATER (6 MONTH MOVING AVERAGE) 17 FIGURE 7: CTE STAFF EXPERIENCES OF USING QUICKBOOKS ACCOUNTING SOFTWARE 18 FIGURE 8: SAEP WATER SUPPLY CONTINUITY (6-MONTH MOVING AVERAGE) 19 FIGURE 9: CTE RESIDUAL CHLORINE CONFORMITY 19 FIGURE 10: FUNCTIONALITY OF CTE MANAGED KIOSKS (6 MONTH MOVING AVERAGE) 20 FIGURE 11: HOUSEHOLD WATER ACCESS FROM HOUSEHOLD CONNECTIONS AND KIOSKS: COVERAGE BY CTE 20 FIGURE 12. CTE ACTIVE SUBSCRIBERS (6 MONTH MOVING AVERAGE) 21 FIGURE 13: CTE COLLECTION EFFICIENCIES (6 MONTH MOVING AVERAGE) 21 iii | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV FIGURE 14: CTE STAFF EXPERIENCES OF USING SIGA 22 TABLE 5: EAF GRANTS AND CURRENT STATUS 30 FIGURE 15: PROJECT ISSUES TIMELINE 38 FIGURE 16: SEXUAL HARRASSMENT TRAINING 39 FIGURE 17: CTE’S MANAGEMENT OF CTE’S GENDER ISSUES BY SEX 39 FIGURE 18: CTE’S MANAGEMENT OF COMMUNITY GENDER ISSUES 40 TABLE 6: DATA OUTLIERS 67 TABLE 7: AVAILABLE DATA 68 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | iv ABBREVIATIONS AND ACRONYMS BSFL Black Solder Fly Larvae CFET Centre de Formation et d’Encadrement Technique COP Chief of Party CTE Centre Technique d’Exploitation/Technical Centre of Exploitation DAI Development Alternatives Incorporated DCOP Deputy Chief of Party DINEPA National Directorate of Potable Water and Sanitation EAF Enterprise Acceleration Fund EQ Evaluation Question ESS Evaluation and Survey Services ET Evaluation Team ETS Ecole de Technologie Superieure FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion FSM Fecal Sludge Management GI Group Interview GOH Government of Haiti HH Household HTG Haitian Gourde IDB Inter-American Development Bank IP Implementing Partner IQR Interquartile Range IRB Internal Review Board KII Key Informant Interview LEA Latrine Emptier Association LWI’S Living Water International LOP Life of Project LSPV Projet de Lits de Séchage Plantés de Végétaux MEL Monitoring, Evaluation, and Learning MFSN Mouvman Fanm Soléy Nó MSME Micro, Small, or Medium-Sized Enterprises MTF Mayoral Task Force NGO Non-Governmental Organization OFATMA Office d’Assurance Accidents du Travail, Maladie et Maternité ONA Office National d’Assurance-Vieillesse v | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV ONEPA Observatoire National de l’Eau Potable et de l’Assainissement OREPA Office Régional de l’Eau Potable et d’Assainissement SAEP Système d’alimentation d’eau potable / drinking water supply system SDG Sustainable Development Goal SI Social Impact SIEPA Système intégré d’Information sur l’Eau Potable et l’Assainissement SIGA Système Intégré de Gestion des Abonnés / CTE customer database SISKLOR Système de Surveillance du Chlore Residuel / Residual Chlorine Surveillance System SME Small and Medium Sized Enterprises TA Technical Assistance TL Team Leader USAID United States Agency for International Development UEH State University of Haiti UniQ University Quisqueya USD US Dollars WASH Water, Sanitation, and Hygiene WATSAN Water and Sanitation WBUTF World Bank Utility Turnaround Framework USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | vi ABSTRACT This evaluation used a mixed-methods methodology to examine the design, approaches, performance and outcomes of the United States Agency for International Development (USAID) Water and Sanitation (WATSAN) Project between December 2017 and December 2021. Project objectives included supporting access to sustainable water supply (target 250,000 people) and sanitation services (target 75,000 people) and strengthening the foundational capacity of water and sanitation service providers for sustainable service delivery. Project targets were mostly met (see Table 1 in the background section). Key findings for the water sector included that the Centres Technique d’Exploitation (CTEs) are better able to manage their businesses using a data-driven approach via the Système intégré d’Information sur l’Eau Potable et l’Assainissement and the mWater platform. Billing and customer service are improved, though some challenges remain with continuity of CTE data. USAID’s approach and these achievements with the CTEs have provided a foundation for sustainable service delivery. Key findings for the sanitation sector included two fecal sludge management sites now operating safely with business plans in place in Les Cayes and Port-au-Prince. Moreover, the Project’s support to the Latrine Emptying Associations and small and medium sized enterprises has empowered them to deliver improved sanitation services. Coordination support to the Mayoral Task Forces on Sanitation has increased access to sanitation services. Recommendations are made on how to capitalize on the Project’s successes, including continued capacity building, improved data management and integration of data platforms, setting service delivery targets, and establishing regulatory frameworks with enforcement for the sanitation sector. vii | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMARY OVERVIEW AND PURPOSE United States Agency for International Development (USAID) Haiti requested that Social Impact’s (SI) Evaluation and Survey Services (ESS) activity design and conduct an independent final performance evaluation of the USAID Water and Sanitation (WATSAN) Project (the Project) implemented by (Development Alternatives Incorporated (DAI) Global, LLC, with implementing partners (IPs) V3 Engineering, Ayiti Nexus, mWater, Centre et Formation et d’Encadrement, and Zanmi Lasante across eight departments in Haiti. The purpose of this evaluation was to determine the effectiveness of the Project in achieving its objectives in order to inform future USAID/Haiti programming similar in scope or approach. The primary audience for the evaluation is USAID, with secondary audiences being Project implementing partners, the Haitian National Directorate of Potable Water and Sanitation (DINEPA), the Government of Haiti (GOH), Haitian private sector entities and donors. This report presents the evaluation methodology, findings, and conclusions for each evaluation question, followed by recommendations. EVALUATION SCOPE AND QUESTIONS The evaluation focused on the ten communes targeted over the life of the Project. For the water sector, the target communities for the evaluation were the customers and staff of the water utility management structures (the Centre Technique d’Exploitation or CTEs) and the governmental and non-governmental organizations (NGOs) supporting their service delivery at either the household or water kiosk level. Additionally, two Enterprise Acceleration Fund (EAF) grants supported water projects (water quality testing and water kiosks). For the sanitation sector, the target communities were the staff and users of the two fecal sludge management (FSM) sites at Morne a Cabri and Fonfred, Latrine Emptying Associations (LEAs), the Mayoral Task Force (MTF) on Sanitation, and small and medium sized enterprises (SMEs) involved in household latrine construction. Additionally, five EAF grants supported sanitation projects. The target populations were 250,000 people for water supply and 75,000 for sanitation. This evaluation addressed the following evaluation questions (EQs): 1. To what extent has WATSAN met its Task Order objectives to build governance capacity at multiple levels (national, regional, local) to improve sustainable water supply and sanitation service delivery? 2. How was the Enterprise Acceleration Fund utilized, and to what extent did those grants support WASH enterprises (such as micro, small, and medium sized enterprises or MSMEs, NGOs, and Bayakous) to move toward sustainable service delivery? 3. How effective was WATSAN’s approach of targeting the end of the sanitation value chain in improving the overall sanitation value chain?1 1 This may also be referred to as the sanitation service chain. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | viii METHODOLOGY The evaluation used a mixed-methods approach consisting of a document review, 55 key informant interviews (KII) and group interviews (GI), six focus group discussions (FGDs), 74 CTE staff surveys and secondary data review of mWater CTE indicator data, DINEPA FSM sites operational data, EAF project performance data, and a CTE customer satisfaction survey conducted in 2021. The Evaluation participants included CTE staff and managers, CTE liaison officers, CTE kiosk managers, members of the LEAs and MTFs, Observatoire National de l’Eau Potable et de l’Assainissement (ONEPA) and Office Régional de l’Eau Potable et d’Assainissement (OREPA) staff, staff of SMEs, EAF grantees, FSM managers and users, staff of implementing partners, and USAID staff. FINDINGS AND CONCLUSIONS EQ1: To what extent has the Project met its Task Order to build governance capacity at multiple levels (national, regional, local) to improve sustainable water supply and sanitation service delivery? FINDINGS (WATER) Organization and Strategy Multiple respondents from the CTEs and IPs reported that before the Project, all CTEs were insolvent and unable to plan and monitor progress, revenues, and expenditures. The 11 mWater key performance indicators (see Table 4 on page 12) used by the Project CTEs have since been adopted by all 29 CTEs, including Pignon drinking water supply system (Système d’Alimentation d’Eau Potable or SAEP). The indicators have underpinned CTE business development across the five performance metrics of Organization and Strategy, Technical Operations, Commercial Operations, Financial Operations and Human Resource (HR) Management, as outlined below. Technical Operations Water supply service continuity across the Project CTEs is at between five and 20 hours/week, with Pignon at around 80 hours/week. An exception is Ouanaminthe, whose continuity in October 2020 leapt from five to 168 hours/week (24/7 supply), where it has remained since, as their management of the system improved. However, there are no established/defined water supply delivery standards for the CTEs, reference to the Sustainable Development Goals (SDGs) or identified milestones to reach them. In Ouanaminthe, use of the mWater mobile app which the Project supported means monthly meter readings were done in ten days instead of taking over a month. There remains variation in reporting between mWater indicators and CTEs in terms of regularity e.g., conformity to residual chlorine levels monitored through residual chlorine surveillance system (Système de Surveillance du Chlore Residuel or SISKLOR) was reported very irregularly in some CTEs. Repairs to CTE-managed water supply infrastructure has been limited by a lack of comprehensive and accessible technical details sufficient to locate the affected water supply infrastructure, but mWater has recently launched a mapping program to address this issue. At the kiosk level, mWater data shows that CTE-managed kiosks have erratic and variable levels of functionality. There are challenges with the interconnectedness of the various software packages the CTEs use. ix | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV Commercial Operations CTE revenues have increased, the largest being Mirebalais where revenue increased from 148,000 Haitian Gourdes (HTG) in July 2019 to 805,000 HTG in January 2022 while revenues doubled for Hinche, Cap Haitien, and Belladere over the same period. CTE FGDs discussed how revenues have increased2 as tariffs have been raised and there is now better billing, revenue collection, and customer service using the Système Intégré de Gestion des Abonnés (SIGA, the CTE customer database) and mWater. Key factors for CTE business sustainability and customer retention included the predictability of water supply and service continuity, together with enhanced billing and customer complaints handling. Financial Operations In 45 percent of cases, the operating ratio (expenses/revenue) was higher than 100 percent for the Project’s CTEs, indicating that expenses were higher than the revenue for close to half of the reported months. A number of interviews with both the donor and the prime contractor indicated that although revenues have increased, “fear of success” was very real. CTEs are not audited, and they have concerns around how they will manage their increased revenues and how to effectively manage expenditure of those funds for future capital infrastructure projects/improvements. Some CTEs are relatively new to using QuickBooks accounting software and are not yet confident in using it. There is no network connection from QuickBooks to the SIEPA (Système Intégré d’Information sur l’Eau Potable et l’Assainissement) system. HR Management The evaluation included a survey of 74 CTE staff (Annex E) with more than 50 percent of respondents having worked at the CTE for four to five years. The survey results, when triangulated with CTE staff interviews and FGDs with a number of CTE stakeholders, found that although CTE staff capacity to use mWater and SIGA software improved over the Project period, enabling better service and billing, there was variation in that capacity i.e. not everyone can use the data for planning purposes. See Annex E and main report for full details. CONCLUSIONS (WATER) The Project has brought data-driven planning and decision-making to each CTE for the first time. The Project enabled the CTEs to grow their customer bases and revenues and increase their business capacity. The project has significantly contributed to the sustainability of the CTEs and reversed the spiral of business decline seen before the Project started. There remains room for improvement in the areas of technical data management for operation and maintenance, consistent and timely reporting of mWater data, SISKLOR data reporting, effective management of future capital expenditure on infrastructure and for some CTEs their capacity to use QuickBooks confidently. FINDINGS (SANITATION) The Project worked with a number of stakeholders in the sanitation sector, including the FSM sites, SMEs/pit emptiers, LEAs, and MTFs. The Project was able to help the SMEs increase capacity, especially 2 All figures relating to CTE performance in this report have come from the secondary data review of mWater data. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | x regarding management of the business (marketing, human resources, contracts, and budgets) through training and encouraged the construction of household toilets and latrines. The Project improved the infrastructure and management capacity of the Fonfred FSM and the Morne a Cabri FSM sites and supported better management practices. Both sites are currently accepting sludge from the surrounding communities, which has reduced complaints about the sites, but there is concern regarding long-term sustainability, especially regarding issues with transportation and not receiving enough sludge to be profitable. CONCLUSIONS (SANITATION) The pit-emptiers were empowered by the work of the Project. This led to them marketing their services, developing official associations, and working together. By working with multiple sanitation stakeholders, the Project was able to assist in the construction of latrines and toilets at the household level, increase the role of the MTFs in their awareness-raising activities regarding the importance and impact of having a latrine, and increase the management and business capacities of the SMEs, LEAs, and FSMs. Through this increased capacity, they are now better able to provide more sanitation services to the communities. EQ 2: How was the enterprise acceleration fund utilized, and to what extent did those grants support WASH enterprises (such as MSME, NGOs, Bayakous) to move toward sustainable service delivery? FINDINGS (EAF) DAI administered seven EAF grants during the Project, four in the sanitation sector and three in the water sector. The grants included micro credit for toilet construction, construction of 14 kiosks, research activities by University Quisqueya (UniQ) on sludge drying and testing, purchase of equipment to improve incineration services, pilot trials of black soldier fly larvae waste to value end products by SOIL and establishing a water testing laboratory at Limonade University. The kiosks provided access to water to 24,000 people, while the micro grants for latrines and toilets provided access to credit for 100 people. About half of the grantees said they did not feel involved in the Project decision-making and use of funds, and many grantees had issues with implementation timelines due to delays in the arrival and quality of equipment. CONCLUSIONS (EAF) The grants were successful in terms of increasing the number of clients for the organizations, both in terms of access to markets and the range of services offered. In addition, the lab at Limonade University can now provide water quality testing in addition to soil testing. The research activities for SOIL and UniQ proved to be a success as activities have been able to finally start. Two of the grants have ongoing issues due to delayed equipment, and one grant has an issue with equipment not functioning. The grantees shared how they were able to expand their services both in terms of the number of clients and types of services provided (but due to delays, this has not happened for all grantees). Due to delays and some challenges, it is difficult to determine if grants improved the sustainability of the organizations. xi | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV EQ 3: How effective was the Project’s approach of targeting the end of the sanitation value chain in improving the overall sanitation value chain? FINDINGS (SANITATION VALUE CHAIN) Before the Project, the Fonfred FSM site in Les Cayes was not operational. The Morne a Cabri site for the Port-au-Prince area was open and accepting waste, but the treatment basins were full and the site was not properly managed. The Project has enabled both sites to now provide a safe and controlled disposal area for sludge waste and illegal sludge dumping has been reduced and the end of the sanitation value chain is now stronger. The FSM sites are now operating with a business model but they are not yet profitable and a key bottleneck is insufficient sludge transport capacity. CONCLUSIONS (SANITATION VALUE CHAIN) The Project focused on the end of the sanitation value chain with the work at the FSM sites; this has ensured the communities of Les Cayes and Port-au-Prince now have a safe fecal sludge disposal facility It is important to note that while the infrastructure and management capacity of the FSM sites improved due to the Project’s activities, sustainability is still an issue as the sites have issues with transportation and do not receive enough customers/sludge each month to be profitable. There are some ongoing concerns that the sites will close without further support from the Project. PRINCIPAL RECOMMENDATIONS FOR USAID/HAITI: 1. Ensure future programming supports CTEs to establish milestones for reaching defined service delivery standards that are linked to the SDGs and tracked in mWater. 2. Engage DINEPA/GOH to enable SIEPA to have better data connectivity between the various software packages for the CTEs. 3. Continue to support FSM organizations in their efforts to reach profitability/sustainability by expanding infrastructure in order to help them increase the amount of sludge they can process. In particular, support greater sludge transportation capacity from FSM users to the FSM sites. 4. Continue to support coordination among sanitation sector stakeholders to build GOH capacity in the development and enforcement of FSM regulations. PRINCIPAL RECOMMENDATIONS FOR FUTURE USAID HAITI IMPLEMENTERS: 1. Undertake a needs assessment with CTEs on how they can grow their financial and asset management capacity to support accountable capital expenditure on infrastructure. 2. Address data continuity/quality issues for data reported on mWater. 3. Support mWater’s capacity to map all water supply infrastructure to improve CTE technical operations. 4. Develop a standardized approach for all CTE-managed kiosks that includes an indicator(s) for kiosks to supplement the current single (technical) indicator of percentage functional kiosks e.g., the hours per day of operation metric used by CTEs for household connections. 5. Use EAF-type grants to support female-lead/owned organizations and projects. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | xii INTRODUCTION The USAID WATSAN Project is a $41.8 million, five-year activity (in total) implemented in collaboration with Haiti’s DINEPA with the overall goal of improving sanitation and water for all Haitians. There was a $4.9 million and six-month extension to the Project (which will now end in December 2022) to primarily focus on helping rebuild the water systems in the South following the August 14, 2021, earthquake.3 The Project reached its goals by focusing on its core values of self-reliance, resilience, and the private sector approach. In total, the Project focused on 12 areas: eight areas hard hit by Haiti’s cholera outbreak (Cap-Haitien, Ouanaminthe, Pignon, Hinche, Mirebalais, Belladere, Lascahobas, and Croix des Bouquets), two areas hard hit by Hurricane Matthew in 2016 (Les Cayes and Jeremie), and now two additional areas hit by the August 14, 2021, earthquake (Aquin and Miragoane). The first five areas that were part of the Project were Cap-Haitien, Mirebalais, Les Cayes, Jeremie, and Croix des Bouquets. This evaluation assesses the Project’s implementation in the first ten communes (Aquin and Miragoane are not part of the evaluation). BACKGROUND HAITI’S WATER AND SANITATION STATUS Access to water and sanitation in Haiti is the worst of any country in the Western Hemisphere. Only 66.7 percent4 of the population has access to basic water sources, a slight increase since 1990, when the rate was 62 percent.5 Increases in services have not been able to keep pace with population growth. However, actual access rates may be lower, as many water systems’ functionality is poor. Only 37.1 percent6 of the Haitian population has access to basic sanitation, a rate that has been relatively constant since 1990. Safe collection, transport, and treatment of human excreta is practically non-existent throughout Haiti. Surveys indicate that the use of improved sanitation facilities has increased from 18 to 28 percent, meaning that much of the population still relies on shared or unimproved sanitation facilities.7 While basic sanitation access rates in urban areas are higher than the national average (37.1 percent), they are still below 50 percent.8 In informal or unplanned settlements, where the poorest and most vulnerable urban populations generally live, urbanization and high localized population density may be accompanied by an increased risk of infectious disease transmission, primarily affecting the poor. The combination of a high fecal-related disease burden and inadequate infrastructure suggests that investment in expanding sanitation access in densely populated urban slums can yield important public health gains. 3 WATSAN Quarterly Report Oct-Dec 2021 4 UNICEF/WHO Joint Monitoring Programme (2020) 5 Ibid. 6 Ibid. 7 Ibid. 8 Ibid. 1 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV It is also important to understand the impact of gender regarding access and use of water and sanitation facilities. For example, women and girls have disproportionate labor and time burdens for household water collection, management, and treatment; also, those who travel long distances to collect water are particularly vulnerable to gender-based violence. This is discussed in detail in the USAID/Haiti Strategic Framework Gender Analysis (November 2020).9 In terms of water management and governance, which the USAID WATSAN activity focuses on, women are often not in decision-making roles and are not visible in both public and private sectors.10 In addition, climate change and other natural and human-made disasters as well as unplanned urbanization threaten water resources and gains made in the water supply and sanitation sector. Building the capacity of utilities and private operators to anticipate needs, plan for and finance improvements, and expand access to underserved communities on a sustainable basis is the strategy promoted by the USAID Water and Development Plan. USAID WATSAN ACTIVITY GOAL AND OBJECTIVES The objective of the USAID WATSAN Project is to build a foundation for long-term, sustainable growth in access to safe drinking water and sanitation in Haiti, where many communities suffer from high incidences of diarrheal disease. The Project’s three primary goals are to: • Help 250,000 people get access to basic or improved water; • Help 75,000 people get access to basic or improved sanitation; and • Lay the foundation for sustainable increases in access to water and sanitation across Haiti. The Project was implemented in collaboration with DINEPA, the branch of the Haitian government tasked with ensuring water and sanitation services for its citizens. At the time of this evaluation, the USAID WATSAN Project covered implementation in 10 communes, located in six departments: Nord, Nord-Est, Centre, Sud, Grand’Anse, and Ouest. USAID WATSAN initially focused on five areas: three areas hard hit by the cholera outbreak11 (Cap Haitien, Mirebalais, and Croix des Bouquets) and two areas hard hit by Hurricane Matthew in 2016 (Les Cayes and Jeremie). During the last two years, it added five additional areas: Ouanaminthe and Hinche in 2020, and Pignon, Lascahobas, and Belladere in 2021. USAID understands that the success of its USAID WATSAN Project rests on these prioritized values. • Self-Reliance: The institutions must be able to generate enough revenue to fully cover their operations without any subsidies. • Resilience: Given the volatile climate in Haiti, the institutions need to design, build, operate, maintain, and upgrade systems that can withstand disruptions and economic uncertainty. 9 USAID Haiti Strategic Framework Gender Analysis (November 2020) pg. 97 10 USAID Haiti Strategic Framework Gender Analysis (November 2020) pg. 98-100 11 The cholera outbreak started in October 2010. USAID provided support that helped in eliminating cholera from Haiti in 2019. USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 2 • Private Sector Approach: The CTEs and fecal sludge managers must adopt a private sector, customer-driven approach. The theory of change is shown in Annex H. USAID WATSAN ACTIVITY INTERVENTIONS AND PHASES The Project’s activities are organized around three closely linked components, with specific tasks for each, shown in Annex H. Component 1: Increasing access to sustainable water services • Task 1.1: Water infrastructure engineering services • Task 1.2: Water infrastructure construction • Task 1.3: Technical assistance for water service providers Component 2: Increased access to sustainable sanitation services • Task 2.1: Support for sanitation enterprises • Task 2.2: Waste treatment and fecal sludge management engineering services • Task 2.3: Wastewater treatment and fecal sludge management construction Component 3: Improving the enabling environment for sustainable implementation, operations, and maintenance of water and sanitation services • Task 3.1: Technical assistance to national and sub-national government structures • Task 3.2: Knowledge dissemination and learning Initially, Component 1 activities focused on Quick Impact Projects and the tailored technical assistance (TA) packaged for the CTEs, but that expanded throughout the Project to include software packages for the CTEs, operation manuals, and support for performance management of directors. Training support to the CTEs was further developed based around the five metrics of organization and strategy, human resources management, financial management, technical operations, and commercial operations (Annex H). To help with implementation in the last quarter of Year Two, the Project moved some of its senior staff into the CTEs as well. In Year Three, the Deputy Chief of Party (DCOP), worked directly with the Cap Haitien CTE. The Senior Advisor and the Utility Specialist worked with the Les Cayes and Jeremie CTEs. The Senior Engineer worked with Mirebalais. By having these senior staff working directly with the CTEs on a daily basis, the Project was able to help the CTEs to implement the required management changes. USAID WATSAN worked with the CTEs to increase the amount of data collected, reported, and used in mWater. DINEPA tracked 11 key indicators that were self-reported by the CTEs as part of the monthly reports. Additional financial, client, and technical information is collected using SIGA, SISKLOR, and Cadastre, and also stored in the mWater database for each CTE, including a limited amount of infrastructure information. 3 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV Throughout the Project, there was a focus on the institutional strengthening program for the Regional Offices for Potable Water and Sanitation (OREPAs) based on the needs identified in the water and sanitation service roadmaps. Support to DINEPA focused on monitoring, especially collaboration with the ONEPA and improving their collection, monitoring, analysis, and archiving portal. Learning activities included developing a learning agenda with sector stakeholders, convening at least one learning event, and completing preliminary design of an online learning platform. Activities under the Project’s Enterprise Acceleration Fund were also essential to the Project’s activities. Initial grants and other mechanisms helped build the capacity of private sector actors working in the water and sanitation sector while also meeting their enterprise development goals and contributing to the Project’s results and impacts. PROJECT PERFORMANCE SUMMARY The key objectives of the Project were to build the governance and financial management capacity of targeted Haitian public utilities and private operators, enable 250,000 people to gain access to new or improved water services, and enable 75,000 people to gain access to basic or safely managed sanitation services. The USAID WATSAN Project supported sector institutions by implementing a combination of targeted infrastructure improvements and technical assistance prescribed by the World Bank Utility Turnaround Framework. The USAID WATSAN Project also supported sector MSMEs and supported water and sanitation innovation projects through its Enterprise Acceleration Fund. The table below summarizes Project achievements against Life of Project (LOP) targets. TABLE 1: SUMMARY OF PERFORMANCE INDICATORS FROM THE PROJECT RESULTS FRAMEWORK OUTCOME INDICATORS LOP TARGET COMMENTS12 safely managed drinking water service250,000 Exceeded by Number of people receiving improved service quality from an existing basic or 24,425 cost recovery9 2 Remaining Number of service providers demonstrating at least a 10 percent increase in “limited” or “basic” service75,000 Exceeded by 1,298 Number of people receiving improved sanitation service quality from an existing services15 0 Number of SMEs demonstrating increased sales of sanitation products and Percentage of staff in target sector institutions self-reporting increased ability to perform effectively in assigned job 50 percent Exceeded by 50 percent service delivery being implemented at national, regional and/or commune level9 0 Percentage of agreed strategies/plans for improved water and/or sanitation Table 1 summarizes the performance outcome indicators from the Project results framework with information from March 2022. Please note that the data was not disaggregated by income group, age, disability, or sex. 12 From the Q1FY22 and Q2FY22 quarterly reports USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 4 EVALUATION QUESTIONS EVALUATION QUESTION 1. To what extent has USAID WATSAN met its Task Order objectives to build governance capacity at multiple levels (national, regional, local) to improve sustainable water supply and sanitation service delivery? To answer this question, the Evaluation Team (ET) considered a number of sub-questions that looked at the CTE water supply, the kiosk water supply, and the sanitation services supported by the USAID WATSAN activity. These included: What is the impact and experience of the data management tools (mWater, SIGA, QuickBooks and Sisklor)? Has CTE customer satisfaction improved? Has the TA improved the service delivery and customer management capacity of the CTEs and led to improvement across the five metrics? For the sanitation aspect, the ET considered if the capacity of the Fonfred and Morne a Cabri FSM management teams has improved. The impact of the business support provided to the SMEs and Latrine Emptier Associations was also examined. EVALUATION QUESTION 2. How was the Enterprise Acceleration Fund utilized, and to what extent did those grants support WASH enterprises (such as MSMEs, NGOs, and Bayakous) to move toward sustainable service delivery? To answer this question, the ET considered a number of sub-questions. Some of these looked at the services and the results from the EAF, such as what success the grants achieved and if there were any recommended modifications. The ET investigated how sustainability improved, if the business or organization expanded (in terms of services and/or customers), and the overall successes and challenges of the EAF grants. EVALUATION QUESTION 3. How effective was WATSAN’s approach of targeting the end of the sanitation value chain in improving the overall sanitation value chain? The ET understood that DINEPA and the FSM management staff will continue the work started under USAID WATSAN. When looking at the overall sanitation value chain, the ET focused on the services the FSM provided to operators and the communities. Some of the sub questions used to investigate the evaluation question included: Is the FSM operating as a business? Are clients using wastewater treatment sites? Are they satisfied? How was the sanitation value chain affected by the Project activities? How does the FSM serve the community and businesses? Has FSM improved and has there been improvement regarding sustainability? And has the opening of the FSM improved disposal of waste and improved businesses for the clients? The ET also investigated the challenges and success of the sanitation activities. METHODOLOGY As part of the evaluation, the ET reviewed USAID WATSAN annual work plans, annual reports, monitoring, evaluation, and learning (MEL) plans, and Activity descriptions and modifications. Following the kick-off meeting on December 15, 2021, the ET continued its review of USAID WATSAN Project documents, contacted DAI for additional documents and stakeholder contact information, and prepared the Inception and Evaluation Design Report. The in-briefing presentation was held with USAID on January 6, 2022, after which the ET made some adjustments to the methodology. The inception report was submitted to USAID on January 31, 2022. The ET had an introductory meeting with mWater and members of the DAI team to gain a better understanding of the Project and the mWater platform. 5 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV DATA COLLECTION OVERVIEW The ET used a mixed-methods evaluation design. The ET collected data on the perspectives and experiences of key stakeholders involved in USAID WATSAN and/or in the sanitation value chain using both purposeful and snowball sampling techniques. The table in Annex C shows all the data collection events undertaken by the ET. Data collection started on February 21, 2022, and ended on April 15, 2022. The data collection tools included guiding questionnaires and associated protocols for KIIs (with one person), for GIs (with two to five people), for FGDs (for six people or more), and a dedicated survey tool for CTE staff. The tools were translated into French and Creole. The KII and GI respondents and FGD participants were selected strategically from a list of key USAID, GOH, DAI, CTE, and partner staff. Respondents likely to have the most information were selected first, balanced by diversity, and supplemented with snowball sampling to fill in gaps in data sources and information Primary data collection included 55 key informant interviews (KIIs) or group interviews (GIs) and six focus group discussions (FGDs) with stakeholders in all 10 communes to address EQs 1, 2, and 3, and a survey of 74 CTE staff in seven communes to address EQ1. The ET also conducted an extensive secondary data review of mWater CTE indicator data, DINEPA FSM sites operational data, EAF Project performance data, and a CTE customer satisfaction survey conducted in 2021. The scope of this data review was substantial for EQ1 (water) and looked at all mWater data collected from December 2017 to December 2021 across the Project communes for the 11 mWater indicators as well as the data from a CTE customer satisfaction survey conducted in 2021. The scope of secondary data review for EQ1 (sanitation), EQ2, and EQ3 was limited to DINEPA FSM sites data and (limited) performance data for the EAF Projects. There was substantially less data available for sanitation than for water supply. The evaluation faced a number of challenges related to the country context and the remote nature of data collection (see Methodological Limitations and Mitigation Strategy section below). DATA COLLECTION METHODS DOCUMENT REVIEW The ET conducted an initial review of more than 50 USAID WATSAN documents, as well as other, third-party resource material, to understand the Activity design and implementation, extract findings relevant to the EQs, and inform the development of data collection questions and tools that appropriately supplemented, or could be cross-checked with, information in the background documents. Documents are listed and fully described in Annex D. SECONDARY DATA COLLECTION AND REVIEW The ET conducted secondary data analysis of the CTE indicators extracted for each CTE from mWater, such as revenue information, subscriber information, kiosk information, and number of staff by gender and contract type. The ET was also able to disaggregate the subscriber information and their satisfaction rating of their CTE’s services by gender from DAI’s CTE Client Satisfaction Survey conducted in March April 2021. There is a delay in the approval process by OREPA, which is needed for the CTE indicator data to be publicly available. The ET followed up with the SAEPs to ensure information was available until December 2021 at a minimum on the mWater platform. The ET had access to data from reports across USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 6 the 10 WATSAN SAEPs ranging between December 2018 and March 2022. The summary of available mWater data is listed in Annex D. The Project completed a CTE customer satisfaction survey as part of a SIGA database update in March – April 2021. The survey was conducted in six communes (Cap Haitien, Mirebalais, Croix des Bouquets, Les Cayes, Jeremie, and Hinche). Using the SIGA client database, the survey attempted to reach all of its active and passive client base and managed to collect responses from a total of 10,525 clients. In addition to updating the client contact information (phone numbers, address, and GPS coordinates of the connection), the survey asked questions on client’s status (active vs. passive) and satisfaction rating of service quality. The ET used the data DAI provided to understand the client base of each CTE and their satisfaction rating. FOCUS GROUP DISCUSSIONS FGDs were undertaken to collect balanced opinions from small groups of key stakeholders, in particular those groups of individuals personally and directly involved in water or sanitation service delivery. These groups included CTE staff, kiosk managers, LEAs, and MTFs. Due to availability of interviewees and connection issues, only six FGDs were conducted of the total 24 planned (all remaining events planned as FGDs became instead KIIs or GIs.) The FGDs were conducted with the LEAs and the managers for the kiosks in Cap Haitien (Living Water kiosks). Thirty-seven people participated in the FGDs, but only six of them were female. Information gathered from FGDs respond to all three evaluation questions. KEY INFORMANT AND GROUP INTERVIEWS The KIIs and GIs allowed the ET to gather information for all three evaluation questions. The ET conducted 55 KIIs and GIs with representatives of key stakeholder groups, including USAID, ONEPA, DAI staff (CTE management staff, EAFs, FSM management teams), FSM users (this includes companies or individual contractors that use the FSM sites but not SMEs), and other working partners. This was more than was originally planned due to challenges faced with schedule and connectivity. The number of KIIs and GIs increased while the number of planned FGDs decreased. Of the participants in the KIIs and GIs, 40 were female and 69 were male, for a total of 109 participants. Table 2 shows the number of each primary data collection event and the type of event that was conducted. A complete breakdown of the events including by gender is in Annex C. TABLE 2: KIIS, GIS, AND FGDS KIIS, GIS (2-5 PAX), FGDS (6 OR MORE PAX) EVENTS TYPE CTE staff 7 GI FSM staff and users 4 GI EAF grants recipients 7 KII, GI Kiosk managers 9 FGD, GI ONEPA/OREPA staff 6 KII DAI staff (COP + Project staff, CTE liaison staff) 11 KII, GI 7 | USAID WATSAN FINAL PERFORMANCE EVALUATION USAID.GOV KIIS, GIS (2-5 PAX), FGDS (6 OR MORE PAX) EVENTS TYPE SMEs 4 FGD Pit Emptier Associations and Mayoral Task Forces 4 GI, FGD Working partners 5 KII Others 1 KII USAID 3 KII Total KIIs/GIs 55 40F/69M Total FGD 6 6F/31M CTE staff surveys for 7 communes 74 13F/60M CTE STAFF SURVEY The ET conducted a survey of lower-level staff at seven CTEs (Cap Haitien, Mirebalais, Croix des Bouquets, Les Cayes, Jeremie, Ouanaminthe, and Hinche). These surveys were conducted by a local data collection partner in Haitian Creole. The targeted sample size for the survey was 70, with 10 respondents per CTE. When conducting the survey, the team was short three people each in Ouanaminthe and Croix des Bouquets. The local data firm then used staff from other cities to reach the target. The survey was carried out via telephone, and an Excel database was shared with the ET. The sampling for this survey was random, with the list of staff and those they were assigned to randomly generated; if someone was not available or had issues with phone connection, the surveyor proceed to the next person on the list. The plan was to have 30 percent of the surveys completed by women, but due to more male staff and connection issues, this was not possible (11.2 percent of survey participants were female)) TABLE 3: CTE STAFF SURVEY DATA COLLECTION DEPARTMENT CTE STAFF WOMEN SURVEYEDMEN SURVEYED North Cap Haitien 22 4 9 North East Ouanaminthe 10 0 7 Center Hinche 16 1 11 West Mirebalais 14 2 8 Croix des Bouquets 8 1 6 South Les Cayes 25 3 12 Grande Anse Jeremie 21 2 8 Total 116 13 61 USAID.GOV USAID WATSAN FINAL PERFORMANCE EVALUATION | 8 DATA ANALYSIS The evaluation Team Lead (TL) oversaw and managed systematic analysis of qualitative and quantitative data. The evaluation included several data collection methods (document review, KIIs, GIs, FGDs, surveys, and secondary data collection) that enabled the ET to conduct triangulation across methods and information sources. DATA ANALYSIS METHODS The ET employed several data analysis methods to identify key findings from the collected data, as well as to draw conclusions and make recommendations for Project follow-up or future potential programming. The ET captured findings, conclusions, and recommendations (FCRs) in an Excel-based matrix that categorized analysis by EQ. The matrix: 1) ensured that the ET prepared a systematic and thorough response to each EQ; 2) compared findings across data collection sources and methods to triangulate primary and secondary data; 3) verified that analysis accounted for gender and social dimensions; 4) identified any gaps where additional clarification or analysis may be necessary; 5) clarified connections between FCRs; and 6) served as the basis for developing the evaluation report. The Encompass Gender Specialist also participated in analysis debriefs and in reviewing the FCR to validate inclusion of sex-disaggregated data. The type of analysis depended on the data being assessed as explained below. QUANTITATIVE ANALYSIS The ET reviewed and analyzed quantitative data from mWater, the 2021 client satisfaction survey, and the primary survey of CTE lower-level staff. mWater Secondary Data Review The ET undertook a review of all available CTE mWater data through December 2021and compared this with key information from primary data collection. The findings and conclusions are discussed in the sections below for each of the five CTE performance metrics. The ET reviewed the secondary mWater data for quality and consistency and used Stata 15.0 software for data cleaning. Outliers identified were addressed for the 11 indicators for each CTE. The ET requested clarification on the outliers from the respective CTEs. The ET received one response from a CTE. Details on the data cleaning process are presented in Annex D. For the analysis, the ET examined