(2020) PERFORMANCE EVALUATION: Haiti Water and Sanitation Project Baseline Survey

PERFORMANCE EVALUATION: HAITI WATER AND SANITATION PROJECT Baseline Survey | Draft Report Haiti Evaluation and Survey Services (ESS) November 2020 CROIX- DES-BOUQUETS CTE OFFICE PHOTO BY MARC LEE STEED PERFORMANCE EVALUATION : Haiti Water and Sanitation Project Baseline Survey | Report Date: November 2020 Submitted to USAID/Haiti Evaluation Mechanism Number: AID- 521-C-17-00002 Haiti Evaluation and Survey Services for USAID/Haiti Contact: Jennifer Mandel, Chief of Party 2300 Clarendon Blvd, Suite 1000 Arlington, VA 22201, USA i | WATSAN FINAL REPORT (BASELINE) USAID.GOV CONTENTS Abstract ............................................................................................................... Executive Summary .......................................................................................... i 1. Introduction ............................................................................................ 1 1.1 Water and Sanitation Project ................................................................................................................ 1 1.2 WATSAN Performance Evaluation ...................................................................................................... 2 2. Desk Review Findings ............................................................................. 3 3. Methodology............................................................................................ 5 3.1 Sampling ....................................................................................................................................................... 5 3.2 Fieldwork Preparations ........................................................................................................................... 6 3.3 Data Collection ......................................................................................................................................... 7 3.4 Data Analysis .............................................................................................................................................. 8 3.5 Indicator Definitions ................................................................................................................................. 9 3.6 Disaggregations ........................................................................................................................................ 11 3.7 Limitations ................................................................................................................................................ 12 4. Findings .................................................................................................. 13 4.1 Description of the Sample .................................................................................................................... 13 4.2 Water......................................................................................................................................................... 15 4.3 Sanitation ................................................................................................................................................... 31 4.4 Hygiene ...................................................................................................................................................... 41 5. Conclusions ........................................................................................... 45 6. References ............................................................................................. 47 7. Annexes ................................................................................................. 49 7.1 Annex A: Household Questionnaire .................................................................................................. 49 7.2 Annex B: Detailed Methodology ......................................................................................................... 79 7.3 Annex C. Detailed Project Context ................................................................................................... 86 7.4 Annex D. Evaluation Scope of Work ................................................................................................. 90 7.5 Annex E. Comparison of JMP vs. PIRS definitions........................................................................... 99 USAID.GOV WATSAN BASELINE SURVEY REPORT | ii FIGURES Figure 1. USAID/Haiti WATSAN target communes ................................................................................................. 1 Figure 2. At least basic drinking water service, by commune (IND 1) ............................................................... 15 Figure 3. Access to basic water service, by SES and commune ............................................................................ 19 Figure 4. Access to piped source on premise, by SES and commune ................................................................. 19 Figure 5. Average round-trip collection time (minutes), by commune and water service ............................ 20 Figure 6. Average round-trip collection time (minutes), travel versus queuing, by commune ..................... 20 Figure 7. Households (%) reporting water “always” available from main source, by commune .................. 21 Figure 8. Households (%) reporting insufficiency last month, by commune...................................................... 21 Figure 9. Main reason for insufficiency last month, by water service and commune ............... 21 Figure 10. Main drinking water is “always” or “mostly” safe, by service level and commune ...................... 23 Figure 11. Correlation between perception of safety and reported treatment of main drinking water .... 23 Figure 12. Plans to apply for a connection, by commune ...................................................................................... 30 Figure 13. Main reason household planning to apply for a connection, by commune .................................... 30 Figure 14. Water expenditures (monthly total and budget share), by commune and SES ............................ 31 Figure 15. Main sanitation facility, percent (%) at least basic, by commune (IND 3) ...................................... 32 Figure 16. Access to basic sanitation facility, by socioeconomic status .............................................................. 34 Figure 17. Level of access to hygiene facility, by commune ................................................................................... 42 Figure 18. Presence of handwashing facility at the household, by commune .................................................... 43 Figure 19. MDES for n=310, across possible baseline values (p1) ....................................................................... 82 Figure 20. Haiti WATSAN results framework ......................................................................................................... 86 Figure 21. Households (%) with at least basic drinking water service, JMP vs. PIRS .................................... 100 TABLES Table 1. Basic water service: USAID PIRS versus JMP service ladder .................................................................. 9 Table 3. Basic sanitation service: USAID PIRS versus JMP service ladder ......................................................... 11 Table 4. Results of total number of survey attempts, by commune .................................................................... 13 Table 5. Sample characteristics (unweighted and weighted) ................................................................................. 14 Table 6. Basic water service breakdown, by commune (IND 1) ......................................................................... 16 Table 7. Breakdown of water sources, by water service and commune ........................................................... 17 Table 8. Problems faced with supply, by service level and commune ................................................................. 22 Table 9. Water treatment methods, by commune .................................................................................................. 24 Table 10. Water storage, by commune ...................................................................................................................... 24 Table 11. Small container observation, by commune ............................................................................................. 25 Table 12. Other sources of water used for drinking, by commune .................................................................... 26 Table 13. Main source of water for other domestic purposes, by commune .................................................. 27 Table 14. Other sources of water used for domestic purposes, by commune ................................................ 28 Table 15. Access to, use of, and supply from tap on premises, by commune .................................................. 29 Table 16. Breakdown of sanitation facilities, by sanitation service and commune........................................... 33 Table 17. Facility location and sharing, by commune .............................................................................................. 35 Table 18. Risks faced when visiting sanitation facility, by commune ................................................................... 35 Table 19. Any member of household trained in relation to open defecation, by commune ........................ 36 Table 20. Sanitation facility leaks or overflows seasonally, by commune .......................................................... 37 Table 21. Access and barriers to accessibility, by commune ................................................................................ 37 Table 22. Cooking, laundry, and bathing water disposal, by commune .............................................................. 38 iii | WATSAN FINAL REPORT (BASELINE) USAID.GOV Table 23. Garbage disposal, by commune ................................................................................................................. 39 Table 24. Desire and willingness to pay for sanitation facility improvements, by commune ........................ 39 Table 25. Desired improvements to sanitation facility, by commune ................................................................. 40 Table 26. Septic tank emptying, by commune .......................................................................................................... 41 Table 27. Hygiene facility by main water source and main sanitation facility .................................................... 43 Table 28. Reported handwashing among households without facilities or unobserved ................................. 44 Table 29. Sample sizes, p1 based on WATSAN data (water) ............................................................................... 83 Table 30. Sample sizes, p1 based on JMP data .......................................................................................................... 83 Table 31. Sample sizes, p1 based on WATSAN data (sanitation) ........................................................................ 83 Table 32. Sample sizes, p1 based on JMP data (sanitation) .................................................................................... 84 Table 33. Population in each commune for post-stratification adjustment ....................................................... 85 Table 34. Households sampled from a single or multiple-household building, by commune ........................ 85 Table 35. WATSAN activity progress to date ......................................................................................................... 87 Table 36. WATSAN activity planned activities for year 3 (2019-2020) ............................................................. 88 Table 37. Basic water service breakdown, by commune (IND 1), PIRS ......................................................... 100 Table 38. Water sources used in Mirebalais, by SES ............................................................................................ 101 USAID.GOV WATSAN BASELINE SURVEY REPORT | iv ACRONYMS ADS Automated Directives System AECID Agency for International Development Cooperation BCC Behavior Change Communications CFET Centre de Formation et d’Encadrement Technique CNIGS Centre National de l’Information Géo-Spatiale CTE Centre Technique d’Exploitation DAI DAI Global, LLC DDL Development Data Library DHS Demographic and Health Survey DINEPA Water and Sanitation National Direction EQ Evaluation Question ESS Evaluation and Survey Services ET Evaluation Team FY Fiscal Year HFC High-Frequency Check IDB Inter-American Development Bank IND Indicator IRB Institutional Review Board JMP Joint Monitoring Programme M&E Monitoring and Evaluation MEL Monitoring, Evaluation, Learning MTPTC Ministry of Publics Works, Transportation, and Communication PCA Principal Component Analysis PE Performance Evaluation PIRS Performance Indicator Reference Sheets SES Socioeconomic Status SI Social Impact SOW Scope of Work SRS Stratified Random Sampling TOC Theory of Change QIP Quick Impact Project UNICEF United Nations International Children’s Emergency Fund USAID United States Agency for International Development WASH Water, Sanitation, and Hygiene WATSAN Water and Sanitation Activity USAID.GOV WATSAN BASELINE SURVEY REPORT | ABSTRACT To improve access to basic water and sanitation in Haiti, USAID is supporting the Water and Sanitation activity (WATSAN), which involves infrastructure building and capacity building for service delivery. WATSAN works in select urban and peri-urban zones within five targeted communes, including Cap- Haitian, Croix-des-Bouquets (Canaan), Mirebalais, Les Cayes, and Jérémie. USAID enlisted Social Impact’s (SI) Haiti Evaluation and Survey Services (ESS) activity to conduct a performance evaluation (PE) of WATSAN. This report details findings from the baseline household survey, which measured two key outcome indicators: percentages of households with access to basic drinking water (IND 1) and basic sanitation facilities (IND 3). Using Joint Monitoring Programme (JMP) definitions of basic water and sanitation services, 94 percent of households in the WATSAN zones have access to at least basic drinking water, and 63 percent of households in WATSAN zones have access to at least basic sanitation. FY19 HL8 PIRS definitions differ from JMP meaningfully (see Section 3.5.1) Apart from reporting on these two indicators, this survey provides valuable additional information. For water, this includes availability and burden of water collection, multiple concurrent source use, access and supply from piped households, and demand for new connections. For sanitation, this includes accessibility and safety, as well as demand and willingness to pay for improvements. Findings provide relevant context toward later evaluation of Water and Sanitation Project achievements as well as for context monitoring by USAID/Haiti. i | WATSAN FINAL REPORT (BASELINE) USAID.GOV EXECUTIVE SUMMARY BACKGROUND & WATSAN ACTIVITY Access to improved water services and sanitation facilities in Haiti remains among the lowest in the Western hemisphere. A ccording to Joint Monitoring Programme (JMP) estimates, access to basic water services has declined over the past two decades, and progress toward basic sanitation has stagnated. 1,2 To enhance access to improved water and sanitation in Haiti, the United States Agency for International Development (USAID) is supporting the Water and Sanitation project (WATSAN) implemented by DAI Global, LLC (DAI). This four-and-a-half-year intervention, which began in 2018 and will run until 2022, aims to increase access to sustainable water and san itation services. Specifically, WATSAN aims to augment access to water and sanitation services in urban and peri-urban communities (referred to as “zones”) in targeted communes that are cholera hotspots or recovering from Hurricane Matthew. These communes include Cap-Haitian, Croix-des-Bouquets (Canaan), Mirebalais, Les Cayes, and Jérémie. The activity has three mutually reinforcing components: increased access to sustainable water supply services (Component 1), increased access to sustainable sanitation services (Component 2), and strengthened enabling environment for sustainable delivery, operation, and maintenance of water, sanitation, and hygiene (WASH) services (Component 3). WATSAN plans to realize these components through an integrated combination of infrastructure building, capacity building for service delivery management, and improving the enabling environment . WATSAN is designed to transition to longer-term, comprehensive interventions that build more sustainable capacity at local and national levels to reduce waterborne diseases in priority geographic zones . ESS PERFORMANCE EVALUATION In adherence with accountability and learning institutional objectives, USAID enlisted Social Impact’s (SI) Haiti Evaluation and Survey Services (ESS) project to conduct a performance evaluation (PE) of WATSAN. 3 The WATSAN PE evaluation questions (EQs), as defined by USAID, are listed below. All three EQs will be answered in full when the performance evaluation is conducted later. This report details findings from the baseline survey which was conducted specifically to collect baseline values that will later be used to answer EQ 1, after subsequent surveys are conducted to measure changes over time. EQ 1) To what extent and in what ways has WATSAN increased access to new or improved basic drinking water and sanitation facilities? EQ 2) To what extent has WATSAN contributed to improved institutional capacity to better manage existing systems and increase revenue? EQ 3) To what extent does the WATSAN Theory of Change (TOC) remain valid and relevant to the activity’s current objectives? As per the ESS baseline survey scope of work (SOW), SI measured two indicators at baseline: (1) the percentage of households in the target communes using basic drinking water services, and (2) the 1 WHO/UNICEF, 2019a. 2 WHO/UNICEF, 2019b. 3 USAID enlisted SI to conduct this evaluation in May 2017. USAID.GOV WATSAN BASELINE SURVEY REPORT | ii percentage of households in the target communes using basic sanitation services. According to its Monitoring and Evaluation Plan, WATSAN aims to improve these indicators by 20 percent each from baseline over the life of the activity. The evaluation team (ET) measured additional indicators, described in this report, to provide context for these key indicators and the household -level WASH environment in WATSAN zones. Baseline findings will serve as a reference point for later PE data collection and allow for measurement of progress toward better access to improved water and sanitation. METHODOLOGY To measure the key indicators, the ET conducted baseline activities between November 2019 and March 2020. The ET worked with a local survey firm to administer a household survey across WATSAN zones in targeted communes (the household questionnaire is in Annex A). Before data collection, the ET led a one-week enumerator training. Field-team enumerators collected data using an ET-designed and electronically programmed survey in January and February 2020. The ET created the sampling frame using GPS data from Haiti’s Centre National de l’ Information Géo-Spatiale (CNIGS) and enumeration area shapefiles from within the WATSAN zones in each of the five targeted communes. Based on sample size calculations, the team collected data from 310 households in each of the five targeted communes, for a total of 1,550 households. These households were drawn specifically from within the WATSAN zones in the targeted communes, rather than from the communes as a whole, to obtain representative estimates of the WATSAN zones. After completing data collection, the ET conducted a robust set of quality checks and finalized the dataset for analysis and reporting. The ET used Stata 15 statistical software to conduct sampling, monitor data quality, and clean and analyze data. “At least basic water service” is defined as water from an improved source that is either on-premises or not more than 30 minutes away by round-trip collection time, including queueing. “At least basic sanitation facility” is defined as an improved facility not shared with other households. This baseline report uses the JMP service ladder classifications and definitions to define water and sanitation services. This is important because it differs from the FY19 HL8 PIRS definition most recently established by USAID, in ways that substantially affect the interpretation of these baseline results. For example, the JMP post-2015 monitoring water service ladder considers carts with small tank or drum improved sources, along with tankers, bottled waters, and, critically, water kiosks. These are all differences compared to the USAID PIRS definition, which is not as generous in classifying such sources as improved (perhaps in line with older JMP definitions). Specifically, in this case, because of the widespread use of private kiosks, the decision to classify them as improved (as implied in the JMP definitions) or unimproved (as implied in the PIRS definition), makes an enormous difference on the resulting estimates of access to basic water service. See Section 3.4 for more on classifications and definitions. The ET disaggregated key outcomes by household head sex and socioeconomic status (SES). SES quintiles are calculated within each commune, rather than across all communes, an analytical decision made based on the priority interest in commune-specific results. Other indicators are disaggregated by the main outcomes (access to basic water and sanitation). Additional detail on the methodology is in Section 3 in the report and Annex B. iii | WATSAN FINAL REPORT (BASELINE) USAID.GOV FINDINGS WATER. Overall, the ET found that 94 percent of households across all WATSAN zones in the targeted communes have access to at least basic drinking water service (using the JMP definition – for results using the PIRS definition, see Annex E). Results show little variation across communes. The result is primarily driven by the widespread use of private kiosks within 30 minutes round-trip collection time. Private kiosks are the most common main drinking water source across all targeted communes. The widespread use of private kiosks, and its effect on the estimated value of this indicator, suggests several points about WATSAN. This estimate would imply that there is almost no room to improve the targeted 20 percent over baseline values. This is relevant to the extent that the project’s performance is tied to this indicator. However, this does not imply that there are no avenues through which the project could meaningfully improve households’ water access or use. Since the majority of households rely on private kiosks, there is still considerable room for source -switching over time toward better-quality water service, such as piped water. Actual experience in terms of water access could further improve if the project were successful in achieving its primary goals of improving service delivery and service quality, such that existing customers experience more reliable and better-quality water, and as additional households potentially switch from private kiosks to piped water. From this perspective, therefore, to understand the potential impact of the project on water access among households in the selected areas, we posit that it will be necessary to look beyond IND 1, even while it is necessary to measure it as a required indicator for this evaluation and as part of the Activity’s Monitoring, Evaluation, and Learning (MEL) Plan. EQ 1 asks not only to what extent access has increased, but also in what ways; further, it asks about “new” access to basic drinking water (understood as those who change from less than basic to at least basic water service) and “improved” basic drinking water (understood as those even within the “at least basic” classification that experienced improvements in their water service ). In this way, we believe that EQ 1 is broad enough to encompass the critical nuance needed to understand not only whether the project is “moving the needle” on IND 1, but also what is happening “under the hood” of that indicator. In this report, we focus on IND 1, but in future phases (midline and endline), an important part of contextualizing any analysis of IND 1 will necessarily involve a close look at what households are experiencing in terms of specific water services used, reliability, etc. Household Head Sex. There were no significant differences in terms of access to basic water service between female- versus male-headed households, across any communes. Socioeconomic Status. As most households have access to basic water service, primarily through private kiosks within 30 minutes round-trip collection time, we do not observe any statistically significant differences in access to basic water service across socioeconomic status. However, there are substantial differences in access to an on-premises tap based on socioeconomic status in all target communes. The only exception is in Canaan, where there are no households reporting their own tap on premises. The most substantial disparity is in Mirebalais . Therefore, despite wide socioeconomic disparities in access to an on-premises tap, a widespread lack of use of those kinds of wate r services, perhaps because of failures of service delivery, leads to widespread reliance on private kiosks across SES quintiles, effectively “equalizing” access to at least basic water sources. Water Collection. Households’ average roundtrip time required to collect water, including travel and queue time, is about 14 minutes across communes. Households with less than basic water service spent substantially more time collecting water than those with basic access (note that this finding is partly USAID.GOV WATSAN BASELINE SURVEY REPORT | iv determined by the fact that basic water service is defined, in part, by collection time). In most cases, households spent more time tr aveling than queueing, regardless of commune or water service. Availability & Problems with Supply. Most households said that water from their main drinking water source was available “always” or “most of the time.” However, many said that they lacked sufficient drinking water at least once in the last month. Common problems reported with main water source include poor quality, time spent collecting water, and bad odor/taste, depending on the location. Most households with at least basic water service reported no problems with their supply. Perception of Safety. Across all areas, most households said they believed that water from their main drinking water source was safe to drink “always” or “most of the time.” This is especially true for those with at least basic service. In line with this, overall, a relatively small proportion report treating their water in any way before consuming. Water treatment does appear to be associated with the perception of safety, which is especially apparent in Jeremie and Les Cayes, as well as among those with at least basic service in Canaan and Mirebalais. Multiple Concurrent Source Use. The survey asked whether households regularly use any other drinking water sources besides their main source. In Cap-Haitian, sachet water was the most mentioned other source of drinking water, by 66 percent of households. Across all other communes, the most common response was “ no other sources” (ranging from 37% in Canaan to 47% in Les Cayes), and sachet water was the second-most common response (ranging from 24% in Les Cayes to 39% in Mirebalais). Acknowledging that households may prioritize better-quality water as accessible for drinking, the survey also asked about households’ main water source for other purposes, such as cooking and handwashing. There was considerable variation across communes. Notably, however, 81 percent of households in Canaan and half the households in Jérémie use tanker truck water for non- drinking purposes. In Cap- Haitian and Mirebalais, the use of a neighbor’s tap was also commonly reported. Access to Piped Water. Another important line of inquiry involved households with a piped connection, regardless of whether they used it as their main source of drinking water. Overall, 12.5 percent of households in WATSAN zones have an on-premises tap in their dwelling or plot; however, of these, only a small minority (11.7%) use it as their main drinking water source. Households in Mirebalais and Les Cayes were more likely to report using their tap as their main drinking wate r source, compared to Cap-Haitian and Jérémie. In Canaan , there were no households in the sample who reported an on-premises tap. Across communes, the most common reason cited for not using a tap as the household’s main drinking water source was that the tap does not supply good quality water. In Cap-Haitian, Jérémie, and Mirebalais, the second most common response was that the tap was inactive or disconnected, e.g. , due to non-payment. In Jérémie and Les Cayes, many households also reported that their main source (usually private kiosks) already provides a regular supply. Intermittency of Piped Supply. All households with an on-premises tap were also asked about the number of days per week that water is supplied through their tap, and on those days, approximately how many hours per day water is supplied. These households reported having water an average of 2.4 days per week, and most (72.8%) reported having only about one to five hours of service on such days. Among communes, households in Les Cayes reported the highest average days of supply per week (3.2) , and those in Mirebalais reported the lowest (1.9). Demand for Connections. The survey asked h ouseholds without a piped connection about their plans to apply for one . The highest levels of intention to connect were in Mirebalais and Canaan, where the vast v | WATSAN FINAL REPORT (BASELINE) USAID.GOV majority (88 and 81 percent, respectively) expressed plans to apply for a connection. Results in Canaan are particularly notable, since no households had a piped connection at the time of the survey. SANITATION. The ET found that 63 percent of households across all WATSAN zones in the targeted communes have access to at least basic sanitation facilities. The highest rate of access is in Les Cayes (70%), and the lowest is in Jérémie and Mirebalais (each 58%). There is no single facility in any commune used by a majority of respondent households; however, the first and second most common latrine types in all communes are “ improved,” per the report’s definition. Among common unimproved sources, pit latrines without slabs are used by 11.5 percent of households in Croix-des-Bouquets. Notably, open defecation is reported by more than 10 percent of households in Jérémie and Mirebalais. When breaking down main sanitation facility by service classification (at least basic versus less than basic), findings show that the majority of basic sources overall are made up of pit latrines with slabs, flush to pit latrines, and flush to septic tanks, all of which are not shared with other households. Overall, “less than basic” sanitation facilities are largely made up of pit latrines with slabs shared with other households (29.5%), open defecation (16.7%), and pit latrines without slabs or open pits (15.1%). Household Head Sex. Overall, there is no clear pattern across all communes regarding whether female - or male-headed households are more likely to report a basic sanitation facility. Differences are not statistically significant, except in Cap-Haitian, where 72 percent of female-headed households reported access to at least basic sanitation, compared with 58 percent of male-headed households. Socioeconomic Status. Overall, a clear pattern emerged showing that wealthier households have substantially greater access to at least basic sanitation facilities, compared to poorer households. The widest disparity observed is in Canaan, where 90 percent of the wealthiest fifth have at least basic sanitation, compared to 37 percent of the poorest fifth. The smallest disparity is in Les Cayes, where 87 percent of the wealthiest have at least basic sanitation , compared to 52 percent of the poorest; this is the only commune where access to basic sanitation exceeds 50 percent among the poorest fifth of households. Facility Sharing, Accessibility, and Risks. Most facilities are on premises, either in the dwelling or on the plot. Households in Canaan report the highest frequency of off-site sanitation, at about 23 percent. Most households do not share sanitation facilities with other households. The highest rate of sharing is in Mirebalais, where about 29 percent of households share sanitation facilities, with the lowest in Cap -Haitian (19%). Overall, households reported that most men and women have equal access to toilet facilities—89 percent across communes. Similarly, most households, 89.3 percent, noted that everyone in their household can access and use their facility any time. Households with at least a basic sanitation facility were more likely to say that they did not face any risks when visiting their sanitation facility, compared to those with less than basic sanitation. Where those with at least basic sanitation mentioned risks, the most common were health related. These were most frequently mentioned in Jérémie and Cap-Haitian. Among those with less than basic sanitation, most also reported health risks, but risks of harassment was also commonly reported. This was especially the case in Mirebalais, Canaan , and Jérémie. Training in Open Defecation. Across communes, most households reported that they have not received training related to open defecation (79.8% total). Cap-Haitian households were most likely to report having received training related to open defecation (32.3%). Demand and Willingness to Pay for Improvements. A majority of households in all communes expressed a desire to make improvements to their sanitation facility. The highest levels of interest were USAID.GOV WATSAN BASELINE SURVEY REPORT | vi in Cap-Haitian (72%) and Jérémie (69%). Among those who said they would like to make improvements, most—at least 80 percent in all communes—expressed willingness to pay for these improvements. Link with Household Hygiene. While it was not one of the two main outcome indicators, the ESS baseline survey also measured household hygiene levels based on JMP definitions. “Basic hygiene” is defined as the presence of a handwashing facility on premises with soap and water. Basic hygiene is highest in Cap- Haitian, where 83 percent of households have a handwashing facility with both soap and water. Looking at hygiene by main water source and main sanitation facility, data showed a stronger relationship between sanitation and hygiene compared to water and hygiene. In total, 66 percent of households with at least basic sanitation have at least basic hygiene, compared with 55 percent of households with less than basic sanitation. CONCLUSIONS This baseline survey measured key indicators IND 1 and IND 3, as related to the USAID- funded WATSAN project in urban Haiti. Overall, the ET found that the vast majority (94%) of households across all WATSAN zones in the targeted communes have access to at least basic drinking water (IND 1), while nearly two-thirds (63%) of households across all WATSAN zones in the targeted communes have access to at least basic sanitation facilities (IND 3). Given how it is defined and measured, IND 1 is already very high and, therefore, does not leave much room for WATSAN to improve it over time. However, beyond measuring these two outcome indicators, the baseline survey provided additional context showing that own-tap connectivity is relatively low, especially for the poorest households, leaving substantial room for improvements in expanding access to on-premises taps across all targeted communes. Likewise, there is room to improve service quality, which WATSAN emphasized for its planned interventions. 1 | WATSAN FINAL REPORT (BASELINE) USAID.GOV 1. INTRODUCTI ON In Haiti, overall access to improved water services has declined over the past two decades, and progress toward improving access to basic sanitation has stagnated. 4,5 Though the number of people with improved access has risen, this has not kept pace with rapid population growth, and urban access to at least basic drinking water sources has declined. 6,7 While access to basic sanitation has increased over time , only 37 percent of urban dwellers had access to basic sanitation by 2015. 8 1.1 WATER AND SANITATION PROJE CT In response, the United States Agency for International Development (USAID) is supporting the Water and Sanitation Project 9, a four-and-half-year intervention (2018–2022), to support access to sustainable water supply and sanitation services and strengthen the enabling environment for sustainable delivery, operation, and maintenance of water and sanitation services. WATSAN focuses on increasing access to water and sanitation services in communes that are cholera hotspots or recovering from Hurricane Matthew. The five target communes include Cap-Haitian, Croix-des-Bouquets (Canaan), Mirebalais, Les Cayes, and Jérémie (Figure 1). WATSAN interventions are directed at specific zones within these five targeted communes, referred to as WATSAN zones. Figure 1. USAID/Haiti WATSAN target communes The activity is an integrated set of interventions organized into three mutually reinforcing components: increased access to sustainable water supply services (Component 1), increased access to sustainable 4 WHO/UNICEF, 2019a. 5 WHO/UNICEF, 2019b. 6 World Bank, 2018. 7 “At least basic” includes new JMP categories of basic and safely managed (see WHO/UNICEF, 2019c). 8 World Bank, 2018. 9 Previously referred to as the WATSAN Activity USAID.GOV WATSAN BASELINE SURVEY REPORT | 2 sanitation services (Component 2), and strengthened enabling environment for sustainable delivery, operation, and maintenance of water, sanitation, and hygiene (WASH) services (Component 3). A key principle of the activity is to transition to longer-term, comprehensive interventions that build more sustainable capacity at the local and national levels to reduce waterborne diseases in priority geographic zones. The WATSAN development hypothesis posits : “If access to improved water and sanitation services in Haiti is expanded through strategic technical assistance and infrastructure to improve the quality and availability of water and sanitation services, while the enabling environment for continued operation, maintenance, oversight, and use of those services is systemically strengthened, then a foundation will be established for sustained service delivery of safe and affordable water and sanitation services in Haiti, leading to reduced prevalence of cholera and other waterbo rne diseases and the improved health and prosperity of the Haitian people.” WATSAN’s results framework is based on the goal to provide 250,000 Haitians with access to new or improved water services over the life of the activity, along with 75,000 Haitians with access to basic or safely managed sanitation services over the same period. The activity results framework is included in the detailed project context in Annex C. Within its three components, WATSAN implements a variety of water and sanitation interventions, which differ from commune to commune, but are primarily in the form of w ater infrastructure engineering services, water infrastructure construction, technical assistance for water service providers, support to sanitation enterprises, wastewater treatment and fecal sludge management engineering services, wastewater treatment and fecal sludge management engineering construction, technical assistance to relevant national and sub- national government units, and knowledge dissemination and learning. 1.2 WATSAN PERFORMANCE EVALUATION In line with its institutional objectives of accountability and learning, USAID/Haiti enlisted Social Impact’s (SI) Haiti Evaluation and Survey Services (ESS) project to conduct a performance evaluation (PE) of WATSAN. The purpose of this PE is to measure whether WATSAN interventions have resulted in access to improved water and sanitation services in the targeted communes. The evaluation questions (EQs) are listed below. All three EQs will be answered in full when the performance evaluation is conducted later. This report details findings from the baseline survey which was conducted specifically to collect baseline values that will later be used to answer EQ 1, after subsequent surveys are conducted to measure changes over time. EQ 1) To what extent and in what ways has WATSAN increased access to new or improved basic drinking water and sanitation facilities? EQ 2) To what extent has WATSAN contributed to improved institutional capacity to better manage existing systems and increase revenue? EQ 3) To what extent does the WATSAN Theory of Change (TOC) remain valid and relevant to the activity’s current objectives? As per the ESS baseline survey scope of work, SI measured two indicators at baseline: IND 1: Percentage increase of households in the target communes using a basic drinking water service. WATSAN aims to increase this by 10 percent by midline and by 20 percent from baseline over the life of the activity. 3 | WATSAN FINAL REPORT (BASELINE) USAID.GOV IND 3: Percentage increase of households in the target communes using a basic sanitation facility. The project aims to increase this by 10 percent by midline and by 20 percent from baseline value over the life of the activity. The baseline survey will serve as a reference point to measure the performance of WATSAN interventions. EQs 2 and 3 will be addressed as part of the future PE. Following the conclusion of the baseline report approval, the evaluation team (ET) will de-identify the baseline dataset to submit to the Development Data Library (DDL) as per contract requirements and USAID Automated Directives System (ADS) 579 open data guidelines. S ee the full scope of work (SOW) in Annex E. The primary stakeholders for this evaluation include USAID/Haiti, WATSAN ’s implementing partner DAI (and DAI’ s partners Ayiti Nexus, Centre de Formation et d’Encadrement Technique [CFET], Zanmi Lasante, V3, and mWater), the Haiti Ministry of Publics Works, Transportation, and Communication (MTPTC), and the Haiti Water and Sanitation National Direction (DINEPA). Other stakeholders include other donors and entities active in Haiti’s WASH sector, such as the Inter-American Development Bank (IDB), the Spanish Agency for International Development Cooperation (AECID), and the World Bank. 2. DESK REVIEW FINDINGS Before launching the baseline survey, the ESS ET reviewed several documents, including the activity’s workplans, progress reports, monitoring, evaluation, and learning ( MEL) plan, internal baseline report, and others. 10 Findings from this desk review informed the baseline survey design and methodology and yielde d important insights related to the interpretation of findings. First, USAID had identified the need to obtain baseline estimates of the key indicators for specific zones within each commune where WATSAN is being implemented. As a result of this explicitly articulated priority, the ESS baseline focuses specifically on the WATSAN zones within each targeted commune. The ESS baseline survey was, thus, designed to be representative of zones where the project is intervening (rather than representative at the commune level as a whole). In addition, interpretation of findings for both baseline and future rounds of data collection must consider the possibility that some benefits have accrued before the ESS baseline survey. For a variety of operational and logistical reasons, some WATSAN activities had already begun by the time this baseline was launched in early 2020. 11 For example, in Les Cayes, WATSAN had begun providing fuel to power well generators in late 2019 and claimed to have observed a 20 percent increase in requests for new connections between then and January 2020. In Canaan, by the end of January 2020, WATSAN was expected to connect several reservoirs or water kiosks to the main distribution pipe before the baseline study. These reservoirs and kiosks had been unimproved water sources (supplied by trucks), but with a connection to the distribution pipe, they would now be considered improved. In Jérémie, the Centre Technique d’Exploitation (CTE) has begun to repair leaks with its own staff using equipment and supplies WATSAN provided. Similar work may have begun in Cap-Haitian in the first quarter of fiscal year 2020. In Mirebalais and Jérémie, starting in May 2019, the project may have been working to promote sanitation and end open defecation through community awareness, support sanitation marketing efforts, and develop ing community sanitation programs. A full list of documented project work to date is in Annex C . 10 See references for full list of reviewed documents. 11 By the time of this baseline, WATSAN was within Year 3 of project implementation. Years 1 and 2 of project implementation were largely dedicated to feasibility studies, assessments, and strategic planning for interventions. USAID.GOV WATSAN BASELINE SURVEY REPORT | 4 Further, even assuming that all WATSAN interventions are implemented successfully, trends in the key outcome indicators over time are unlikely to be strictly the result of any one a ctivity or funding stream, but more likely the result of multiple donors working together to support CTEs in their objectives of expanding access to and improving the quality of key services. S ome WATSAN interventions are interrelated with and interdependent on interventions implemented by other donors. As noted in a recent workplan, WATSAN is “ leveraging the investments being made by the [IDB], AECID, and World Bank. The activity is counting on the World Bank to rehabilitate the Fonfred FSM site, AECID to rehabilitate the water distribution system in Jérémie and Cap-Haitian, and the [ IDB] to expand the water distribution systems in Cap-Haitian and Les Cayes.” 12 Lastly, WATSAN interventions largely focused on capacity- building for CTEs and the enabling environment. Associated interventions, in many cases, are expected to lead to improvements in service quality rather than increasing access to piped water. The ET’s review of activity documentation and discussions with WATSAN staff reinforced that WATSAN is mostly a technical capacity building intervention, focused on strengthening local CTEs and sludge management facilities and their ability to provide quality and regular water and sanitation services. Taking this into consideration, WATSAN does not expect many of its interventions to provide new access to basic water, but rather result in improvements in service quality (e.g., reliability, continuity of service, and water quality) among households already connected to piped water. For sanitation, the main focus is on improving treatment of fecal sludge waste. In other words, many WATSAN interventions are not likely to “move the needle” on IND 1 and IND 3, though that is the main focus of the ESS baseline survey, as per the evaluation SOW. Discussions with USAID, WATSAN, and ESS led to an agreement that ESS would include select questions in the baseline survey that measure service quality for those households already connected. These are explored in the Findings section of this report. WATSAN Communes The baseline survey obtained commune-specific results (for the WATSAN zones within each targeted commune), and the report presents commune-specific results throughout. Part of the reason for this is that the context in each commune is substantially different: not all are at the same “ starting point” concerning WATSAN programming. The content below is taken directly from the WATSAN internal baseline report 13 and provides an informative context for each commune, which aids in interpreting the different results observed between communes. Cap Haitian is Haiti’ s second-largest city. It is a port city in the north of the country and was Haiti’s original capital. Cap Haitian has a rich aquifer that is easily reached by drilled wells, and, because the public water network has fallen into disrepair, most people get their water from wells. In the downtown area, these are drilled wells that provide reasonably safe water. In the outlying, poorer areas, the wells are shallow and dug by hand. Nearly everyone in Cap-Haitian relies on dry toilets, with a small percentage having flush toilets and a small percentage having no toilet. The only fecal sludge management is provided by a nonprofit that uses containerized toilets. Canaan (Croix de Bouquets commune), Haiti’s most rapidly growing area, began as an informal settlement after the earthquake and has grown to a population estimated at 216,000. 12 DAI, October 2019. 13 DAI, 2019a. [... middle sections omitted for long document ...] 97 | WATSAN FINAL REPORT (BASELINE) USAID.GOV Task Estimated Due Date Inception Report February 15, 2019 Evaluation Design February 22, 2019 Inception Report and Evaluation Design approved February 28, 2019 In Briefing January 28, 2018 Data collection and analysis March 2019 Out-Briefing Presentation TBD Baseline Draft Report submitted to USAID TBD USAID Feedback on Baseline Draft Report TBD Baseline Final Report submitted to USAID TBD Endline Data Collection and Analysis June – August 2021 9. Final Report Format The baseline final report should include an abstract of no more than 250 words; executive summary of no more than 5 pages; background on the local context and strategies/projects/activities being evaluated; the baseline purpose and evaluation questions; the methodology or methodologies; study limitations; findings, conclusions, and recommendations. For more detail, see “ How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the activity being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in detail in the report. Limitations to the evaluation shall be disclosed in the report, with attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information properly identified and listed; and • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. • Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’ s conclusion. 10. Other Requirements USAID.GOV WATSAN BASELINE SURVEY REPORT | 98 All quantitative data collected by the evaluation team must be provided in machine-readable, non- proprietary formats as required by USAID’ s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the activity or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Baseline Report. 99 | WATSAN FINAL REPORT (BASELINE) USAID.GOV 7.5 ANNEX E. COMPARISON OF JMP VS. PIRS DEFINIT IONS As described in section 3.5.1., the different definitions used by JMP and PIRS lead to disagreement in the interpretation of baseline data, in that JMP allows for a greater number of sources to be categorized as improved relative to the PIRS definition. Moreover, the widespread use of private kiosks most heavily influences these differences, since the JMP and PIRS classify them differently – while the PIRS does not explicitly list private kiosks as an unimproved source, it includes language that “all o ther sources” other than those listed are considered unimproved, and this classification was confirmed via further consultations between our team, USAID Program and Technical offices, and the Water and Sanitation project team. Applying the PIRS definition and comparing with the JMP, we see that they agree in about 23% of cases – about 17% of cases where the household’s drinking water service is at least Basic, and nearly 6% of cases where the household’s drinking water service is less than basic. There is disagreement in about 77% of cases (i.e. 1200 households) where the JMP would classify a household’s drinking water service as “at least basic”, whereas PIRS would classify it as “less than basic”. As described earlier, this disagreement is driven heavily by the decision of whether to classify private kiosks as improved or not – among those 1200 cases of disagreement, nearly 77% of them are accounted for by private kiosks. Another approximatel y 10% are accounted for by the different classification of “cart with small tank/drum”. JMP vs. PIRS: Frequency and cell percentage (unweighted) Water | service | Water service (PIRS) (JMP) | Basic+ <Basic | Total -----------+ ----------------------+ ---------- Basic+ | 263 1,200 | 1,463 | 16.97 77.42 | 94.39 -----------+ ----------------------+ ---------- <Basic | 0 87 | 87 | 0.00 5.61 | 5.61 -----------+ ----------------------+ ---------- Total | 263 1,287 | 1,550 | 16.97 83.03 | 100.00 Main drinking water source in cases of JMP vs. PIRS disagreement (unweighted) Main DW source | Freq. Percent Cum. --------------------------+ ------------------------------------ Tanker truck | 52 4.33 4.33 Cart with small tank/drum | 124 10.33 14.67 Public Water Kiosk | 71 5.92 20.58 Private Water Kiosk | 919 76.58 97.17 Bottled water* | 28 2.33 99.50 Sachet water* | 6 0.50 100.00 --------------------------+ ------------------------------------ Total | 1,200 100.00 *Note that the new JMP definition considers bottled and sachet water improved, while the PIRS considers them improved only if sources used for other household activities are also improved. Thus, in this table, these are cases where bottled or sachet water are used as the household’s main drinking water source and unimproved sources are used for other household activities. Below we provide comparisons of the main results for JMP vs. PIRS definitions, we see the impact that the classification of private kiosks in particular has on the possible interpretation of results, with JMP definitions clearly leading to much more optimistic estimates of access than the PIRS. Of course, the ability to differentiate between the two types of private kiosks in further phases of the evaluation may assist USAID.GOV WATSAN BASELINE SURVEY REPORT | 100 classification and interpretation, though the limitation of the data at baseline will prevent direct comparisons in later stages with the two types of private kiosks at baseline. Results using the PIRS definition show the greatest access to basic drinking water service in Les Cayes (41% of households), followed by Mirebalais (19%), Cap-Haitian (13%), Jeremie (10%), then Canaan (1%). Figure 21. Households (%) with at least basic drinking water service, JMP vs. PIRS Post-stratification weights applied. Analytical sample size is 301 for all communes. Breaking the PIRS definition of basic water service down below in Table 37, in all communes, the majority of improved sources are those other than a household’s own tap on premises. The commune with the greatest reported use of own tap on premises in Les Cayes, with 10% of households reporting the use of their own tap as their main source of drinking water. Note that the underlying water sources reportedly used by all households has not changed – the only difference between these results using the PIRS definition and the JMP definition is the classification of sources as improved or unimp roved. Table 36. Basic water service breakdown, by commune (IND 1), PIRS Cap- Haitian Canaan Jeremie Les Cayes Mirebalais Total At Least Basic (IND 1) Own tap on premises 0% 0% 2% 10% 5% 2% Improved source, <=30 min RT 13% 1% 8% 31% 15% 12% Less than Basic Water Service Improved source, >30 min RT 1% 0% 1% 1% 1% 1% Unimproved source, <=30 min RT 80% 95% 85% 54% 76% 80% Unimproved source, >30 min RT 6% 4% 4% 4% 4% 5% Total 100% 100% 100% 100% 100% 100% Post-stratification weights applied. Analytical sample size is 301 for all communes. Compares with Table 6 in body of the report. By head of household sex and by socioeconomic status. Just as with the JMP definition, there are n o statistically significant differences in basic water service for female- versus male-headed households in the WATSAN areas of the five communes. Likewise, using the PIRS definition of basic water service, there are no statistically significant differences in access based on socioeconomic status across the five communes, except in the case of Mirebalais, though the results are the opposite of what may be expected, with those in the highest socioeconomic quintile supposedly reporting substantially lower access (9% at least basic) 13% 1% 10% 41% 19% 15% 93% 96% 94% 94% 95% 94% 0% 20% 40% 60% 80% 100% Cap-Haitian Canaan Jeremie Les Cayes Mirebalais Total PIRS definitionJMP definition 101 | WATSAN FINAL REPORT (BASELINE) USAID.GOV to improved sources compared with those in the lowest socioeconomic quintile (35% at least basic) – looking specifically at the sources of drinking water reportedly used by those in Mirebalais at different levels of socioeconomic status (see Table 38 below), this appears to be driven by households of lower SES being more likely to report getting piped water from a neighbor, which is classified as improved, in combination with those of higher SES being more likely to report using water from private kiosks and carts, which by PIRS definition are both considered unimproved. Table 37. Water sources used in Mirebalais, by SES Mirebalais, SES Quintile Water Source (%) 1 (lowest) 2 3 4 5 (highest) Piped into dwelling 1.6 1.6 1.6 4.8 3.2 Piped into plot 1.6 0 4.8 0 3.2 Piped to neighbor 22.6 8.1 11.3 1.6 0 Public tap/standpipe 0 9.7 1.6 0 0 Borehole or tubewell 6.5 1.6 3.2 1.6 1.6 Protected well 1.6 1.6 0 0 0 Protected spring 1.6 0 0 0 0 Unprotected spring 0 0 1.6 0 0 Cart with small tank/drum 9.7 16.1 12.9 14.5 12.9 Public Water Kiosk 1.6 0 0 1.6 0 Private Water Kiosk 48.4 59.7 62.9 74.2 77.4 Sachet water 3.2 1.6 0 1.6 1.6 Surface water 1.6 0 0 0 0 Total 100 100 100 100 100 Post-stratification weights applied. Analytical sample size is 301 in the commune. A full reproduction of all the tables and disaggregations in this report by the PIRS definition has not been undertaken for the purpose of this annex but is available upon request.