Rapport initial de l'évaluation finale du programme EFSP de WV Haïti à La Gonâve

Emergency Food Security Program in La Gonâve USAID EFSP #72DFFP19GR00074 Final Evaluation Study Final Report Submitted by: Amisial LEDIX, Economist-Statistician/Monitoring & Evaluation Specialist In collaboration with: Amos MONTREUIL JEAN , Agronomist/Food Security and Nutrition Specialist Obed JULES , IT Technology, Economic Development Document Version Date Author (s) Final Evaluation ’ s Initial Report 1.0 August 28 th , 2021 Amisial Ledix & Col. World Vision International Haiti USAID EFSP 2 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Table of Contents 1 Acknowledgements 3 List of Tables, Pictures and Charts 4 List of Abbreviations and Acronyms 6 Executive Summary 7 1. Final Evaluation’s Background 10 2. Purpose of the Final Evaluation 11 3. Work Methodology 11 3.1. Quantitative Survey 14 a) Calculation of Sample Size 17 b) Processing and analysis of quantitative data 22 3.2. Qualitative Survey 23 a) Focus Groups 24 b) Key Interviews 24 c) Evaluation of the Program’s Effects on the Markets 25 d) Processing and analysis of qualitative data 25 4. Constraints and Limitations of the Study 26 5. Analysis of the Final Evaluation’s Findings 26 5.1 Broad Characteristics of Beneficiary Households 26 5.2 Analysis of Evaluative Questions 33 5.2.1 Relevance 33 5.2.2 Profitability 37 5.2.3 Effectiveness 38 5.2.4 Impact 49 5.2.5 Sustainability 52 5.2.6 Linkages, Overlaps and Exit Strategies 54 5.3 Analysis of the program’s effects on local markets and certain interest groups (women and men; youth population; boys and girls, etc.) 57 6. Conclusion and Recommendations 60 7. Bibliographical References 65 8. Appendices 66 Appendix I: FCS Indicator 66 Appendix II: HHS Indicator 67 Appendix III: Reduced Coping Strategy Index (rCSI) 68 Appendix IV: Data collection tools (in separate files) 70 USAID EFSP 3 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Acknowledgements We, the team of consultants in charge of this study would like to express, in a very special manner, our gratitude to all those who contributed to the completion of this work. In spite of the various challenges and constraints, we were able to succeed thanks to your active and efficient participation. First, we thank the Monitoring and Evaluation team at the World Vision Haiti’s National Office for their open and fruitful cooperation. We had very useful interactions with those we discussed with, and that helped achieve a smooth working process. We also wish to thank the EFSP Project Manager with whom we also spoke during the meeting that set this study’s guidelines. In addition, we thank the Monitoring and Evaluation Officer of the La Gonâve Office for his support in making the required information available to facilitate field operations. We also thank the investigators and supervisors who were in the field to collect data from beneficiaries within the project’s various intervention communities in spite of the fact that this activity started exactly at the same time the country was hit by the devastating August 14 th , 2021 earthquake, in the larger South. The team of consultants still believes that this study’s conclusions and recommendations will adequately help the EFSP project team face the challenges and achieve the expected results for the benefit of beneficiaries living in the organization’s intervention areas. USAID EFSP 4 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation List of Tables, Pictures and Charts Table 1 : Number of beneficiaries by commune and communal section ........................................ 19 Table 2: Breakdown of the number of beneficiary households to be surveyed in Anse-à-Galets by communal section ................................................................................................................................ 20 Table 3: Breakdown of the number of beneficiary households to be surveyed in Pointe-à- Raquettes by communal section ......................................................................................................... 21 Table 4: Breakdown of the sample of children aged 0-59 months to be surveyed in Anse-à-Galets ............................................................................................................................................................... 22 Table 5: Breakdown of the sample of children aged 0-59 months to be surveyed in Pointe-à- Raquettes .............................................................................................................................................. 22 Table 6: Number of focus groups per commune and category of beneficiaries ............................. 24 Table 7: Number of key interviews by commune and category of stakeholder ............................. 24 Table 8 : Distribution of respondents by commune according to the type of habitat (n = 726) ... 30 Table 9 : Distribution of respondents by commune according to the head of household’s main activity .................................................................................................................................................. 30 Table 10 : Distribution of respondents by commune according to the head of household’s second activity .................................................................................................................................................. 31 Table 11 : Key indicators measured and their progress against target .......................................... 39 Table 12 : Food consumption score of project beneficiaries by commune .................................... 40 Table 13 : Hunger score of project beneficiaries by department .................................................... 41 Table 14 : Dietary Diversity Score by department ........................................................................... 42 Table 15 : Percentage of food use by type of project beneficiaries for the island of La Gonâve and the communes ............................................................................................................................... 42 Table 16 : Percentage of households where adults and children eat at least 2 meals a day (adults and children) ........................................................................................................................................ 43 Table 17 : Reduced Coping Strategies Index (ISSr/rCSI) ............................................................... 43 Table 18 : Percentage of targeted households using and/or benefiting from community assets created/rehabilitated ........................................................................................................................... 44 Table 19 : Percentage of targeted households reporting an increase in the amount saved through the program ......................................................................................................................................... 45 Table 20 : Percentage of households adopting best practices in health and nutrition .................. 46 Table 21 : Prevalence of acute malnutrition by commune .............................................................. 46 Table 22 : Comparison of Baseline and Final Evaluation Results .................................................. 50 Table 23: Level of satisfaction in terms of meeting criteria ............................................................ 56 Chart 1 : Age pyramid of EFSP Program beneficiaries in the commune of Anse-à-Galets ........................ 27 Chart 2 : Age pyramid of EFSP Program beneficiaries in the commune of Pointe-à-Raquettes ................ 27 Chart 3 : Distribution of respondents by commune according to their sex (n=726) .................................... 27 Chart 4 : Distribution of respondents by commune according to household status (n= 726) ..................... 27 Chart 5 : Distribution of heads of households by commune according to their education level (n=726) .. 28 Chart 6 : Distribution of heads of household by commune according to their marital status (n= 726) ..... 28 Chart 7: Distribution of respondents by commune according to whether or not there is at least one orphan in the household ..................................................................................................................................... 28 Chart 8 : Distribution of respondents by commune according to whether or not there is a pregnant woman in the household ..................................................................................................................................... 29 Chart 9 : Distribution of respondents by commune according to whether or not there is at least one disabled person in the household ....................................................................................................................... 29 USAID EFSP 5 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Chart 10 : Distribution of respondents by commune according to the monthly household income bracket .............................................................................................................................................................................. 32 Chart 11 : Distribution of respondents by commune according to the first item of annual household expenditure .......................................................................................................................................................... 32 Chart 12 : Distribution of respondents by commune according to the monthly household income bracket 32 Chart 13 : Hunger score of project beneficiaries by department .................................................................. 39 USAID EFSP 6 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation List of Abbreviations and Acronyms LA Local Authorities ASCP Agent Sanitaire Communautaire Polyvalent ( Community Health Worker ) FVFW Food Voucher for Work EFSP Emergency Food Security Program FCS Food Consumption Score FFP Food for Peace GPS Global Positioning System HDDS Household Dietary Diversity Score HHS Household Hunger Scale IPC Integrated Phase Classification OECD Organization for Economic Cooperation and Development rCSI Reduced Coping Strategies Index S4T Savings for Transformation SPSS Statistical Package for the Social Sciences ToR Terms of Reference USAID U.S. Agency for International Development UVT Unconditional Voucher Transfer VFW Voucher for Work USAID EFSP 7 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Executive Summary The Emergency Food Security Program (EFSP), implemented by WV in two of the West department’s communes, specifically on the island of Gonâve (Pointe-à-Raquettes and Anse-à- Galets), aims to provide immediate access to food to communities in crises and emergencies (IPC 3 and 4) and help them access training on good health and nutritional practices. This 15- month Emergency Food Security Program in La Gonâve provided support to 5,770 vulnerable households (28,850 people) to alleviate the crisis-level food insecurity as a result of depleted harvests and eroded livelihoods. This report presents an analysis of the findings of this program’s evaluation by reviewing the achievement level of the goal, objectives and outputs of the EFSP project and the extent to which they were achieved. To carry out this work, we considered both quantitative and qualitative approach and, to collect the data required by this evaluation, a representative sample of 726 beneficiaries (365 in Anse- à-Galets and 361 in Pointe-à-Raquettes) were surveyed. On the other hand, to implement the qualitative approach, 25 key interviews and 14 focus groups were conducted with other project stakeholders (project managers, government institutions, community leaders, local authorities, pregnant women, nursing women, individuals with reduced mobility, members of community savings and credit groups, and young people). The results of this evaluation show a study universe characterized by the following aspects: • The 0-24-year age groups are the most represented in the 2 intervention communes of beneficiary households’ populations. Data also show that in both communes, over 24 years of age, there are slightly more women than men in beneficiary households. • More than 25% of households are in fact led by women with slightly more cases in Anse-à-Galets. • In Anse-à-Galets, just as in Pointe-à-Raquettes, more than 70% of beneficiaries surveyed are in a common-law relationship or are married. • Beneficiaries’ houses are mostly made of tin roofs, which are respectively 77% in Anse- à-Galets and 87% in Pointe-à-Raquettes. This is consistent with the surveyed beneficiaries’ level of vulnerability. • Unfortunately, more people are in charcoal production (6% in Anse-à-Galets and 7% in Pointe-à-Raquettes) than in fishing, even though we are on an island (4% of fishermen in Anse-à-Galets and 6% in Pointe-à-Raquettes). • Most people are living in poverty with a monthly income of less than 5,000 HTG (around 50 USD), which is less than 1.65 USD a day in both communes. By analyzing the quantitative and qualitative data collected to provide answers to evaluation questions, the overall findings show the project fulfilled very well the relevance criterion in the sense that the priorities of targeted vulnerable households were taken into account in the activities . USAID EFSP 8 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation The profitability/efficiency criterion, on the other hand, was not met by the project. Internal systems slowed down the implementation of activities, particularly in the nutritional component. The recruitment process for 2 nurses was unsuccessful. The slow delivery of vendor checks negatively affected their resupplying. In terms of project effectiveness , only 1 indicator (percentage of targeted households with an acceptable food consumption score (FCS) did not increase according to the desired trend despite the intervention of the project among those mentioned in the terms of reference. It is noteworthy that all of them reached at least 50% of their target. So, the project has met this criterion in a way. The project’s impact criterion was met insofar as the 5,770 households were reached in the various distribution cycles. In addition, the difference test shows at least 6 indicators, out of the 10 mentioned in the terms of reference, with significant change in their value in the desired trend, when comparing the situation at the start and at the end of the project. The sustainability criterion, for its part, was moderately met within this project in the sense that certain project actions such as community savings and loan groups can continue past the discontinuation of donor funding. However, it is not sure that ASCP will keep supporting the communities, given the fact that they have been let down by the government. Some are not even included in the government’s budget. Finally, regarding linkages, overlaps and exit strategies , we can say that the project was successful in the sense that it received investments from other programs in the use of SIMAST for targeting but especially with the presence community savings groups created since Kore Lavi that made the overall action sustainable. USAID EFSP 9 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Summary of project indicator results Indicators Island of la Gonâve (%) ± CI Baseline values Endline values Desired trend Current trend Target Explanations Percentage of targeted households with an acceptable food consumption score (FCS) 48.2 ± 6.2% 46%±7.2% + - 75% Throughout the project intervention, the country's macroeconomic situation deteriorated due to socio-political crises and the slowdown in activities with COVID19. There was high inflation, the intervention of the project was not enough in this context to improve the household food consumption score. Prevalence of households with little or no hunger (Household Hunger Scale - HHS) (score 0-1) 15.5 ±4.7% 47.3%±7.3% + + 70% The desired trend is maintained but the target has not been reached due to the shocks mentioned above. Households that were at the bottom of the vulnerability scale despite the intervention. Proportion of households consuming at least 6 food groups in the previous month 73.9±5.6% 89.5±2.2% + + 100% Idem Percentage of food use by type (household consumption) 95±2.8% 98.9% ±1.6% + + 100% Idem Percentage of households where adults and children eat at least 2 meals per day (adults and children) 67.2%±2.8% 95% ±3.1% + + 70% Reduced Coping Strategies Index (% of households with a less serious or moderate index) 62.5±6.3% 68.8±6.45% + + 70% Idem Percentage of targeted households using and/or benefiting from community assets created/rehabilitated. 0% 71.6±3.3% + + 60% Percentage of targeted households reporting an increase in the amount saved thanks to the program. 0% 87.2±10.5% + + 60% Percentage of households adopting best practices in health and nutrition 0% 31%±6.70% + + 60% Prevalence of Global acute malnutrition (GAM) 6% 5.8±2.2% - - 10% + : increased trend ; - : decreased trend USAID EFSP 10 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation 1. Final Evaluation’s Background La Gonâve, one of the most food insecure regions in Haiti, is currently experiencing extreme food insecurity in its rural areas. The poor harvests for spring and fall of the 2017-2018 season caused 45% of households (around 40,000 people in IPC 3 and 4) to be at risk of a serious food crisis if urgent aid is not provided to this population which barely manages to meet its basic food needs (OCHA Haiti, 2019) 1 . The sorghum and pigeon pea harvests, the two most dominant crops of the fall season in La Gonâve, are estimated at less than 40 percent of what is traditionally considered a normal fall/winter harvest. The last failed harvest had an impact on food security in rural areas and, according to the latest surveys by CNSA and FEWSNET, the crisis will most likely be prolonged until the next spring harvest. Haiti’s precarious economic situation, characterized by currency depreciation and inflation, continues to drive up the prices of staple foods, especially grains (IPC Haiti) 2 . Access to food is even more difficult for most of the inhabitants, as 60% of the food consumed is purchased outside the island. To cope with this situation, people sell their livestock and non-productive assets (FEWSNET, 2019) 3 . Other coping strategies include migration to the Dominican Republic, South America, particularly Chile or Brazil, labor sale, prostitution, fishing, and some households have already consumed their seed reserves (World Vision Haiti) 4 . This is the context in which World Vision Haiti, thanks to funding from USAID, implemented a 15-month Emergency Food Security Program in La Gonâve providing support to 5,770 vulnerable households (28,850 people) to alleviate critical food insecurity resulting from depleted harvests and eroded livelihoods. This project targeted the 2 communes of the island of La Gonâve (in IPC 3 & 4 and a crisis and humanitarian emergency situation) and included the following activities: ➔ Unconditional electronic voucher transfer (UVT) ➔ Conditional voucher Transfer-Food Voucher for Work (VFW) ➔ Essential complementary activities: agriculture, livelihoods, savings for transformation (S4T) and nutrition This document serves as the report for this final evaluation of the EFSP program, analyzing the field findings from the intervention communities according to evaluation questions, in the light of OECD criteria. 1 https://www.humanitarianresponse.info/en/operations/haiti/document/haiti-2019-humanitarian-needs-overview-2019-summary- january-2019 2 http://www.ipcinfo.org/ipc-country-analysis/details-map/en/c/1068538/?iso3=HTI 3 http://fews.net/node/22705 4 https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/assessments/la-gonave-one-pager.pdf USAID EFSP 11 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation 2. Purpose of the Final Evaluation The overall purpose of this final evaluation is to analyze the level of achievement of the goal, objectives and outcomes of the EFSP project and the extent to which they were achieved. This is a performance review. More specifically, the final evaluation aims at the following specific objectives: ● Evaluate program’s achievements with respect to goal, objectives, outcomes and targets. ● Assess program’s effects on local markets and certain interest groups (women and men; youth population; boys and girls, etc.). ● Evaluate the effectiveness and relevance of the modality, transfers and complementary interventions to achieve program’s outcomes. ● Identify best practices, lessons learned, strengths and challenges in the program’s design, including the logical framework and implementation to achieve project’s achievements. ● Recommended strategies for other projects or new interventions. 3. Work Methodology To achieve the objectives of this final evaluation, a dual quantitative and qualitative approach was considered. This approach combined the collection of project-related secondary data and primary data among project partners and participants in the intervention communities. The evaluation questions mentioned in the Terms of Reference are based on OECD criteria. The following tables clarify the data sources that helped address the different questions according to the Relevance-Effectiveness-Efficiency-Impact-Sustainability criteria: USAID EFSP 12 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Crit eria Evaluative Questions Data Sources /Collection Methods Relevance What are the opinions of stakeholders on the nature and quality of implementation? Were relevant government officials involved? Has the project implementation strategy been adjusted to take into account the realities on the ground? If so, in what way? Were program activities and outputs consistent with the expected impacts and effects? Did they meet the needs and priorities of the most vulnerable and targeted pregnant and nursing women? - Quantitative survey of beneficiaries - Key interviews with project manager and/or project component managers - Key interviews with the project’s key partners (Local Authorities (Mayors / CASEC), MAST, MSPP, Wesleyan Hospital, Mother-leaders, ...) - Focus group with beneficiaries from vulnerable groups (pregnant women, nursing women, people with reduced mobility, etc.), mothers’ clubs and ASCP Profitability Did the project have adequate and appropriate resources (human, financial and capital) for its implementation? If not, how was it addressed? Were quality control and accountability measures in place and consistently applied during the review, approval, funding disbursement, monitoring, and reporting phases? Do recipient comments indicate widespread instances where funds (vouchers) were taxed or stolen, or where receiving a voucher represented a protection risk? - Key interviews with project manager and/or project component managers - Focus group with beneficiaries from vulnerable groups (pregnant women, nursing women, people with reduced mobility, etc.), mothers’ clubs and ASCP - Consultation of project’s financial reports - Key interviews with the project’s key partners (Local Authorities (Mayors / CASEC), MAST, MSPP, Wesleyan Hospital, Mother-leaders, ...) Efficiency To what extent were the objectives achieved? What were the main factors that did or did not influence the achievement of the objectives? Did the M&E system provide appropriate and reliable quality information to measure the planned indicators? - Quantitative survey of beneficiaries - Key interviews with project manager and/or project component managers - Key interviews with the project’s key partners (Local Authorities (Mayors / CASEC), MAST, MSPP, Wesleyan Hospital, Mother-leaders, ...) USAID EFSP 13 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Crit eria Evaluative Questions Data S ources /Collection Methods Effic iency (continued) How effective was the project model in terms of design, relevance, management and accountability? How effective was the program in terms of implementation (coordination, cooperation, effectiveness, standardization)? Have humanitarian standards been met and humanitarian principles complied with (SPHERE, HAPs, and Codes of conduct)? What measures were taken to identify and reduce possible negative effects? -Consultation of M&E tools/periodic reports (Post-Distribution Monitoring if available, Monitoring plan, Indicator Tracking Table, etc.) -Focus group with beneficiaries from vulnerable groups (pregnant women, nursing women, people with reduced mobility, etc.), mothers’ club and ASCP Impact Did the project reach the expected number of beneficiaries and territorial coverage? To what extent has the project contributed to reducing beneficiaries’ vulnerability level? What are the unintended positive and negative impacts of project implementation? How satisfied are the communities with the response? Did the program require more time from the women than from men? What gender-specific issues were addressed? Did the voucher project affect the market and context in any way (did the voucher assistance have an impact on inflation?) Did the voucher aid affect food availability on the markets? How did the voucher aid affect local trade? - Quantitative survey of beneficiaries - Consultation of beneficiaries’ monitoring databases - Key interviews with project manager and/or project component managers/M&E manager - Key interviews with the project’s key partners (Local Authorities (Mayors / CASEC), MAST, MSPP, Wesleyan Hospital, Mother-leaders, ...) - Key interviews with some vendors from the voucher component and a few others not involved with the project - Focus group with beneficiaries from vulnerable groups (pregnant women, nursing women, people with reduced mobility, etc.), mothers’ club and ASCP USAID EFSP 14 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Crit eria Evaluative Questions Data Sources /Collection Methods Sustainability To what extent will project benefits continue after donor funding ends? Are positive effects sustainable? To what extent did the project take into account factors that, in experience, have a major influence on sustainability such as economic, ecological, social and cultural aspects? What sustainability drivers are obvious (local ownership, partnership, transformed relationships, household and family resilience)? - Key interviews with project manager and/or project component managers - Key interviews with key project partners (Local Authorities (Mayors / CASEC), MAST, MSPP, Wesleyan Hospital, Mother- leaders ...) - Focus group with beneficiaries from vulnerable groups (pregnant women, nursing women, people with reduced mobility, etc.), mothers’ club and ASCP Linkages, overlaps and exit strategies To what extent did the project take advantage of other US Government (USG) and non-USG investments in the same area to facilitate linkages with complementary services, overlaying previous investments and implementing exit strategies to minimize reliance on external support? To what extent has the project aligned and integrated with the host country’s service provision strategy/policy for social protection? - Consultation of government policy document related to the issues addressed by the project - Key interviews with project manager and/or project component managers/M&E manager - Key interviews with the project’s key partners (Local Authorities (Mayors / CASEC), MAST, MSPP, Wesleyan Hospital, Mother-leaders, and NGO directors implementing similar projects...) - Focus group with beneficiaries from vulnerable groups (pregnant women, nursing women, people with reduced mobility, etc.), mothers’ club and ASCP The collection tools that were developed in relation to data sources listed in the tables above have taken into account lessons and major challenges, as well as strengths, weaknesses, opportunities and threats experienced by the project during its implementation and the way they were addressed. The evaluation was then able to examine the effects or potential impacts of the project on participants and their community knowledge, attitudes and practices. 3.1. Quantitative Survey A quantitative sampling survey was conducted among project beneficiaries in the island of La Gonâve’s intervention communes, which are Anse-à-Galets and Pointe-à-Raquettes. As mentioned in the ToRs, the same sampling methodology used for the baseline was applied. Thus, a random stratified with Proportional Probability by Size (PPS) Cluster Sampling was used in each communal section, and a simple random sampling was used to select beneficiaries from the predefined list. One must say that this strategy is made possible by the existence of beneficiary lists serving as the survey basis. USAID EFSP 15 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation The structured questionnaire used in the project baseline for the collection of quantitative data is used in this study with additions taking into account the evaluative questions based on OECD criteria. The questions take into account the calculation method for project’s key indicators as presented: Key I ndicat ors D e finition of Indicator s Data Collection Method Percentage of targeted households with an acceptable food consumption score (FCS) The frequency-weighted dietary diversity score is calculated from the consumption frequency for the various food groups consumed ( Appendix 1 ) by a household during the 7 days preceding the survey . Quantitative survey of project beneficiary households Prevalence of households with little or no hunger (household hunger scale – HHS) Is essentially a behavioral measure that tends to capture more serious behaviors such as: - Was there never anything to eat in your house? For lack of resources to get food? - Did you or a member of your family fall asleep at night hungry because there was not enough food? - Did you or a member of your family go all day and night without eating anything because there was not enough food? Quantitative survey of project beneficiary households Proportion of households consuming at least 6 food groups in the previous month Household Dietary Diversity Scale (HDDS): Dietary diversity represents the number of different foods or food groups consumed in a given reference period. - Similar to the FCS, but usually with a 24-hour recall period with no information on frequency or weighted categorical thresholds - It is an indirect measure of household access to food. Quantitative survey of project beneficiary households Percentage of food use by type (household consumption, sale, exchange, livestock feed) This indicator takes into account foods generally available in the household, regardless of its origin and quantity. It analyzes the different modes of use, namely: a. Self-consumption b. The sale c. Exchange d. Livestock feed The priority use of food can inform on the economic situation of the household. Quantitative survey of project beneficiary households USAID EFSP 16 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Key Indicat or s Definition of Indicators Data Collection Methods Reduced Coping Strategies Index (rCSI) rCSI measures behavior: What people do when they don’t have access to enough food by answering the question: What do you do when you don’t have enough to eat and you don’t have enough money to buy food? - Measures the adjustments households make in consumption and livelihoods. These may be changes in consumption, reduced spending, or income growth; - rCSI tends to measure less severe coping behaviors. - rCSI uses the five most common strategies with standardized weights: 1- Focus on less preferred and less expensive foods? 2- Borrow food or rely on the help of a friend or relative? 3- Limit portion size at mealtime? 4- Restrict adult consumption so that young children can eat? 5- Reduce the number of meals consumed per day? ( Appendix III ) Quantitative household survey Percentage of target households using and/or benefiting from community assets created/rehabilitated. This indicator takes into account households living in areas using and/or benefiting from community infrastructure (stone cord, earthen dike (soil conservation works/watershed protection), etc.) that were created or rehabilitated by the project. Quantitative household surveys Percentage of targeted households reporting an increase in the amount saved because of the program. This indicator refers to the building of cash assets for the families benefiting from the project, whether through cash transfers, IGAs, agricultural production activities, remittances and S4Ts which aim to increase household income, reduce expenditure items (such as food), strengthen savings and thus promote asset growth. Quantitative household surveys Percentage of households adopting best practices in health and nutrition This indicator analyzes the understanding and application of nutritional and family health practices within households as a direct result of the program. The analyzes relate to: -For a pregnant woman, a household adopting good health and nutrition practices if it takes into consideration one or more of these aspects: 1. Iron and folic acid supplementation 2. Advice on mother and/or child nutrition 3. Calcium supplementation 4. Multiple micronutrient supplementation Quantitative household surveys USAID EFSP 17 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation 5. Direct food aid from fortified/specialized food products (i.e., CSB+, Super cereal Plus, RUTF, RUTF, etc.) - Children under 5, including one or more of these aspects: • Immediate, exclusive and continuous breastfeeding • Appropriate, adequate and safe complementary foods from 6 to 24 months • Vitamin A supplementation during the last 6 months • Zinc supplementation during episodes of diarrhea • Multiple micronutrient powder supplementation (MNP) • Treatment of severe acute malnutrition • Treatment of moderate acute malnutrition • Direct food aid from fortified/specialty food products (i.e., CSB+, Super cereal Plus, RUTF, ASPE, etc.) (Ref: FFP indicators, page 54) Key Indicat ors D e finition of I ndicat o rs M e thod of Data Collection Prevalence of Global Acute Malnutrition (GAM) To measure GAM, anthropometric measurement is taken (weight-for-height or MUAC) for children aged 0 to 59 months to calculate wasting. All children with a weight-to-weight Z score less than -2 standard deviation and/or edema are classified as acutely malnourished. Anthropometric data from a quantitative survey of a sub-sample of children aged 0 to 59 months from project beneficiaries. The indicator values found in this final study were compared with those established during the baseline study. A comparison test of means or proportions was carried out to see if there are statistically significant differences or not in order to confirm whether the project had the desired effects and impacts. a) Calculation of Sample Size To determine the size of the beneficiary sample, the calculation formula is based on the FFP/USAID protocol in that the required minimum number ( 339 per commune ) of beneficiaries to be surveyed was met for food security indicators (FCS, HHS, rCSI). One-step random sampling ( One-stage Single Random Sample (SRS)) is used just as it was at the baseline, considering each commune as a layer. The values of the indicators were established by commune. The formula for calculating the survey’s sample size is as follows: 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = 𝐷𝐷 𝑒𝑒𝑒𝑒 𝑖𝑖 ⎣ ⎢ ⎢ ⎡𝑍𝑍 1−∝ � 2 𝑃𝑃� 1 − 𝑃𝑃� + 𝑍𝑍 1−𝛽𝛽 �𝑃𝑃 1 , 𝑒𝑒𝑒𝑒 𝑖𝑖 � 1 − 𝑃𝑃 1 , 𝑒𝑒𝑒𝑒 𝑖𝑖 � + 𝑃𝑃 2 , 𝑒𝑒𝑒𝑒 𝑖𝑖 � 1 − 𝑃𝑃 2 , 𝑒𝑒𝑒𝑒 𝑖𝑖 � 𝛿𝛿 ⎦ ⎥ ⎥ ⎤ 2 Where 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = is the initial sample size required by surveys for each one of the two-stage points 𝛿𝛿 = 𝑃𝑃 1 , 𝑒𝑒𝑒𝑒 𝑖𝑖 - 𝑃𝑃 2 , 𝑒𝑒𝑒𝑒 𝑖𝑖 = minimum effect size to be achieved over the period specified by both surveys USAID EFSP 18 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation - The calculation of the sample size for Anse-à-Galets 𝑷𝑷 𝟏𝟏 , 𝒆𝒆𝒆𝒆𝒆𝒆 = 0.27 proportion of the population with a less severe rCSI at baseline . FCS is chosen over the other 2 (namely the FCS and HHS) as the one that yielded the most suitable sample, although insufficient for the measurement of food insecurity indicators. 𝑷𝑷 𝟐𝟐 , 𝒆𝒆𝒆𝒆𝒆𝒆 = 0.37 proportion of the population with a less serious rCSI in Anse-à-Galets in the 3rd MDP carried out in March 2021. 𝑃𝑃 = 𝑃𝑃 1 , 𝑒𝑒𝑒𝑒 𝑖𝑖 + 𝑃𝑃 2 , 𝑒𝑒𝑒𝑒 𝑖𝑖 2 = 0.27 + 0.37 2 = 0.32 𝑍𝑍 1−∝ is the value of the normal probability distribution corresponding to a confidence level of 1- 𝛽𝛽 . For 1- 𝛽𝛽 =0.95, the corresponding value is 𝑍𝑍 0 , 95 = 1,64. 𝑍𝑍 1−𝛽𝛽 is the value of the normal probability distribution corresponding to a confidence level of 1-. 𝛽𝛽 . For 1- 𝛽𝛽 =0.80, the corresponding value is 𝑍𝑍 0 , 80 = 0,84. 𝐷𝐷 𝑒𝑒𝑒𝑒 𝑖𝑖 : It is the estimated design effect (DEFF) of the survey which is 1 in this sampling plan ( One-stage Simple Random Sample (SRS) ). Hence 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = 1 ∗ � 1.64 ∗ � 2 ∗ 0.32(1 − 0.32) + 0.84 ∗ � 0.27(1 − 0.27) + 0.37(1 − 0.37) 0.27 − 0.37 � 2 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = 267 There is indeed a non-response rate (generally set at 1.1, corresponding to 10%, but may change depending on the context). 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 267 ∗ 1.1 = 294 𝑏𝑏𝑏𝑏 𝑛𝑛 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 - The calculation of the sample size for Pointe-à-Raquettes 𝑷𝑷 𝟏𝟏 , 𝒆𝒆𝒆𝒆𝒆𝒆 = 0.12 proportion of the population with a less severe rCSI in baseline . 𝑷𝑷 𝟐𝟐 , 𝒆𝒆𝒆𝒆𝒆𝒆 = 0.27 proportion of the population with a less severe rCSI in Anse-à-Galets in 3rd MDP carried out in March 2021. 𝑃𝑃 = 𝑃𝑃 1 , 𝑒𝑒𝑒𝑒 𝑖𝑖 + 𝑃𝑃 2 , 𝑒𝑒𝑒𝑒 𝑖𝑖 2 = 0.12 + 0.27 2 = 0.195 𝑍𝑍 1−∝ is the value of the normal probability distribution corresponding to a confidence level 1- 𝛽𝛽 . For 1- 𝛽𝛽 =0,95, the corresponding value is 𝑍𝑍 0 , 95 = 1,64. USAID EFSP 19 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation 𝑍𝑍 1−𝛽𝛽 is the value of the normal probability distribution corresponding to a confidence level 1- 𝛽𝛽 . For 1- 𝛽𝛽 =0,80, the corresponding value is 𝑍𝑍 0 , 80 = 0,84. 𝐷𝐷 𝑒𝑒𝑒𝑒 𝑖𝑖 : It is the estimated design effect (DEFF) of the survey which is 1 in this sampling plan ( One-stage Simple Random Sample (SRS) ). Hence 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = 1 ∗ � 1.64 ∗ � 2 ∗ 0.195(1 − 0.195) + 0.84 ∗ � 0.12(1 − 0.12) + 0.27(1 − 0.27) 0.12 − 0.27 � 2 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = 85 There is indeed a non-response rate (generally set at 1.1 corresponding to 10% but may change depending on the context). 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 85 ∗ 1.1 = 94 𝑏𝑏𝑏𝑏 𝑛𝑛 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 As for each commune, to be able to measure FCS, HHS, rCSI in this final evaluation, we need at least 339 units, therefore the sample size per commune is adjusted to 339. So, overall we should survey on the island of La Gonâve at least 678 beneficiary households in the two project intervention communes in this final evaluation. The following table breaks down the total number of beneficiaries by commune, communal section (including target communities): Table 1 : Number of beneficiaries by commune and communal section Communes Communal Sections/Town Target Communities Number of Beneficiaries Anse-à-Galets Ville Ville 875 1 st Palma Palma 168 2 nd Petite Source Fortuna, Mare Sucrin, Port Frégard 566 3 rd Grande Source Bois Brûlé, Bois Noir, Les Étroits, Nan Café, Plaine Mapou, Grande Source 1063 4 th Grand Lagon Trou Louis Jeune, Zabricot 866 5 th Picmy Picmy 97 6 th Section Petite Anse Petite Anse 164 Total 3799 Communes Communal Sections /Town Target Commun iti es N umber of Bene ficiar i es Pointe-à-Raquettes 1 st Section la Source Plaine La Source, Latanier 84 2 nd Section Grand Vide Grand vide, Tamarin 456 USAID EFSP 20 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation 3 rd Section Trou Louis Dan Griyen, Deux Frères, Morne Trou Louis, Port-de-Bonheur, Port- Trou-Louis 621 4 th Section Pointe-à-Raquettes La Palmiste, Lotoré, Plaisance, Terre Sèche, Ti Palmiste, Bois Pin 539 5 th Section Gros Mangle Bel Platon, Gros Mangles 134 Town Town of Pointe-à-Raquettes 137 Total 1971 Grand total 5770 By considering the lists of beneficiaries by commune as the sampling basis, the sample of households to be surveyed was drawn according to a simple random selection by grouping the target communities by communal section. The a léa () function in MS Excel was used in the lists to select households to be surveyed. This approach ensures a geographical coverage more representative of the beneficiaries’ location. To access the randomly selected households, it was expected that GPS coordinates be used to locate them. However, these not being correct, we had to rely on guides who know the communities well to identify the randomly selected households. The table below shows the allocation of households to be surveyed by communal section/town using the Probability Proportional to Size (PPS) method: Table 2: Breakdown of the number of beneficiary households to be surveyed in Anse-à-Galets by communal section Communal Sections/Town Target Communities Number of Beneficiaries Demograph ic Weight Sample allocation by communal section Number of surveys actually carried out Town Ville 875 0.23 78 78 1 st Palma Palma 168 0.04 15 16 2 nd Petite Source Fortuna, Mare Sucrin, Port Frégard 566 0.15 51 55 3 rd Grande Source Bois Brûlé, Bois Noir, Les Étroits, Nan Café, Plaine Mapou, Grande Source 1063 0.28 95 95 4 th Grand Lagon Trou Louis Jeune, Zabricot 866 0.23 77 77 5 th Picmy Picmy 97 0.03 9 14 6 th Section Petite Anse Petite Anse 164 0.04 15 30 3799 1.00 339 365 USAID EFSP 21 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Table 3: Breakdown of the number of beneficiary households to be surveyed in Pointe-à-Raquettes by communal section Communal Sections/Town Target Communities Number of Beneficiaries Demographic Weight Sample allocation by communal section Number of surveys actually carried out 1 st Section la Source Plaine La Source, Latanier 84 0.04 14 10 2 nd Section Grand Vide Grand Vide, Tamarin 456 0.23 78 80 3 rd Section Trou Louis Dan Griyen, Deux Frères, Morne Trou Louis, Port-de- Bonheur, Port-Trou-Louis 621 0.32 107 114 4 th Section Pointe-à- Raquettes La Palmiste, Lotore, Plaisance, Terre Sèche, Ti Palmiste, Bois Pin 539 0.27 93 94 5 th Section Gros Mangle Bel Platon, Gros Mangles 134 0.07 23 34 Town Town of Pointe-à-Raquettes 137 0.07 24 29 1971 1.00 339 361 Note: 726 beneficiaries were surveyed instead of the planned minimum quantity 678. This surplus is explained by the fact that each interviewer was asked to interview 3 to 5 respondents as a precaution. In fact, when clearing and processing the data, the research team may need to eliminate forms deemed to be inconsistent. To measure the GAM indicator (Prevalence of Global Acute Malnutrition), one of the 3 approaches proposed by the FFP protocol was used, in this case approach B. We looked at the official statistics for the West department on the proportion of the population of children under 5 years old. This proportion is 0.12 (Statistical Report, MSPP, November 2019) and the respective average household sizes for Anse-à-Galets and Pointe-à-Raquettes are 2.2 and 2.4 ( according to the list of beneficiaries by commune ). Using these parameters for each commune in the USAID’s sample size calculator (USAID, population-based survey sample size calculator) 5 , we then have the respective minimum sizes of children under 5 to be considered for the measurement of the GAM indicator in each of the communes: - n (Anse-à-Galets) = 109 children from 0 to 59 months - n (Pointe-à-Raquettes) = 118 children from 0 to 59 months These samples of children under 5 years were sought among the samples of surveyed households. It should be noted that, within the same household, more than one child under 5 could be considered. So the samples of children were broken down according to the demographic weight of households by commune. The following tables break down the number of children to be considered to measure the GAM by commune: 5 https://www.usaid.gov/documents/1866/population-based-survey-sample-size-calculator USAID EFSP 22 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation Table 4: Breakdown of the sample of children aged 0-59 months to be surveyed in Anse-à-Galets Communal Sections/Town Target Communities Demographic Weight Allocation of less than 5 years old samples by communal section Number of children less than 5 actually observed Town Town 0.23 25 30 1 st P alma Palma 0.04 5 7 2 nd Petite Source Fortuna, Mare Sucrin, Port Frégard 0.15 16 25 3 rd Grande Source Bois Brûlé, Bois Noir, Les Étroits, Nan Café, Plaine Mapou, Grande Source 0.28 30 60 4 th Grand Lagon Trou Louis Jeune, Zabricot 0.23 25 39 5 th Picmy Picmy 0.03 3 12 6 th Section Petite Anse Petite Anse 0.04 5 13 Total 1.00 109 196 Table 5: Breakdown of the sample of children aged 0-59 months to be surveyed in Pointe-à-Raquettes Communal Sections/Town Target Communities Number of Beneficiaries Demographic Weight Allocation of less than 5 years old samples by communal section Number of children less than 5 actually observed 1 st Section la Source Plaine La Source, Latanier 84 0.04 5 9 2 nd Section Grand Vide Grand Vide, Tamarin 456 0.23 27 66 3 rd Section Trou Louis Dan Griyen, Deux Frères, Morne Trou Louis, Port-de- Bonheur, Port-Trou- Louis 621 0.32 37 76 4 th Section Pointe- à-Raquettes La Palmiste, Lotoré, Plaisance, Terre Sèche, Ti Palmiste, Bois Pin 539 0.27 32 48 5 th Section Gros Mangle Bel Platon, Gros Mangles 134 0.07 8 19 Town Town of Pointe-à- Raquettes 137 0.07 8 16 Total 1971 1.00 118 234 b) Processing and analysis of quantitative data The processing and analysis of quantitative data helped respond mainly to issues of project effectiveness, impact, sustainability and relevance and also check whether there are unexpected outcomes. Moreover, the indicator values were established in this final study. The values USAID EFSP 23 Initial Report of the WV Haiti La Gonâve EFSP Final Evaluation found were compared with those established during the study baseline. A comparison test of means or proportions was performed on MS Excel to see whether there are statistically significant differences or not to actually confirm if the program reached the desired effects and impact as mentioned in section 3.1. Here are the steps followed in implementing the test: 1) H0: P1=P2, there is no significant difference between both values H1: P1 ≠P2 , there is a significant difference between both values P1 : indicator value at baseline P2 : indicator value during the final study 2) Materiality threshold : α=5% 3) Conditions to implement the test : case of large samples n1>30 and n2 >30 4) The test statistics are : �𝑃𝑃 �1−P �2�− ( p1−p2 ) � 𝑝𝑝1𝑞𝑞1 𝑛𝑛1 + 𝑝𝑝2𝑞𝑞2 𝑛𝑛2 ~ N(0,1) (follows a centered and reduced normal). 5) Decision rule : if the test statistic is greater than 1.96 (Table value of N (0.1) for a risk α = 5%), then no rejection of the null hypothesis, i.e. H0 . Or we use the pvalue by comparing it with the risk α = 5%. A pvalue> α = 5% => DH0 (non -rejection of the null hypothesis). In EXCEL, the pvalue is obtained using the function: NORMSDIST (test statistic) 6) Decision and conclusion: We can therefore conclude, with a risk of error α = 5%, that the proportion of the final evaluation is not significantly different from that of the baseline. The difference observed between the two is due to sampling fluctuations. In order to have the best possible data quality, technological tools were used to guarantee the validity, reliability, integrity, accuracy and availability of the information collected. In the case of quantitative data, the Ona digital survey platform was used. The survey questionnaire was designed using the XLSForm syntax 6 and then uploaded to the platform. Each enumerator was therefore provided with an Android digital tablet on which the data collection tool was installed. Once the collected data was cleaned and validated, the analysis phase was launched. The main data analysis tools that were used during this phase are the SPSS software 7 and MS Excel. 3.2. Qualitative Survey The qualitative approach, in the final evaluation, was essential not only to complete the quantitative data collected from project beneficiaries but also to validate the findings of the study through information triangulation. It is also imp