(2022) Results-Based Financing (RBF) in Haiti’s Health System: Impact Evaluation Report

Report No: AUS0002763 RESULTS-BASED FINANCING (RBF) IN HAITI’S HEALTH SYSTEM Impact Evaluation Report Health, Nutrition and Population Global Practice April 21, 2022 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized © 2017 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: “World Bank. 2022. Results-based Financing in Haiti’s Health System. © World Bank.” All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. i TABLE OF CONTENTS EXECUTIVE SUMMARY ____________________________________________________________ 1 1. BACKGROUND ______________________________________________________________ 5 Background to the study ______________________________________________________ 5 1.2 Objectives and Research Questions Related to Impact Evaluation _____________________ 8 2. METHODOLOGY OF THE STUDY _______________________________________________ 9 Experimental Approach ______________________________________________________ 9 2.1.1 Implementation of the Interventions __________________________________________ 12 2.2 Data Sources: _____________________________________________________________ 15 2.2.1 Household surveys ______________________________________________________ 15 2.2.2 Surveys of health facilities _________________________________________________ 16 2.3 RBF Impact Evaluation Indicators _____________________________________________ 19 2.4 Statistical Methods _________________________________________________________ 20 2.4.1 Balance of the study groups for the baseline survey _____________________________ 20 2.4.2 Econometric models______________________________________________________ 24 3. IMPACT EVALUATION RESULTS ______________________________________________ 26 3.1 Impacts on the Governance of Health Facilities ___________________________________ 26 3.2 Impacts on the Quality of Health Services _______________________________________ 27 3.2.1 Basic Infrastructure at Health Facilities _______________________________________ 27 3.2.2 Availability of Health Services ______________________________________________ 28 3.2.3 Amenities and Basic Care _________________________________________________ 29 3.2.4 Malaria Care Services ____________________________________________________ 31 3.2.5 Medical Equipment and Supplies ____________________________________________ 31 3.2.6 Prenatal Care ___________________________________________________________ 34 3.2.7 Under-five Child Health Care _______________________________________________ 35 3.3 Impacts on Human Resources ________________________________________________ 36 3.4 Impacts on the Use of Reproductive Health Services _______________________________ 39 3.4.1 Family Planning Behaviors _________________________________________________ 39 3.4.2 Prenatal Care Behaviors __________________________________________________ 40 3.4.3 Delivery and Postnatal Care Behaviors _______________________________________ 41 3.5 Impacts on the Use of Child Preventive Health Services ____________________________ 42 3.6 Impacts on the Health Status of the Population ___________________________________ 43 4. DISCUSSION _______________________________________________________________ 48 4.1 Children’s health ___________________________________________________________ 49 4.2 Coverage and quality of priority health services related to maternal and child health _______ 49 4.3 Health system in Haiti: governance, human resources and health financing _____________ 50 4.4 Health expenditure _________________________________________________________ 50 4.5 Technical assistance, coverage and quality of health services ________________________ 51 4.6 Summary comments ________________________________________________________ 52 ii 4.7 Study limitations ___________________________________________________________ 52 5. CONCLUSION ______________________________________________________________ 54 6. BIBLIOGRAPHIC REFERENCES _______________________________________________ 56 Annex A: Basic Statistics for Different Sub-Samples __________________________________ 59 Annex B: Household Healthcare Seeking Behavior Analysis ____________________________ 68 Annex C: Qualitative Study on Findings from Quantitative Impact Evaluation of the Results- Based Financing Program (RBF) in Haiti _____________________________________________ 72 iii TABLES Table 1: Key SDG for Maternal and Child Health Services ...................................................................... 8 Table 2: Mechanisms by Which RBF Can Potentially Work .................................................................... 9 Table 3: Summary of Sample by Intervention Arm ................................................................................ 10 Table 4: Tariffs (in HTG) of Indicators Incentivized Under the RBF Program in Haiti † .......................... 13 Table 5: Characteristics of Households Surveys ................................................................................... 15 Table 6: Characteristics of Health Facility Surveys ................................................................................ 18 Table 7: Geographic Distribution of Health Facilities Included in the IE and Surveyed During the Two Periods of the Study .............................................................................................................................. 18 Table 8: List of RBF IE Variables ........................................................................................................... 19 Table 9: Characteristics of Households and Individuals in the Baseline Study Sample ......................... 21 Table 10: Characteristics of Health Facilities in the Baseline Study Sample ......................................... 23 Table 11: Provision of Health Services in the Baseline Study ............................................................... 24 Table 12: Impacts of RBF on Administration and Management of Health Facilities † ............................ 27 Table 13: Impacts of RBF Initiatives on Basic Infrastructure at Health Facilities † ................................ 28 Table 14: Impacts of RBF on the Availability of Health Services at the Health Facility † ....................... 29 Table 15: Impacts of RBF on the Quality of Health Services Provided at Health Facilities † ................. 30 Table 16: Impacts of RBF on Malaria Care Services at Health Facilities † ............................................ 31 Table 17: Impacts of RBF on the Provision of Medical Equipment and Supplies to Health Facilities † . 33 Table 18: Impacts of RBF on the Quality of ANC Consultations and Related Satisfaction † ................. 35 Table 19: Impacts of RBF on the Quality of Consultations for Children Under Five and the Related Levels of Satisfaction † .......................................................................................................................... 36 Table 20: Impacts of RBF on the Delivery of Health Care Services in Health Facilities † ...................... 37 Table 21: Impacts of RBF on the Wages and Motivation of Health Care Personnel in Health Facilities † .............................................................................................................................................................. 38 Table 22: Impacts of RBF on the Level of Satisfaction of Health Care Personnel in Health Facilities † 38 Table 23: Impacts of RBF on the Reproductive Health Behaviors of Women Between Ages of 15 and 49 † ....................................................................................................................................................... 39 Table 24: Impacts of RBF on the Use of Prenatal Care Services Among Recently Pregnant Women Between the Ages of 15 and 49 (24 Months Preceding Data Collection) † ........................................... 40 Table 25: Impacts of RBF on the Use of Obstetric and Postnatal Care Services Among Recently Pregnant Women Between the Ages of 15 and 49 (24 Months Preceding Data Collection) † ............... 42 Table 26: Impacts of RBF on the Immunization of Children Under Five † ............................................. 43 Table 27: Impact of RBF on the Nutritional Status of Children Between the Ages of 6 and 59 Months † .............................................................................................................................................................. 44 Table 28: Impacts of RBF on Household Health Expenditure † ............................................................. 45 Table 29: Summary of RBF IE Results in Haiti ...................................................................................... 45 FIGURES Figure 1: Health indicators in Haiti and SDG targets, 2015 ..................................................................... 5 Figure 2: Location of the different HFs with interventions (RBF and TA Groups) and the Control Group facilities of the RBF program in Haiti ...................................................................................................... 12 Figure 3: Evolution of Total RBF Payments (in US Dollars) to RBF Health Facilities Over the Period 2016-2019 ............................................................................................................................................. 34 iv ABBREVIATIONS AND ACRONYMS AEV External Verification Agency (Agence Externe de Vérification) ANC Antenatal Care BCG Bacille Calmette Guerin (Tuberculosis Vaccine) CAL Centre de Santé Avec Lit (Health Center With Bed) CHE Catastrophic Health Expenditure CHW Community Health Worker CSL Centre de Santé Sans Lit (Health Center Without Bed) DDSs Departmental Health Directorates DTP Diphtheria, Tetanus and Pertussis ECVMAS Survey of Household Living Conditions After the Earthquake (Enquête sur les Conditions de Vie des Ménages Après le Séisme) EMMUS Mortality, Morbidity and Use of Health Services Survey (Enquête sur la Mortalité, la Morbidité et l'Utilisation des services de Santé) FP Family Planning HCR Community Reference Hospital (Hôpitaux Communautaires de Reference) HF Health Facility HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HTG Haitian Gourdes HRITF Health Results Innovation Trust Fund IE Impact Evaluation IMCI Integrated Management of Childhood Illness LIC Low-Income Country MMR Maternal Mortality Ratio in Haiti MSPP Ministry of Public Health and Population (Ministère de la Santé Publique et de la Population) NGOs Non-governmental Organizations OPV Oral Poliovirus Vaccine PNS National Health Policy (Politique Nationale de Santé) RBF Results-Based Financing RDT Rapid Diagnostic Test SDGs Sustainable Development Goals SSQH Quality Health Services for Haiti (Service de Santé de Qualité Pour Haïti) U5MR Under-Five Mortality Rate USAID United States Agency for International Development WB World Bank WHO World Health Organization 1 EXECUTIVE SUMMARY This report is about the main results of the impact assessment conducted for the Result-Based Financing (RBF) of Haiti’s health system. The decisions on the design of this assessment have been taken by the Ministry of Public Health and the World Bank. Data collection for this impact assessment was undertaken by Haitian companies, supported by a technical team of the World Bank. The analysis of the data and the drafting of this report was undertaken by a World Bank technical team. As a country of Latin America and the Caribbean, Haiti is still far behind on regional averages for many health indicators, particularly those related to maternal, newborn and child health. Despite the efforts that have been made in recent years, many challenges remain in trying to reach the Sustainable Development Goals as far as health is concerned. To address this situation, Haiti designated the implementation of its RBF program as a strategic axis of its National Health Policy (Politique Nationale de Santé, PNS) in order to improve both the supply and demand for health. The country had had a positive experience with RBF between 1999 and 2001, with the Non-Governmental Organization Management Sciences for Health, in certain Departments. This led the country to develop the current RBF model, and to launch it in 2014 with a pilot involving a limited number of health facilities (HFs) in the Northeast Department. One year later, the RBF strategy was scaled up to 7 other Departments, with many more HFs included. A quasi-experimental study was implemented to assess the impact of this program on primary health care outcomes and on the functioning of Haiti’s health system. For this purpose, two packages of interventions (RBF with technical assistance, and technical assistance only (TA)), and three groups under the study (RBF, TA only and Control), were established. The study covered 170 HFs divided into three groups. Data was collected for the entire sample of HFs as well as for households in localities served by these HFs (with one locality chosen randomly for every HF, among all areas that it covers). In every locality, interviews were conducted with 28 randomly chosen households with at least one woman pregnant at the time of the survey or one woman that has interrupted her pregnancy within the last 24 months preceding the survey. Data collected in 2015 (baseline survey) and in 2019 (endline survey) were used to assess impacts by using the approach of difference in differences, in accordance with the Health Results Innovation Trust Fund (HRITF) standard strategy for impact evaluations concerning RBF programs. The following table summarizes the main findings. Summary of the Results of the Impact Evaluation of the RBF Program in Haiti INDICATORS RELATED TO: IMPACTS OF RBF (RELATIVE TO CONTROL GROUP) IMPACTS OF TA (RELATIVE TO CONTROL GROUP) COMMENTS ON IMPACT OF RBF (RELATIVE TO CONTROL GROUP) Administration and management of HFs No impact No impact These results could be explained by the fact that these indicators were not among the quality indicators incentivized under the RBF program in Haiti. Basic infrastructure of HFs Significant positive impact on the availability of a medical evacuation system. No impact on availability of permanent drinking water and electricity. No impact Note that the indicators on the availability of permanent drinking water and electricity were introduced quite late during the implementation of the RBF program. Availability of different types of health care services at HFs No impact No impact For antenatal care and immunizations, the lack of significant impact could be explained by the high proportion in the baseline study of HFs that were already offering these services, controlled health services that already had those services. In addition, note that in practice, if a HF wants to start offering a new type of health service, it would require support (to ensure provision of the needed human resources, inputs, technical assistance, etc.) from higher-level decision-making authorities, and this is not always 2 easy to obtain. Hence, in practice health workers at HFs have little control over this set of indicators. Quality of services offered at HFs Significant and positive impact on adherence to universal protocols, capacity to conduct lab tests, availability of vaccines and availability of medicines in stock No impact Various types of equipment at HFs Significant and positive impact on the availability of general equipment, as well as equipment for immunizations, and for deliveries and postnatal care. No impact on the availability of equipment for sterilization and antenatal care. No impact The lack of significant impact on equipment for sterilization and antenatal care could be explained by the high costs of these types of equipment. RBF payments – especially at the beginning of the implementation period when RBF tariffs were substantially lower (until the first quarter of 2018) – were too low to allow HFs to purchase these types of more expensive equipment. Quality of antenatal consultations and related satisfaction Significant and positive impact on patient education (during the antenatal consultations) as well as on pregnant women’s satisfaction regarding the quality and cost of these consultations. Significant and positive impact on pregnant women’s satisfaction regarding the quality of antenatal consultations (in rural areas) and regarding their cost (in overall sample). Quality of consultations for children aged under 5 years and related satisfaction Significant and positive impact on satisfaction regarding the cost of children’s consultations. No impact on various measures of the quality of children’s consultations. No impact The lack of impact on the quality of these consultations could be explained by the fact that these indicators were not incentivized under the RBF program in Haiti. Size of workforce and number of hours worked No impact Significant and positive impact on the numbers of health workers and negative for the hours of work (the expected direction) The lack of impact could be explained by the fact that these indicators were not directly incentivized under the RBF program in Haiti. Salaries and motivation of health care staff Significant and positive impact on the regularity of salary payments of health workers. No impact on personal motivation of health workers Significant and positive impact on personal motivation of health workers, in urban areas It should be noted that these indicators were not directly incentivized under the RBF program. Satisfaction of health workers Significant and positive impact on satisfaction of health workers regarding their work environment and regarding their management and supervision at HFs (in rural areas for the latter). But no impact on health workers’ satisfaction regarding their salaries. No impact It should be noted that indicators related to health workers’ satisfaction were not directly incentivized under the RBF program. Reproductive behavior amongst women aged between 15-49 years. No impact Significant and positive impact on unmet family planning needs; no impact on the utilization of contraceptive methods. According to qualitative work carried out in 2021 in conjunction with the quantitative analysis, some cultural attitudes, preconceptions and low education levels still constitute barriers to the use of contraceptive methods in Haiti, especially modern methods. Antenatal consultations by women aged 15-49 years who were recently pregnant Significant and positive impact on having 4 antenatal consultations and on having at least one antenatal consultation in the first first 4 months of pregnancy (in rural areas). No impact on having at least one antenatal consultation and on having at least two doses of antitetanus vaccine during pregnancy. Significant and negative impact (at 10% level of significance) on having at least one antenatal consultation (in urban areas only). Significant and positive impact on having at least two doses of antitetanus vaccine during pregnancy (in urban areas only). The lack of significant impact (in the RBF group) on having at least one antenatal consultation could be explained by the high degree of attainment of this indicator that was already observed at the time of the baseline study (81.2% in the overall sample). Obstetric and postnatal care amongst women aged between 15-49 years and recently pregnant. Significant and positive impact on assisted deliveries and on deliveries at HFs (in rural areas). No impact on postnatal consultations by women at HFs and on iron supplementation after delivery. Significant and positive impact on the iron supplementation after delivery. Postnatal consultations as defined under this study (i.e. postnatal consultations by pregnant women at HFs) were not incentivized under the program. Instead, the related indicator incentivized under the program was the number of postnatal visits by Community Health Agents to the home of a woman who recently gave birth, within 3 days of the delivery. Immunization of young children Significant and positive impact on immunization coverage of children aged between 12-23 months. Significant and positive impact on immunization coverage of children aged between 12-23 months in urban areas. Nutritional status of children aged between 0-59 months. Significant and negative impact on the proportion of children that suffer from underweight (improvement of nutritional status), in rural areas. No impact on stunting and wasting. Significant and positive impact on stunting (deterioration of nutritional status). No impact on the proportion of children suffering from underweight and wasting. These indicators were not directly incentivized under the RBF program. (The related indicator directly incentivized under the RBF program was the number of children that underwent nutritional screening.) However, these indicators could have been influenced by other indicators directly incentivized under the RBF program, and likely by other factors operating over several years. Health care expenses of households (financial protection) Significant and negative impact on health care expenses (improved financial protection). No impact It should be noted that indicators of health care expenses were not directly incentivized under the RBF program. The results of this assessment show that in general, RBF interventions together with technical assistance improved several indicators of coverage and the quality of maternal and child health in Haiti, relative to the Control Group. With this approach, significant improvements (relative to the Control Group) were seen with immunization coverage and the nutritional status of children; availability of basic infrastructure as well as of inputs and equipment; adherence to universal protocols; capacity to realize lab tests; quality of antenatal 3 consultations; and confidence by pregnant women to have deliveries at HFs – especially in rural areas. All of these tend to improve the health conditions of communities through better quality of health services, hence guaranteeing better financial protection of households because of reduced health expenses (which improved due to RBF relative to the Control Group). Although indicators relatively to governance and human resources were not directly incentivized under the RBF program, Haiti’s health system generally showed relative improvement for these indicators – with higher health worker satisfaction due to RBF (relative to the control group) with the work environment, and with management and supervision at HFs, and with the regularity of salary payments. An important feature of the analysis is that for some indicators where RBF was found to have a positive impact relative to the Control Group, there was sometimes an underlying trend of deterioration to begin with – which can be expected given the context of the worsening economic and security situation in Haiti. In these cases, there was a deterioration in the indicator in all groups, but the deterioration was significantly less in the RBF areas than in the Control Group areas. An example of this type of indicator is immunization coverage. (In some cases, RBF led to a reversal of the negative trend, so that improvements were seen in the RBF areas while a deterioration was seen in the Control Group areas, but this was not as common.) There were also indicators such as assisted deliveries in rural areas, where RBF was found to have a positive impact relative to the Control Group and there was an improving time trend to begin with. In these cases, improvements were seen in both the RBF areas and the Control Group areas, but the former were significantly larger than the latter. In addition to the above, the study reveals that technical assistance alone is not sufficient to improve the Haitian health system. Its impact on the coverage of quality health services was found to be mixed. Under technical assistance alone, improvements were seen with the following relative to the Control Group: (i) unmet family planning needs of women; (ii) women having at least two doses of anti-tetanus vaccine during pregnancy (in urban areas); (iii) iron supplementation after birth; (iii) satisfaction by pregnant women with the quality of antenatal consultations (in rural areas) and their cost; (iv) the size of the workforce and the number of hours worked by health workers; and (v) health worker motivation (in urban areas). On the other hand, deteriorations (relative to the Control Group) were observed with women having at least one antenatal consultation (in urban areas), and on children’s nutritional status (stunting), and no impacts were observed on governance, availability and quality of health care services, availability of basic equipment and infrastructures, the quality of children’s consultations, overall satisfaction of health workers and financial protection of households. The fact that health worker motivation improved in urban areas due to technical assistance only (relative to the Control Group), but not under RBF, could potentially be explained by the following two factors, based on consultations undertaken by the team: First, health workers in the group that had technical assistance only may have believed that they would soon start receiving RBF payments – which is what happened in practice after the evaluation period – which may have been a motivating factor for them. At the same time, there were significant delays in RBF payments to HFs, which worsened over time due to the deteriorating security situation (which delayed the verification process), and this could have been a demotivating factor. Some limitations to the present study have been identified, which are useful to note for follow up analysis. Key among these are: (i) the fact that there were many other ongoing health interventions not covered by the RBF program in the areas covered by the study, which could not all be controlled for (although major ones 4 were identified and addressed during the analysis); (ii) the non-controlled effects of the economic and security crisis that the country had to face during the implementation of the program; (iii) contagion bias in the household data (explained in Annex B); and (iv) the fact that this study does not allow a comparison with other interventions on the supply-side (e.g., direct facility financing) or the demand-side (e.g., cash transfers or maternal vouchers) that would have similarly increased the budgets of facilities or households but without using the specific RBF mechanisms. Additional research is necessary to evaluate to what extent the results of the present study may change if these effects are addressed. In addition, additional qualitative work is necessary to better understand some of the mechanisms by which RBF has worked in Haiti, going beyond what has already been done (see Annex C). Based on the findings of this study, the following are key recommendations made:  Consider introducing incentives specifically for consultations for under-five children (with appropriate protocols to be followed). The present Impact Evaluation exercise found that the RBF program in Haiti had no impact on the quality of child consultations, which is not surprising since this was not incentivized in the case of Haiti. Given the importance of basic child health services in any health system, it is suggested that this is explicitly included and incentivized in Haiti’s RBF program, as in the case of many RBF programs worldwide.  For indicators incentivized under the RBF program over which HF workers and administrators have little control – such as rapid diagnostic test (RDT) kits and the availability of certain types of services at a facility – it is advisable to reconsider whether these should indeed be incentivized under the program.  Make efforts to substantially reduce the delays in making RBF payments to HFs. These delays have been significant in the case of Haiti, especially towards the end of the IE implementation period, when security and other issues led to substantial delays in the verification process. The qualitative analysis found that these delays have substantially affected performance. One way of reducing these delays is to make advance payments to HFs for each quarter even before the verification process for that quarter has been completed – especially in the case of delays in the verification – which would be adjusted accordingly after the verification is completed.  Test out one or two small RBF pilots in urban areas, with modifications to see if the impact from RBF can be increased there. Currently, the impact from RBF has been significant in rural areas for most variables, but less so in urban areas. 1  Introduce RBF at the community level, initially on a pilot basis. This may be particularly helpful in the case of the family planning indicators, where RBF does not seem to have a statistically significant impact. The qualitative work done – see Annex C – indicates that actions to stimulate behavior change are key here, and this is where community-level actions by Community Health Workers (CHWs) could be especially helpful. These would be incentivized under RBF at the community level. RBF at the community level may also help boost indicators in urban areas. 1 It is possible that this is due to the smaller sample size in urban areas. It is possible that the impacts in urban areas – where the relevant coefficients in the regressions have been generally of the right sign but not statistically significant – would have been statistically significant with a larger sample size. However, a priori the evidence so far indicates substantially lower impact in urban areas. 5 1. BACKGROUND Background to the study The Republic of Haiti covers an area of 27,750 km 2 , subdivided into the following 10 Departments: Ouest, Sud, Sud-Est, Grand’Anse, Nippes, Nord, Nord-Ouest, Nord-Est, Centre, and Artibonite, as well as 140 communes and 570 communal sections. In 2017, the population stood at 11,085,919, with a male ratio of 0.98, reflecting a slightly higher number of women in the total population (5,486,970 men versus 5,598,949 women). The Haitian population is very young—the median age is 23 and life expectancy is 62.2 years. It is estimated that 35.9 percent of the total population is under age 15. In 2017, there were 278,862 live births and 97,000 deaths. The annual growth rate for the period 2017-2018 is estimated at 1.37 percent. Economically, a large percentage of the Haitian population is poor. While noting a decline in monetary and multidimensional poverty rates since 2000, the World Bank (WB) Report established that in 2012, more than one in two Haitians was poor, living on less than $2.41 a day, and one in four persons was living below the national extreme poverty line of $1.23 a day 2 . Urban areas have, in relative terms, fared better than rural areas, owing to nonagricultural employment opportunities, private transfers, greater access to critical goods and services, and narrowing inequality. The performance of the health system is poor, with high morbidity and mortality rates. Despite the progress made by Haiti in achieving the Millennium Development Goals, many challenges remain with achievement of the new Sustainable Development Goals (SDGs), especially for the poorest. Between 1990 and 2015, the maternal mortality ratio (MMR) in Haiti fell from 670 deaths per 100,000 live births to 359 deaths per 100,000 live births. The infant mortality rate and the under-five mortality rate (U5MR) are 59 percent and 81 percent, respectively EMMUS, 2017 (Mortality, Morbidity and Use of Health Services Survey (Enquête sur la Mortalité, la Morbidité et l'Utilisation des services de Santé). These figures still fall well short of the SDG targets of reducing the MMR to fewer than 70 maternal deaths per 100,000 live births and the U5MR to 25 deaths per 1,000 live births by 2030 (Figure 1). Figure 1: Health indicators in Haiti and SDG targets, 2015 Maternal Mortality Ratio Under-five Mortality Rate Sources: WHO 2016; EMMUS 2000, 2005–06, 2012 2 https://www.worldbank.org/en/topic/poverty/publication/beyond-poverty-haiti 0 100 200 300 400 500 600 700 800 1990 2000 2013 2015SDG 2030 0 20 40 60 80 100 120 140 160 1990200020132015SDG 2030 6 According to various EMMUS 3 , trends in the nutritional status of children under age 5 do not seem to be improving in a regular and sustained manner over time. After remaining stable (29 percent) between 2000 and 2006, stunting among children fell to 22 percent in 2012 and in 2016-2017. Wasting and underweight figures seem to have fluctuated during the same period—the percentage of wasting among children doubled between 2000 and 2006 (from 5 percent in 2000 to 10 percent in 2006) before falling again to 5 percent in 2012 and to 4 percent in 2016-2017. The percentage of underweight children increased from 14 percent in 2000 to 18 percent in 2006 before falling to 11 percent in 2012 and to 10 percent in 2016-2017. More recently, the SMART survey conducted in December 2019 by the Ministry of Health with UNICEF support, showed that 2.1% of children had severe acute malnutrition. Considering this situation, the Ministry of Public Health and Population (Ministère de la Santé Publique et de la Population, MSPP) set out to improve health services supply and demand by establishing a results-based financing (RBF) program (MSPP, 2012). RBF is a health system intervention aimed at increasing the use of health services (especially for maternal and child health) by improving their quality and quantity with particular emphasis on efficiency and equity. This approach is based on the granting of incentives when certain health indicators related to providers or patient outcomes are achieved by making use of the resources (financial, material and human) available (Musgrove, 2011; Fritsche et al., 2014). In Haiti, RBF 4 is part of the National Health Policy (Politique Nationale de Santé, PNS) and national health financing strategies. The principal aim of the PNS is to reduce morbidity and mortality through an appropriate, efficient, accessible, and universal health system. This includes the rational use of available resources through donor alignment with national priorities, in the context of a partnership based on performance and accountability. Encouraged by the positive experience of the United States Agency for International Development (USAID) 5 with the establishment of a performance-based payment system between 1999 and 2012, the MSPP, seeking to extend this approach at the national level, identified a strategic pillar in the PNS for the establishment of a performance-based financing system. Furthermore, since 2012, the WB and USAID (through the Leadership, Management and Governance Project and the Quality Health Services for Haiti Project (Service de Santé de Qualité Pour Haïti, SSQH) have jointly started a series of discussions to assist the MSPP with the implementation of a national RBF program. In 2013, the contracting unit was created to supervise implementation of the RBF at the national level. The RBF program was launched in March 2016 by the contracting unit of the MSPP, with WB financing in the Sud, Nord-Ouest, and Centre Departments and with USAID financing in the Grand’Anse, Nippes, Nord-Est, and Nord Departments. Several early studies have pointed to a positive effect of RBF on the efficiency of the health system and the provision of health services, including increases in maternal and child health services coverage (Soeters et al. 2006; Sabri et al. 2007; Rusa et al. 2009; Basinga et al. 2011). A recent synthesis of the evidence concludes that, when compared with business-as-usual, in low-income settings with centralized health systems RBF can result in substantial gains in effective coverage, but that the relative benefits of PBF—the 3 Better known as Demographic and Health Surveys. 4 See the definition of the term and its relevance in the proposal submitted to the Ethics Committee in July 2015. 5 In 2013 Zeng et al evaluated the effect of RBF combined with technical assistance (but also separately) provided by the SDSH program with respect to key maternal and child health indicators. They found that RBF, along with technical assistance (training, monitoring and evaluation, and coaching), increased the quantity of primary health care services by 39 percent in NGO-run facilities between 2008 and 2010. In addition, the increase in health services for children under 12 months and pregnant women between 2008 and 2010 was statistically greater compared to facilities not using the RBF approach. 7 performance pay component in particular-- are less clear when it is compared to two alternative approaches: (i) direct facility financing which provides operating budgets to frontline health services with facility autonomy on allocation, but not performance pay, and (ii) demand-side financial support for health services (that is, conditional cash transfers and vouchers) (de Walque, Kandpal, Wagstaff et al. 2022). The WB (2016) has shown that all the HFs in Haiti have very low technical efficiency scores at all levels of the health system pyramid and much lower scores than other low-income countries (LICs) (Hernandez, 2013; Akzali et al. 2011; Sébastian et al. 2007; Marshall et al. 2011; Kirigia, 2013; Osmani, 2015, Jéhu-Appiah, 2014). This, among other things, could explain the mediocre productivity of medical staff (on average, staff providing medical care work for four hours per day and have six consultations per day in the primary health care sector in 2013), absenteeism (which is estimated to result in a loss to the health system of Haitian Gourdes (HTG) 3 billion), and the limited availability of basic inputs (only 32 percent of HFs (n = 907) provide essential drugs, and only 31 percent of HFs have basic medical equipment) (WB, 2016). Furthermore, the same study also revealed that quality processes related to maternal and child health services continue to be substandard (only 30 percent of health professionals follow proper clinical protocols during prenatal consultations). Lastly, RBF is a tool for improving health services coverage, especially among the poorest. Health services coverage in Haiti continues to be low, according to the World Health Organization (WHO) statistics (2015) shown in Table 1 below. This table shows that all preventive maternal and child health indicators are lower than those in LICs. When compared with LICs, the proportion of assisted deliveries in Haiti (37 percent) is 14 percentage points lower; unmet family planning needs in Haiti (35 percent) are 13 percentage points higher; and the proportion of children under 24 months who have received all doses of the diphtheria, tetanus, and pertussis vaccine in Haiti (68 percent) is 12 percentage points lower. Significant disparities remain in maternal and child health (with the exception of immunizations) between the poor and non-poor (EMMUS, 2012)— 9 percent of pregnant women in the lowest wealth quintile give birth in a HF compared to 76 percent in the highest wealth quintile; 23 percent of children with acute respiratory infections receive treatment in the lowest wealth quintile against 52 percent of children in the highest wealth quintile. As financial resources are the main constraint to use of health services for the poor 6 , RBF reforms could increase the use of health services by reducing the consultation costs for several maternal and child health services or by promoting mobile clinics (WB, 2016). 6 It should be noted that domestic Government spending for health is just 0.52% of GDP – very low when compared to other LICs – and this is a significant constraint. 8 Table 1: Key SDG for Maternal and Child Health Services Unmet need for family planning (%) % of pregnant women receiving four antenatal care (ANC) visits % pregnant women who gave birth with qualified personnel % children having received three doses of DTP % population using an improved water source (%) Population using improved toilets (%) Low Income Countries 22 48 51 80 69 37 Latin America 9 90 96 90 96 88 Haiti 35 67 37 68 62 24 El Salvador 18 80 100 92 90 70 Honduras 11 89 83 87 93 80 Jamaica 10 86 99 93 93 80 Costa Rica 8 90 99 95 97 94 Dominican Republic 11 95 99 83 81 82 Guatemala 21 … 59 85 94 80 Nicaragua 11 88 88 98 85 52 Benign 25 58 81 69 76 14 Rwanda 21 35 69 98 71 64 Burkina Faso 33 34 66 88 82 19 Ethiopia 26 19 10 72 52 24 Malawi 26 46 87 89 85 10 Tanzania 25 43 49 91 53 12 Source: WHO Statistics, 2015 Rising political and economic instability, COVID-19 shock and falling resources for health sector. Starting around 2017, Haiti started facing growing political instability as well as insecurity and violence, including periods of national lockdown notably the “Peys Lock” period between September and December 2019. This all occurred in the context of falling resources for the health sector due to a number of factors including poor economic conditions and a sharp fall in donor financing for health. All of this negatively impacted the health sector during the period of implementation of this Impact Evaluation (IE), and accounts for some of the declining trends observed in key variables over this period (see below for more details). Unfortunately, the situation in terms of exogenous factors affecting the health sector has only worsened since the implementation period of the study – i.e. after 2019 – due to the COVID-19 pandemic as well as rising political instability culminating in the assassination of the President in July 2021. 1.2 Objectives and Research Questions Related to Impact Evaluation To measure the effectiveness of this intervention over the long term and its operationalization in the context of Haiti, the MSPP and its partners decided to evaluate the impact of the RBF approach. The objective of this IE is to measure the impact of the national RBF program on priority health outcomes, household health expenditures, and the key functions of the Haitian health system, thus allowing the MSPP and its partners to determine whether this health policy is effective relative to facilities that have not adopted RBF or only receive technical assistance (coaching/local supervision). This overall objective leads to six main research questions: 1. Does the RBF program contribute to better infant health outcomes? 2. Does the RBF program contribute to the coverage and quality of priority health services related to maternal and child health? 3. What are the effects of the RBF program on the Haitian health system, in particular regarding human resources, health financing, and governance? 4. What impact does RBF have on catastrophic household expenditure? Does RBF provide households with greater financial protection? A household is said to incur catastrophic health 9 expenditures (CHE) if 25 percent or more of its nonfood expenses are allocated to health (WHO, WB, 2014). 5. Does technical assistance or coaching/local supervision alone improve priority health services coverage and quality related to maternal and child health more or as much as RBF accompanied by technical assistance? The table below provides the chain of RBF impact assumptions. Table 2: Mechanisms by Which RBF Can Potentially Work Problems identified RBF hypothesis RBF Impact Study Questions Health system supply/inputs Low motivation, absenteeism, and low productivity of medical staff - RBF stimulates better governance and accountability of staff vis-à-vis patients and increases the motivation of medical staff because teamwork is better perceived and rewarded. This contributes to better productivity as well. (3) Low availability of medicine and basic equipment RBF institutions have more resources at their disposal to improve the availability of essential inputs. 7 Low quality of maternal and child health services Medical staff adhere better to clinical protocols for maternal and childcare because they are more accountable to patients. (2); (5) Demand Low coverage of maternal and child health indicators, especially for the poorest RBF should encourage medical personnel to produce more, by implementing strategies to target and cover a greater number of patients and therefore increase demand. The RBF aims more to cover the poorest, many times those facilities are dispensaries, which are by nature located in rural and poor areas. (2) (5). Financial barriers for the poorest quintile was particularly high The CHE of the lowest quintiles are expected to decrease because - incentives received under RBF can be used to subsidize user fees for certain maternal and child health services - the implementation of advanced strategies to increase the coverage of services including the establishment of mobile clinics and assembly stations. Since nearly 75% of patients seen during this type of strategy are poor (ECVMAS, 2013), RBF is expected to contribute to increasing coverage of the poorest. (4) Malnutrition indicators The increase in the coverage of maternal and child health services contributes to an improvement in the nutritional status of children under 5 years old. (1) Source: RBF baseline study report, 2015 2. METHODOLOGY OF THE STUDY Experimental Approach A quasi-experimental study was conducted to evaluate the impact of the RBF program on Haiti’s health system. To do this, HFs operating at the primary care level were surveyed: Health Centers With Beds (Centre de Santé Avec Lit, CALs); Health Centers Without Beds (CSLs); Dispensaries; and Community Reference Hospitals (Hôpitaux Communautaires de Reference, HCRs). Furthermore, these were divided into three groups before the implementation of the study (Table 3) according to the methods described below. There are two interventions: (i) the first intervention that provides to HFs both RBF and technical assistance (TA); and (ii) the second intervention that provides TA only without RBF. This second intervention is justified by the results of the study measuring the effects of performance-based financing in Management Sciences for Health sites in 2012, which demonstrated a positive effect of TA alone on the coverage of maternal and child health services (Zeng et al, 2013). The third group consisted of a Control Group. This study focused on 196 catchment areas distributed in nine out of ten Departments in the country. A catchment area is made up of a health institution (HF) and the localities it serves. The HFs in each of the intervention groups and the Control Group are described below. 7 In theory, a similar effect could be obtained by providing to HFs lump-sum unconditional cash amounts for spending on essential items. But it would be important for the HFs to spend these resources in accordance with some sort of HF spending plan for these resources, which is a critical part of the RBF program of Haiti where close, hands-on TA is provided for the HF spending plans (as for many countries). In addition, 10 Table 3: Summary of Sample by Intervention Arm Intervention One Intervention Two Control Group Description RBF + Technical Assistance Technical Assistance Only Neither RBF nor Technical Assistance Number of sites 48 HFs WB 23 SSQH/USAID sites* 52 HFs WB 23 SSQH/USAID sites 50 sites - neither SSQH/USAID nor WB Department Northwest, South, Center Grand-Anse, Nippes, Northeast, North Northwest, South, Center Grand-Anse, Nippes, North-East, North In municipalities and Departments that have neither a USAID site nor a BM site *These are the sites of the SSQH project which are managed by various Non-governmental Organizations (NGOs) or American firms. - HFs benefiting from the entire RBF package with technical assistance or TA (Group One – labelled the “RBF” Group): This consisted of 48 HFs financed by the WB (in the Northwest, South and Center Departments) and twenty financed by USAID under its SSQH program (in Grand 'Anse, Nippes, Northeast, Nord, and Sud). Within each Department, the WB sites were randomly selected by type of establishment and technical capability score (between two and three) according to a method of proportionality in relation to size. The notion of “technical score 8 ” was inserted because the WB wanted to ensure that the sites selected were functional and of sufficient quality to provide a basic minimum level of services. As for the USAID sites, they fowere not randomly selected because they had already been preselected based on criteria defined by USAID and MSPP. However, applying the “technical score” criteria of WB sites to USAID sites, USAID sites were found to be comparable to WB sites. - HFs benefiting from TA (coaching) only (Group Two – labelled the “TA” Group): These comprised 52 HFs under the WB (in the North-West, South and Center Departments) and twenty- three under USAID (all Departments, except Grand'Anse, Sud, Sud-Est and Ouest). The WB sites were randomly selected by Department, type of establishment and technical score according to a method of proportionality with respect to size. The SSQH/USAID sites were selected using a matching method to ensure that these sites had comparable characteristics between intervention groups one and two. Ideally the matching method of selection would have been by Department, level and technical score. (For example, one would have paired a dispensary under "SSQH/USAID", from the Northeast Department, with a score of three, from intervention Group One (RBF+ coaching/TA) with a dispensary from SSQH/USAID, in the same Department and with an equivalent score from intervention Group Two.) However, in practice, it was not always possible to match SSQH/USAID sites from both groups in the same Department. For example, in the case of Grand’Anse, there are four SSQH/USAID sites implementing RBF (Group One), but there are no SSQH/USAID sites in that Department from Group Two. In this case, HFs of the same level and score were selected from another neighboring Department (Nippes and Southeast) to be paired with the SSQH/USAID HFs from Group One in the Grand'Anse Department. 8 The technical score was created based on a composite indicator taking into consideration the following elements: availability of basic services, availability of basic technical staff, regular submission of reports, availability of medicines, number of patients served, regular source of water and electricity, and basic equipment. The criteria have a different definition depending on the level of service. The criteria “availability of basic services”, “technical staff” and “electricity and water” are the most important criteria, representing 2/3 of the total score. Sites are classified into three categories based on their total score which can reach a maximum of 45. Sites that score below the 1st quartile (25% - or the 25% of Sites with the lowest score get 1, those above or equal to the 1st quartile but below the 3rd quartile get a score of 2 and those equal to or above the 3rd quartile (or 25% of sites with the highest score) get 3. 11 - HFs not benefiting from any intervention (Control Group): These were randomly selected outside the intervention areas of USAID and the WB (outside the communes with USAID sites, and outside the North- West, Center and South Departments where the WB sites were concentrated). The HFs here were randomly assigned to the control group considering the Department, the level and the technical score via a method of proportionality to the size. Table 3b: Geographic Distribution of HFs Covered by RBF in 2015 Group 1 (RBF) Group 2 (TA) Group 3 (Control) All Proportion of all HFs in Department 9 BM USAID BM USAID Artibonite 6 16 22 15.9% North 8 2 9 19 17.9% Northeast 9 2 11 26.2% Nippes 3 1 6 10 31.3% Grand Anse 3 7 10 18.9% Southeast 1 12 13 16.5% South 21 2 18 2 41 53.2% Center 8 8 5 21 32.8% Northwest 19 26 4 49 52.7% All 48 23 52 23 50 196 10 28.7% Source: Baseline study report, 2015 Figure 2 below provides a map of the locations of the different HFs with interventions (RBF and TA) and the different control group HFs. 9 Based on then numbers of HFs in each Department at the time of the baseline surveys. 10 It should be noted that due to site closures and other reasons, only 178 health institutions remained in the program and therefore in the sample of the final RBF evaluation study in 2019. 12 Figure 2: Location of the different HFs with interventions (RBF and TA Groups) and the Control Group facilities of the RBF program in Haiti In 2015, a baseline survey was conducted in the 196 service areas selected in order to obtain the baseline situation prior to implementation of the program. This survey covered HFs but also households in the districts served by the HFs in the sample. To this end, for every HF selected, one district was randomly selected among the districts served. In the district selected, a complete census was conducted to identify households with at least one woman who was pregnant or had been pregnant in the 24 months preceding the survey (eligible households), following which 28 eligible households, also randomly selected, were interviewed. RBF implementation in Haiti first started with selecting the Departments for the program, followed by contracting the Departmental Health Directorates (DDSs) in the Departments selected and, lastly, the HFs in the Departments selected. The HFs selected via the method described above. 2.1.1 Implementation of the Interventions o Technical Assistance At the start of the implementation period, all the contracted HFs in the Technical Assistance group (Group Two) received technical support to perform their functions (planning, programming, monitoring/evaluation, supervision, financial management, capacity building and strengthening of good practices, etc.). In addition, advanced strategies were encouraged so as to reach the vulnerable population groups. 13 o Results-Based Financing combined with Technical Assistance In the case of HFs in the RBF group (Group One), training was first provided on the RBF approach and different tools, as well as assistance with drafting their annual work plan prior to the signing of a memorandum of understanding with the DDS. The signed contractual documents provide the indicators to which incentives should be applied in the context of program implementation. Table 4 below shows the different indicators to which incentives apply and the payments, depending on the type of HF. The incentives were given in the form of: (i) fee-for-service payments for the incentivized quantitative indicators (number of deliveries etc.), and (ii) payments based on a quality score for each HF, consisting of observable characteristics of quality (e.g. availability of equipment, essential drugs, etc.) Startup assistance funds were then allocated to them, along with investment funds, granted on the principle of accountable advances for the purchase of major equipment and inputs. Table 4: Tariffs (in HTG) of Indicators Incentivized Under the RBF Program in Haiti † No. Indicators Q3 2016 Q4 2016 – Q1 2018 T2 – Q3 2018 Index Q4 2018 Q4 2018 Index Q1 - Q3 2019 Q1 2019 Q2 2019 Q3 2019 US Dollar exchange rate 80 Gourdes 90 Gourdes 93.36 Gourdes 91.68 Gourdes CSLs, CALs, and Dispensaries 1 New curative consultations 17 17 32.5 0.5 40 1 90 93.4 91.7 2 Cases of diarrhea treated 34 34 65 N / A N / A N / A N / A N / A N / A 3 Referrals to higher level 169 169 325 5 400 7.5 675 700.2 687.6 4 Children aged 6-59 months that had nutritional screening 17 17 32.5 0.5 40 1 90 93.4 91.7 5 Deworming of children aged 12 - 59 months 17 17 32.5 0.5 40 1 90 93.4 91.7 6 Children aged 6 – 59 months who received vitamin A supplementation 17 17 32.5 0.5 40 1 90 93.4 91.7 7 Microscopic screening of Tuberculosis 236 236 455 7 560 10.5 945 980.3 962.6 8 Fully immunized children aged <12 months 270 270 520 8 640 12 1080 1120.3 1100.1 9 Institutional deliveries 405 405 780 12 960 18 1620 1680.5 1650.2 10 Pregnant women with at least one prenatal consultation 101 101 195 3 240 4.5 405 420.1 412.6 11 Pregnant women with four prenatal consultations 203 203 390 6 480 9 810 840.2 825.1 12 Pregnant women with two doses of anti-tetanus vaccine 34 34 65 1 80 1.5 135 140.0 137.5 13 Pregnant women counseled and screened for HIV/AIDS and syphilis 101 101 195 3 240 4.5 405 420.1 412.6 14 Pregnant woman with HIV/AIDS+ on treatment 101 101 195 3 240 4.5 405 420.1 412.6 15 HIV/AIDS+ pregnant woman referred to a higher level for care 101 101 195 3 240 4.5 405 420.1 412.6 16 Postnatal home visits performed 0-3 days 68 68 130 2 160 3 270 280.1 275.0 17 Use of modern contraceptives 34 34 65 1 80 1.5 135 140.0 137.5 18 Number of Tuberculosis + cases treated N / A 101 195 3 240 4.5 405 420.1 412.6 19 Number of Tuberculosis+ cases lost to follow-up N / A 101 195 3 240 4.5 405 420.1 412.6 20 Number of children born to HIVAIDS-positive mothers put on antiretroviral prophylaxis within 72 hours N / A 101 195 3 240 4.5 405 420.1 412.6 HCRs 1 New consultation in outpatient clinic referred by a lower level and management 243 243 260 4 320 6 540 560.2 550.1 2 Client referred by a lower echelon hospitalized 122 122 130 2 160 3 270 280.1 275.0 3 Institutional childbirth 730 730 780 12 960 18 1620 1680.5 1650.2 4 Caesarean section 912 912 975 15 1200 22.5 2025 2100.6 2062.8 5 Counter reference 243 243 260 4 320 6 540 560.2 550.1 6 Number of TB+ cases treated N / A 366 390 6 480 9 810 840.2 825.1 7 Number of TB+ cases lost to follow-up found N / A 366 390 6 480 9 810 840.2 825.1 8 Number of children born to HIV/AIDS-positive mothers put on ARV Prophylaxis within 72 hours N / A 366 390 6 480 9 810 840.2 825.1 Source: MSPP Contracting Unit, 2021 † It should be noted that at the beginning of the intervention period, the prices were directly fixed in Gourdes. From Q4 2018 onwards, the prices were adjusted by an index linked to the dollar, so that they would remain the same in dollar terms over time. Like the HFs in the TA group, those in the RBF group received technical support to execute the tasks within their purview (planning, programing, monitoring/evaluation, supervision, financial management, capacity building and strengthening of good practices, etc.). The development of more advanced strategies was also encouraged. 14 Regular checks are conducted at the HFs at the end of each quarter by an external and independent agency, namely the Agence Externe de Vérification (AEV). There are three types of checks: - Quantity checks, which focus on the agreed quantitative indicators (verification of data provided by the HFs) and are conducted using a check sheet. - Quality checks/assessments using quality verification /assessment checklists that are adapted to each category of HF. - A community verification component that focuses on patient satisfaction, using a specially designed questionnaire that is administered to a sample of patients, the size of which is based on the level of the HF. At the end of each verification exercise and after the results have been validated by the Departmental Steering Committee, an invoice is prepared for each HF. The AEV then transmits these invoices to the contracting unit, which then submits a payment request to the fiduciary agencies responsible for accounting/financial control and for issuing the bonuses to the beneficiary HFs. Out of all bonuses received, 78% went to the HFs, and the remainder went to the DDS/CDAI (Departmental Center for Supply of Inputs). Out of the former, health workers were allowed to use up to 50% for salary topups. It should be noted, however, that there were a number of hiccups in the implementation of these procedures, especially over the past few years, may have affected the efficient and effective implementation of the program. These include: - The recurrence of political, economic, and security crises in the country over the past few years. - Numerous revisions of the tools used in the process of verifying and modifying the list of indicators. - The frequent changes in the verification teams on the ground, which add to the problem of interpreting and therefore validating the data and result in delays to the process described above. - The problem of slow decision-making. This is due, among other reasons, to the fact that it is difficult for the technical group to meet and arrive at a decision on a problem that requires the input of various stakeholders. Another reason is the absence of an effective flow of communication among the stakeholders. - The difficulty of using the indicator tools and the staff evaluation mechanisms, arising from the variety of approaches and different degrees of rigor applied to the evaluation of staff from one Departmental health directorate to another. - Payment delays to HFs because of the problems mentioned above, but also because supporting documents are either unavailable or invalid, information sharing is deficient, or because of the absence of a bank account, among other reasons. Three years after the baseline survey and the implementation of the program, the MSPP, in collaboration with the WB, embarked in 2019 upon the final study to evaluate the impact of the RBF program itself. The aim of the final survey is to establish the indicators to be used for evaluating the impact of the RBF program interventions on maternal and child health, such as quality of health care, use of health services, household health expenditures, and the key functions of the Haitian health system. With this in view, the final evaluation survey was again conducted at a random number of HFs that were still in operation in the districts studied in 2015. Households that were tracked from the 2015 survey and which were still eligible for the study were 15 again surveyed in 2019. However, dropout households and those that continued to be tracked but were no longer eligible in 2019 were replaced by neighboring households that met the eligibility criteria. 2.2 Data Sources: The IE used two main data sources to address the research questions identified above: - The household surveys conducted in 2015 before the implementation of the RBF program, and those done later in 2019, a little over three years after the implementation of the program. - HF surveys: these institutions were also surveyed at the start and end of the program implementation process. 2.2.1 Household surveys Household surveys were conducted in the areas served by the HFs included in the study (196 in the baseline study and 178 in the final study). These service areas are the primary survey units. A complete census of the households in each service area was conducted. A total of 28 eligible households (those that include at least one pregnant woman or a woman who had given live birth in the 24 months preceding the survey) participated in the baseline study in 2015. Of the 5,488 households targeted, 5,347 participated in the survey, representing a completion rate of 97 percent. In conducting the final study in 2019, the aim was to track the same households in each service area that had been surveyed in the baseline study. The households that were tracked in the baseline study and continued to meet the eligibility criteria were then surveyed, while drop out households and those that were tracked but no longer eligible were replaced by eligible, neighboring households. In 2019, 73.5 percent of the households in the baseline study were tracked, but only 37.2 percent of them were still eligible, and thus surveyed. Households are therefore regarded as the secondary survey units. The data collection tools were adapted from the survey instruments used in the Health Results Innovation Trust Fund (HRITF), which were designed for IEs of this kind. The household survey methods are summarized in Table 5 below. Table 5: Characteristics of Households Surveys Survey Types Respondents Level Period Description Household survey – socio-economic component Head of household or his representative All household members Housework Baseline and Endline Socio-economic characteristics, household health, health expenditure n= 28 households per service area Household survey – maternal and child health component Women who had a child during the 2 years preceding the survey Mothers or guardians of children under 5 Housework Baseline and Endline Health behaviors (pregnancy, prenatal care, care during childbirth, postnatal care, etc.) Vaccination, anthropometric measurements Source: Author 16 2.2.2 Surveys of health facilities For the purposes of the baseline study, surveys of HFs were conducted at 196 HFs. However, owing to security issues and the closure of a number of HFs, data collection for the final study was carried out at only 170 HFs. Data collection was carried out at each HF for the purpose of evaluating the facility, while data were also collected on health workers, including directly observed data on interactions between patients (women at their first prenatal consultation, and children under 5) and health providers; data on patient satisfaction after their consultation; and data on community health workers (CHWs) deployed in the areas served by each HF. o Survey on the evaluation of health facilities The same evaluation questionnaire was administered to all HFs. The questionnaire was directed mainly to the person in charge of the HF at the time of the survey. The following were the main issues addressed: administration and management; human resources; staff; laboratories; services provided and use of health services; the health information system; user fees; level of autonomy; availability of medical equipment and drugs. For the baseline study, 195 of the 196 HFs in the sample were surveyed. However, in view of the security considerations referred to earlier, only 170 HFs from the initial sample were surveyed for the final study in 2019. o Survey of health workers A list of health personnel responsible for prenatal consultations, deliveries, and maternal and childcare was prepared for each HF surveyed. The list was compiled from the staff information provided by each HF in the questionnaire referred to above on the evaluation of HFs. If the list included more than four health workers, then four of these workers were selected at random to participate in the survey. Otherwise, all the health workers listed were surveyed. The interviews focused mainly on: the training and knowledge of workers; workers’ wages; their income and other forms of compensation or benefits that they received; and their on- the-job supervision, motivation, and satisfaction. In 2015, 542 health workers were surveyed from the sample of 784 staff members in the 196 HFs, representing a coverage rate of 69.13 percent. In 2019, only 559 of the 712 health workers identified in the 170 HFs included in the revised sample were surveyed, representing a coverage rate of 78.51 percent. The high non-response rate may be attributed to the following factors, among others: the workload at the HFs; the fact that the number of health workers at each HF is lower than the size of the staff complement requested by each facility; absenteeism; and, at times, the refusal of health workers to participate. o Service provider-patient interactions This module relates to children under five who come in for consultations, accompanied by their parent, as a result of a new episode of morbidity, and pregnant women visiting for their first prenatal consultation. For each of these instances of service provider-patient interaction, three (3) randomly selected consultations (in the best-case scenario) were observed at each HF. 17 For the interactions of service providers with women visiting for their first prenatal consultation, the direct observation exercise focused on the following: case history of previous pregnancies; the condition of the expectant mother; and counseling or treatment provided at the time of the survey. In 2015, a total of 543 prenatal consultations were observed from the 588 consultations (92.4% coverage rate) that were targeted in the 196 HFs identified in the baseline survey. In 2019, 468 prenatal consultations were observed from the 534 visits targeted in the 178 HFs identified in the sample for the final study. In relation to the interaction of service providers with children under 5 coming in for consultations for a new morbidity, the observation focused on: medical history and examination, counseling, and treatment of the disease. In 2015, a total of 546 children’s consultations were observed from the 588 visits targeted at 196 HFs identified in the baseline survey. In 2019, a total of 500 consultations were observed from the 534 visits (87.6% coverage rate) targeted in the 178 HFs in the sample for the final study. o Post-consultation interviews Patients (parents of children under 5 visiting a HF for a new morbidity and pregnant women visiting for their first prenatal consultation) who were observed during the consultation (service provider-patient interaction) were approached at the end of the visit for an exit interview, focusing mainly on: the perception and level of satisfaction of the patient with the care and advice received during the consultation; and their level of health expenditure. For the baseline study, all 543 women observed during their prenatal consultation and 545 of the 546 children observed participated in exit surveys after the consultation. In the final study, 466 of the 468 women observed during their prenatal consultations and all 500 children observed, plus three children who were not observed, participated in exit surveys after the consultation. o Interviews with community health workers (CHWs) In each service area, one CHW, selected randomly where there was more than one such worker, was surveyed. These interviews focused on: training and the services provided by CHWs; their remuneration and other benefits; their level of on-the-job supervision, motivation, and satisfaction. In 2015, 304 CHWs were surveyed, as against 137 in 2019. In effect, for both the 2015 baseline survey and the 2019 final study, the target of one CHW per HF was not met on the ground. Indeed, a total of 196 CHWs from the 196 HFs surveyed were slated to participate in the baseline study. The survey methods used to gather data from the HFs are summarized in Table 6 below. 18 Table 6: Characteristics of Health Facility Surveys Survey Types Respondents Level Period Description HF survey Head of HF Health institution Baseline and Endline Facility staffing, infrastructure, drug supply, equipment, supervision, health information system, management reports, user fees, facility funding/income sources n=1 per HF Human Resources Survey medical staff Health institution Staff workload, compensation, satisfaction, staff motivation n=4 per HF Baseline and Endline Observation of the interaction between patients and medical staff New prenatal consultation clients Health institution Baseline and Endline Patient treatment and counseling n=3 per HF New children under 5 years old n=3 per HF Interview at the end of the prenatal and under- five childcare visits New prenatal consultation clients Health institution Baseline and Endline Perception of patients on the quality of health services n=3 per HF New children under 5 years old n=3 per HF CHW CSA Health institution Baseline and Endline Work, hours, motivation, satisfaction Source: Author Table 7 below shows the distribution of HFs visited in both the baseline and final studies, disaggregated by type of facility and status. It may be noted that 61.76 percent of these HFs are public bodies, mainly dispensaries, health centers without beds, and CALs. Table 7: Geographic Distribution of Health Facilities Included in the IE 11 and Surveyed During the Two Periods of the Study Type of HF Facility HF Status Total Department HCRs CALs CSLs Dispensary Public Private for profit Private not for profit Mixed Artibonite 2 5 2 11 16 3 1 0 20 Center 2 8 2 7 11 1 1 6 19 Grand Anse 1 1 4 4 4 1 1 4 10 Nippes 1 3 1 5 4 2 0 4 10 North 2 4 9 2 10 3 2 2 17 Northeast 0 4 1 3 8 0 0 0 8 Northwest 1 7 2 30 27 2 2 9 40 South 4 4 10 19 21 3 2 11 37 Southeast 0 0 4 5 8 0 0 1 9 All 13 36 35 86 109 15 9 37 170 12 HCRs, CALs, CSLs Source: Author, calculations based on the IE data, 2015 and 2019 11 Includes RBF and TA sites, TA only sites, and control sites. 12 The following 8 HFs, although appearing in both the samples of the baseline HF survey of 2015 and the final HF survey of 2019, could not be surveyed in both periods for various reasons: 1. Palmists in the North-West Department (refusal by the head of the HF to be surveyed); 2. Seguin dispensary in the South-East (temporarily closed during the collection period in 2019); 3. Claire Heureuse in Artibonite (security reasons); 4. La Chapelle in Artibonite (security reasons); 5. Marebriole dispensary in the South-East (infrastructure destroyed); 6. Jean-Clair in the North West (security reasons); 7. Dupont in the North West (security reasons); and 8. Dispensaire la Montagne in the South-East (temporarily closed during the data collection period in 2019). 19 2.3 RBF Impact Evaluation Indicators In relation to the five research questions referred to above, a corresponding list of 53 indicators has been developed. Indicators on monitoring the implementation of the RBF program in Haiti; alignment with quality standards for health care and with national and international coverage; alignment with RBF change theory; and trends observed in the baseline study (such as indicators with extremely high values in the baseline study). The availability and quality of data (data collected during the 2015 and 2019 surveys) have, among other things, helped determine the choice of these indicators. Table 8: List of RBF IE Variables # Indicators Data source Human Resources Indicators 1 Proportion of nursing staff receiving their salary on time Interview with the nursing staff 2 Intrinsic motivation of nursing staff Interview with the nursing staff 3 Satisfaction of caregivers with the work environment Interview with the nursing staff 4 Satisfaction of caregivers regarding their remuneration Interview with the nursing staff 5 Satisfaction of caregivers with management and supervision in HF Interview with the nursing staff Indicators Relating to the Quality of Health Services 6 Proportion of HFs with a functional management committee HF assessment 7 Proportion of HFs having terms of reference for all workstations HF assessment 8 Proportion of HFs whose funding is conditional on the results of general performance evaluations HF assessment 9 Proportion of HFs with permanent availability of drinking water in the last 7 days HF assessment 10 Proportion of HFs with permanent electricity availability in the last 7 days HF assessment 11 Proportion of HFs with an operational waste disposal system HF assessment 12 Proportion of nurses with prenatal care services HF assessment 13 Proportion of nurses receiving childbirth care HF assessment 14 Proportion of nurses with postpartum care service HF assessment 15 Proportion of nursing homes with a vaccination service HF assessment 16 Proportion of HFs respecting universal precautionary measures HF assessment 17 Proportion of nurses able to perform laboratory tests HF assessment 18 Proportion of HFs with vaccines in stock HF assessment 19 Proportion of HFs with generic drugs in stock HF assessment 20 Proportion of HFs with antimalarials in stock HF assessment 21 Proportion of HFs with malaria rapid diagnostic kits in stock HF assessment 22 Proportion of HFs with general equipment in good condition HF assessment 23 Proportion of nurses with sterilization equipment in good condition HF assessment 24 Proportion of health care providers with vaccination equipment in good condition HF assessment 25 Proportion of nurses with prenatal care equipment in good condition HF assessment 26 Proportion of nursing homes with childbirth and postnatal care equipment in good condition HF assessment 27 Proportion of antenatal care (ANC) visits with top three routine services (BP control, weight control, iron/folic acid prescription) ANC direct observation 28 Proportion of ANC cases with patient education on the four key elements (birth assistance/skilled birth plan, at least two pregnancy danger signs, nutrition, exclusive breastfeeding) ANC direct observation 29 Proportion of observed women who are satisfied with ANC Interview at the end of the prenatal care visit 30 Proportion of women observed perceiving the quality of ANC adequate on seven key elements Interview at the end of the prenatal care visit 31 Proportion of child curative consultations with assessment of all Integrated Management of Childhood Illness (IMCI) danger signs according to IMCI guidelines (provider asks if child can drink or breastfeed; provider asks if child vomits everything s/he eats /drinks; provider asks for symptoms of lethargy or change in level of consciousness; provider of seizure symptoms) Direct observation of childcare visit 32 Proportion of child curative consultations with assessment of common childhood symptoms according to IMCI guidelines (CHW asks about: fever, cough, diarrhea, ear problems; CHW checks: weight, temperature, signs of anemia (conjunctiva, palms), vaccination status) Direct observation of childcare visit 33 Proportion of children observed who are satisfied with their curative consultation Direct observation of childcare visit 20 34 Proportion of children observed perceiving adequate quality of care on seven key elements of HFs (cleanliness, opening hours, waiting time, time with the provider, appropriate explanation, respectful treatment, privacy) Direct observation of childcare visit Indicators Related to the Use of Reproductive Health Services 35 Women's unmet need for family planning Household 36 Proportion of women of reproductive age (15-49) who use a modern contraceptive method Household 37 Proportion of married women aged 15-49 who use a modern method of contraception Household 38 Proportion of women of reproductive age (15-49) who use family planning (FP) methods from any source Household 39 Proportion of recently pregnant women who had at least one ANC visit Household 40 Proportion of recently pregnant women who had at least four ANC visits Household 41 Proportion of recently pregnant women who had their first ANC visit within the first four months of pregnancy Household 42 Proportion of recently pregnant women who received at least 2 doses of tetanus vaccine during pregnancy Household 43 Proportion of recently pregnant women whose delivery was assisted by a health professional Household 44 Proportion of recently pregnant women who gave birth in a HF Household 45 Proportion of recently pregnant women who had at least one postnatal visit within 2 months after delivery Household 46 Proportion of recently pregnant women who received iron supplementation after delivery Household Indicators Related to the Use of Preventive Health Services for Children 47 Proportion of children aged 12-23 months fully immunized (i.e., having received one dose of Bacille Calmette Guerin (Tuberculosis Vaccine, BCG), four doses of oral poliovirus vaccine (OPV), three doses of pentavalent vaccine, one dose of measles) Household Indicators Relating to the Health Status of the Population 48 Proportion of children 0-59 months with severe stunting Household 49 Proportion of children aged 0-59 months who are underweight Household 50 Proportion of children 0-59 months with wasting Household 51 Catastrophic health expenses Household 52 Proportion of households whose health expenditure exceeded their financial means Household 53 Proportion of people who have been ill who took the prescribed drugs Household 2.4 Statistical Methods 2.4.1 Balance of the study groups for the baseline survey As indicated above, a quasi-experimental study with three groups was conducted to evaluate the impact of RBF on Haiti’s health system. While the distribution between the three groups was done randomly for HFs financed by the WB project, the matching distribution method was used for the HFs financed by USAID. In order to ensure that the method being used to analyze the IE exercise is a sound one, this hybrid distribution method should be borne in mind when determining the extent to which the different groups involved in the study display the same characteristics at the start of program implementation. In this regard, Table 9 below shows the balance tests between the different groups of the study for a certain number of indicators. The averages of the different groups are compared individually using two-sample statistical hypothesis testing. Tables 8 and 9 show that there were several statistically significant differences for certain variables (p-value < 0.1) between the two groups in the baseline study, at both the household and individual levels. The RBF and control groups were not balanced with respect to residence in rural areas, household size, the poorest households, the wealthiest households, the average age of household members, the number of children under 5 in households, marital status (common-law relationships) of household members, assisted delivery, average age of children under 5, and stunting. The TA and control groups were unbalanced with respect to residence in rural areas, education level (from no formal education to higher education), health spending on hospitalization, reproductive health behaviors of women between 15 and 49 years of age (unmet family planning needs, use of modern contraceptive methods), obstetric care (first 21 prenatal consultation in the first four months of pregnancy, assisted delivery by a professional health worker), and stunting in children under 5. The two intervention groups (RBF and TA) were also statistically different in relation to rural households, the wealthiest households, individual characteristics (average age, women 15- 49 years, children under 60 months, marital status (common-law relationship), education level (from no formal education to higher education), reproductive health behaviors of women between 15 and 49 years of age (unmet family planning needs, use of modern contraceptive methods), first prenatal consultation visit in the first four months of pregnancy, full immunization coverage for children between the ages of 12 and 23 months (from all sources). Nevertheless, it should be noted that the three groups in the study were balanced in relation to CHE, health spending that exceeds the household’s ability to pay, level of primary education, use of health services, purchase of medicine to treat diseases, total health expenditure on disease treatment, health expenditure to purchase medicine, average age of women of childbearing age, having at least one prenatal consultation at a HF, having at least four prenatal consultation at a HF, use of postnatal care, children under five who are underweight or suffer from wasting. Table 9: Characteristics of Households and Individuals in the Baseline Study Sample Variables Average RBF Medium AT Medium Ctrl Total average p-value RBF/Ctrl p-value AT/Ctrl p-value RBF/AT NOT Household characteristics Rural 0.430 0.594 0.716 0.558 < 0.001 < 0.001 < 0.001 5347 Household size 5.052 5.002 4.938 5.006 0.089 0.355 0.418 5347 Poorer 0.190 0.199 0.218 0.200 0.042 0.179 0.492 5347 2nd quintile of poverty 0.197 0.199 0.207 0.200 0.465 0.559 0.892 5347 3rd quintile of poverty 0.193 0.213 0.194 0.200 0.945 0.186 0.115 5347 4th quintile of poverty 0.201 0.195 0.206 0.200 0.720 0.452 0.647 5347 Richer 0.220 0.195 0.175 0.200 0.002 0.160 0.056 5347 CHE 0.044 0.050 0.053 0.049 0.210 0.697 0.358 5347 Health costs exceeding the financial means of the household 0.107 0.099 0.104 0.103 0.791 0.617 0.393 5347 Characteristics of household members Age (in completed years) 21.104 21.075 21.769 21.259 0.022 0.020 0.910 26773 Women aged 15-49 1.553 1.574 1.562 1.562 0.493 0.405 0.090 26773 Children 0-59 months 1.305 1.270 1.258 1.281 0.000 0.349 0.002 26773 Women who are married or in stable union 0.577 0.602 0.605 0.593 0.004 0.724 0.005 16435 No education 0.300 0.274 0.303 0.292 0.691 0.001 0.000 20535 Primary level 0.364 0.375 0.371 0.369 0.399 0.650 0.149 20535 Secondary level or higher 0.329 0.345 0.318 0.332 0.209 0.002 0.035 20535 Use of health services in the previous month 0.456 0.409 0.481 685.059 0.841 0.694 0.839 1123 Purchasing/taking medication to treat the disease 0.419 0.389 0.444 291.894 0.532 0.841 0.517 492 Health expenditure for illness treatment 689.392 632.651 758.616 452.934 0.412 0.123 0.391 1123 Health expenditure for hospitalization 450.704 168.340 142.259 0.446 0.518 0.079 0.168 1122 Health expenditure for drugs 453.200 333.622 640.227 0.414 0.514 0.170 0.367 1127 Reproductive health behaviors of women aged 15-49 Average age of women of reproductive age 27.873 27.942 27.999 27.929 0.6083 0.8235 0.7659 7355 Unmet need for family planning 0.312 0.349 0.316 0.326 0.7511 0.0206 0.0032 7355 Use of modern contraceptive methods 0.434 0.371 0.408 0.406 0.1946 0.0779 0.0008 3787 ANC deliveries and postnatal care At least one prenatal visit 0.831 0.838 0.816 0.830 0.3528 0.1989 0.6696 3254 At least four prenatal visits 0.597 0.611 0.579 0.598 0.4097 0.1561 0.4871 3254 First ANC visit in the first 4 months of pregnancy 0.702 0.734 0.672 0.706 0.147 0.0029 0.0795 3254 22 Delivery assisted by a health professional 0.541 0.529 0.455 0.515 0.0012 0.0063 0.6235 2301 Use of postnatal care 0.406 0.416 0.396 0.407 0.6604 0.3799 0.61 3254 Infant care Average age of children (in months) 21.384 20.764 20.182 20.876 0.0201 0.2699 0.1914 6238 Children aged 12-23 months completely vaccinated according to the vaccination card 0.418 0.381 0.397 0.400 0.4517 0.584 0.1441 2011 Children 12-23 months fully vaccinated according to all sources 0.423 0.381 0.397 0.402 0.3548 0.584 0.0984 2011 Stunting 0.257 0.254 0.313 0.271 0.0069 0.006 0.8696 2997 Underweight 0.142 0.145 0.158 0.147 0.3174 0.4594 0.8015 2883 Wasting 0.140 0.157 0.136 0.145 0.8362 0.2467 0.2765 2883 Standard errors clustered by HF Source: Author’s calculations based on RBF IE (2015) Tables 10 and 11 below, which outline the balance tests for certain characteristics of HFs and health services coverage that were applied in the baseline survey between the different groups of the study, show that the RBF and control groups were statistically different in the baseline study in relation to governance (existence of ToR for each senior position at a HF), continuous supply of water and electricity, availability of delivery care services and postnatal care, capacity to conduct laboratory tests, financing based on the sale of medicines, motivation of health workers, number of BCG vaccines, number of third doses of pentavalent vaccines, and number of first injections of measles vaccine administered to children under five. The TA and control groups were unbalanced with respect to the continuous supply of electricity, the availability of postnatal care, and the number of health personnel. The RBF and TA groups were unbalanced in the baseline study with respect to governance (existence of ToR for each senior position at a HF), availability of prenatal care services, financing based on the sale of medicines, number of health personnel, motivation of health workers, as well as the number of first injections of the measles vaccine administered to children under five. The different groups used in the study appear, however, to be balanced in relation to RBF tied to overall output evaluation, availability of means of transport, existence of a system for operational waste disposal, compliance with universal precautions, availability of vaccination services, availability of medicines and vaccine supplies, availability of medical supplies and equipment, number of patients who have used a modern contraceptive method, number of assisted deliveries, number of postnatal consultations, number of third doses of OPV administered to children under five, number of curative care treatments for children, number of growth monitoring and nutritional advice services for children, number of tetanus shots administered to pregnant women, number of curative treatments for children of five years and above, financing received from Haiti’s MSPP and fees paid by patients, total financing in 2015 and average number of hours devoted to health care. Going beyond the issue of balance between different groups, the table provides useful information. As an illustration, the table shows that on average, 90.4% of HFs in the overall sample had vaccination equipment, but only 36.6% had vaccines in stock. Just 34.4% had availability to drinking water at all times, and just 36.9% had availability to electricity at all times. 23 Table 10: Characteristics of Health Facilities in the Baseline Study Sample Variables Average RBF Average TA Average Ctrl Average total p-value RBF/Ctrl p-value TA/Ctrl p-value RBF/TA N Characteristics of HF Existence of a functional management committee 0.153 0.178 0.160 0.164 0.915 0.796 0.684 195 Existence of terms of reference for all workstations 0.583 0.431 0.380 0.474 0.027 0.580 0.068 194 Funding linked to the results of general performance evaluations 0.028 0.014 0.020 0.021 0.787 0.796 0.563 194 Availability of a means of transport 0.208 0.247 0.160 0.210 0.506 0.252 0.586 195 Availability of drinking water at all times 0.417 0.356 0.220 0.344 0.024 0.108 0.458 195 Availability of electricity at all times 0.403 0.425 0.240 0.369 0.062 0.035 0.791 195 Existence of an operational waste disposal system 0.847 0.877 0.880 0.867 0.611 0.957 0.610 195 Compliance with universal precautionary measures 0.706 0.690 0.654 0.687 0.143 0.291 0.607 194 Availability of ANC services 0.931 1.000 0.940 0.959 0.838 0.036 0.023 194 Availability of childbirth care services 0.583 0.486 0.340 0.485 0.008 0.110 0.245 194 Availability of postpartum care services 0.736 0.625 0.460 0.624 0.002 0.072 0.155 194 Availability of immunization services 0.972 0.944 0.920 0.948 0.193 0.595 0.408 194 Ability to perform laboratory tests 0.428 0.351 0.270 0.359 0.018 0.214 0.207 195 Vaccines in stock 0.361 0.384 0.346 0.366 0.807 0.563 0.703 195 Generic drugs in stock 0.616 0.600 0.559 0.595 0.379 0.487 0.769 195 Antimalarials in stock 0.500 0.534 0.533 0.521 0.603 0.988 0.538 195 Basic HF equipment 0.733 0.700 0.710 0.715 0.534 0.785 0.326 194 Sterilization equipment material 0.271 0.250 0.220 0.250 0.413 0.635 0.714 194 Vaccination equipment 0.894 0.940 0.867 0.904 0.572 0.112 0.196 194 Postnatal care equipment 0.906 0.844 0.840 0.866 0.195 0.946 0.167 194 Delivery equipment 0.345 0.304 0.265 0.309 0.201 0.548 0.473 194 HF receives funds from the Ministry of Health 13 0.153 0.164 0.240 0.179 0.229 0.303 0.850 195 HF receives funds from fees paid by patients 0.931 0.973 0.960 0.954 0.496 0.702 0.241 195 HF receives funds from the sale of drugs 0.722 0.890 0.900 0.831 0.017 0.866 0.010 195 HF receives funds from Faith-Based Bodies 0.028 0.027 0.000 0.021 0.238 0.242 0.989 195 HF receives funds from private organizations 0.028 0.000 0.000 0.010 0.238 - 0.154 195 HF receives funds from NGOs 0.111 0.041 0.040 0.067 0.162 0.976 0.113 195 HF receives funds from donors 0.083 0.027 0.080 0.062 0.948 0.186 0.142 195 Other source of HF funding 0.125 0.164 0.140 0.144 0.811 0.716 0.504 195 Amount received (in Gourdes) 14 in 2015 1,154,463 620,600 1,276,596 987,806 0.892 0.273 0.411 194 Characteristics of Human Resources Number of staff administering care 8.518 4.671 4.640 6.432 0.000 0.888 0.000 1784 Average number of hours of care per day 4.833 4.943 5.087 4.922 0.292 0.571 0.597 1784 CHW satisfaction 0.771 0.743 0.745 0.754 0.002 0.848 0.001 539 Standard errors clustered by HF Source: Author’s calculations based on RBF IE (2015) 13 For this item (and for the ones directly below it), the entries are based on a question asking whether the HF receives funds from the Ministry of Health, from user fees, etc. 14 Refers to amounts received in user fees and other sources of cash, at baseline (before the RBF program started). 24 Table 11: Provision of Health Services in the Baseline Study Variables Average RBF Average TA Average Ctrl Average total p-value RBF/Ctrl p-value TA/Ctrl p-value RBF/TA N Number of patients who used a modern contraceptive method 7646 4657 5840 6007 0.583 0.548 0.251 163 Number of assisted deliveries 90 103 52 86 0.444 0.264 0.780 87 Number of postnatal care consultations 217 88 105 147 0.574 0.813 0.398 87 Number of BCG doses administered 259 231 130 215 0.091 0.332 0.791 177 Number of 3 rd Pentavalent vaccine doses administered 279 241 121 224 0.081 0.453 0.806 177 Number of 3 rd Poliovirus vaccine doses administered 261 141 134 182 0.171 0.850 0.129 177 Number of first measles vaccines administered 219 127 114 156 0.087 0.718 0.091 177 Number of tetanus vaccinations administered to pregnant women 176 196 137 173 0.476 0.361 0.762 177 Number of curative care consultations for children under five 43 76 33 53 0.592 0.131 0.209 195 Number of curative care consultations for all over five years of age 212 135 120 159 0.355 0.796 0.372 195 Number of children who underwent growth monitoring and nutrition counseling 35 31 26 31 0.660 0.597 0.843 166 Standard errors clustered by HF Source: Author’s calculations based on RBF IE (2015) The analyses of this IE of the RBF program on Haiti’s health system will focus solely on the data collected in service areas (for households) and on the HFs that participated in both the baseline study and the final survey. The analyses will therefore not include service areas and HFs that were visited over only one period. 2.4.2 Econometric models In accordance with the standard HRITF analysis strategy for RBF IEs, a difference-in-differences approach was used for all impact analyses. This methodology is even more appropriate for this study, given the quasi- experimental distribution (hybrid approach using randomization and matching) and the basic imbalances between the groups observed in Tables 9 and 11. Indeed, the advantage of this method is that it makes it easier to compare the changes in the outcomes between the intervention and control groups. By comparing these changes, the observed and unobserved characteristics that did not change over time are controlled in the models. In this way, the targeted impact of the RBF program will be estimated based on the parameters of interest &#3627409207; &#3627408524; outlined in the following equation: &#3627408512; &#3627408522;&#3627408523;������= &#3627409206; &#3627408523;+ &#3627409208; &#3627409360;&#3627409358;&#3627409359;������+ &#3627409207; &#3627409359;(������&#3627408489;������ &#3627408523;������ &#3627409360;&#3627409358;&#3627409359;������)+ &#3627409207; &#3627409360;(&#3627408488;������ &#3627408523;������ &#3627409360;&#3627409358;&#3627409359;������) +������&#3627408511; &#3627408522;������ + ������ &#3627408522;&#3627408523;������ where &#3627408512; &#3627408522;&#3627408523;������ represents a variable of interest assigned to an individual i in a HF or service area j for the period t. &#3627409206; &#3627408523; is the fixed effect of service area j. &#3627409208; &#3627409360;&#3627409358;&#3627409359;������ refers to the fixed effect for the 2019 period of the study. ������&#3627408489;������ &#3627408523; (TAj respectively) is a dummy variable indicating the treatment received in the context of RBF program implementation and which takes the value “1” if the district is served by a HF j of the RBF group (of the TA group, respectively) and “0” where this is not the case. ������ &#3627409360;&#3627409358;&#3627409359;������ is a dummy variable indicating the data collection period, and which takes the value “1” in relation to the 2019 IE survey and “0” for the baseline study. RBFjI2019 and TAjI2019 are interaction terms between each of the RBF and TA groups and the data collection period indicator. The parameters ������ ������ related to these terms measure the changes observed over the course of the study period between the control group and the corresponding treatment group. They therefore measure the impact of the treatment in each group. &#3627408511; &#3627408522;������ represents the vector of the control variables (for household data, this will refer to the characteristics of individual i; characteristics of the 25 household and service area j). When analyzing data on HFs, the control variables will include the type of HF, the source of funding for the intervention (WB or USAID), the Department, and the location of the HF, among others. It should therefore be noted that the effects of the other health programs and projects implemented in service areas are controlled in the different models 15 ) and ������ &#3627408522;&#3627408523;������ is the error term. Furthermore, we calculate cluster-robust standard errors in relation to service areas in order to take account of the potential correlation between the error terms across different parameters and time periods. 15 A dichotomous variable was developed for each health program and project implemented in the area covered by the study, namely PRISMA UNICEF CARITAS KORE-LAVI UNFPA PAHO/WHO. This variable will take the value “1” if the project or program is implemented in a service area and, where this is not the case, the value “0.” 26 3. IMPACT EVALUATION RESULTS This chapter will first examine households’ health-seeking behaviors. It will then outline the main results obtained using the estimates of the equation presented above. These estimates are presented according to place of residence, to capture any instances of social inequality in relation to health. It should be noted that in the tables below, the “post indicator” measures the effects between the baseline study and the final study observed in the control group, while “RBF/post-interaction” and “TA/post-interaction” measure the impact on the relevant indicator of RBF interventions accompanied by technical assistance and of interventions related exclusively to technical assistance, respectively. These may be interpreted as the difference in use of health services over the course of the study period between the control group and each of the treatment groups. The control variables included in the models are listed below each table. 3.1 Impacts on the Governance of Health Facilities Information on the administration and management of HFs was collected during the RBF evaluation surveys. The survey questions were posed directly to the person in charge of the particular HF or to their deputy, and focused, among other things, on the existence and operations of a management committee at the HF; the existence of terms of reference describing the duties and responsibilities of each position in the HF; and evaluations and their impact on funding for the particular HF. One of the strategies for achieving improved outcomes in the health system targeted by the RBF program is to improve the governance of HFs. Table 12 below shows the impact of RBF on the governance indicators selected for the purposes of this study. It indicates that neither RBF nor TA has had any significant impact on the operations of management committees at HFs nor on the existence of ToR for each senior position. There has also been no discernible impact on the level of HF funding as a result of the evaluations made of overall output. These results may be because these indicators are not included in Haiti’s RBF Program quality indicator list. 27 Table 12: Impacts of RBF on Administration and Management of Health Facilities † Existence of a functional management committee Existence of terms of reference for each position HF funding based on general performance evaluations (1) (2) (3) (4) (5) (6) (7) (8) (9) VARIABLES All Urban Rural All Urban Rural All Urban Rural Post dummy ( 1=endline) 0.0485 0.303 -0.0333 0.159 0.130 0.166 0.0706 -0.0475 0.110 (0.112) (0.226) (0.120) (0.153) (0.242) (0.195) (0.0760) (0.137) (0.0789) RBF/post-interaction 0.215 0.0310 0.249 0.111 0.133 0.103 0.131 0.318 -0.0261 (0.154) (0.252) (0.212) (0.190) (0.283) (0.270) (0.111) (0.193) (0.132) TA/post-interaction 0.00124 -0.298 0.136 -0.0744 -0.134 -0.0319 0.00976 0.169 -0.0685 (0.145) (0.280) (0.163) (0.189) (0.312) (0.239) (0.0900) (0.177) (0.0942) Constant 0.328 1.028*** 0.404 1.724*** -0.355 0.536** -1.186*** -0.391 0.116 (0.269) (0.375) (0.252) (0.288) (0.346) (0.262) (0.233) (0.305) (0.145) Number of observations 333 146 187 333 146 187 333 146 187 R-squared 0.601 0.628 0.604 0.596 0.604 0.575 0.537 0.562 0.517 P-value RBF vs TA 0.136 0.124 0.596 0.249 0.269 0.564 0.199 0.344 0.702 Mean for control group in the baseline sample 0.178 0.133 0.200 0.378 0.467 0.333 0.0222 0.0667 0 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 3.2 Impacts on the Quality of Health Services In order to increase the use of the health system, it is vital that quality health services are provided. Quality generally refers to factors such as safe and effective care that is in keeping with the preferences and needs of the people and communities being served; care that is provided on a timely basis in the context of an agreed framework, and that is dispensed equitably across populations; care that is coordinated along the entire health continuum and throughout the life course of individuals, while minimizing resource waste (WHO, 2019). The same study shows that poor quality care would not only be harmful to patients but would also wastes precious resources that could otherwise have been invested in other key sectors that promote economic and social development in order to improve the living conditions of citizens. The RBF program places special emphasis on the quality of health services. This is precisely why it is so vital to regularly verify the activities being carried out at the targeted HFs. In addition, RBF surveys have been used to collect information on service quality (infrastructure and availability). 3.2.1 Basic Infrastructure at Health Facilities The availability and quality of certain installations and amenities at HFs play a key role in determining the quality of care that they can provide. For example, the availability of a continuous supply of water and electricity in a HF and the existence of a waste disposal system provide health care staff with a clean and hygienic environment in which to dispense quality care. In addition, the availability of means of transport would make it easier to transfer patients who have been referred to a higher-level health care facility. Heads of HFs have been requested to provide information on their main source of water and electricity, and disruptions in/lack of access to water and electricity services during the preceding seven days, as well as their duration. This information has facilitated a better understanding of the extent to which HFs benefit from 28 a continuous supply of water and electricity. Information was also collected on the availability of a waste disposal system. While these indicators were later added to Haiti’s list of RBF indicators, the results show that, following various RBF interventions, there have been improvements (relative to the control group) in the availability of a regular supply of potable water (+29.4 percentage points as a whole and +39.5 percentage points in rural areas) and electricity (+14.5 percentage points for all areas included in the study). RBF initiatives have also had a positive and significant impact on the availability of a functioning waste disposal system in HFs (+17.2 percentage points as a whole and +30.1 percentage points in urban areas). This impact is not statistically different from that of the TA group (RBF vs TA p-value > 0.1). 16 Table 13: Impacts of RBF Initiatives on Basic Infrastructure at Health Facilities † Availability of drinking water at all times in the last 7 days Availability of electricity permanently in the last 7 days Availability of an evacuation system (1) (2) (3) (4) (5) (6) (7) (8) (9) VARIABLES All Urban Rural All Urban Rural All Urban Rural Post dummy ( 1=endline) 0.0483 0.326** -0.109 0.315*** 0.203 0.375*** 0.000695 -0.102 0.0321 (0.0767) (0.133) (0.0871) (0.0806) (0.140) (0.0975) (0.0800) (0.103) (0.0956) RBF/post-interaction 0.294*** 0.0917 0.395*** 0.145* 0.211 0.0849 0.172* 0.301** 0.0824 (0.0928) (0.139) (0.120) (0.0872) (0.134) (0.119) (0.113) (0.146) (0.178) TA/post-interaction 0.108 -0.0595 0.183 0.0609 0.0310 0.0860 0.0368 0.138 -0.00686 (0.0964) (0.164) (0.117) (0.0951) (0.170) (0.112) (0.115) (0.154) (0.155) Constant 0.317** 0.241 0.481* 0.535*** 0.549*** 0.316 1.071*** 0.268 0.949*** (0.127) (0.149) (0.273) (0.135) (0.188) (0.216) (0.225) (0.215) (0.163) Number of observations 333 146 187 333 146 187 333 146 187 R-squared 0.120 0.191 0.083 0.217 0.184 0.249 0.513 0.548 0.501 P-value RBF vs TA 0.0378 0.237 0.0911 0.299 0.165 0.992 0.229 0.271 0.619 Mean for control group in the baseline sample 0.222 0.200 0.233 0.222 0.267 0.200 0.889 0.933 0.867 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 3.2.2 Availability of Health Services In order to ensure that the provision of health care is timely and equitable across populations, it is important that certain maternal and child health services be made available at HFs. These include prenatal health services, delivery care, postnatal care, and immunization services. During the RBF surveys, heads of HFs were requested, among other things, to provide information on the availability and operations of the services mentioned above. The outcomes show that there was no statistically significant impact on the availability of these services as a result of either the RBF or the TA programs (Table 14). The lack of a significant impact on prenatal care and immunization services could be due to the high proportion of HFs in the control group that were found in the baseline study to be already offering these services. Furthermore, this proportion appears to have improved over the 2015-2019 period with the free distribution of specific material to deliver these services in the HFs covered by the program (in all three study groups). Also, it is important to note that in practice, the availability of a new health service in 16 There were some concurrent capital investments in structures for potable water and for electricity (especially the former) during this period, across the country but in a rather haphazard way, with some sites benefitting more than others (for all 3 groups in the study). 29 a HF would require the support (human resources, inputs, technical assistance, among many others) of higher-level decision-making entities of the State, which is difficult to obtain. Table 14: Impacts of RBF on the Availability of Health Services at the Health Facility † Availability of ANC services Availability of delivery services Availability of postpartum care services Availability of immunization services (1) (2) (3) (1) (2) (3) (1) (2) (3) (1) (2) (3) VARIABLES All Urban Rural All Urban Rural All Urban Rural All Urban Rural Post dummy ( 1=endline) 0.0234 0.0734 0.00922 -0.0428 0.154 -0.165 0.118 -0.0272 0.158 0.00098 5 -0.0103 0.00553 (0.0553) (0.107) (0.0681) (0.0895) (0.146) (0.100) (0.118) (0.159) (0.159) (0.0452) (0.0214) (0.0646) RBF/post-interaction 0.00690 -0.0993 0.113 0.0850 -0.134 0.205 -0.143 -0.110 -0.0639 0.0128 0.00444 0.0368 (0.0797) (0.110) (0.139) (0.126) (0.173) (0.193) (0.147) (0.197) (0.193) (0.0597) (0.0549) (0.0861) TA/post-interaction -0.0537 -0.0728 -0.0498 0.00566 -0.128 0.0593 -0.146 0.0413 -0.230 0.0720 0.189 0.00254 (0.0653) (0.104) (0.0859) (0.113) (0.161) (0.145) (0.154) (0.212) (0.207) (0.0673) (0.124) (0.0693) Constant 0.871*** 1.027*** 0.926*** 1.543*** 0.859*** 1.655*** 1.402*** -0.213 0.330 1.288*** 1.240*** 0.996*** (0.112) (0.0654) (0.145) (0.238) (0.252) (0.202) (0.240) (0.270) (0.211) (0.0758) (0.156) (0.0557) Number of observations 333 146 187 333 146 187 333 146 187 333 146 187 R-squared 0.568 0.518 0.592 0.814 0.872 0.741 0.714 0.689 0.719 0.655 0.653 0.700 P-value RBF vs TA 0.383 0.508 0.238 0.480 0.955 0.457 0.982 0.426 0.337 0.354 0.172 0.532 Mean for control group in the baseline sample 0.933 0.933 0.933 0.356 0.400 0.333 0.467 0.600 0.400 0.911 0.867 0.933 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 3.2.3 Amenities and Basic Care Compliance with precautionary measures, the capacity to conduct laboratory tests, the availability of vaccines and stocks of generic drugs are also key elements for ensuring the quality of a health service. The evaluation questionnaire that was administered to HFs in the course of the RBF surveys was used, among other things, to collect information on compliance by these facilities with universal precautionary measures. This information included the availability of a consultation room, a safety box in the consultation room for sharp or pointed waste, running water and soap, constant supplies of disinfectant products, and incinerators, as well as compliance with protocols for sterilizing medical equipment and devices. All these factors were used to develop an indicator reflecting the extent to which HFs complied with universal protocols, and a score was assigned to each HF. Information was also collected from the medical testing services available at HFs to determine their capacity to conduct certain laboratory tests or analyses (16, in total), including red and white blood cell count, estimation of hemoglobin (hematocrit) levels, blood group, and diagnosis of malaria (thick drops and smears or rapid diagnostic testing), etc. These 16 laboratory tests were used to develop a composite indicator and, thereafter, to assign a score to each facility, reflecting their capacity to conduct laboratory tests. According to WHO, setting up systems for the procurement and stock management of medicines, vaccines, and supplies necessary to deliver all the required services is a key criterion for achieving global standards for quality health care. For this reason, pharmacists and heads of HFs were requested to provide information on the stocks of available vaccines (BCG); OPV (oral drops); tetanus vaccines (injection); hepatitis B vaccine; measles and rubella vaccines; HiB vaccines; and pentavalent vaccines. Each of these vaccines is assigned 30 the value “1,” if in stock, or the value “0,” if out of stock. Average scores reflecting the extent to which stocks of all the vaccines listed were available were then calculated for each HF. Questions were also posed about the availability of supplies of common drugs, such as Tetracycline 1% eye ointment; paracetamol 500 mg tablets; paracetamol 125 mg/5 ml syrup; ibuprofen 400 mg tablets; ibuprofen 100 mg/5 ml syrup; amoxicillin 500 mg tablets; amoxicillin 125 mg/5 ml suspension; oral rehydration salts powder; iron 60 mg tablets; folic acid 5 mg tablets; co-trimoxazole 400 mg/80 mg tablets; metronidazole 500 mg tablets; metronidazole 125 mg/5 ml suspension; vitamin A gel capsules 200,000 IU; and albendazole 400 mg tablets As was done in relation to vaccines, average scores reflecting the extent to which stocks of all the common drugs listed were available were then calculated for each HF. Table 15: Impacts of RBF on the Quality of Health Services Provided at Health Facilities † Compliance with universal hygiene protocols Ability to perform laboratory tests Vaccines in stock Generic drugs in stock (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES All Urban Rural All Urban Rural All Urban Rural All Urban Rural Post dummy ( 1=endline) -0.0202 0.0740 -0.0764 -0.0301 -0.0441 -0.0300 0.0525 0.239 -0.0154 -0.220*** -0.133 -0.262*** (0.0493) (0.0678) (0.0595) (0.0258) (0.0419) (0.0345) (0.0809) (0.170) (0.0945) (0.0812) (0.187) (0.0878) RBF/post-interaction 0.0989** -0.0513 0.218*** 0.0992** 0.0410 0.191** 0.144* -0.0550 0.257* 0.177** 0.0144 0.325*** (0.0605) (0.0791) (0.0818) (0.0528) (0.0696) (0.0771) (0.104) (0.181) (0.133) (0.105) (0.201) (0.134) TA/post-interaction 0.0308 -0.128 0.116 0.0169 0.0357 0.0105 -0.00362 -0.127 0.0392 0.0520 0.0242 0.0572 (0.0660) (0.0926) (0.0842) (0.0400) (0.0607) (0.0526) (0.111) (0.213) (0.126) (0.101) (0.209) (0.115) Constant 1.062*** 0.510*** 1.134*** 0.157** -0.195*** 1.229*** 0.376** 1.137*** 0.655*** 0.859*** 0.445** 1.458*** (0.110) (0.130) (0.0847) (0.0764) (0.0659) (0.0848) (0.170) (0.225) (0.178) (0.169) (0.180) (0.156) Number of observations 333 146 187 333 146 187 333 146 187 333 146 187 R-squared 0.663 0.539 0.638 0.926 0.892 0.906 0.620 0.644 0.625 0.622 0.591 0.647 P-value RBF vs TA 0.223 0.327 0.203 0.144 0.943 0.0293 0.147 0.656 0.110 0.172 0.940 0.0434 Mean for control group in the baseline sample 0.663 0.724 0.633 0.296 0.537 0.175 0.352 0.181 0.438 0.575 0.529 0.598 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. The results (Table 15) show that, in general, RBF has a positive and significant impact on amenities and the provision of basic health care, especially in rural areas. The following may therefore be noted in the rural HFs of the RBF group: increased compliance with universal protocols for the delivery of health care (+21.8 percentage points as against +9.9 percentage points for all HFs of the group combined); greater capacity to conduct laboratory tests and examinations (+19.1 percentage points versus +9.9 percentage points for all HFs combined); increased availability of vaccine supplies (+25.7 percentage points versus +14.4 percentage points for all HFs combined) and supplies of generic drugs (+32.5 percentage points as against +17.7 percentage points for all HFs combined). (These are all increases relative to the control group). These positive outcomes could be attributed to the RBF approach, which stipulates that all grants are contingent on compliance with certain explicit procedures. The RBF payments to facilities enabled them to purchase various items and supplies, and to finance the transport to the facilities of items such as vaccines where the items themselves were provided for free, but not transport costs to the facilities – hence enabling 31 the positive results seen in Table 15. The RBF approach also includes a system of regular inspections and audits to ensure compliance and improve the quality of health care services (Musgrove, 2011). 3.2.4 Malaria Care Services Malaria is one of the endemic diseases in Haiti where epidemic outbreaks sometimes occur after the rainy seasons. Once considered a rural disease, malaria, which appears to be spreading at an alarming rate, can now be found in urban areas. Treatment at HFs involves a rapid microscopic test and a combination treatment of chloroquine and primaquine. The quality of antimalarial care (availability of stocks of antimalarials and malaria rapid diagnostic test (RDT) kits would undoubtedly contribute to efforts to achieve the country’s goal of eliminating the disease by 2020. Table 16: Impacts of RBF on Malaria Care Services at Health Facilities † Antimalarial drugs in stock Antimalarial kits in stock (1) (2) (3) (4) (5) (6) VARIABLES All Urban Rural All Urban Rural Post dummy ( 1=endline) -0.185** -0.184 -0.170 -0.283* -0.111 -0.371** (0.0877) (0.151) (0.112) (0.151) (0.281) (0.176) RBF/post-interaction 0.280*** 0.197 0.403*** 0.215 0.0327 0.241 (0.121) (0.187) (0.163) (0.197) (0.320) (0.269) TA/post-interaction 0.0799 0.232 -0.0131 0.0773 -0.101 0.140 (0.124) (0.215) (0.149) (0.182) (0.332) (0.213) Constant 0.744*** 0.602** 0.286 -4.012*** 0.0648 3.410*** (0.218) (0.282) (0.186) (0.801) (0.421) (0.405) Number of observations 333 146 187 333 146 187 R-squared 0.576 0.575 0.611 0.578 0.546 0.626 P-value RBF vs TA 0.101 0.854 0.0111 0.434 0.618 0.693 Mean for control group in the baseline sample 0.548 0.533 0.556 0.644 0.467 0.733 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. The results presented in Table 16 above show that during the period of the study, the stock of antimalarial drugs increased by 28 percentage points in HFs in the RBF group, and even more (40.3 percentage points) in rural HFs. Furthermore, the effects observed in HFs in the RBF group differ considerably from those observed in HFs in the TA group in rural areas (RBF vs TA p-values= 0.0111< 0.1). The results nevertheless reveal that RBF interventions had no significant impact on the stock availability of RDT kits. The availability of RDT kits in HFs does not depend on the HFs themselves; rather, interventions for other programs such as the Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome (HIV/AIDS) and Tuberculosis programs have a greater impact on the availability of these kits. 3.2.5 Medical Equipment and Supplies The standards relating to the characteristics of HFs for the delivery of quality health care services focus on the importance of systems that ensure the availability of equipment, supplies, and technologies needed to ensure the delivery of efficient services. As part of the IE surveys of the RBF program, the medical equipment 32 and supplies in the HFs were verified and inspected and information on their quantities and functionality was collected. These primarily include general equipment and supplies, as well as equipment and supplies for sterilization, immunization, prenatal care, delivery care, and neonatal care. 17 For each category of medical equipment and supplies, consideration was given to the most critical ones for HFs in Haiti, in accordance with the manual for the package of essential services, developed by the MSPP in 2015. A composite indicator that provides average scores reflecting the equipment and supply levels of HFs was then calculated for each category of equipment and supplies. Table 17 below shows the impact of the RBF program on the provision of medical equipment and supplies to HFs. RBF had a significant and positive impact on the provision of general equipment and supplies to HFs (+11.7 percentage points in all facilities and +6.3 percentage points in rural areas). With respect to immunization equipment and supplies, HFs in the RBF group (+10.6 percentage points), particularly those located in rural areas (+19.5 percentage points), were better equipped as a result of the program’s interventions. Intervention packages in the RBF group also improved the provision of delivery care and postnatal medical equipment and supplies to HFs (+5.5 percentage points overall). (These are all increases relative to the control group). In addition, the p-values (> 0.1 in all models) in the RBF and TA groups show no significant statistical differences between the two intervention groups. 17 Supplies and smaller types of equipment largely had to be financed by the HFs themselves out of their own funds including funds from RBF payments and user fees – but sometimes donors financed these items, under certain types of donor programs. The latter occurred for some of the HFs in all three groups under the study, and these effects have been captured by dummy variables (at the HF level) for the main donor programs in the country. 33 Table 17: Impacts of RBF on the Provision of Medical Equipment and Supplies to Health Facilities † General Equipment Sterilization Equipment Vaccination Equipment Prenatal Care Equipment Equipment For Delivery and Postnatal Care (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) VARIABLES All Urban Rural All Urban Rural All Urban Rural All Urban Rural All Urban Rural Post dummy ( 1=endline) -0.215*** -0.154 -0.223*** 0.0814 -0.0543 0.128 -0.0820 0.0043 2 -0.126 -0.0208 0.0137 0.000720 -0.0319 0.0104 -0.0339 (0.0539) (0.136) (0.0488) (0.0864) (0.200) (0.0884) (0.0634 ) (0.0697 ) (0.0864 ) (0.0693) (0.125) (0.0903) (0.0647) (0.109) (0.0818 ) RBF/post-interaction 0.117** 0.121 0.0632* -0.0139 0.0235 0.0530 0.106* -0.0154 0.195** 0.0618 -0.0489 0.145 0.0552* -0.0349 0.147 (0.0714) (0.144) (0.0876) (0.134) (0.246) (0.162) (0.0794 ) (0.0967 ) (0.106) (0.0838) (0.124) (0.124) (0.0801) (0.119) (0.120) TA/post-interaction 0.0562 0.0276 0.0592 -0.0378 0.105 -0.0981 0.0722 0.0046 3 0.101 0.0724 0.0288 0.0834 0.0729 0.0327 0.0898 (0.0714) (0.161) (0.0713) (0.116) (0.247) (0.128) (0.0766 ) (0.115) (0.0969 ) (0.0839) (0.134) (0.108) (0.0833) (0.125) (0.109) Constant 0.414*** 0.785** * 0.618*** 1.044*** 1.136*** 0.134 1.077*** 1.574** * 0.818** * 0.672** 0.955*** 1.175*** 0.693*** 0.874*** 0.645** * (0.113) (0.166) (0.0889) (0.210) (0.266) (0.149) (0.109) (0.180) (0.0941 ) (0.310) (0.193) (0.219) (0.132) (0.148) (0.136) Number of observations 333 146 187 333 146 187 333 146 187 333 146 187 333 146 187 R-squared 0.687 0.574 0.754 0.543 0.472 0.580 0.641 0.636 0.660 0.665 0.636 0.692 0.658 0.629 0.669 P-value RBF vs TA 0.362 0.399 0.963 0.855 0.688 0.373 0.611 0.857 0.235 0.882 0.302 0.600 0.802 0.413 0.617 Mean for control group in the baseline sample 0.693 0.707 0.687 0.211 0.400 0.117 0.881 0.800 0.922 0.850 0.900 0.825 0.850 0.900 0.825 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. The absence of significant impacts on sterilization equipment and prenatal care equipment could be explained by the high costs of these types of equipment, relative to the RBF tariffs and payment amounts, especially at the start of program implementation when these tariffs were low (see Figure 3). 18 18 In theory, these larger types of more costly equipment should be financed by parties other than the HFs, in particular the Departmental Directorates of Health. In practice, there are gaps in financing of this equipment. In some cases, donors’ step in and finance this equipment but in other cases, the HFs may have to be finance the equipment on their own if feasible. For the larger HFs, this may be feasible but not for the smaller ones. 34 Figure 3: Evolution of Total Quarterly RBF Payments (in US Dollars) to RBF Health Facilities 19 Over the Period 2016-2019 Source: Haiti RBF Contracting Unit, 2020 3.2.6 Prenatal Care With 359 maternal deaths per 100,000 live births in 2015, the MMR remains high in Haiti. Most of these maternal deaths, which are often due to obstetrical complications, could be avoided if medical care were to be provided during pregnancy, delivery, and in the postnatal period. All pregnant women must therefore have access to prenatal care during pregnancy. However, financial constraints, geographic inaccessibility, and the quality of health services can often be barriers that deny certain women, especially women in rural areas, access to this prenatal care. During the RBF surveys conducted in the HFs, several women who were visiting the facility for their first prenatal consultation were directly observed during the consultation and interviewed once it was completed. The information collected was used to assess the quality of prenatal consultations and the level of satisfaction of the expectant mothers. The results of Table 18 below show an improvement (+24.4 percentage points overall) in the quality of prenatal consultations (as the following four key/recommended areas were broached/discussed with the patients: (i) skilled birth attendance or delivery plan, (ii) at least two danger signs during pregnancy, (iii) nutrition, and (iv) exclusive breastfeeding) in HFs in the RBF group. The impact obtained in theHFs in the RBF group are statistically different from those observed in HFs in the TA group (RBF vs TA p-value > 0.01). The improvement in the quality of prenatal consultations observed in HFs in the RBF group may explain the level of satisfaction noted by expectant mothers interviewed after their consultation (14.7 percentage points). This level of satisfaction is even higher in rural facilities (+16.9 percentage points in the RBF group and +50.6 percentage points in the TA group, relative to the control group). The satisfaction levels of expectant mothers with the cost of prenatal consultations and the associated medical tests also increased (+25.8 percentage points in the RBF group and +48.5 percentage points in the TA group, relative to the control group). 19 Only including the HFs in the IE under the WB and not USAID, since data on the latter were not available. The weighted average tariffs was calculated using the total quantities of the year 2017 as weights. 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 0 20000 40000 60000 80000 100000 120000 140000 160000 Weighted Average of RBF Tariffs Total RBF Payments Quarter Weighted Average of RBF Tariffs Total RBF Payments 35 Table 18: Impacts of RBF on the Quality of ANC Consultations and Related Satisfaction † ANC with the three main related routine services (blood pressure, weight, and iron/folic acid supplementation) ANC with patient education on the four key elements (childbirth plan, danger signs, nutrition, and exclusive breastfeeding) ANC patient satisfaction Satisfaction regarding the costs of ANC and related medical procedures (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES All Urban Rural All Urban Rural All Urban Rural All Urban Rural Post dummy (1= endline) 0.0170 -0.0290 0.0440 0.224** 0.0846 0.254* -0.225** -0.256 -0.230 -0.121 -0.00461 -0.280 (0.0464) (0.111) (0.0431) (0.108) (0.146) (0.150) (0.0970) (0.196) (0.184) (0.355) (0.478) (0.314) RBF/post-interaction -0.0256 -0.0136 -0.00547 0.244** 0.178 0.188 0.147* 0.167 0.169*** 0.258* 0.304 0.177 (0.0572) (0.105) (0.0591) (0.111) (0.121) (0.197) (0.100) (0.152) (0.0383) (0.297) (0.371) (0.342) TA/post-interaction -0.0364 0.0118 -0.0668 0.0243 0.0401 0.0832 0.172 0.180 0.506* 0.485* 0.402 0.622 (0.0685) (0.144) (0.0702) (0.130) (0.179) (0.175) (0.133) (0.144) (0.292) (0.272) (0.317) (0.617) Constant 1.019*** 0.656*** 0.819*** 0.333 0.915*** 0.675** 0.282** 0.632*** 0.836*** 0.244 0.366 1.728* (0.122) (0.147) (0.131) (0.255) (0.261) (0.279) (0.135) (0.206) (0.238) (0.735) (0.296) (1.021) Number of observations 943 501 442 943 501 442 940 499 441 940 499 441 R-squared 0.453 0.459 0.485 0.543 0.558 0.537 0.820 0.498 0.951 0.617 0.278 0.932 P-value RBF vs TA 0.861 0.812 0.280 0.0246 0.199 0.158 0.852 0.932 0.209 0.448 0.779 0.498 Mean for control group in the baseline sample 0.948 0.953 0.944 0.257 0.256 0.258 0.894 0.857 0.955 0.673 0.581 0.810 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 3.2.7 Under-five Child Health Care According to the EMMUS-VI, approximately 1 in 12 children in Haiti dies before the age of five. Very little progress appears to have been made in the area of child health care services in Haiti, as the infant and child mortality rate remains extremely high across the country. Between 2012 and 2017, the infant and child mortality rate remained virtually stable: the risk of death between birth and a child’s fifth birthday is 81 per 1,000 live births, making Haiti the country with the highest child mortality rate in the Latin America and Caribbean region. To improve child survival rates, the country must focus in particular on access to health services and the quality of child health care, regardless of their socioeconomic background. Curative consultations for children under the age of five were observed firsthand with a view to assessing the quality of child health care under the RBF program. The opinions of parents who accompanied their children to a HF were also gathered at the end of the consultation. The results in Table 19 below do not show any significant statistical impacts on the quality of curative consultations for children (relating mainly to the IMCI guidelines). This could be attributed to the fact that these indicators on curative consultations for children under five were not incentivized under the RBF program. However, parents expressed greater satisfaction (+4.1 percentage points in HFs in the RBF group, relative to the control group) with the cost of these consultations. 36 Table 19: Impacts of RBF on the Quality of Consultations for Children Under Five and the Related Levels of Satisfaction † Curative consultations with assessment of all IMCI danger signs according to IMCI guidelines Curative consultations with assessment of common childhood symptoms according to IMCI guidelines Patient satisfaction with the consultation Satisfied with consultation fees (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES All Urban Rural All Urban Rural All Urban Rural All Urban Rural Post dummy (1= endline) -0.294** -0.148 -0.360** -0.00744 -0.0186 0.0112 -0.0432 -0.0413 -0.0471 -0.0269 -0.0633 -0.0135 (0.133) (0.196) (0.143) (0.0317) (0.0796) (0.0208) (0.0268) (0.0317) (0.0412) (0.0584) (0.115) (0.0635) RBF/post-interaction -0.0849 -0.308 0.118 0.0233 0.01000 0.0475 -0.00664 -0.00997 0.0388 0.0410* 0.0525 0.0823 (0.160) (0.208) (0.198) (0.0516) (0.0923) (0.0442) (0.0308) (0.0326) (0.0806) (0.0685) (0.114) (0.0793) TA/post-interaction -0.153 -0.272 0.0478 0.0217 -0.00487 0.0710 0.0189 0.0181 0.0362 0.0227 0.0598 0.0101 (0.170) (0.252) (0.183) (0.0553) (0.108) (0.0583) (0.0349) (0.0389) (0.114) (0.0803) (0.158) (0.0798) Constant 0.612** 0.532*** 1.360*** -0.218*** 0.274*** 0.164 0.369*** 0.371*** 0.373*** 1.078*** 1.302*** 1.421*** (0.305) (0.146) (0.339) (0.0725) (0.0590) (0.113) (0.00743) (0.0134) (0.0494) (0.0909) (0.173) (0.0947) Number of observations 980 505 475 980 505 475 376 245 131 911 475 436 R-squared 0.490 0.494 0.570 0.292 0.293 0.324 0.589 0.549 0.688 0.340 0.276 0.432 P-value RBF vs TA 0.615 0.855 0.719 0.974 0.849 0.689 0.376 0.376 0.976 0.805 0.953 0.205 Mean for control group in the baseline sample 0.389 0.216 0.500 0.0153 0.0196 0.0125 0.404 0.389 0.417 0.900 0.837 0.944 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 3.3 Impacts on Human Resources The knowledge, skills, and attitudes of health care providers are critical to ensuring the delivery of quality services (WHO, 2015). Providing care in accordance with recommendations is key to improving the use of health services. During RBF surveys, data on sociodemographic (age, gender, marital status, level of education, etc.) and professional (grade, position, salary, training, knowledge, time devoted to health care, WHO well-being index, motivation and level of satisfaction) characteristics of health care personnel were collected. The results of the evaluation of the program’s impact on health care personnel presented in Table 20 reveal a positive and significant impact of TA on the number of health care personnel in HFs (+6.9 percentage points), especially in urban HFs (+18.9 percentage points). It bears noting, therefore, that the effects observed in urban HFs in the TA group differ greatly from those observed in HFs in the RBF group (RBF vs TA p-values < 0.01). This increase in the number of health care providers in HFs in the TA group has, without a doubt, had an impact on the distribution and organization of work within HFs, which could lead to a reduction in the average number of hours devoted to health care each day: -14 percentage points in all HFs in the TA group and -24.7 percentage points in rural areas. These impacts of TA on the average number of hours of health care do not differ significantly from those observed in HFs in the RBF group (RBF vs TA p-values > 0.01). It should be noted that the indicators on staff size and the organization of work were not incentivized under the RBF program. This, among other factors, could explain the absence of any noted significant impact in Table 20. 37 Table 20: Impacts of RBF on the Delivery of Health Care Services in Health Facilities † Number of staff administering care Average hours of care per day (1) (2) (3) (4) (5) (6) VARIABLES All Urban Rur All Urban Rur Post dummy (1= endline) -0.144*** -0.157** -0.129** -0.0882 -0.200 -0.0290 (0.0450) (0.0614) (0.0603) (0.0686) (0.149) (0.0739) RBF/post-interaction 0.00477 0.0320 -0.0126 -0.0421 0.0238 -0.0508 (0.0555) (0.0746) (0.0745) (0.0893) (0.151) (0.113) TA/post-interaction 0.0694* 0.189*** 0.00131 -0.140* 0.0637 -0.247** (0.0542) (0.0674) (0.0726) (0.0999) (0.167) (0.119) Constant -0.172*** 0.102 0.219*** 1.234*** 0.578*** 0.278** (0.0481) (0.0887) (0.0686) (0.102) (0.207) (0.140) Number of observations 4.111 1.804 2.307 4.090 1.799 2.291 R-squared 0.179 0.168 0.192 0.339 0.208 0.422 P-value RBF vs TA 0.127 0.00818 0.805 0.316 0.754 0.141 Mean for control group in the baseline sample 0.460 0.375 0.496 0.645 0.742 0.603 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. The results on the wages and motivation of health care personnel in HFs (Table 21) show a significant and positive impact of RBF on the timely payment of wages in urban HFs only (+9.4 percentage points and +15 percentage points in urban facilities, relative to the control group). However, in the control group HFs, there appears to be an increasing delay in the payment of wages (-73.7 percentage points in all facilities). 20 TA also had a positive and significant impact on the motivation of health workers in urban HFs (+6.6 percentage points, relative to the control group). The observed effects on the motivation of health care personnel in HFs in the TA group and those observed in HFs in the RBF group are statistically different (RBF vs TA p-value = 0.0666 < 0.1) in this urban area. Further qualitative work to better understand these issues and the related mechanisms by which RBF works – looking at rural and urban areas separately – would be warranted (going beyond the qualitative work already done and summarized in Annex C.) 20 This represents a time trend observed in all groups. 38 Table 21: Impacts of RBF on the Wages and Motivation of Health Care Personnel in Health Facilities † Payment of wages on time Personal motivation (1) (2) (3) (10) (11) (12) VARIABLES All Urban Rural All Urban Rural Post dummy (1= endline) -0.737*** -0.891*** -0.616*** 0.0456** 0.0167 0.0598** (0.0719) (0.0764) (0.114) (0.0189) (0.0288) (0.0251) RBF/post-interaction 0.0940* 0.150** 0.115 -0.0133 0.0149 -0.0433 (0.0691) (0.0720) (0.125) (0.0209) (0.0281) (0.0313) TA/post-interaction 0.00760 0.0923 -0.000266 0.0178 0.0663* -0.0383 (0.0730) (0.0836) (0.111) (0.0241) (0.0360) (0.0292) Constant 0.744*** 0.939*** 1.199*** 0.819*** 0.684*** 0.759*** (0.108) (0.163) (0.235) (0.0284) (0.0475) (0.0575) Number of observations 1.016 571 445 1.015 571 444 R-squared 0.666 0.684 0.679 0.484 0.503 0.519 P-value RBF vs TA 0.190 0.467 0.329 0.132 0.0666 0.871 Mean for control group in the baseline sample 0.797 0.877 0.732 0.740 0.735 0.744 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. Table 22 below shows that, as a result of RBF interventions, health workers are also increasingly satisfied with their work environment in HFs (+2.6 percentage points) especially in rural areas (+12.3 percentage points). Health workers, especially those in rural HFs in the RBF group also have a favorable view (+9.3 percentage points) of the management and oversight required under the approach. These impacts are statically different from those observed in rural HFs in the TA group (RBF vs TA p-values <0.01). We note that, irrespective of the type of program intervention, the wage expectations of health care personnel were not met. Table 22: Impacts of RBF on the Level of Satisfaction of Health Care Personnel in Health Facilities † Work environment satisfaction Salary satisfaction Satisfaction with management and supervision at HF (1) (2) (3) (4) (5) (6) (7) (8) (9) VARIABLES All Urban Rural All Urban Rural All Urban Rural Post dummy (1= endline) 0.0217 0.0210 0.0318 0.0141 -0.0409 0.0368 0.0472 0.0279 0.0754* (0.0208) (0.0275) (0.0302) (0.0255) (0.0349) (0.0368) (0.0332) (0.0565) (0.0403) RBF/post-interaction 0.0261* 0.0204 0.123*** -0.0107 0.00854 0.0166 -0.0251 -0.0136 0.0925* (0.0252) (0.0322) (0.0410) (0.0316) (0.0322) (0.0602) (0.0373) (0.0577) (0.0524) TA/post-interaction -0.00355 -0.0339 0.00856 -0.0279 0.0285 -0.0712 -0.0462 -0.0261 -0.0762 (0.0265) (0.0333) (0.0354) (0.0321) (0.0307) (0.0551) (0.0379) (0.0606) (0.0493) Constant 0.569*** 0.778*** 0.546*** 0.0108 0.171 -0.0279 0.653*** 0.841*** 0.198** (0.0369) (0.0637) (0.0751) (0.0731) (0.104) (0.132) (0.0538) (0.0765) (0.100) Number of observations 1.015 571 444 1.016 571 445 1.015 571 444 R-squared 0.447 0.384 0.595 0.197 0.219 0.202 0.343 0.303 0.436 P-value RBF vs TA 0.403 0.0983 0.000688 0.563 0.577 0.115 0.482 0.294 0.769 Mean for control group in the baseline sample 0.563 0.565 0.562 0.0156 0.0175 0.0141 0.467 0.497 0.443 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 39 3.4 Impacts on the Use of Reproductive Health Services In addition to surveys in HFs, surveys were also conducted in communities (households) served by the surveyed HFs in order to measure the impact of the RBF program on the demand for health services. Apart from the sociodemographic and economic characteristics of households, data on the reproductive health behaviors of all women between the ages of 15 and 49 living in the surveyed households, as well as on the prenatal, delivery, and postnatal care behaviors of women between the ages of 15 and 49 who recently gave birth, were collected. 3.4.1 Family Planning Behaviors The indicators on reproductive health behaviors identified during this study are: unmet family planning needs (i.e., the proportion of women who are not using contraceptive methods and for whom a pregnancy could pose a problem; the use of modern contraceptive methods; the use of modern contraceptive methods among women in a common-law relationship and the use of any form of contraception. The results of Table 23 below show that the program’s interventions have led to a reduction in unmet family planning needs, particularly in the TA service areas (-9.8 percentage points in all HFs and -10.4 percentage points in rural areas, relative to the control group). However, in the control group facilities, these unmet family planning needs increased (+17.3 percentage points in all HFs: +17.7 percentage points in urban areas and +17.5 percentage points in rural areas, relative to the control group). 21 Furthermore, we also note that the effects observed in the two intervention groups are statically different (RBF vs TA p-values being <0.1). Table 23: Impacts of RBF on the Reproductive Health Behaviors of Women Between Ages of 15 and 49 † Unmet need for family planning Use of modern contraceptive methods Use of modern contraceptive methods among married women or in stable union Use of any contraceptive methods (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES All Urban Rural All Urban Rural All Urban Rural All Urban Rural Post dummy (1= endline) 0.173*** 0.177*** 0.175*** -0.0520 -0.0224 -0.0724* -0.0419 0.00915 -0.0728* 0.0124 0.00306 0.0124 (0.0247) (0.0508) (0.0257) (0.0334) (0.0600) (0.0403) (0.0349) (0.0594) (0.0429) (0.0301) (0.0503) (0.0371) RBF/post-interaction -0.0273 -0.0208 -0.0420 -0.0142 -0.0414 0.0153 0.00605 -0.0416 0.0489 -0.0450 -0.0352 -0.0339 (0.0282) (0.0510) (0.0371) (0.0373) (0.0619) (0.0480) (0.0413) (0.0643) (0.0519) (0.0393) (0.0569) (0.0575) TA/post-interaction -0.0975*** -0.0880 -0.104*** 0.0499 0.0152 0.0708 0.0507 0.00315 0.0743 0.0149 0.0511 -0.00325 (0.0272) (0.0542) (0.0292) (0.0374) (0.0657) (0.0451) (0.0400) (0.0669) (0.0492) (0.0421) (0.0725) (0.0512) Constant 0.119*** -0.142** 0.0187 0.530*** 0.118 0.850*** 0.997*** 0.0485 0.476*** 0.317*** -0.760*** 0.278*** (0.0365) (0.0621) (0.0466) (0.0514) (0.0739) (0.0564) (0.0542) (0.0958) (0.0632) (0.0470) (0.0692) (0.0554) Number of observations 12.867 5.863 7.004 6.847 3.156 3.691 5.214 2.310 2.904 9.822 4.531 5.291 R-squared 0.059 0.067 0.058 0.078 0.076 0.080 0.086 0.097 0.081 0.072 0.079 0.067 P-value RBF vs TA 0.00289 0.0688 0.0613 0.0307 0.210 0.155 0.185 0.394 0.543 0.155 0.195 0.603 Mean for control group in the baseline sample 0.305 0.337 0.288 0.416 0.419 0.415 0.416 0.411 0.419 0.324 0.343 0.314 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. Taboos in Haitian culture still constitute a barrier to access reproductive health care despite the progress made in the use of modern family planning methods. A qualitative study conducted to explain the findings of 21 This represents a time trend observed in all groups. 40 the IE of RBF – see Annex C – indicated that certain cultural conceptions, preconceived ideas and the level of education are still barriers to the use of family planning methods, especially modern methods in Haiti 22 . Only three out of ten women aged 15-49 used these methods in 2017 (EMMUS, 2017), and this trend remained unchanged in 2019. This could explain the absence of a statistically significant impact on the use of modern family planning methods among women of childbearing age whether they are in stable unions or not. In addition, indications are that individuals in Haiti – especially women – are often reluctant to answer questions about family planning honestly, especially in front of others (e.g. their husbands). This could potentially lead to inaccurate findings. 3.4.2 Prenatal Care Behaviors The RBF IE indicators for the prenatal care behaviors of women who recently gave birth are: at least one prenatal consultation; at least four prenatal consultations; first prenatal consultation in the first four months of the pregnancy and at least two tetanus shots during pregnancy. The results of Table 29 below show a significant albeit negative impact of TA in urban areas (-5.8 percentage points, relative to the control group) with respect to having at least one prenatal consultation, while it is on the rise in the control groups (the coefficients of the post indicator being positive and significant regardless of place of residence) during the 2015-2019 implementation period of the program. 23 The absence of a significant impact in the RBF group of the program with respect to at least one prenatal consultation could be attributable to the high level (81.2 percent overall) of this indicator in the control group for the baseline study, and the challenges related to the ‘last mile’ of reaching the remaining 20%, which may relate more to demand-side constraints. Table 24: Impacts of RBF on the Use of Prenatal Care Services Among Recently Pregnant Women Between the Ages of 15 and 49 (24 Months Preceding Data Collection) † Had at least one prenatal visit Had at least four prenatal visits Had the first prenatal visit in the first four months of pregnancy Had two doses of anti-tetanus vaccine during pregnancy (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES All Urban Rural All Urban Rural All Urban Rural All Urban Rural Post dummy (1= endline) 0.116*** 0.109*** 0.116*** 0.167*** 0.224*** 0.136*** 0.105*** 0.112** 0.103*** 0.230*** 0.189*** 0.255*** (0.0193) (0.0238) (0.0260) (0.0307) (0.0414) (0.0408) (0.0292) (0.0496) (0.0363) (0.0298) (0.0405) (0.0400) RBF/post-interaction 0.00390 0.0185 -0.00241 0.00971 -0.0357 0.0273* -0.0223 -0.0527 0.0172* 0.00460 0.0168 0.0288 (0.0267) (0.0333) (0.0383) (0.0381) (0.0478) (0.0634) (0.0369) (0.0574) (0.0479) (0.0341) (0.0452) (0.0530) TA/post-interaction -0.0223 -0.0577* 0.00178 0.00619 -0.0968 0.0645 -0.0578 -0.0698 -0.0528 -0.00641 0.104** -0.0680 (0.0268) (0.0302) (0.0364) (0.0399) (0.0621) (0.0504) (0.0374) (0.0562) (0.0490) (0.0369) (0.0527) (0.0476) Constant 0.843*** 0.918*** 0.678*** 0.379*** 0.304*** 0.297*** 0.830*** 0.841*** 0.848*** 0.0760 -0.0166 0.442*** (0.0415) (0.0438) (0.0573) (0.0534) (0.0729) (0.0625) (0.0555) (0.0656) (0.0673) (0.0574) (0.0831) (0.0736) Number of observations 6.241 2.835 3.406 6.241 2.835 3.406 6.241 2.835 3.406 6.241 2.835 3.406 R-squared 0.140 0.117 0.162 0.122 0.114 0.132 0.096 0.086 0.100 0.113 0.107 0.124 P-value RBF vs TA 0.331 0.0186 0.912 0.922 0.244 0.531 0.317 0.710 0.153 0.736 0.0428 0.0578 Mean for control group in the baseline sample 0.814 0.833 0.803 0.579 0.579 0.579 0.665 0.679 0.658 0.485 0.533 0.459 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 22 Also, in Haiti's RBF program, the definition of family planning indicators changed significantly in the period of the IE study. The original indicator was “Number of users of any modern FP method” with the definition “Number of women aged 15-49 who use or whose partner uses any modern FP method”. It has been replaced by the indicator "Number of patients who accept the use any modern FP method" with the following definition "Number of people, men aged 15-59 or women aged 15-49, who agree to use for the first time in her life a modern method of contraception within the framework of a structured public or private FP program”. 23 Reflecting a rising time trend in all Groups. 41 The results show a significant and positive impact (+2.8 percentage points, relative to the control group) in rural HFs in the RBF group among recently pregnant women between the ages of 15 and 49 who had at least four prenatal consultations. This is also the case for recently pregnant women between the ages of 15 and 49 who had their first prenatal consultation in the first four months of pregnancy. The results also show that during the period of the study (2015-2019), the proportion of women who had at least four prenatal consultations and of women who had their first prenatal consultation in the first four months of pregnancy increased significantly (positive and significant coefficients of the post indicator) in the facilities in the control group. 24 The number of these women who received two tetanus shots during pregnancy also increased (+10.4 percentage points) in the HFs in the TA group. No significant impact of RBF or TA on the other indicators was observed. 3.4.3 Delivery and Postnatal Care Behaviors The following indicators will be used to evaluate the impact of the program on the use of delivery and postnatal care services: delivery assisted by a health professional; delivery in a HF; at least one postnatal checkup within two months after delivery; and iron supplementation after delivery. The results of Table 25 below show that RBF increased the share of recently pregnant women between the ages of 15 and 49 who gave birth with the assistance of a health professional (+9.2 percentage points, relative to the control group) and in a modern HF (approximately +4 percentage points) in rural areas. These observed positive and significant impacts in the RBF group are statically different from the impacts observed in the TA group (RBF vs TA p- values being <0.1). However, TA was associated with a positive and significant impact on postnatal iron supplementation among recently pregnant women between the ages of 15 and 49 (+10.8 percentage points overall and +13.6 percentage points in rural areas, relative to the control group). 24 Reflecting a positive and significant time trend across all groups. 42 Table 25: Impacts of RBF on the Use of Obstetric and Postnatal Care Services Among Recently Pregnant Women Between the Ages of 15 and 49 (24 Months Preceding Data Collection) † Delivery assisted by a health professional Delivery at a HF At least one postnatal visit within two months after delivery Taking iron supplements after childbirth (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES All Urban Rural All Urban Rural All Urban Rural All Urban Rural Post dummy (1= endline) 0.0294 0.0807 0.00838 0.108*** 0.142** 0.0912** 0.0495 0.0856 0.0293 -0.0195 0.0412 -0.0420 (0.0332) (0.0610) (0.0394) (0.0372) (0.0655) (0.0453) (0.0353) (0.0645) (0.0431) (0.0317) (0.0497) (0.0391) RBF/post-interaction 0.0285 -0.0544 0.0912** -0.0135 -0.0706 0.0397* 0.00558 0.0188 -0.0381 0.0546 -0.000551 0.0588 (0.0413) (0.0682) (0.0523) (0.0458) (0.0734) (0.0588) (0.0433) (0.0680) (0.0647) (0.0391) (0.0549) (0.0543) TA/post-interaction -0.0295 -0.0759 -0.00783 -0.0619 -0.114 -0.0330 0.0492 0.0903 0.0277 0.108*** 0.0397 0.136** (0.0416) (0.0762) (0.0493) (0.0453) (0.0825) (0.0538) (0.0447) (0.0800) (0.0531) (0.0404) (0.0577) (0.0525) Constant 0.0546 0.480*** 0.752*** -0.0855 0.399*** 0.612*** 0.222*** 0.261*** 0.404*** 0.107** -0.111 0.299*** (0.0657) (0.0942) (0.0712) (0.0653) (0.0843) (0.0743) (0.0534) (0.0835) (0.0774) (0.0484) (0.0754) (0.0646) Number of observations 5.391 2.409 2.982 5.391 2.409 2.982 5.573 2.502 3.071 5.573 2.502 3.071 R-squared 0.264 0.211 0.240 0.275 0.219 0.251 0.100 0.104 0.105 0.119 0.114 0.118 P-value RBF vs TA 0.155 0.745 0.0562 0.250 0.537 0.170 0.316 0.274 0.304 0.170 0.435 0.182 Mean for control group in the baseline sample 0.453 0.533 0.414 0.364 0.467 0.314 0.528 0.515 0.535 0.277 0.287 0.272 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. As for postnatal consultations, it should be noted that this indicator was not incentivized under the RBF program in Haiti, and these visits are generally carried out by CHWs in households within 72 hours after childbirth. This might explain the lack of significant impact of the program on postnatal care as defined in the table above. In this table and in the IE surveys, postnatal consultations were defined as postnatal visits by women to a HF within two months of a delivery. 3.5 Impacts on the Use of Child Preventive Health Services Preventive health services for children under five primarily entailed immunizations. To assess the impact of the RBF program on the use of these services, information on the immunization status of all children under the age of five living in the surveyed households was obtained from an immunization card and/or the mother. This information was then used to calculate the proportion of fully immunized children under the age of one year based on their immunization cards. This was achieved by taking account of the immunization schedule in Haiti to include only the vaccines to be administered for that age group. These vaccines are BCG, polio 0, 1, and 2, and Hib 1, 2, and 3. However, according to the recommended immunization schedule, most childhood vaccines should be received before a child’s first birthday. To avoid including infants who, because of their age, should not have yet completed their immunization schedule, we also calculated the proportion of fully immunized children between the ages of 12 and 23 months, based on their immunization cards. To that end, we added the polio 3 and measles vaccines to the abovementioned list of vaccines for children under the age of one year. While immunization coverage generally declined in the control areas between 2015 and 2019 25 (negative and significant coefficients of the post indicator), the results presented in Table 26 below reveal a significant and positive impact of RBF on full immunization coverage among children between the ages of 12 and 23 25 Reflecting a declining time trend across all Groups. 43 months based on their immunization card (+6.5 percentage points overall and +15.9 percentage points in urban areas, relative to the control group). 26 This immunization coverage improved for this age group of children in urban areas with TA (+18.3 percentage points, relative to the control group). The absence of a significant impact in rural areas may be attributable to a number of constraints related to the cold chain, lack of a reliable power supply needed for the storage of vaccines, all common challenges in these areas. The absence of an impact of the program on immunization coverage for children between the ages of 0-11 months could be explained by the fact that some infants would not yet have received all their vaccines because of their age. Furthermore, the results in Table 30 also show, in broad terms, a decline in full immunization over the period of the study in the control areas for children between the ages of 0-11 months and for children between the ages of 12 and 23 months. Table 26: Impacts of RBF on the Immunization of Children Under Five † Children 0-11 months fully vaccinated based on vaccination records Children 12-23 months fully vaccinated based on vaccination records (1) (2) (3) (4) (5) (6) VARIABLES All Urban Rural All Urban Rural Post dummy (1= endline) -0.202*** -0.215*** -0.204*** -0.228*** -0.342*** -0.171*** (0.0408) (0.0547) (0.0546) (0.0440) (0.0712) (0.0537) RBF/post-interaction -0.0171 -0.0256 0.0256 0.0644* 0.159** 0.0302 (0.0463) (0.0588) (0.0666) (0.0520) (0.0767) (0.0769) TA/post-interaction 0.00175 -0.00956 0.00464 0.0193 0.183** -0.0705 (0.0477) (0.0748) (0.0601) (0.0516) (0.0842) (0.0612) Constant 0.179** 0.0631 0.885*** 0.177** 0.0661 1.031*** (0.0724) (0.136) (0.0687) (0.0876) (0.142) (0.112) Number of observations 3.740 1.687 2.053 3.097 1.401 1.696 R-squared 0.216 0.226 0.213 0.142 0.150 0.148 P-value RBF vs TA 0.651 0.808 0.714 0.320 0.710 0.139 Mean for control group in the baseline sample 0.404 0.432 0.390 0.402 0.442 0.381 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 3.6 Impacts on the Health Status of the Population The health status of the population refers here to the nutritional status of children under the age of five; to health expenditure in households and health-seeking behaviors. To assess the impact of the RBF program on the nutritional status of children under five, data on the height and weight of all children under five were collected from households during the two surveys conducted in 2015 and 2019. We have used the WHO Child Growth Standards to construct the related indicators. We have thus calculated a z-score for the height- for-age, weight-for-age, and weight-for-height indicators for children. These z-scores represent the nutritional status of each child in accordance with international standards for the healthy development of children. We used these scores to calculate the prevalence of stunting (height-for-age is below -2 standard deviations of 26 The coefficient of the “Post dummy” variable for these regressions is 0.228 the full sample, and 0.342 for the urban-only sample – indicating overall reductions (in all groups as a whole) of 22.8% and 34.2% respectively. Hence the number of children fully vaccinated still decreased in the RBF areas, but by less than in other areas. 44 the median), underweight (weight-for-age is below -2 standard deviations of the median) and wasting (weight- for-age is below -3 standard deviations of the median). The results of the IE in Table 27 below show an increase in the prevalence of stunting (+9.6 percentage points overall and +9.9 percentage points in rural areas, relative to the control group) with TA. However, the prevalence of underweight declined (-5.6 percentage points) in rural areas with RBF. While a decline in the prevalence of wasting was also observed with RBF (negative interaction term), particularly in rural areas, this decrease was not statistically significant. These mixed impacts on the nutritional status of children under five years-old in Haiti could be explained, among other things, by the relatively short duration (three years) of program implementation, but also by the effects of other uncontrolled factors in the framework of this study (such as factors related to food insecurity). Table 27: Impact of RBF on the Nutritional Status of Children Between the Ages of 6 and 59 Months † Stunting Underweight Wasting (1) (2) (3) (4) (5) (6) (7) (8) (9) VARIABLES All Urban Rural All Urban Rural All Urban Rural Post dummy (1= endline) -0.00289 0.0324 -0.0194 0.0187 -0.0106 0.0349 0.0196 0.0120 0.0238 (0.0360) (0.0457) (0.0481) (0.0395) (0.0488) (0.0541) (0.0381) (0.0673) (0.0459) RBF/post-interaction 0.0648 0.0378 0.0882 0.00698 0.0560 -0.0551* -0.0106 0.00237 -0.0276 (0.0427) (0.0502) (0.0668) (0.0416) (0.0545) (0.0082) (0.0419) (0.0691) (0.0551) TA/post-interaction 0.0961** 0.0883 0.0991* 0.0203 0.0342 0.0216 -0.00664 -0.0133 0.00536 (0.0479) (0.0747) (0.0595) (0.0428) (0.0572) (0.0580) (0.0436) (0.0751) (0.0522) Constant 0.731*** 0.770*** 0.495*** 0.336*** 0.395*** 0.572*** 0.332*** 0.362*** 0.762*** (0.0620) (0.0926) (0.0893) (0.0559) (0.0803) (0.0612) (0.0569) (0.0759) (0.0670) Number of observations 4.429 2.041 2.388 4.275 1.983 2.292 4.275 1.983 2.292 R-squared 0.083 0.100 0.081 0.072 0.074 0.080 0.082 0.081 0.094 P-value RBF vs TA 0.471 0.479 0.863 0.677 0.581 0.109 0.895 0.737 0.415 Mean for control group in the baseline sample 0.307 0.264 0.329 0.158 0.145 0.164 0.142 0.140 0.144 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. Table 28 shows that, as a result of RBF, there are fewer and fewer households with health expenditure exceeding their income, particularly in rural areas (-5.1 percentage points and -2 percentage points overall, relative to the control group). The results also show that the observed impacts in the RBF and TA groups are statistically different in all sites (RBF vs TA p-values = 0.0185 <0.1) as well as in rural areas (RBF vs TA p-values <0.01). Moreover, in the control group areas, the proportion of households with CHE or health costs that exceeded their usual income increased between 2015 and 2019, regardless of place of residence (positive and significant coefficient of the post indicator). 45 Table 28: Impacts of RBF on Household Health Expenditure † Costs exceeding the financial means of the household (4) (5) (6) VARIABLES All Urban Rural Post dummy (1= endline) 0.0734*** 0.0666** 0.0757*** (0.0148) (0.0260) (0.0181) RBF/post-interaction -0.0203* 0.00462 -0.0509** (0.0200) (0.0317) (0.0268) TA/post-interaction 0.0324 0.0503 0.0236 (0.0231) (0.0424) (0.0274) Constant 0.0131 0.0737*** -0.0713*** (0.0174) (0.0263) (0.0213) Number of observations 9.356 4.125 5.231 R-squared 0.055 0.049 0.060 P-value RBF vs TA 0.0184 0.229 0.00964 Mean for control group in the baseline sample 0.0993 0.115 0.0920 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. Table 29: Summary of RBF IE Results in Haiti RBF IE Indicators Incentivized under the RBF program? RBF TA All Urban Rural All Urban Rural Comments (for the results regarding RBF) Administration and Management of HFs Existence of a functional management committee No These results could be explained by the fact that these indicators were not part of the quality indicators incentivized in Haiti’s RBF Program. Existence of terms of reference for each position No HF funding based on general performance assessments No Basic Infrastructure of HFs Availability of drinking water at all times in the last seven days Yes The indicator was introduced late in the RBF implementation period Availability of electricity permanently in the last seven days Yes The indicator was introduced late in the RBF implementation period Availability of an evacuation system Yes Availability of Health Services Availability of ANC services Yes For prenatal care and vaccination services, this lack of significant impact could be explained by the high proportion in the baseline study of control HFs already offering these services. In addition, this proportion seemed to have improved during the period 2015-2019. Furthermore, it is important to note that in practice, offering a new health service would require the support (human resources, inputs, technical assistance, etc.) of higher-level decision- making entities of the State. Availability of delivery services Yes Availability of postpartum care services Yes Availability of immunization services Yes Quality of Service in HFs Compliance with universal hygiene protocols Yes Ability to perform laboratory tests Yes Vaccines in stock Yes Generic drugs in stock Yes Antimalarial drugs in stock Yes Equipment at HFs Basic HF equipment Yes 46 RBF IE Indicators Incentivized under the RBF program? RBF TA All Urban Rural All Urban Rural Comments (for the results regarding RBF) Sterilization equipment Yes The absence of significant impacts on sterilization equipment and prenatal care equipment could be explained by the high costs of these types of equipment, relative to the RBF tariffs and payment amounts, especially at the start of program implementation when these tariffs were low. Vaccination equipment Yes Prenatal care equipment Yes The absence of significant impacts on sterilization equipment and prenatal care equipment could be explained by the high costs of these types of equipment, relative to the RBF tariffs and payment amounts, especially at the start of program implementation when these tariffs were low. Childbirth and postnatal care equipment Yes Quality of Prenatal Consultations and Patient Satisfaction ANC services with the three main routine services (blood pressure control, weight measurement, and folic acid and iron supplementation) Yes ANC services with patient education on the four key elements (childbirth plan, danger signs, nutrition, exclusive breastfeeding) Yes ANC services patient satisfaction No Satisfaction for the costs of ANC services No Quality of Consultations for Children Under Five Years and Patient Satisfaction Curative consultations with assessment of all IMCI danger signs according to IMCI guidelines No The lack of impact could be explained by the fact that these indicators were not incentivized under the RBF program in Haiti. Curative consultations with assessment of common childhood symptoms according to IMCI guidelines No Patient satisfaction related to services No Patient satisfaction related to fees No Human Resources Number of staff administering care No Average number of hours of care No Payment of wages on time No Satisfaction with the work environment No Personal motivation Salary satisfaction No Satisfaction with management and supervision at HFs No Reproductive Health Behaviors of Women Aged 15-49 Unmet need for family planning Yes Use of modern contraceptive methods Yes A qualitative study conducted to explain the findings of the IE of RBF – see Annex C – indicated that certain cultural conceptions, preconceived ideas and the level of education are still barriers to the use of family planning methods, especially modern methods in Haiti. Use of modern contraceptive methods among married women or in stable union Yes Use of any contraceptive methods Yes Use of Prenatal Care by Recently Pregnant Women Aged 15-49 Pregnant women with at least one prenatal consultation Yes The absence of a significant impact in the RBF group on the number of pregnant women having at least one ANC visit could be explained by the high coverage (81.2% overall) of this indicator in the control group at baseline. Pregnant women with four prenatal consultations Yes Accessing the first ANC visit in the first four months of pregnancy Yes 47 RBF IE Indicators Incentivized under the RBF program? RBF TA All Urban Rural All Urban Rural Comments (for the results regarding RBF) Receiving at least two doses of anti- tetanus vaccine during pregnancy Yes Obstetric and Postnatal Care for Recently Pregnant Women Aged 15-49 Delivery assisted by a health professional Yes Delivery at a HF Yes Use of at least one postnatal visit within two months after delivery No This indicator is not incentivized within the framework of the RBF program in Haiti, and these are generally carried out by CHWs in households within 72 hours after childbirth Taking iron supplements after delivery No Vaccination of Children Under Five Years Children 0-11 months fully vaccinated based on vaccination records Yes Children 12-23 months fully vaccinated based on vaccination records Yes Nutritional Status of Children Under Five Years Stunting No These indicators were not among the indicators incentivized in the RBF program. These mixed impacts on the nutritional status of children under five years-old in Haiti could be explained, among other things, by the relatively short duration (three years) of program implementation, but also by the effects of other uncontrolled factors in the framework of this study. Underweight No Wasting No Household Health Expenditure Health costs exceeding the financial means of the household No Impact (*** p<0.01, ** p<0.05) Impact (* p<0.1) Impact (*** p<0.01, ** p<0.05) Impact (* p<0.1) 48 4. DISCUSSION The Republic of Haiti occupies the Western portion of the island of Hispaniola. It has a young population; children under-five years of age represent 11.3% of an estimated population of 11,411,527 inhabitants in 2018 and children between 0 and 14 years old represent 32.8%. The median age of the population is 23. Women of reproductive age represent 26.8% of the population. With a Human Development Index of 0.498 in 2018 (United Nations Development Programme, 2018), a poverty rate of 59%, and a Gini index of 0.61 in 2012, Haiti is a LIC where health indicators (determined by the quantity and quality of health services, the use of maternal health services, etc.) – even if they have been improved over the last years – remain at a low level. The challenges of the healthcare system to reach the SDG are still numerous. On the other hand, the country has aimed to improve supply and demand of health care services through the implementation of a Results Based Financing program (RBF) (MSPP, 2012). RBF is in fact a managerial tool of health care systems. It was designed to increase the efficiency of inputs in the health care system in order to improve the quality of primary maternal health care services. This is done by paying bonuses to teams of health care workers to incentivize the quality and quantity of their provides services. In order to investigate the impact of the reforms that have been provided by the RBF program in the Haitian health care system, a quasi-experimental design has been introduced (a baseline survey in 2015, with randomization or matching of HFs in study groups – RBF, AT and Control – and an endline survey in 2019). The IE of the program has been based on the five questions below: 1. Does the RBF program improve children’s health (nutrition)? 2. Does the RBF program improve coverage and quality of priority health services related to maternal and child health? 3. What are the effects of the RBF program on the Haitian health care system, as far as human resources are concerned, as well as the financing of health care services and its governance? 4. What is the impact of the RBF program on catastrophic expenses of households? Does the RBF program contribute to improve financial protection of the households? A household is considered to incur catastrophic health care expenses when 25% or more of its non-food expenses are spent on health (WHO, WB, 2014). 5. Does technical assistance or local coaching/supervision alone improve the coverage and quality of priority health services related to maternal and child health compared to RBF together with technical assistance? The findings of this study are summarized below, focusing on the impacts of RBF together with TA (Group 1 of the study), as well as the impacts of TA only (Group 2 of the study). The size of these impacts (if any) was quantified in the study and are cited in percentage points below. (They are derived from the coefficients of the RBF and TA variables in the regressions reported above.) It is important to note that these are all impacts relative to the Control Group. Hence the percentage figures presented below, showing the impacts of RBF 49 (Group 1) and of TA (Group 2), are all percentage changes in the RBF areas or the TA areas relative to those in the Control Group. 4.1 Children’s health In terms of child health, this study found that RBF interventions accompanied by technical assistance have improved the use of health services, especially in rural areas. Indeed, the results reveal a decrease in the prevalence of underweight of -5.5 percentage points in rural areas of the RBF group (relative to the Control Group). 27 (In this document, the “RBF Group” refers to those that had RBF together with TA, while the “TA Group” refers to those that had TA only.) There was also a relative decrease in the prevalence of wasting, but not significantly. The proportion of children aged 12-23 months that was fully vaccinated has deteriorated overall in all areas over the study period, but it deteriorated less in the RBF areas. In the latter, there was an improvement relative to the overall deteriorating time trend (i.e. relative to the control group) of +6.5 percentage points overall, and +15.9 percentage points in urban areas. 4.2 Coverage and quality of priority health services related to maternal and child health RBF has generally improved the quality of health service provision, especially in rural areas. The results show that in HFs that were part of the RBF group, there were improvements (relative to the Control Group) in the evacuation system (+17.2 percentage points, and +30.1 percentage points in urban areas); respect for universal hygiene protocols and standards in the administration of care (+9.9 percentage points, and +21.8 percentage points in rural areas); and capability of carrying out laboratory tests (+10.0 points percentage points, and +19.1 percentage points in rural areas). 28 Furthermore, the availability of health-related supplies has improved (relative to the Control Group) due to RBF: vaccines (+14.4 percentage points, and +25.7 percentage points in rural areas); drugs (+17.7 percentage points, and +32.5 percentage points in rural areas); and antimalarials (+28.0 percentage points, and +40.3 percentage points in rural areas). In absolute terms, however, the availability of drugs still fell overall in RBF areas in the overall sample – but less than in other areas. 29 The RBF program (i.e. RBF together with TA) also positively impacted the availability of equipment, especially in rural HFs where general equipment availability improved due to RBF (+11.7 percentage points, and +6.3 percentage points in rural areas, relative to the control group) while in the Control group, the same variable deteriorated over the period of the study. (In absolute terms, this variable declined in RBF areas, but less than in other areas. 30 ) Availability of vaccination equipment also improved due to RBF (+10.6 percentage points, and +19.5 percentage points in rural areas); and availability of delivery and postnatal care equipment increased (+5.5 percentage points). All of which would help explain the significant improvement observed in rural areas in deliveries assisted by a health professional (+9.1 percentage points) and in deliveries at HFs 27 Overall, there was no overall time trend observed over the study period. Hence, this constitutes an overall reduction in absolute as well as relative terms in underweight due to RBF in rural areas. 28 In all these cases, no time trend was observed over the study period. Hence, these illustrate an overall improvement due to RBF in absolute as well as relative terms, over the study period. 29 As seen from Table 15, for generic drugs in stock, there was an overall time trend of -22.0% in the overall larger sample – exceeding the relative improvement of +17.7% due to RBF. However, in rural areas, for drugs, the relative improvement due to RBF (32.5%) exceeded the overall time trend of -26.2%. Similarly, for vaccines in stock and antimalarials, no significant time trend was observed. 30 This is indicated by the fact that the overall time trend observed for general equipment availability was -21.5% overall, and -22.3% in rural areas – exceeding the relative improvement in RBF areas (+11.7% and +6.3% respectively). As for the other measures of equipment availability, there was no overall time trend observed over the study period. 50 (nearly +4.0 percentage points). 31 As for the lower impact observed on deliveries at a HF, this could be explained by the substantial increase in the proportion of women who gave birth at a HF in the entire sample as a whole over the study period, including in control areas. RBF has also affected ANC behaviors. We note a statistically significant but relatively modest increase in the proportion of recently pregnant rural women aged 15-49 (over 24 months preceding data collection) having accessed at least four ANC visits (nearly +3.0 percentage points) and the proportion of those who had their first ANC visit within the first four months of pregnancy (nearly +2.0 percentage points). 32 In terms of quality of the health services, the results show that in the RBF group, prenatal care visits improved by +24.4 percentage points 33 , which would justify the overall improvement of satisfaction observed in pregnant women/clients (+14.6 percentage points) and especially in rural areas (+16.9 percentage points) after a prenatal consultation. 34 Also, the costs of ANC and medical laboratories related to ANC have become more accessible: pregnant women/clients are more satisfied regarding this (+25.8 percentage points). 35 Regarding children’s curative consultations, RBF led to an increase in the satisfaction of the parents with regard to financial accessibility to HFs in the SIs of the RBF group (+4.1 percentage points) 36 , although the improvement in the quality of these curative consultations was not found to be statistically significant. 4.3 Health system in Haiti: governance, human resources and health financing Regarding the motivation of health personnel, the study found that RBF (relative to the Control Group) has had a positive impact on the regularity and decreased delays in salary payments (+9.4 percentage points, including +15.0 percentage points in urban areas) while these deteriorated in the control HFs over the period 2015-2020. (Overall, the regularity and delays in salary payments deteriorated even in the RBF areas, but less than in the overall sample. Further qualitative work, beyond what has already been done – see Annex C – is warranted to better understand these findings regarding the impact of RBF.) We also note, due to RBF, an improvement in the working environment (+2.6 percentage points of health workers overall, and +12.3 percentage points for workers in rural areas); as well as increased satisfaction with management and supervision in rural HFs (+9.3 percentage points). 37 Furthermore, the quality and physical condition of the equipment made available to CHWs in rural areas also improved (+6.3 percentage points). 4.4 Health expenditure Overall, RBF has improved the financial protection of households. It was noted that the proportion of households with health expenditure exceeding their income decreased (-2.1 percentage points), especially in rural areas (-5.1 percentage points) due to RBF (relative to the Control Group). However, in the control areas, the financial protection of households deteriorated significantly regardless of the place of residence, which could be partially explained by the endogenous and exogenous shocks (economic and security crises) that the country has been facing. (The degree of financial protection still decreased in the RBF areas, but 31 For this variable, no time trend was observed over the study period. 32 All on top of significant time trends observed for these variables in all areas. 33 A substantial additional improvement on top of a significant time trend in the overall sample of 22.4%. 34 For this variable, no time trend was observed over the study period. 35 For this variable, no time trend was observed over the study period. 36 For this variable, no time trend was observed over the study period. 37 The latter occurred on top of a significant time trend in the overall sample. For the former, there was no significant time trend observed in the overall sample. 51 less than in the overall sample. 38 ) There are two ways in which RBF in Haiti may have led to increased financial protection of households. First, RBF has improved the coverage and quality of health services provided (as seen above), and this has likely improved the health status of households, hence leading to a reduced need for households to incur health spending. Second, the above results show that RBF has improved the availability of essential medicines at HFs, and this has likely led to a reduced need for individuals to purchase medicines at private pharmacies – hence, in turn, improving the financial protection of households. Further qualitative work (beyond what has already been done – see Annex C) would be useful to be able to better understand, among others, the pathways by which the RBF program improved the financial protection of households. 4.5 Technical assistance, coverage and quality of health services As for the areas where HFs underwent TA only, results revealed mixed impacts on the coverage and quality of health services in Haiti (relative to the Control Group). While the prevalence of stunting increased (+9.6 percentage points including +9.9 percentage points in rural areas 39 ), vaccination coverage among children aged 12-23 months in urban areas has improved significantly relatively to the Control Group and to the overall time trend (+18.3 percentage points). There was still a deterioration in vaccination coverage in urban areas in the TA-only group, but less than in the overall sample. (However, stunting is the result of many factors accumulated over many years, and there are likely many unobserved confounding factors here in the analysis). In terms of the reproductive health of women aged 15-49, TA significantly reduced unmet needs for family planning, relative to the Control Group and to the overall time trend (-9.8 percentage points, of which -10.4 percentage points in rural areas). Unmet needs for family planning still increased in TA-only areas, but less than in other areas. 40 Compared to the behaviors of recently pregnant women aged 15-49 in terms of prenatal, obstetric and postnatal care, our findings show a negative and significant impact due to TA – statistically significant at the 10% level – on women that had at least one prenatal consultation (-5.8 percentage points). 41 However, the study found improvements due to TA on taking at least 2 doses of anti-tetanus vaccine during pregnancy in urban areas (+10.4 percentage points); and also, on taking iron supplementation after delivery (+10.9 percentage points, and +13.6 percentage points in rural areas). 42 Regarding the impact on the quality of the health services, we can also note a positive and significant impact of TA on the general satisfaction of pregnant women/customers after having an antenatal consultation (+5.1 percentage points), and on financial access of pregnant women/customers regarding ANC and related medical procedures: (+48.5 percentage points). 43 38 As seen from Table 28, for this variable there was an overall time trend of 7.3% in the overall larger sample (7.6% in rural areas) – exceeding the relative improvement of +-2.1% (-5.1% in rural areas) due to RBF. 39 For this variable, no time trend was observed over the study period. 40 As seen from Table 23, for this variable there was an overall time trend of 17.3% in the overall larger sample (17.5% in rural areas) – exceeding the relative improvement of +-9.8% (-10.4% in rural areas) due to TA only. 41 However, due to a significant time trend of +10.9% (larger in absolute value than -5.8%), this variable still declined in TA areas – but less than in the overall sample. 42 On top of a significant time trend of +18.9% for the former. There was no significant time trend observed for the latter. 43 For this variable, there was no significant time trend. 52 There was also a relative increase in the number of staff at HFs due to TA (+6.9 percentage points, and +18.9 percentage points in urban areas 44 ) with the immediate consequence of reducing the average number of hours of care per day and per CHW (-14.0 percentage points, and -24.7 percentage points in rural areas 45 ), potentially due to a better distribution and organization of work within the IS. The personal motivation of health workers also due to TA, especially in urban areas (+6.6 percentage points). 4.6 Summary comments These positive impacts of program interventions under RBF, noted especially in rural areas, allow us to assert that, for several indicators, RBF improved the utilization and quality of child health services, the coverage and quality of priority health services related to maternal health, as well as the salary treatment of health workers. It also contributed to the financial protection of households. These results are all in line with the findings of previous IE studies examining the effects of RBF, in particular Soeters et al (2006), Sabri et al. (2007), Rusa et al. (2009), Basinga et al, (2011), Saïdou (2017) and De Walque et al (2021). As for the unexpected results and the absence of significant impact observed for other indicators, they are also found in other studies which overall find mixed evidence in favor of RBF (de Walque, Kandpal, Wagstaff et al. 2022). This could be explained – as mentioned above – by the limits, flaws and shortcomings in the implementation of the program, as well as the economic and security crises that hit the country during the program implementation period. In particular, the delays in the payment of performance-related bonuses and subsidies in the HFs of the RBF group also appear to have been substantial constraints to the efficient and effective implementation of RBF in Haiti (see Annex C). 4.7 Study limitations - The effects of other non-RBF health programs implemented in the study areas were difficult to monitor due to the lack of information on the type of interventions and support they provide as well as on the financial resources allocated to them. - The lack of sufficient information to assess and better control the effects of the economic and security crises that the country faced during the implementation of the program. - Statistical contamination in household data following the fact that patients do not always go to the HF assigned to them, as clearly shown by the analyses of household therapeutic behaviors during the implementation of the RBF program (Annex B). - While this study compares RBF to business-as-usual and technical assistance, it does not allow a comparison with other interventions on the supply-side (e.g., direct facility financing) or the demand- side (e.g., cash transfers or maternal vouchers) that would have similarly increased the budgets of facilities or households but without using the specific RBF mechanisms. - Inability under this quantitative IE study to understand the mechanisms by which the RBF program has impacted key variables, in particular those not incentivized under the program, such as the degree of financial protection of households (which was found to have improved due to RBF, despite not being directly incentivized under the RBF program.) For this reason, some qualitative work has 44 The former was less in magnitude than the time trend which was significant (-14.4%), but the former was greater in magnitude than the time trend which was also significant (-12.9%). Hence, for the overall sample, the number of staff still declined in TA-only areas, but less than in the overall sample. 45 For this variable, there was no significant time trend. 53 been done to better understand these mechanisms – see Annex C – and further detailed work along these lines is recommended below. 54 5. CONCLUSION The results of the RBF IE in Haiti reveal that the reforms brought about by the RBF program have led to improvements in the use and quality of certain child health services, an increase in the coverage and quality of priority health services related to maternal health as well as salary treatment of health workers – despite the deteriorating socioeconomic and political situation (worsening poverty, poor performance of the health system, growing violence and lack of security etc.). They also contributed to the financial protection of households. All this was observed despite numerous challenges facing the country and its health sector during this period, including rising insecurity, political instability and substantially falling overall resources for the health sector. But these reforms have not significantly impacted the availability of a number of health services, due to various reasons as mentioned above (see Table 29) including high initial coverage in some cases at the beginning of the implementation period. (The latter implies “last-mile” challenges of making further marginal improvements in cases where the coverage is already very high). Also, RBF (accompanied by TA and supervision) affects the coverage and quality of health services more than TA alone. In addition, the reforms brought about by the RBF seem to have more positive impacts on the supply of health services than on the demand for these services. This could be explained by insufficient incentives on the demand side. To this end, a policy discussion on the combination of demand-side and supply-side incentives would be helpful. As for the unexpected results observed within the framework of this study, they could be explained by the limits, faults and insufficiencies in the implementation of the program as well as by the limitations of the RBF approach. Thus, in view of these results, the following recommendations are made: On Analytical Aspects - Collect data to better monitor the effect of other non-RBF health interventions implemented by other donors. - Collect additional data to better appreciate the effects of the economic and security crises which would undoubtedly have jeopardized the effective and efficient implementation of these different interventions. - Conduct further qualitative work to better understand some of the mechanisms by which RBF has worked in Haiti (as discussed in various areas of the report above). On design and implementation aspects for the RBF program in Haiti 46 :  Consider introducing incentives specifically for consultations for under-five children (with appropriate protocols to be followed). The present IE exercise found that the RBF program in Haiti had no impact on the quality of child consultations, which is not surprising since this was not incentivized in the case of Haiti. Given the importance of basic child health services in any health system, it is suggested that this is explicitly included and incentivized in Haiti’s RBF program, as in the case of many RBF programs worldwide. 46 The program is still continuing, and with ongoing donor support as per during the IE implementation period. 55  For indicators incentivized under the RBF program over which HF workers and administrators have little control – such as RDT kits and the availability of certain types of services at a facility – it is advisable to reconsider whether these should indeed be incentivized under the program.  Make efforts to substantially reduce the delays in making RBF payments to HFs. These delays have been significant in the case of Haiti, especially towards the end of the IE implementation period, when security and other issues led to substantial delays in the verification process. The qualitative analysis found that these delays have substantially affected performance. One way of reducing these delays is to make advance payments to HFs for each quarter even before the verification process for that quarter has been completed – especially in the case of delays in the verification – which would be adjusted accordingly after the verification is completed.  Test out one or two small RBF pilots in urban areas, with modifications to see if the impact from RBF can be increased there. Currently, the impact from RBF has been significant in rural areas for most variables, but less so in urban areas. 47  Introduce RBF at the community level, initially on a pilot basis. This may be particularly helpful in the case of the family planning indicators, where RBF does not seem to have a statistically significant impact. The qualitative work done – see Annex C – indicates that actions to stimulate behavior change are key here, and this is where community-level actions by CHWs could be especially helpful. These would be incentivized under RBF at the community level. RBF at the community level may also help boost indicators in urban areas. 47 It is possible that this is due to the smaller sample size in urban areas. It is possible that the impacts in urban areas – where the relevant coefficients in the regressions have been generally of the right sign but not statistically significant – would have been statistically significant with a larger sample size. However, a priori the evidence so far indicates substantially lower impact in urban areas. 56 6. BIBLIOGRAPHIC REFERENCES Angrist, D.A. and J.-S. Pischke (2009): Mostly Harmless Econometrics: An empiricist’s companion. Princeton University Press. Basinga P, Gertler PJ, Binagwaho A et al. 2011. Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation. The Lancet 377: 1421–8. Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How much should we trust differences-in-differences estimates?. The Quarterly journal of economics, 119(1), 249-275. Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90(3), 414-427. Cameron, A. C., & Miller, D. L. (2015). A practitioner’s guide to cluster-robust inference.Carter, A. V., Schnepel, K. T., & Steigerwald, D. G. (2013). Asymptotic behavior of at test robust to cluster heterogeneity. Review of Economics and Statistics, (0). De Walque, D. B., Robyn, P. J., Saidou, H., Sorgho, G., & Steenland, M. W. (2017). Looking into the performance-based financing black box: evidence from an impact evaluation in the health sector in Cameroon. Health Policy and Planning 1-13. De Walque, Damien, Eeshani Kandpal, Adam Wagstaff, Jed Friedman, Sven Neelsen, Moritz Piatti- Fünfkirchen, Anja Sautmann, Gil Shapira, and Ellen Van de Poel. Forthcoming 2022. Improving Effective Coverage in Health: Do Financial Incentives Work? Policy Research Report. Washington, DC: World Bank. https://doi.org/10.1596/978-1-4648-1825-7. Eichler R, Auxila P, Pollock J. 2001. Output-based health care: paying for performance in Haiti. Public Policy in the Private Sector, Note 236. Washington, DC: World Bank. Online at: http://wwwwds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2002/05/14/000094946_0204 2504015950/Rendered/PDF/multi0page.pdf Eichler R, Levine R. 2009. Performance Incentives for Global Health: Potential and Pitfalls. Washington, DC: Center for Global Development. Imbens, G. W., & Kolesar, M. (2016). Robust standard errors in small samples: Some practical advice. Review of Economics and Statistics, 98(4), 701-712. Josephson M, et al. 2008. The same factors influence job turnover and long spells of sick leave–a 3-year ollow-up of Swedish nurses. Eur J Public Health, 18:380–385. Kivimaki M, et al. 2001. Sickness absence in hospital physicians: 2-year follow up study on determinants. Occup Environ Med, 58:361–366. Meessen B, Kashala JP, Musango L. 2007. ‘Output-based payment to boost staff productivity in public health centres: contracting in Kabutare district, Rwanda’. Bulletin of the WHO 85: 108–15 57 Ministère de la Santé et de la Population. 2014. Operational Manual for the implementation of RBF in Haiti. Unpublished document by MSPP. Ministère de la Santé et de la Population. 2012. Politique Nationale de la Sante d’Haïti. Port-au-Prince. Haiti: Ministère de la Sante Publique et de la Population, République d’Haïti. Online at: http://mspp.gouv.ht/site/downloads/PNS%2021juillet%20version%20finale.pdf Ministère de la Santé et de la Population. 2016. Le Paquet Essential de Services. Port-au-Prince. Haiti. https://mspp.gouv.ht/site/downloads/Manuel%20du%20PES%20Lancement%201er%20Septembre%20201 6%20compressed.pdf Musgrove, P. (2011) Financial and Other Rewards for Good Performance or Results: A Guided Tour of Concepts and Terms and a Short Glossary (Washington: Banque mondiale), https://www.rbfhealth.org/sites/rbf/files/RBFglossarylongrevised_0.pdf. OMS (2015). Compétences de base en santé et développement de l’adolescent pour les prestataires de soins primaires. Genève : Organisation mondiale de la Santé. OMS (2016), Normes mondiales pour la qualité des services de santé destinés aux adolescents : Guide pour la mise en œuvre d’une stratégie fondée sur des normes afin d’améliorer la qualité des services de santé pour les adolescents Volume 1: Normes et critères. Genève : Organisation mondiale de la Santé. OMS (2019), La qualité des services de santé: un impératif mondial en vue de la couverture santé universelle. Genève : Organisation mondiale de la Santé. Nyathi M, Jooste K. 2008. Working conditions that contribute to absenteeism among nurses in a provincial hospital in the Limpopo Province. Curationis, 31:28–37. Plant M. 2003. Primary care nurses’ attitude to sickness absence: a study. Br J Community Nurs, 8:421–427. Sabri B, Siddiqi S, Ahmed AM, Kakar FK, Perrot J. 2007. Towards sustainable delivery of health services in Afghanistan: options for the future. Bulletin of the WHO 85: 712–18. Saïdou H (2017), Impact du Financement Basé sur la Performance sur les recours thérapeutiques en cas de paludisme au Cameroun, thèse de Doctorat, Université Paris-Dauphine. Soeters R, Griffiths F. 2003. ‘Improving government health services through contract management: a case from Cambodia’. Health Policy and Planning 18: 74–83 Soeters R, Habineza C, Peerenboom PB. 2006. ‘Performance based financing and changing the district health system: experience from Rwanda’. Bulletin of the WHO 84: 884–9. Williams ES, Rondeau KV, Xiao Q, Francescutti LH. 2007. Heavy physician workloads: impact on physician attitudes and outcomes. Health Serv Manage Res 2007, 20:261–269. 58 Witter, Sophie; Toonen, Jurrien, Meessen, Bruno; Kagubare, Jean; Fritsche, Gyorgy; Vaughan, Kelsey. 2013. Performance-based financing as a health system reform: mapping the key dimensions for monitoring and evaluation. BMC Health Services Research. 13:367. World Bank. 2016. Health Financing System Assessment. Forthcoming. World Bank. 2013. Project Appraisal Document: Improving Maternal and Child Health through Integrated Social Service Project. Online at http://documents.worldbank.org/curated/en/2013/04/17659693/haiti- improving-maternal-child-health-through-integrated-social-services-project (last accessed May 8, 2014) World Bank. 2012. World Development Indicators. Online at: http://data.worldbank.org/data-catalog/world- development-indicators (last accessed May 8, 2014). Zeng W., Cros M., Dilley K., Shepard D.2012. Impact of performance-based financing on primary health care services in Haiti. Health Policy and Planning. 59 Annex A: Basic Statistics for Different Sub-Samples Table A1: Evolution of the proportion of medication taken in the event of illness between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.43 0.50 414 0.41 0.49 3704 -0.02 TA 0.37 0.48 356 0.44 0.50 3481 0.07 Control 0.44 0.50 223 0.41 0.49 2510 -0.03 Total 0.41 0.49 993 0.42 0.49 9695 0.01 Table A2: Evolution of the proportion of women with unmet need for family planning between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Differenc e RBF 0.32 0.46 2445 0.47 0.50 2648 0.15 TA 0.35 0.48 2287 0.44 0.50 2448 0.09 Control 0.31 0.46 1727 0.48 0.50 1727 0.17 Total 0.33 0.47 6459 0.46 0.50 6823 0.14 Table A3: Evolution of the proportion of women aged 15-49 using a modern contraceptive method between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.43 0.50 1263 0.33 0.47 1495 -0.10 TA 0.37 0.48 1153 0.34 0.47 1305 -0.03 Control 0.41 0.49 899 0.33 0.47 954 -0.08 Total 0.40 0.49 3315 0.34 0.47 3754 -0.07 Table A4: Evolution of the proportion of women aged 15-49 in stable union using a modern contraceptive method between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.41 0.49 955 0.33 0.47 1104 -0.08 TA 0.36 0.48 931 0.33 0.47 971 -0.02 Control 0.41 0.49 688 0.34 0.47 732 -0.08 Total 0.39 0.49 2574 0.33 0.47 2807 -0.06 Table A5: Evolution of the proportion of women aged 15-49 using any contraceptive method between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.37 0.48 2445 0.35 0.48 1477 -0.03 TA 0.32 0.47 2287 0.37 0.48 1273 0.04 Control 0.32 0.47 1727 0.35 0.48 943 0.02 Total 0.34 0.47 6459 0.35 0.48 3693 0.01 60 Table A6: Evolution of the proportion of recently pregnant women aged 15-49 who had recourse to at least one prenatal visit during their last pregnancy between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.83 0.38 1029 0.98 0.15 1491 0.15 TA 0.83 0.37 1065 0.95 0.22 1327 0.11 Control 0.81 0.39 754 0.95 0.21 971 0.14 Total 0.82 0.38 2848 0.96 0.20 3789 0.14 Table A7: Evolution of the proportion of recently pregnant women aged 15-49 who had recourse to at least four prenatal visits during their last pregnancy between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.59 0.49 1029 0.79 0.40 1491 0.21 TA 0.60 0.49 1065 0.79 0.41 1327 0.18 Control 0.58 0.49 754 0.76 0.43 971 0.18 Total 0.59 0.49 2848 0.78 0.41 3789 0.19 Table A8: Evolution of the proportion of recently pregnant women aged 15-49 who had the first prenatal visit in the first four months of their pregnancy between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.70 0.46 1029 0.80 0.40 1491 0.10 TA 0.72 0.45 1065 0.78 0.42 1327 0.05 Control 0.67 0.47 754 0.78 0.41 971 0.11 Total 0.70 0.46 2848 0.79 0.41 3789 0.09 Table A9: Evolution of the proportion of recently pregnant women aged 15-49 who took at least two doses of tetanus vaccine during their last pregnancy between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.50 0.50 1029 0.74 0.44 1491 0.25 TA 0.49 0.50 1065 0.73 0.45 1327 0.23 Control 0.48 0.50 754 0.73 0.44 971 0.25 Total 0.49 0.50 2848 0.73 0.44 3789 0.24 Table A10: Evolution of the proportion of recently pregnant women aged 15-49 whose delivery was attended by a health professional between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.54 0.50 695 0.62 0.48 1449 0.09 TA 0.52 0.50 748 0.51 0.50 1286 -0.01 Control 0.45 0.50 551 0.49 0.50 951 0.04 Total 0.51 0.50 1994 0.55 0.50 3686 0.04 61 Table A11: Evolution of the proportion of recently pregnant women aged 15-49 whose delivery took place in a health institution between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.48 0.50 695 0.60 0.49 1449 0.12 TA 0.45 0.50 748 0.48 0.50 1286 0.02 Control 0.36 0.48 551 0.47 0.50 951 0.11 Total 0.44 0.50 1994 0.52 0.50 3686 0.09 Table A12: Evolution of the proportion of recently pregnant women aged 15-49 who received at least one postnatal visit within 2 months after delivery between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.54 0.50 753 0.61 0.49 1491 0.07 TA 0.55 0.50 803 0.66 0.47 1327 0.11 Control 0.53 0.50 570 0.59 0.49 971 0.06 Total 0.54 0.50 2126 0.62 0.48 3789 0.08 Table A13: Evolution of the proportion of recently pregnant women aged 15-49 who took iron supplementation after childbirth between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.29 0.45 753 0.35 0.48 1491 0.07 TA 0.25 0.43 803 0.35 0.48 1327 0.10 Control 0.29 0.45 570 0.28 0.45 971 -0.01 Total 0.27 0.45 2126 0.33 0.47 3789 0.06 Table A14: Evolution of the proportion of children 0-11 months fully vaccinated based on vaccination cards between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.44 0.50 675 0.25 0.43 900 -0.19 TA 0.42 0.49 614 0.23 0.42 829 -0.19 Control 0.40 0.49 467 0.19 0.39 573 -0.21 Total 0.42 0.49 1756 0.23 0.42 2302 -0.19 Table A15: Evolution of the proportion of children aged 12-23 months fully vaccinated based on vaccination cards between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.42 0.49 648 0.31 0.46 641 -0.11 TA 0.38 0.49 653 0.23 0.42 555 -0.16 Control 0.39 0.49 450 0.23 0.42 429 -0.16 Total 0.40 0.49 1751 0.26 0.44 1625 -0.14 62 Table A16: Evolution of the proportion of children under five years-old presenting stunting between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.24 0.43 993 0.35 0.48 872 0.11 TA 0.25 0.43 941 0.36 0.48 858 0.11 Control 0.32 0.47 708 0.34 0.47 465 0.02 Total 0.26 0.44 2642 0.35 0.48 2195 0.09 Table A17: Evolution of the proportion of children under five years-old suffering from underweight between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.13 0.34 957 0.17 0.37 854 0.03 TA 0.15 0.36 910 0.19 0.40 823 0.04 Control 0.16 0.36 670 0.18 0.38 449 0.02 Total 0.15 0.35 2537 0.18 0.38 2126 0.03 Table A18: Changes in the proportion of children under 5 suffering from wasting between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.13 0.34 957 0.13 0.34 854 0.00 TA 0.16 0.37 910 0.18 0.38 823 0.02 Control 0.13 0.34 670 0.17 0.37 449 0.03 Total 0.14 0.35 2537 0.16 0.36 2126 0.02 Table A19: Evolution of the availability of drinking water at all times in the last seven days in the health facilities between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.40 0.49 57 0.69 0.47 65 0.29 TA 0.36 0.48 67 0.47 0.50 60 0.11 Control 0.22 0.42 45 0.36 0.49 44 0.14 Total 0.34 0.47 169 0.53 0.50 169 0.19 Table A20: Evolution of permanent electricity availability in the last seven days at health facilities between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.42 0.50 57 0.82 0.39 65 0.39 TA 0.37 0.49 67 0.72 0.45 60 0.34 Control 0.22 0.42 45 0.64 0.49 44 0.41 Total 0.35 0.48 169 0.73 0.44 169 0.38 63 Table A21: Evolution of the availability of an evacuation system at health facilities between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.81 0.40 57 0.95 0.21 65 0.15 TA 0.87 0.34 67 0.92 0.28 60 0.05 Control 0.89 0.32 45 0.91 0.29 44 0.02 Total 0.85 0.36 169 0.93 0.26 169 0.08 Table A22: Evolution of the proportion of health facilities respecting universal 63ygiene protocols between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.69 0.20 57 0.78 0.14 64 0.09 TA 0.68 0.18 66 0.68 0.19 59 0.00 Control 0.66 0.18 45 0.64 0.21 44 -0.02 Total 0.68 0.19 168 0.71 0.19 167 0.03 Table A23: Evolution of the proportion of nurses providing ANC between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.93 0.26 57 0.97 0.18 64 0.04 TA 1.00 0.00 66 0.97 0.18 60 -0.03 Control 0.93 0.25 45 0.95 0.21 44 0.02 Total 0.96 0.20 168 0.96 0.19 168 0.01 Table A24: Evolution of the proportion of nurses providing delivery care between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.56 0.50 57 0.61 0.49 64 0.05 TA 0.47 0.50 66 0.38 0.49 60 -0.09 Control 0.36 0.48 45 0.32 0.47 44 -0.04 Total 0.47 0.50 168 0.45 0.50 168 -0.02 Table A25: Changes in the proportion of health facilities providing postpartum care between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.77 0.42 57 0.72 0.45 64 -0.05 TA 0.62 0.49 66 0.58 0.50 60 -0.04 Control 0.47 0.50 45 0.57 0.50 44 0.10 Total 0.63 0.48 168 0.63 0.48 168 0.00 64 Table A26: Evolution of the proportion of health care providers having a vaccination service between the surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.98 0.13 57 0.98 0.13 64 0.00 TA 0.94 0.24 66 1.00 0.00 60 0.06 Control 0.91 0.29 45 0.93 0.25 44 0.02 Total 0.95 0.23 168 0.98 0.15 168 0.03 Table A27: Evolution of the proportion of health facilities able to perform laboratory tests between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.42 0.37 57 0.49 0.33 65 0.07 TA 0.34 0.36 67 0.31 0.34 60 -0.03 Control 0.30 0.35 45 0.26 0.33 44 -0.04 Total 0.35 0.36 169 0.37 0.35 169 0.01 Table A28: Evolution of the proportion of health facilities having vaccines in stock between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.36 0.34 57 0.51 0.25 65 0.15 TA 0.36 0.37 67 0.43 0.25 60 0.06 Control 0.35 0.34 45 0.41 0.26 44 0.06 Total 0.36 0.35 169 0.46 0.26 169 0.10 Table A29: Changes in the proportion of health facilities with generic drugs in stock between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.60 0.34 57 0.53 0.22 65 -0.08 TA 0.58 0.30 67 0.43 0.24 60 -0.16 Control 0.57 0.36 45 0.37 0.21 44 -0.21 Total 0.59 0.33 169 0.45 0.23 169 -0.14 Table A30: Evolution of the proportion of health facilities having antimalarial drugs in stock between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.50 0.36 57 0.57 0.33 65 0.08 TA 0.54 0.32 67 0.42 0.37 60 -0.12 Control 0.55 0.34 45 0.37 0.32 44 -0.18 Total 0.53 0.34 169 0.47 0.35 169 -0.06 65 Table A31: Evolution of the proportion of health care providers having malaria RDT kits in stock between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.58 0.50 57 0.54 0.50 65 -0.04 TA 0.52 0.50 67 0.38 0.49 60 -0.14 Control 0.64 0.48 45 0.43 0.50 44 -0.21 Total 0.57 0.50 169 0.46 0.50 169 -0.12 Table A32: Evolution of the proportion of health facilities equipped with general equipment between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.74 0.20 57 0.65 0.22 65 -0.09 TA 0.69 0.20 66 0.52 0.21 60 -0.17 Control 0.69 0.19 45 0.48 0.23 44 -0.21 Total 0.71 0.20 168 0.56 0.23 169 -0.15 Table A33: Evolution of the proportion of nurses equipped with sterilization equipment between surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.28 0.35 57 0.38 0.37 65 0.10 TA 0.26 0.35 66 0.25 0.33 60 -0.01 Control 0.21 0.33 45 0.30 0.36 44 0.08 Total 0.25 0.35 168 0.31 0.36 169 0.06 Table A34: Evolution of the proportion of healthcare workers equipped with vaccination equipment between the two surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.92 0.16 57 0.93 0.18 65 0.02 TA 0.95 0.18 66 0.93 0.15 60 -0.02 Control 0.88 0.29 45 0.82 0.33 44 -0.06 Total 0.92 0.21 168 0.90 0.22 169 -0.02 Table A35: Evolution of the proportion of nurses equipped with prenatal care equipment between the two surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.93 0.21 57 0.93 0.17 65 0.00 TA 0.84 0.28 66 0.90 0.19 60 0.06 Control 0.85 0.30 45 0.81 0.27 44 -0.04 Total 0.88 0.27 168 0.89 0.21 169 0.02 66 Table A36: Evolution of the proportion of nurses equipped with childbirth and postnatal care equipment between the two surveys Mean baseline Standard deviation baseline N baseline Mean endline Standard deviation endline N endline Difference RBF 0.34 0.32 57 0.48 0.37 65 0.14 TA 0.30 0.35 66 0.31 0.36 60 0.01 Control 0.24 0.33 45 0.25 0.34 44 0.02 Total 0.30 0.34 168 0.36 0.37 169 0.06 67 Table A37: Amount of RBF Tariffs Paid to World Bank Sites August 2016 to September 2019 Department Health Institutions Q3_2016 Q4_2016 Q1_2017 Q2_2017 Q3_2017 Q4_2017 Q1_2018 Q2_2018 Q3_2018 Q4_2018 Q1_2019 Q2_2019 Q3_2019 Center Boc Banic Health Center 13365.0 33278.0 41394.0 30161.0 45760.0 37565.0 43777.0 132055.0 129391.0 116324.7 163482.0 Center Saint Joseph de Thomassique Clinic 65367.0 111356.0 118319.0 140026.0 149902.0 179205.0 197489.0 375167.0 416577.0 493293.3 790226.6 761938.3 744947.8 Center Cerca Carvajal Health Center 33472.0 81697.0 52684.0 70559.0 59779.0 35848.0 94886.0 229648.0 229864.0 277643.5 363584.6 376880.4 422056.6 Center Thomonde Health Center 70408.0 240605.0 120314.0 171145.0 272391.0 281368.0 322159.0 628663.0 652651.0 391604.2 843269.7 746364.4 719070.5 Center St Michel de Boucan Carré Health Center 153040.0 328263.0 224778.0 368487.0 430665.0 431703.0 458511.0 1038148.0 1082099.0 562557.6 988533.1 1059132.1 1077917.5 Center Sarazin Dispensary 15676.0 24992.0 20236.0 27007.0 16338.0 18730.0 41819.0 168656.0 156375.0 178708.9 Center Mardi Gras cup dispensary 42054.0 38413.0 39414.0 53482.0 72153.0 86125.0 88137.0 177903.0 114744.0 288757.9 Center Saut d'Eau Health Center 7873.0 45653.0 21076.0 24366.0 31046.0 26542.0 60680.0 203399.0 84558.0 99807.5 99520.8 302536.2 247451.7 South Saint Anthony of Padua dispensary 1867.0 0.0 15072.0 9388.0 0.0 0.0 1527.0 0.0 0.0 0.0 0.0 0.0 0.0 South Charles Pasqua Health Center 3392.0 8770.0 11311.0 13331.0 9501.0 16319.0 12251.0 64281.0 66360.0 54886.4 104185.8 55641.2 92428.9 South Aquin Reference Community Hospital 208686.0 193289.0 461436.0 494858.0 510918.0 458466.0 499576.0 517760.0 1007092.8 1312670.6 1287683.1 1032283.0 South Sunlight Mission La Colline Health Center 0.0 4321.0 8155.0 4522.0 8328.0 1769.0 8110.0 35986.0 73823.0 51724.4 98751.6 102898.1 107290.1 South Cavaillon Health Center 8211.0 19803.0 20441.0 19156.0 25750.0 27934.0 112161.0 129888.0 162773.1 293133.1 352024.7 333372.6 South Henry Sugar Dispensary 4817.0 21089.0 31262.0 34624.0 32936.0 30156.0 30232.0 84425.0 75318.0 68086.1 140349.9 118422.3 87506.2 South St Louis du Sud Health Center 16663.0 23842.0 21699.0 21379.0 48671.0 44825.0 107372.0 145954.0 193279.0 261647.4 226781.7 266314.6 South 4 Paths Health Center 16029.0 24528.0 682.0 12488.0 47323.0 45833.0 69404.0 234928.0 243410.0 0.0 0.0 0.0 247714.4 South Mika Cance Health Center/ World Vision 237.0 1527.0 1249.0 165.0 1774.0 1473.0 8729.0 3101.0 10639.0 3196.0 9936.7 56962.7 49860.4 South Charpentier Sacred Heart Dispensary 10715.0 46783.0 62523.0 57994.0 44750.0 61637.0 61339.0 151461.0 110137.0 172065.5 447883.7 365556.2 478158.3 South Sainte Anne de Camp Perrin Health Center 17554.0 89291.0 35958.0 44210.0 47327.0 90195.0 81735.0 0.0 268467.0 169479.2 282381.7 0.0 354992.7 South Eben Ezer dispensary in Guichard 14266.0 21393.0 26282.0 30189.0 28649.0 36127.0 76584.0 90717.0 0.0 0.0 0.0 109948.7 South Chantal Health Center 7073.0 25299.0 2133.0 44974.0 45461.0 22486.0 21189.0 133114.0 157281.0 89474.9 146159.6 139052.3 238693.8 South Melon Dispensary 1322.0 2701.0 4636.0 0.0 0.0 3108.0 14073.0 43438.0 0.0 0.0 0.0 40846.7 South Maillard Good Samaritan Dispensary 4909.0 6186.0 2765.0 11054.0 0.0 0.0 3476.0 28846.0 37729.0 0.0 0.0 0.0 37678.3 South Dispensary Ferme Leblanc 4330.0 2761.0 6631.0 10550.0 6032.0 14602.0 19160.0 40934.0 47275.0 0.0 0.0 0.0 44731.5 South Arniquet Health Center 1005.0 3687.0 45232.0 37325.0 46921.0 38934.0 8634.0 113710.0 123971.0 0.0 0.0 0.0 138383.3 South Dispensary of Labo Pharma Chateau 1059.0 7420.0 16201.0 11505.0 8122.0 12252.0 36282.0 42736.0 0.0 0.0 0.0 71955.2 South Port Salut Community Referral Hospital 326766.0 416131.0 438942.0 407154.0 467166.0 472824.0 518007.0 661735.0 635930.0 1052293.8 1084434.9 0.0 1317671.9 South Health Center of St Jean du Sud 3037.0 12485.0 22166.0 10688.0 7303.0 4290.0 1710.0 54800.0 0.0 42433.3 80468.6 1268223.9 63946.5 Northwest CSL Beauchamps 6475.0 13165.0 18188.0 23901.3 24900.8 24279.9 67651.4 68293.0 92738.0 151997.0 162490.0 172868.0 Northwest DP Three Rivers 36303.0 48640.0 50216.0 73860.9 79721.1 63326.4 165698.0 154833.0 215235.0 383684.0 280372.0 442736.0 Northwest CAL Mole St Nicolas 2596.0 4484.0 25228.0 19559.0 20271.5 48459.7 134676.1 70893.0 93487.8 178861.0 141531.5 178286.0 Northwest Chansolme CAL 11543.0 30213.0 47855.0 55045.6 58285.9 59967.8 158659.2 159454.0 182651.0 260853.0 289950.8 325545.0 Northwest DP Lady Mary 6653.0 2568.0 22269.0 2988.9 7909.7 0.0 48228.4 49626.0 50779.0 45712.0 51,491.00* 55313.0 Northwest Crève DP 800.0 4221.0 2598.0 1744.6 2164.9 867.0 18766.8 15326.0 103273.0 40684.0 78884.0 Northwest DP Desgranges 15405.0 19310.0 37254.0 32453.7 7166.5 39967.4 95847.7 96168.0 96751.0 138644.0 164408.0 136878.0 Northwest DP Dugas 932.0 5726.0 19571.0 7936.2 6758.3 10149.7 30040.4 20342.0 23247.0 57576.0 70247.0 Northwest CAL NWCHM 0.0 68198.0 41100.0 - 79557.2 110483.1 273764.4 315425.0 342109.0 Northwest DP High Mosquito 5853.0 15365.0 15330.0 6775.5 9118.6 34399.5 57782.4 54140.0 91945.0 105989.0 100689.0 104059.0 Northwest La Perriere 10927.0 0.0 2534.0 4112.5 9088.9 12415.8 8073.6 31934.0 - 58013.0 51539.0 Northwest D DP Roche Fort 8354.0 17888.0 28980.0 15855.1 30366.5 46805.4 85449.0 79898.0 122877.0 260155.0 260155.0 111794.0 Northwest CAL Sentrain 20171.0 70764.0 76399.0 113474.6 112185.6 100748.8 0.0 213953.0 255242.0 450938.0 423408.0 461656.0 Northwest DP Guichard 1245.0 16305.0 8735.0 29402.8 29255.0 27440.0 37357.0 38789.1 38902.0 Northwest DP Gaspard 2460.0 8318.0 12891.0 7166.5 27017.6 42914.6 53771.3 101891.0 157203.0 98620.0 Northwest UNHCR Austrian Haitian 1045502.0 739718.0 0.0 901596.8 942773.3 721430.9 551369.4 795841.0 1013322.0 1353558.0 1174115.0 1541055.0 68 Annex B: Household Healthcare Seeking Behavior Analysis According to the quasi-experimental design of the study described below, we make the initial assumption that the members of the household are treated in the HF closest to their home, in the HF that serves their locality of residence. During the impact assessment surveys, information was collected on the place (including the names of the HF in the event of seeking care in modern health structures) of recourse to health care by the members of the households surveyed. But at the baseline study, the information on the exact names of the HF where care was sought was not always well collected as in 2019. It had to be reconstructed based on certain variables on the characteristics of the HF. From this information, we created variables that measure the therapeutic behaviors of household members with the following modalities: 0=if the household member did not seek care; 1=if the household member sought care in the HF to which his household is assigned; 2= if the household member sought care in an HF in a study group other than the one to which his household is assigned; 3= if the household member sought care in an HF outside the study area. When the household seeks care in its HF to which it belongs, it is a therapeutic recourse carried out according to our initial hypothesis. However, when the household is treated in another IS, this therapeutic recourse potentially poses a problem of contagion, leading to a measurement error. This measurement error would introduce statistical bias into the analysis and reduce our ability to measure the potential impacts of interventions (mitigation bias). This is a substantial limitation of household survey analysis that should be kept in mind. Table B1 shows that before the implementation of the RBF program, households did not always seek care in the nearest HF. It can thus be noted that only 8.5% of household members seek curative care from their assigned HF against 28.9% in an unassigned HF and 49.6% do not seek such care at all. For prenatal care, only 17.3% of households seek it in their assigned HF against 65.3% in an unassigned HF and 14.8% do not seek it at all. Only 3.5% of women gave birth in their assigned HF against 21.0% in an unassigned HF, 36.9% in a HF outside the study area and for 34.4% of women, the HF d childbirth has not been completed. As for postnatal care, only 5.9% of women use it in their assigned HF as opposed to a 14.4% delivered in an unassigned HF. 69 Table B1: Health care seeking behaviors of households at baseline and endline Baseline study Curative care ANC Delivery care Postnatal care % N % N % N % N Not seeking care 49.6 556 14.8 422 4.2 119 34.3 978 Search for care in the assigned HF 8.5 95 17.3 493 3.5 99 5.9 168 Seeking care in an unassigned HF 28.9 324 65.3 1861 21.0 599 14.4 410 Seeking care in a HF outside the study area 1.5 17 2.5 72 36.9 1052 13.4 381 HF not known 11.6 130 0.0 0 34.4 979 32.0 911 Number of observations 1122 48 2848 2848 2848 Endline study Curative care ANC Childbirth care Postnatal care % N % N % N % N Not seeking care 74.1 7321 2.9 111 2.7 103 37.7 1428 Search for care in the assigned HF 15.2 1499 56.8 2153 11.6 438 28.8 1091 Seeking care in an unassigned HF 7.6 750 38.7 1467 83.8 3174 32.7 1239 Seeking care in a HF outside the study area 1.3 125 1.5 58 2.0 74 0.8 31 HF not known 1.9 188 0.0 0 0.0 0 Number of observations 9883 3789 3789 3789 In the final study, we observe that the proportions of households seeking care in their assigned HF increased, but not significantly. Indeed, the proportion of household members who use curative care in their assigned HF increased to 15.2% in 2019; while the proportion of women who used ANC in their assigned HF increased to 56.8%; the proportion of women who delivered in their assigned HF rose to 11.6%; and that of women having used postnatal care rose to 28.8%. However, the results of the multinomial logit difference-in-differences regression models, presented in Tables B2 to B5 show that with RBF interventions, households seek more care in their assigned IS: +10.4 percentage points for curative care in their HF in the TA group, and households that do not seek curative care there at all are less and less numerous (-12.0 percentage points); +13.1 percentage points on the use of ANC in assigned HF in the RBF group with a significant decrease (-14.9 percentage points) in the proportion of women who seek care in a HF not assigned to them assigned. For childbirth and postnatal care, the program interventions (RBF and TA) did not significantly affect households' treatment behaviors in favor of their assigned HF. 48 This small sample of people surveyed who were ill in the last 4 weeks preceding the visit in 2015 represents only 4.2% of people surveyed. This low morbidity rate could be explained by the seasonality of the survey but also by the quality of the data collected on health behaviors. 70 Table B2: Impacts of the RBF program on seeking care † (1) (2) (3) (4) VARIABLES Not seeking curative care Seeking care in the assigned HF Seeking care in an unassigned HF Seeking care in a HF outside the study area Post dummy ( 1=endline) 0.234*** 0.0185 -0.236*** -0.0166 (0.0418) (0.0369) (0.0393) (0.0134) RBF/post-interaction -0.0275 0.0341 -0.0246 0.0180 (0.0542) (0.0427) (0.0505) (0.0155) TA/post-interaction -0.120** 0.104** 0.00306 0.0126 (0.0568) (0.0435) (0.0495) (0.0152) Constant 0.493*** 0.229*** 0.274*** 0.00414 (0.0395) (0.0256) (0.0311) (0.00922) Number of observations 10,406 10,406 10,406 10,406 R-squared 0.088 0.090 0.098 0.032 P-value RBF vs TA 0.0745 0.0273 0.529 0.612 Mean for control group in the baseline sample 0.514 0.142 0.321 0.0229 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. Table B3: Impacts of the RBF program on ANC seeking† (1) (2) (3) (4) VARIABLES Not seeking ANC Seeking ANC in the assigned HF Seeking ANC in an unassigned HF Seeking ANC in a HF outside the study area Post dummy ( 1=endline) -0.103*** 0.317*** -0.194*** -0.0204** (0.0179) (0.0520) (0.0545) (0.00847) RBF/post-interaction -0.0133 0.131** -0.149** 0.0312 (0.0254) (0.0653) (0.0660) (0.0106) TA/post-interaction -0.00102 0.0783 -0.0894 0.0121 (0.0267) (0.0663) (0.0726) (0.00993) Constant 0.124*** -0.236*** 1.080*** 0.0316* (0.0374) (0.0786) (0.0780) (0.0177) Number of observations 6,241 6,241 6,241 6,241 R-squared 0.134 0.321 0.241 0.042 P-value RBF vs TA 0.644 0.374 0.360 0.0399 Mean for control group in the baseline sample 0.153 0.204 0.607 0.0364 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 71 Table B4: Impacts of the RBF program on care-seeking at delivery† (1) (2) (3) (4) VARIABLES Not delivering at a HF Seeking to deliver in the assigned HF Seeking to deliver in an unassigned HF Seeking to deliver in a HF outside the study area Post dummy ( 1=endline) -0.0112 0.0435 0.551*** -0.583*** (0.00941) (0.0264) (0.0529) (0.0445) RBF/post-interaction -0.0239 0.0293 -0.101 0.0952 (0.0155) (0.0328) (0.0695) (0.0584) TA/post-interaction -0.0166 0.0210 0.00477 -0.00914 (0.0170) (0.0370) (0.0851) (0.0653) Constant 0.0858*** 0.509*** -0.0549 0.460*** (0.0244) (0.0458) (0.0772) (0.0602) Number of observations 5,344 5,344 5,344 5,344 R-squared 0.049 0.196 0.351 0.523 P-value RBF vs TA 0.701 0.810 0.206 0.103 Mean for control group in the baseline sample 0.0308 0.0493 0.308 0.612 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. Table B5: Impacts of the RBF program on seeking postnatal care† (1) (2) (3) (4) VARIABLES Not seeking postnatal care Seeking postnatal in the assigned HF Seeking postnatal care in an unassigned HF Seeking postnatal care in a HF outside the study area Post dummy ( 1=endline) -0.0732* 0.163*** 0.131*** -0.220*** (0.0378) (0.0391) (0.0411) (0.0300) RBF/post-interaction -0.0130 0.0209 -0.0845 0.0766 (0.0499) (0.0486) (0.0519) (0.0350) TA/post-interaction -0.0979 0.0691 -0.00398 0.0327 (0.0508) (0.0532) (0.0638) (0.0389) Constant 0.689*** -0.00388 0.355*** -0.0402 (0.0655) (0.0628) (0.0677) (0.0414) Number of observations 5,404 5,404 5,404 5,404 R-squared 0.110 0.208 0.122 0.188 P-value RBF vs TA 0.103 0.305 0.181 0.185 Mean for control group in the baseline sample 0.498 0.0714 0.190 0.241 Robust standard errors per cluster in parentheses ***p<0.01, **p<0.05, *p<0.1 † The control variables introduced into the models and whose coefficients are not presented in the table above are: type of HF, status of HF, provision of care 24 hours a day, mean distance from a higher level of HF, Department, rural dummy, source of funding for RBF, other health program in catchment area, cluster dummy. 72 Annex C: Qualitative Study on Findings from Quantitative Impact Evaluation of the Results-Based Financing Program (RBF) in Haiti A qualitative study was undertaken to explore in more depth the findings of the quantitative IE described in this document. 49 This study carried out thirty-eight key informant interviews with four different types of stakeholders: implementers, departmental staff, healthcare providers, and technical advisors. More specifically, the study focused on identifying additional factors that help explain the impact of the Results Based Financing (RBF) program by looking into the heterogeneity of HFs, human resources, infrastructure and health service availability, as well as cultural practices and beliefs of the population. Findings on the quality of health services: The quantitative IE showed that the overall quality of health services was impacted by the RBF program to a statistically significant degree, based on several measurable quality indicators including the availability of medicines and vaccines, the capacity to carry out laboratory tests, and the compliance of universal hygiene protocols. The key-informant interviews in the qualitative study showed that most health providers agreed that the RBF program had a positive influence on the quality of health services. In particular:  Interviewees mentioned that there were instances in which the program facilitated the procurement of specialized equipment (from the RBF payments) that proved to be key for the good functioning of the HF and provision of better-quality services.  Interviewees also mentioned that the availability of medicines significantly improved thanks to the RBF program, which confirmed the same salient finding from the quantitative IE.  Most importantly, the RBF program sparked a lot of interest among the health providers to think about ways to organize the functioning of the HFs and the healthcare team itself to improve the quality of health services. the RBF program boosted the health workers motivation to get better scores in the final evaluation, and the associated financial reward. Reasons for more impact of RBF program in rural areas: The quantitative IE showed that, overall, the RBF program had a stronger impact in rural areas than in urban areas, in terms of statistically significant findings. The qualitative study found that health providers in HFs in rural areas had previously been exposed to more challenges than in urban areas, – in terms of lower availability of functioning equipment, and poor infrastructure. Hence the health personnel in rural facilities were more inclined to make sacrifices since the impacts were potentially larger in the rural areas (which were starting from a “lower base”). On prenatal care and assisted deliveries: The quantitative IE found that there was an overall statistically significant improvement due to the RBF program in the utilization (quantity) as well as quality of prenatal visits, particularly in rural areas. Assisted deliveries also were found to have improved due to the RBF program (on top of a significant positive time trend for the overall sample), but to a statistically significant degree only in rural areas. Based on the interviews under the qualitative study, the impacts of the RBF program could have been even stronger if not for several constraining cultural factors. First, there are several misconceptions among women related to accessing care for deliveries at HFs. For example, health providers mentioned that some women think that they should only seek care until the gender of their child is detectable. Second, key informants reported that there is a myriad of preconceived notions about giving birth at a HF. For example, it has been observed that women prefer to give birth at home because that would provide an extra spiritual protection from their deceased loved ones. Others believe that “matrons” (Traditional Birth Attendants) can provide quality care and are prepared to act upon any complication. 49 “L’Evaluation Finale de l’Impacte du Programme de Financement Basé sur les Résultats – Etude Qualitative. Rapport Final.” Draft. WB, February 2022. 73 On reproductive health indicators: According to the IE, the RBF program did not have a statistically significant impact on any reproductive health indicators for women (15 to 49 years-old). This could be explained by the findings from the qualitative study, where health providers mentioned that family planning remains a taboo subject amongst patients from both rural and urban HFs. This provides a plausible explanation for the lack of significant change in unmet family planning needs, and use of modern contraceptives from the IE report. More specifically, key-informants mentioned that there are religious leaders that explicitly oppose to its use, or other instances in which women have preconceived notions about the consequences of using birth control methods before marriage. Child Immunization: There was an overall statistically significant positive impact due to the RBF program in child immunization, for children aged of 12-23 months. (The overall trend for this variable was declining, but much less so in RBF areas.) Key informant interviews showed that most health providers highlighted the role of CHWs as a key factor for the successful immunization programs. More specifically, CHWs advocate for the importance of immunization while also providing immunization services. In particular, rural areas have significantly benefitted from the CHWs because these CHWs are able to target hard-to-reach populations. Delays in RBF payments: One of the main findings from the qualitative study was that the delay in RBF payments to HFs has significantly affected the implementation and impact of RBF. (In turn, this was due to delays in verification due to the deteriorating security situation, among others.) The impact of the program would have been stronger if the RBF payments had been made consistently on time.