Finansman Baze sou Rezilta (FBR) nan Sistèm Sante Ayiti a: Rapò Evalyasyon Enpak

Report No: AUS0002763 RESULTS - BASED FINANCING (RBF) IN HAITI’S HEALTH SYSTEM Impact Evaluation Report Health, Nutrition and Population Global Practice April 2 1 , 2022 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized © 2017 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202 - 473 - 1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territ ory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for no ncommercial purposes as long as full attribution to this work is given. Attribution — Please cite the work as follows: “World Bank. 2022 . Results - based Financing in Haiti ’s Health System . © World Bank.” All queries on rights and licenses, including subsi diary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202 - 522 - 2625; e - mail: pubrights@worldbank.org . i TABLE OF CONTENTS EXECUTIVE SUMMARY ____________________________________________________________ 1 1. BACKGROUND ______________________________________________________________ 5 Background to the study ________________________________ ______________________ 5 1.2 Objectives and Research Questions Related to Impact Evaluation _____________________ 8 2. METHODOLOGY OF THE STUDY _______________________________________________ 9 Experimental Approach ________________________________ ______________________ 9 2.1.1 Implementation of the Interventions __________________________________________ 12 2.2 Data Sources: _____________________________________________________________ 15 2.2.1 Household surveys ______________________________________________________ 15 2.2.2 Surveys of health facilities _________________________________________________ 16 2.3 RBF Impact Evaluation Indicators _____________________________________________ 19 2.4 Statistical Methods _________________________________________________________ 20 2.4.1 Balance of the study groups for the baseline survey _____________________________ 20 2.4.2 Econometric models______________________________________________________ 24 3. IMPACT EVALUATION RESULTS ______________________________________________ 26 3.1 Impacts on the Governance of Health Facilities ___________________________________ 26 3.2 Impacts on the Quality of Health Services _______________________________________ 27 3.2.1 Basic Infrastructure at Health Facilities _______________________________________ 27 3.2.2 Availability of Health Services ______________________________________________ 28 3.2.3 Amenities and Basic Care _________________________________________________ 29 3.2.4 Malaria Care Services ____________________________________________________ 31 3.2.5 Medical Equipment and Supplies ____________________________________________ 31 3.2.6 Prenatal Care ___________________________________________________________ 34 3.2.7 Under-five Child Health Care _______________________________________________ 35 3.3 Impacts on Human Resources ________________________________________________ 36 3.4 Impacts on the Use of Reproductive Health Services _______________________________ 39 3.4.1 Family Planning Behaviors_________________________________________________ 39 3.4.2 Prenatal Care Behaviors __________________________________________________ 40 3.4.3 Delivery and Postnatal Care Behaviors _______________________________________ 41 3.5 Impacts on the Use of Child Preventive Health Services ____________________________ 42 3.6 Impacts on the Health Status of the Population ___________________________________ 43 4. DISCUSSION _______________________________________________________________ 48 4.1 Children’s health ___________________________________________________________ 49 4.2 Coverage and quality of priority health services related to maternal and child health _______ 49 4.3 Health system in Haiti: governance, human resources and health financing _____________ 50 4.4 Health expenditure _________________________________________________________ 50 4.5 Technical assistance, coverage and quality of health services ________________________ 51 4.6 Summary comments ________________________________________________________ 52 ii 4.7 Study limitations ___________________________________________________________ 52 5. CONCLUSION ______________________________________________________________ 54 6. BIBLIOGRAPHIC REFERENCES _______________________________________________ 56 Annex A: Basic Statistics for Different Sub-Samples __________________________________ 59 Annex B: Household Healthcare Seeking Behavior Analysis ____________________________ 68 Annex C: Qualitative Study on Findings from Quantitative Impact Evaluation of the Results- Based Financing Program (RBF) in Haiti _____________________________________________ 72 iii TABLES Table 1: Key SDG for Maternal and Child Health Services ...................................................................... 8 Table 2: Mechanisms by Which RBF Can Potentially Work .................................................................... 9 Table 3: Summary of Sample by Intervention Arm ................................................................................ 10 Table 4: Tariffs (in HTG) of Indicators Incentivized Under the RBF Program in Haiti † .......................... 13 Table 5: Characteristics of Households Surveys ................................................................................... 15 Table 6: Characteristics of Health Facility Surveys ................................................................................ 18 Table 7: Geographic Distribution of Health Facilities Included in the IE and Surveyed During the Two Periods of the Study .............................................................................................................................. 18 Table 8: List of RBF IE Variables ........................................................................................................... 19 Table 9: Characteristics of Households and Individuals in the Baseline Study Sample ......................... 21 Table 10: Characteristics of Health Facilities in the Baseline Study Sample ......................................... 23 Table 11: Provision of Health Services in the Baseline Study ............................................................... 24 Table 12: Impacts of RBF on Administration and Management of Health Facilities † ............................ 27 Table 13: Impacts of RBF Initiatives on Basic Infrastructure at Health Facilities † ................................ 28 Table 14: Impacts of RBF on the Availability of Health Services at the Health Facility † ....................... 29 Table 15: Impacts of RBF on the Quality of Health Services Provided at Health Facilities † ................. 30 Table 16: Impacts of RBF on Malaria Care Services at Health Facilities † ............................................ 31 Table 17: Impacts of RBF on the Provision of Medical Equipment and Supplies to Health Facilities † . 33 Table 18: Impacts of RBF on the Quality of ANC Consultations and Related Satisfaction † ................. 35 Table 19: Impacts of RBF on the Quality of Consultations for Children Under Five and the Related Levels of Satisfaction † .......................................................................................................................... 36 Table 20: Impacts of RBF on the Delivery of Health Care Services in Health Facilities † ...................... 37 Table 21: Impacts of RBF on the Wages and Motivation of Health Care Personnel in Health Facilities † .............................................................................................................................................................. 38 Table 22: Impacts of RBF on the Level of Satisfaction of Health Care Personnel in Health Facilities † 38 Table 23: Impacts of RBF on the Reproductive Health Behaviors of Women Between Ages of 15 and 49 † ....................................................................................................................................................... 39 Table 24: Impacts of RBF on the Use of Prenatal Care Services Among Recently Pregnant Women Between the Ages of 15 and 49 (24 Months Preceding Data Collection) † ........................................... 40 Table 25: Impacts of RBF on the Use of Obstetric and Postnatal Care Services Among Recently Pregnant Women Between the Ages of 15 and 49 (24 Months Preceding Data Collection) † ............... 42 Table 26: Impacts of RBF on the Immunization of Children Under Five † ............................................. 43 Table 27: Impact of RBF on the Nutritional Status of Children Between the Ages of 6 and 59 Months † .............................................................................................................................................................. 44 Table 28: Impacts of RBF on Household Health Expenditure † ............................................................. 45 Table 29: Summary of RBF IE Results in Haiti ...................................................................................... 45 FIGURES Figure 1: Health indicators in Haiti and SDG targets, 2015 ..................................................................... 5 Figure 2: Location of the different HFs with interventions (RBF and TA Groups) and the Control Group facilities of the RBF program in Haiti...................................................................................................... 12 Figure 3: Evolution of Total RBF Payments (in US Dollars) to RBF Health Facilities Over the Period 2016-2019 ............................................................................................................................................. 34 iv ABBREVIATIONS AND ACRONYMS A EV External Verification Agency ( Agence Externe de Vérification ) ANC Antenatal Care BCG Bacille Calmette Guerin (Tuberculosis Vaccine) CAL Centre de S anté A vec L it ( H ealth C enter W ith B ed) CHE Catastrophic Health Expenditure CHW Community Health Worker CSL Centre de Santé Sans Lit ( Health Center W ithout Bed ) DDSs Departmental Health Directorates DTP Diphtheria, Tetanus and Pertussis ECVMAS Survey of H ousehold L iving C onditions A fter the E arthquake ( Enquête sur les C onditions de V ie des M énages A près le S éisme ) EMMUS Mortality, Morbidity and Use of Health Services Survey (Enquête sur la M ortalité, la M orbidité et l' U tilisation des services de S anté) FP Family Planning HCR Community Reference Hospital ( Hôpitaux Communautaires de Reference ) HF Health Facility HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HTG Haitian G ourdes HRITF Health Results Innovation Trust Fund IE Impact Evaluation IMCI Integrated Management of Childhood Illness LIC Low - Income Country MMR Maternal Mortality Ratio in Haiti MSPP Ministry of Public Health and Population ( Ministère de la Santé Publique et de la Population ) NGOs Non - governmental Organizations OPV Oral Poliovirus Vaccine PNS National Health Policy ( Politique Nationale de Santé ) RBF Results - Based Financing RDT R apid Di agnostic T est SDGs Sustainable Development Goals SSQH Quality Health Services for Haiti ( Service de Santé de Qualité Pour Haïti) U5MR Under - Five Mortality Rate USAID United States Agency for International Development WB World Bank WHO World Health Organization 1 EXECUTIVE SUMMARY This report is about the main results of the impact assessment conducted for the Result-Based Financing (RBF) of Haiti’s health system. The decisions on the design of this assessment have been taken by the Ministry of Public Health and the World Bank. Data collection for this impact assessment was undertaken by Haitian companies, supported by a technical team of the World Bank. The analysis of the data and the drafting of this report was undertaken by a World Bank technical team. As a country of Latin America and the Caribbean, Haiti is still far behind on regional averages for many health indicators, particularly those related to maternal, newborn and child health. Despite the efforts that have been made in recent years, many challenges remain in trying to reach the Sustainable Development Goals as far as health is concerned. To address this situation, Haiti designated the implementation of its RBF program as a strategic axis of its National Health Policy ( Politique Nationale de Santé , PNS) in order to improve both the supply and demand for health. The country had had a positive experience with RBF between 1999 and 2001, with the Non-Governmental Organization Management Sciences for Health, in certain Departments. This led the country to develop the current RBF model, and to launch it in 2014 with a pilot involving a limited number of health facilities (HFs) in the Northeast Department. One year later, the RBF strategy was scaled up to 7 other Departments, with many more HFs included. A quasi-experimental study was implemented to assess the impact of this program on primary health care outcomes and on the functioning of Hait i’s health system. For this purpose, two packages of interventions (RBF with technical assistance, and technical assistance only (TA)), and three groups under the study (RBF, TA only and Control), were established. The study covered 170 HFs divided into three groups. Data was collected for the entire sample of HFs as well as for households in localities served by these HFs (with one locality chosen randomly for every HF, among all areas that it covers). In every locality, interviews were conducted with 28 randomly chosen households with at least one woman pregnant at the time of the survey or one woman that has interrupted her pregnancy within the last 24 months preceding the survey. Data collected in 2015 (baseline survey) and in 2019 (endline survey) were used to assess impacts by using the approach of difference in differences, in accordance with the Health Results Innovation Trust Fund (HRITF) standard strategy for impact evaluations concerning RBF programs. The following table summarizes the main findings. Summary of the Results of the Impact Evaluation of the RBF Program in Haiti INDICATORS RELATED TO: IMPACTS OF RBF (RELATIVE TO CONTROL GROUP) IMPACTS OF TA (RELATIVE TO CONTROL GROUP) C OMMENTS ON IMPACT OF RBF (RELATIVE TO CONTROL GROUP) Administration and management of H Fs No impact No impact These results could be explained by the fact that these indicators w ere not among the quality indicators incentivized under the RBF program in Haiti. Basic infrastructure of H Fs Significant positive impact on the availability of a medical evacuation system. No impact on availability of permanent drinking water and electricity. No impact Note that the indicators on the availability of permanent drinking water and electricity were introduced quite late during the implementation of the RBF program. Availability of different types of health care services at HFs No impact No impact For antenatal care and immunizations, the lack of significant impact could be explained by the high proportion in the baseline study of HFs that were already offering these services, controlled health services that already had those services. In addition, note that in practice, if a HF wants to start offering a new type of health service, it would requir e support (to ensure provision of the needed human resources, inputs, technical assistance, etc.) from higher - level decision - making authorities, and this is not always 2 easy to obtain. Hence, in practice health workers at HFs have little control over this s et of indicators. Quality of services offered at HFs Significant and positive impact on adherence to universal protocols, capacity to conduct lab tests, availability of vaccines and availability of medicines in stock No impact Various types of equipment at HFs Significant and positive impact on the availability of general equipment, as well as equipment for immunizations, and for deliveries and postnatal care. No impact on the availability of equipment for sterilization and antenatal care. No impact The lack of significant impact on equipment for sterilization and antenatal care could be explained by the high costs of these types of equipment. RBF payments – especially at the beginning of the implementation period when RBF tariffs were subs tantially lower (until the first quarter of 2018) – were too low to allow HFs to purchase these types of more expensive equipment. Quality of antenatal consultations and related satisfaction Significant and positive impact on patient education (during the antenatal consultations) as well as on pregnant women’s satisfaction regarding the quality and cost of these consultations. Significant and positive impact on pregnant women’s satisfaction regarding the quality of antenatal consultations (in rural area s) and regarding their cost (in overall sample). Quality of consultations for children aged under 5 years and related satisfaction Significant and positive impact on satisfaction regarding the cost of children’s consultations. No impact on various measures of the quality of children’s consultations. No impact The lack of impact on the quality of these consultations could be explained by the fact that these indicators were not incentivized under the RBF program in Haiti. Size of workforce and number of hours worked No impact Significant and positive impact on the numbers of health workers and negative for the hours of work (the expected direction) The lack of impact could be explained by the fact that these indicators were not directly incentivized u nder the RBF program in Haiti. Salaries and motivation of health care staff Significant and positive impact on the regularity of salary payments of health workers. No impact on personal motivation of health workers Significant and positive impact on personal motivation of health workers, in urban areas It should be noted that these indicators were not directly incentivized under the RBF program. Satisfaction of health workers Significant and positive impact on satisfaction of health workers regardin g their work environment and regarding their management and supervision at HFs (in rural areas for the latter). But no impact on health workers’ satisfaction regarding their salaries. No impact It should be noted that indicators related to health workers’ satisfaction were not directly incentivized under the RBF program. Reproductive behavior amongst women aged between 15 - 49 years. No impact Significant and positive impact on unmet family planning needs; no impact on the utilization of contraceptive methods. According to qualitative work carried out in 2021 in conjunction with the quantitative analysis, some cultural attitudes, preconceptions and low education levels still constitute barriers to the use of contraceptive methods in Haiti, especially mo dern methods. Antenatal consultations by women aged 15 - 49 years who were recently pregnant Significant and positive impact on having 4 antenatal consultations and on having at least one antenatal consultation in the first first 4 months of pregnancy (in rural areas). No impact on having at least one antenatal consultation and on having at least two doses of antitetanus vaccine during pregnancy. Significant and negative impact (at 10% level of significance) on having at least one antenatal consultation (i n urban areas only). Significant and positive impact on having at least two doses of antitetanus vaccine during pregnancy (in urban areas only). The lack of significant impact (in the RBF group) on having at least one antenatal consultation could be explained by the high degree of attainment of this indicator that was already observed at the time of the baseline study (81.2% in the overall sample). Obstetric and postnatal care amongst women aged between 15 - 49 years and recently pregnant. Significant and positive impact on assisted deliveries and on deliveries at HFs (in rural areas). No impact on postnatal consultations by women at HFs and on iron supplementation after delivery. Significant and positive impact on the iron supplementation a fter delivery. Postnatal consultations as defined under this study (i.e. postnatal consultations by pregnant women at HFs) were not incentivized under the program. Instead, the related indicator incentivized under the program was the number of postnatal vi sits by Community Health Agents to the home of a woman who recently gave birth, within 3 days of the delivery. Immunization of young children Significant and positive impact on immunization coverage of children aged between 12 - 23 months. Significant and positive impact on immunization coverage of children aged between 12 - 23 months in urban areas. Nutritional status of children aged between 0 - 59 months. Significant and negative impact on the proportion of children that suffer from underweight (improvement of nutritional status), in rural areas. No impact on stunting and wasting. Significant and positive impact on stunting (deterioration of nutritional s tatus). No impact on the proportion of children suffering from underweight and wasting. These indicators were not directly incentivized under the RBF program. (The related indicator directly incentivized under the RBF program was the number of children tha t underwent nutritional screening.) However, these indicators could have been influenced by other indicators directly incentivized under the RBF program, and likely by other factors operating over several years. Health care expenses of households (financial protection) Significant and negative impact on health care expenses (improved financial protection). No impact It should be noted that indicators of health care expenses were not directly incentivized under the RBF program. The results of this assessment show that in general, RBF interventions together with technical assistance improved several indicators of coverage and the quality of maternal and child health in Haiti, relative to the Control Group. With this approach, significant improvements (relative to the Control Group) were seen with immunization coverage and the nutritional status of children; availability of basic infrastructure as well as of inputs and equipment; adherence to universal protocols; capacity to realize lab tests; quality of antenatal 3 consultations; and confidence by pregnant women to have deliveries at HFs – especially in rural areas. All of these tend to improve the health conditions of communities through better quality of health services, hence guaranteeing better financial protection of households because of reduced health expenses (which improved due to RBF relative to the Control Group). Although indicators relatively to governance and human resources were not directly incentivized under the RBF program, Haiti’s health system generally showed relative improvement for these indicators – with higher health worker satisfaction due to RBF (relative to the control group) with the work environment, and with management and supervision at HFs, and with the regularity of salary payments. An important feature of the analysis is that for some indicators where RBF was found to have a positive impact relative to the Control Group, there was sometimes an underlying trend of deterioration to begin with – which can be expected given the context of the worsening economic and security situation in Haiti. In these cases, there was a deterioration in the indicator in all groups, but the deterioration was significantly less in the RBF areas than in the Control Group areas. An example of this type of indicator is immunization coverage. (In some cases, RBF led to a reversal of the negative trend, so that improvements were seen in the RBF areas while a deterioration was seen in the Control Group areas, but this was not as common.) There were also indicators such as assisted deliveries in rural areas, where RBF was found to have a positive impact relative to the Control Group and there was an improving time trend to begin with. In these cases, improvements were seen in both the RBF areas and the Control Group areas, but the former were significantly larger than the latter. In addition to the above, the study reveals that technical assistance alone is not sufficient to improve the Haitian health system. Its impact on the coverage of quality health services was found to be mixed. Under technical assistance alone, improvements were seen with the following relative to the Control Group: (i) unmet family planning needs of women; (ii) women having at least two doses of anti-tetanus vaccine during pregnancy (in urban areas); (iii) iron supplementation after birth; (iii) satisfaction by pregnant women with the quality of antenatal consultations (in rural areas) and their cost; (iv) the size of the workforce and the number of hours worked by health workers; and (v) health worker motivation (in urban areas). On the other hand, deteriorations (relative to the Control Group) were observed with women having at least one antenatal consultation (in urban areas), and on children’s nutritional status (stunting), and no i mpacts were observed on governance, availability and quality of health care services, availability of basic equipment and infrastructures, the quality of children’s consultations, overall satisfaction of health workers and financial protection of households. The fact that health worker motivation improved in urban areas due to technical assistance only (relative to the Control Group), but not under RBF, could potentially be explained by the following two factors, based on consultations undertaken by the team: First, health workers in the group that had technical assistance only may have believed that they would soon start receiving RBF payments – which is what happened in practice after the evaluation period – which may have been a motivating factor for them. At the same time, there were significant delays in RBF payments to HFs, which worsened over time due to the deteriorating security situation (which delayed the verification process), and this could have been a demotivating factor. Some limitations to the present study have been identified, which are useful to note for follow up analysis. Key among these are: (i) the fact that there were many other ongoing health interventions not covered by the RBF program in the areas covered by the study, which could not all be controlled for (although major ones 4 were identified and addressed during the analysis); (ii) the non-controlled effects of the economic and security crisis that the country had to face during the implementation of the program; (iii) contagion bias in the household data (explained in Annex B); and (iv) the fact that this study does not allow a comparison with other interventions on the supply-side (e.g., direct facility financing) or the demand-side (e.g., cash transfers or maternal vouchers) that would have similarly increased the budgets of facilities or households but without using the specific RBF mechanisms. Additional research is necessary to evaluate to what extent the results of the present study may change if these effects are addressed. In addition, additional qualitative work is necessary to better understand some of the mechanisms by which RBF has worked in Haiti, going beyond what has already been done (see Annex C). Based on the findings of this study, the following are key recommendations made:  Consider introducing incentives specifically for consultations for under-five children (with appropriate protocols to be followed). The present Impact Evaluation exercise found that the RBF program in Haiti had no impact on the quality of child consultations, which is not surprising since this was not incentivized in the case of Haiti. Given the importance of basic child health services in any health system, it is suggested that this is explicitly included and incentivized in Haiti’s RBF program , as in the case of many RBF programs worldwide.  For indicators incentivized under the RBF program over which HF workers and administrators have little control – such as rapid diagnostic test (RDT) kits and the availability of certain types of services at a facility – it is advisable to reconsider whether these should indeed be incentivized under the program.  Make efforts to substantially reduce the delays in making RBF payments to HFs. These delays have been significant in the case of Haiti, especially towards the end of the IE implementation period, when security and other issues led to substantial delays in the verification process. The qualitative analysis found that these delays have substantially affected performance. One way of reducing these delays is to make advance payments to HFs for each quarter even before the verification process for that quarter has been completed – especially in the case of delays in the verification – which would be adjusted accordingly after the verification is completed.  Test out one or two small RBF pilots in urban areas, with modifications to see if the impact from RBF can be increased there. Currently, the impact from RBF has been significant in rural areas for most variables, but less so in urban areas. 1  Introduce RBF at the community level, initially on a pilot basis. This may be particularly helpful in the case of the family planning indicators, where RBF does not seem to have a statistically significant impact. The qualitative work done – see Annex C – indicates that actions to stimulate behavior change are key here, and this is where community-level actions by Community Health Workers (CHWs) could be especially helpful. These would be incentivized under RBF at the community level. RBF at the community level may also help boost indicators in urban areas. 1 It is possible that this is due to the smaller sample size in urban areas. It is possible that the impacts in urban areas – where the relevant coefficients in the regressions have been generally of the right sign but not statistically significant – would have been statistically significant with a larger sample size. However, a priori the evidence so far indicates substantially lower impact in urban areas. 5 1. BACKGROUND Background to the study The Republic of Haiti covers an area of 27,750 km 2 , subdivided into the following 10 Departments: Ouest, Sud, Sud-Est, Grand ’ Anse, Nippes, Nord, Nord-Ouest, Nord-Est, Centre, and Artibonite, as well as 140 communes and 570 communal sections. In 2017, the population stood at 11,085,919, with a male ratio of 0.98, reflecting a slightly higher number of women in the total population (5,486,970 men versus 5,598,949 women). The Haitian population is very young — the median age is 23 and life expectancy is 62.2 years. It is estimated that 35.9 percent of the total population is under age 15. In 2017, there were 278,862 live births and 97,000 deaths. The annual growth rate for the period 2017-2018 is estimated at 1.37 percent. Economically, a large percentage of the Haitian population is poor. While noting a decline in monetary and multidimensional poverty rates since 2000, the World Bank (WB) Report established that in 2012, more than one in two Haitians was poor, living on less than $2.41 a day, and one in four persons was living below the national extreme poverty line of $1.23 a day 2 . Urban areas have, in relative terms, fared better than rural areas, owing to nonagricultural employment opportunities, private transfers, great er access to critical goods and services, and narrowing inequality. The performance of the health system is poor, with high morbidity and mortality rates. Despite the progress made by Haiti in achieving the Millennium Development Goals, many challenges remain with achievement of the new Sustainable Development Goals (SDGs), especially for the poorest. Between 1990 and 2015, the maternal mortality ratio (MMR) in Haiti fell from 670 deaths per 100,000 live births to 359 deaths per 100,000 live births. The infant mortality rate and the under-five mortality rate (U5MR) are 59 percent and 81 percent, respectively EMMUS, 2017 (Mortality, Morbidity and Use of Health Services Survey ( Enquête sur la Mortalité, la Morbidité et l'Utilisation des services de Santé ). These figures still fall well short of the SDG targets of reducing the MMR to fewer than 70 maternal deaths per 100,000 live births and the U5MR to 25 deaths per 1,000 live births by 2030 (Figure 1). Figure 1: Health indicators in Haiti and SDG targets, 2015 Maternal Mortality Ratio Under - five Mortality Rate Sources: WHO 2016; EMMUS 2000, 2005 – 06, 2012 2 https://www.worldbank.org/en/topic/poverty/publication/beyond-poverty-haiti 0 100 200 300 400 500 600 700 800 1990 2000 2013 2015 SDG 2030 0 20 40 60 80 100 120 140 160 1990 2000 2013 2015 SDG 2030 6 According to various EMMUS 3 , trends in the nutritional status of children under age 5 do not seem to be improving in a regular and sustained manner over time. After remaining stable (29 percent) between 2000 and 2006, stunting among children fell to 22 percent in 2012 and in 2016-2017. Wasting and underweight figures seem to have fluctuated during the same period — the percentage of wasting among children doubled between 2000 and 2006 (from 5 percent in 2000 to 10 percent in 2006) before falling again to 5 percent in 2012 and to 4 percent in 2016-2017. The percentage of underweight children increased from 14 percent in 2000 to 18 percent in 2006 before falling to 11 percent in 2012 and to 10 percent in 2016-2017. More recently, the SMART survey conducted in December 2019 by the Ministry of Health with UNICEF support, showed that 2.1% of children had severe acute malnutrition. Considering this situation, the Ministry of Public Health and Population ( Ministère de la Santé Publique et de la Population, MSPP) set out to improve health services supply and demand by establishing a results-based financing (RBF) program (MSPP, 2012). RBF is a health system intervention aimed at increasing the use of health services (especially for maternal and child health) by improving their quality and quantity with particular emphasis on efficiency and equity. This approach is based on the granting of incentives when certain health indicators related to providers or patient outcomes are achieved by making use of the resources (financial, material and human) available (Musgrove, 2011; Fritsche et al., 2014). In Haiti, RBF 4 is part of the National Health Policy ( Politique Nationale de Santé, PNS) and national health financing strategie s. The principal aim of the PNS is to reduce morbidity and mortality through an appropriate, efficient, accessible, and universal health system. This includes the rational use of available resources through donor alignment with national priorities, in the context of a partnership based on performance and accountability. Encouraged by the positive experience of the United States Agency for International Development ( USAID ) 5 with the establishment of a performance - based payment system between 1999 and 2012, the MSPP, seeking to extend this approach at the national level, identified a strategic pillar in the PNS for the establishment of a performance - based financing system. Furthermore, since 2012, the WB and USAID (through the Leadership, Management and Governance Project and the Quality Health Services for Haiti Project ( Service de Santé de Qualité Pour Haïti , SSQH) have jointly started a series of discussions to assist the MSPP with the implementation of a national RBF program. In 2013, the contracting unit was created to supervise implementation of the RBF at the national level. The RBF program was launched in March 2016 by the contracting unit of the MSPP, with WB financing in the Sud, Nord-Ouest, and Centre D epartments and with USAID financing in the Grand’Anse, Nippes, Nord -Est, and Nord Departments. Several early studies have pointed to a positive effect of RBF on the efficiency of the health system and the provision of health services, including increases in maternal and child health services coverage (Soeters et al. 2006; Sabri et al. 2007; Rusa et al. 2009; Basinga et al. 2011). A recent synthesis of the evidence concludes that, when compared with business-as-usual, in low-income settings with centralized health systems RBF can result in substantial gains in effective coverage, but that the relative benefits of PBF — the 3 Better known as Demographic and Health Surveys. 4 See the definition of the term and its relevance in the proposal submitted to the Ethics Committee in July 2015. 5 In 2013 Zeng et al evaluated the effect of RBF combined with technical assistance (but also separately) provided by the SDSH program with respect to key maternal and child health indicators. They found that RBF, along with technical assistance (training, monitoring and evaluation, and coaching), increased the quantity of primary health care services by 39 percent in NGO-run facilities between 2008 and 2010. In addition, the increase in health services for children under 12 months and pregnant women between 2008 and 2010 was statistically greater compared to facilities not using the RBF approach. 7 performance pay component in particular-- are less clear when it is compared to two alternative approaches: (i) direct facility financing which provides operating budgets to frontline health services with facility autonomy on allocation, but not performance pay, and (ii) demand-side financial support for health services (that is, conditional cash transfers and vouchers) (de Walque, Kandpal, Wagstaff et al. 2022). The WB (2016) has shown that all the HFs in Haiti have very low technical efficiency scores at all levels of the health system pyramid and much lower scores than other low-income countries (LICs) (Hernandez, 2013; Akzali et al. 2011; Sébastian et al. 2007; Marshall et al. 2011; Kirigia, 2013; Osmani, 2015, Jéhu-Appiah, 2014). This, among other things, could explain the mediocre productivity of medical staff (on average, staff providing medical care work for four hours per day and have six consultations per day in the primary health care sector in 2013), absenteeism (which is estimated to result in a loss to the health system of Haitian Gourdes (HTG) 3 billion), and the limited availability of basic inputs (only 32 percent of HFs (n = 907) provide essential drugs, and only 31 percent of HFs have basic medical equipment) (WB, 2016). Furthermore, the same study also revealed that quality processes related to maternal and child health services continue to be substandard (only 30 percent of health professionals follow proper clinical protocols during prenatal consultations). Lastly, RBF is a tool for improving health services coverage, especially among the poorest. Health services coverage in Haiti continues to be low, according to the World Health Organization (WHO) statistics (2015) shown in Table 1 below. This table shows that all preventive maternal and child health indicators are lower than those in LICs. When compared with LICs, the proportion of assisted deliveries in Haiti (37 percent) is 14 percentage points lower; unmet family planning needs in Haiti (35 percent) are 13 percentage points higher; and the proportion of children under 24 months who have received all doses of the diphtheria, tetanus, and pertussis vaccine in Haiti (68 percent) is 12 percentage points lower. Significant disparities remain in maternal and child health (with the exception of immunizations) between the poor and non-poor (EMMUS, 2012) — 9 percent of pregnant women in the lowest wealth quintile give birth in a HF compared to 76 percent in the highest wealth quintile; 23 percent of children with acute respiratory infections receive treatment in the lowest wealth quintile against 52 percent of children in the highest wealth quintile. As financial resources are the main constraint to use of health services for the poor 6 , RBF reforms could increase the use of health services by reducing the consultation costs for several maternal and child health services or by promoting mobile clinics (WB, 2016). 6 It should be noted that domestic Government spending for health is just 0.52% of GDP – very low when compared to other LICs – and this is a significant constraint. 8 Table 1: Key SDG for Maternal and Child Health Services Unmet need for family planning (%) % of pregnant women receiving four antenatal care (ANC) visits % pregnant women who gave birth with qualified personnel % children having received three doses of DTP % population using an improved water source (%) Population using improved toilets (%) Low Income Countries 22 48 51 80 69 37 Latin America 9 90 96 90 96 88 Haiti 35 67 37 68 62 24 El Salvador 18 80 100 92 90 70 Honduras 11 89 83 87 93 80 Jamaica 10 86 99 93 93 80 Costa Rica 8 90 99 95 97 94 Dominican Republic 11 95 99 83 81 82 Guatemala 21 … 59 85 94 80 Nicaragua 11 88 88 98 85 52 Benign 25 58 81 69 76 14 Rwanda 21 35 69 98 71 64 Burkina Faso 33 34 66 88 82 19 Ethiopia 26 19 10 72 52 24 Malawi 26 46 87 89 85 10 Tanzania 25 43 49 91 53 12 Source: WHO Statistics, 2015 Rising political and economic instability, COVID-19 shock and falling resources for health sector. Starting around 2017, Haiti started facing growing political instability as well as insecurity and violence, including periods of national lockdown notably the “Peys Lock” period between September and December 2019. This all occurred in the context of falling resources for the health sector due to a number of factors including poor economic conditions and a sharp fall in donor financing for health. All of this negatively impacted the health sector during the period of implementation of this Impact Evaluation (IE), and accounts for some of the declining trends observed in key variables over this period (see below for more details). Unfortunately, the situation in terms of exogenous factors affecting the health sector has only worsened since the implementation period of the study – i.e. after 2019 – due to the COVID-19 pandemic as well as rising political instability culminating in the assassination of the President in July 2021. 1.2 Objective s and Research Questions Related to Impact Evaluation To measure the effectiveness of this intervention over the long term and its operationalization in the context of Haiti, the MSPP and its partners decided to evaluate the impact of the RBF approach. The objective of this IE is to measure the impact of the national RBF program on priority health outcomes, household health expenditures, and the key functions of the Haitian health system, thus allowing the MSPP and its partners to determine whether this health policy is effective relative to facilities that have not adopted RBF or only receive technical assistance (coaching/local supervision). This overall objective leads to six main research questions: 1. Does the RBF program contribute to better infant health outcomes? 2. Does the RBF program contribute to the coverage and quality of priority health services related to maternal and child health? 3. What are the effects of the RBF program on the Haitian health system, in particular regarding human resources, health financing, and governance? 4. What impact does RBF have on catastrophic household expenditure? Does RBF provide households with greater financial protection? A household is said to incur catastrophic health 9 expenditures (CHE) if 25 percent or more of its nonfood expenses are allocated to health (WHO, WB, 2014). 5. Does technical assistance or coaching/local supervision alone improve priority health services coverage and quality related to maternal and child health more or as much as RBF accompanied by technical assistance? The table below provides the chain of RBF impact assumptions. Table 2: Mechanisms by Which RBF Can Potentially Work Problems identified RBF hypothesis RBF Impact Study Questions Health system supply/inputs Low motivation, absenteeism, and low productivity of medical staff - RBF stimulates better governance and accountability of staff vis - à - vis patients and increases the motivation of medical staff because teamwork is better perceived and rewarded. This contributes to better productivity as well. (3) Low availability of medicine and basic equipment RBF institutions have more resources at their disposal to improve the availability of essential inputs. 7 Low quality of maternal and child health services Medical staff adhere better to clinical protocols for maternal and childcare because they a re more accountable to patients. (2); (5) Demand Low coverage of maternal and child health indicators, especially for the poorest RBF should encourage medical personnel to produce more, by implementing strategies to target and cover a greater number of patients and therefore increase demand. The RBF aims more to cover the poorest, many times those facilities are dispensaries, which are by nature located in rural and poor areas. (2) (5). Financial barriers for the poorest quintile was particularly high The CHE of the lowest quintiles are expected to decrease because - incentives received under RBF can be used to subsidize user fees for certain maternal and child health services - the implementation of advanced strategies to increase the coverage of services inc luding the establishment of mobile clinics and assembly stations. Since nearly 75% of patients seen during this type of strategy are poor (ECVMAS, 2013), RBF is expected to contribute to increasing coverage of the poorest. (4) Malnutrition indicators The increase in the coverage of maternal and child health services contributes to an improvement in the nutritional status of children under 5 years old. (1) Source : RBF baseline study report, 2015 2. METHODOLOGY OF THE STUDY Experimental Approach A quasi- experimental study was conducted to evaluate the impact of the RBF program on Haiti’s health system. To do this, HFs operating at the primary care level were surveyed: Health Centers With Beds (Centre de Santé Avec Lit, CALs); Health Centers Without Beds (CSLs); Dispensaries; and Community Reference Hospitals ( Hôpitaux Communautaires de Reference, HCRs). Furthermore, these were divided into three groups before the implementation of the study (Table 3) according to the methods described below. There are two interventions: (i) the first intervention that provides to HFs both RBF and technical assistance (TA); and (ii) the second intervention that provides TA only without RBF. This second intervention is justified by the results of the study measuring the effects of performance-based financing in Management Sciences for Health sites in 2012, which demonstrated a positive effect of TA alone on the coverage of maternal and child health services (Zeng et al, 2013). The third group consisted of a Control Group. This study focused on 196 catchment areas distributed in nine out of ten Departments in the country. A catchment area is made up of a health institution (HF) and the localities it serves. The HFs in each of the intervention groups and the Control Group are described below. 7 In theory, a similar effect could be obtained by providing to HFs lump-sum unconditional cash amounts for spending on essential items. But it would be important for the HFs to spend these resources in accordance with some sort of HF spending plan for these resources, which is a critical part of the RBF program of Haiti where close, hands-on TA is provided for the HF spending plans (as for many countries). In addition, 10 Table 3: Summary of Sample by Intervention Arm Intervention One Intervention Two Control Group Description RBF + Technical Assistance Technical Assistance Only Neither RBF nor Technical Assistance Number of sites 48 HFs WB 23 SSQH/USAID sites* 5 2 HFs WB 23 SSQH/USAID sites 50 sites - neither SSQH/USAID nor WB Department North w est, South, Center Grand - Anse, Nippes, North e ast, North North w est, South, Center Grand - Anse, Nippes, North - East, North In municipalities and Department s that have neither a USAID site nor a BM site *These are the sites of the SSQH project which are managed by various Non-government