Évaluation finale de la performance de l'activité Limiyé ak Òganizasyon pou Kolektivite yo Ale Lwen (LOKAL+)

FINAL PERFORMANCE EVALUATION OF LOKAL+ MAKIN PROGRESS O THE LIMIYÉ AK ÒGANIZASYON POU OUTCOME 1 B SUPPORTING TH DELMAS LOCA KOLEKTIVITE YO ALE LWEN (LOKAL+) GOVERNMENT O IMPROVING ROAD WITHIN THEI ACTIVITY COMMUNIT SOURCE: USAID LOKAL FINAL EVALUATION REPORT July 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the Haiti Evaluation and Survey Services of Social Impact, Inc., the USAID/Haiti Contractor. G N Y E L N S R Y. + FINAL PERFORMANCE EVALUATION OF THE LIMIYÉ AK ÒGANIZASYON POU KOLEKTIVITE YO ALE LWEN (LOKAL+) ACTIVITY FINAL EVALUATION REPORT July 2018 Evaluation Mechanism Number: AID-521-C-17-00002 Haiti Evaluation and Survey Services for USAID/Haiti DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV EVALUATION DESIGN| i ABSTRACT Executed by Tetra Tech Associates in Rural Development from January 2013 to October 2018, United States Agency for International Development/Haiti’s Limiyé ak Òganizasyon pou Kolektivite yo ale Lwen (LOKAL+) project sought to increase communal government legitimacy through improved governance and improved service delivery in nine target communes by increasing local government technical and management capacity. This evaluation assessed the project’s: 1) relevance to Government of Haiti concerns and needs; 2) effectiveness in improving three domains: accountability, transparency (see Annex I for definitions), and service delivery; and 3) sustainability of improving governance in target communes. Methods included a document review, 29 key and group informant interviews (involving 50 officials), 35 focus group discussions (involving 162 female and 168 male citizens), and 12 direct observation site visits in target communes. Almost all stakeholders stated that LOKAL+ was very effective in improving accountability, transparency, and service delivery, especially for tax collection and sanitation. However, except for a sizeable minority in St. Marc, Carrefour, and Cap Haitian, respondents felt that communes have no capacity to independently continue improved governance in these three domains. Nonetheless, given Haiti’s historically highly centralized and opaque local governance, it is important to note three fundamentally important changes brought about by the project: 1) LOKAL+ has demonstrated that it is possible for communes to be financially sound and administratively autonomous; 2) it has introduced the notion of transparency to municipal citizens; and 3) it has successfully improved critical administrative systems such as tax collection administration, housing census, and database management. ii | FINAL EVALUATION REPORT USAID.GOV ACKNOWLEDGEMENTS The evaluation team would like to thank the Government of Haiti representatives who made time to engage with them for this evaluation, particularly the mayors in the target communes and staff at the Ministère de l’Interieur et des Collictivités Territoriales. The evaluation team also thanks the scores of citizens in all the target communes, who were so willing and open to discussing their concerns and dreams for improved governance in their communities. We are also grateful to the LOKAL+ senior management team for putting aside time to meet with us and for their quick responses to our many requests for documents and information during their very demanding schedule, as well as to the United States Agency for International Development team for assistance and guidance during the preparation and execution of the evaluation. USAID.GOV FINAL EVALUATION REPORT | iii TABLE OF CONTENTS Abstract i Acknowledgements ii Table of Contents iii List of Tables and Figures iv Acronyms v Executive Summary vi Activity Summary vi Evaluation Purpose and Questions vi Evaluation Design, Methods, and Challenges vi Findings and Conclusions vii Recommendations x LOKAL+ Background 1 Activity Summary 1 Evaluation Background 3 Evaluation Purpose and Audience 3 Evaluation Questions 3 Evaluation Team and Support Structure 4 Evaluation Methods and Challenges 4 Phase One: Desk Review 5 Phase Two: Field Data Collection 5 Phase Three: Data Processing and Analysis 8 Challenges and Mitigation Strategies 9 Findings and Conclusions 10 EQ1 10 EQ2 12 EQ3 1 Recommendations 4 Annexes 6 Annex A: Evaluation Statement of Work 7 Annex B: Evaluation Matrix 14 Annex C: List of Persons Interviewed 16 iv | FINAL EVALUATION REPORT USAID.GOV Annex D: Consent Form 22 Annex E: Data Collection Protocols 23 Annex F: Key Informant/Group Interview & FGD Protocols 29 Annex G: Site Observation Diagnostic Sheet 48 Annex H: EQ 2 Results by Category of Commune 50 Annex I: Definition of Terms 62 Annex J: Disclosure of Conflict of Interest 64 Annex K: Information on Increased Tax Revenues 66 Annex L: Documents consulted in Desk Review 74 Annex M: Sites for Direct Observation Visits 75 Annex N: LOKAL+ Intermediate Results 77 LIST OF TABLES AND FIGURES Table 1: Connection between LOKAL+'s Five IRs and MICT's "plan d'action" 11 Table 2: Presentation of Increased Tax Revenues 13 Figure 1: Overview of the LOKAL+ Activity 1 Figure 2: LOKAL+ Theory of Change 2 Figure 3: Evaluation Phases 5 Figure 4: Data Collection Methods Used and Gender Breakdown 5 USAID.GOV FINAL EVALUATION REPORT | v ACRONYMS AA Action Area APM Advanced Participation Method ARD Associates in Rural Development BRIDES Bureau de Recherche en Informatique et en Développement Économiques et Social CBO Community-based Organization CCC Consultative Citizens Committees CFPB Contribution Foncière des Propriétés Bâties COR Contracting Officer’s Representative CSO Civil Society Organization CSCCA Superior Court of Auditors and Administrative Disputes DGI Direction Générale des Impôts DO Direct Observation EQ Evaluation Question ESS Evaluation and Survey Services ET Evaluation Team FENAMH Association of Mayors FGD Focus Group Discussion FGDCT Territorial Communities Management and Development Fund FY Fiscal Year FOCAS Functional Organizational Capacity Assessment GI Group Interview GoH Government of Haiti IFOS Institut de Formation du Sud IR Intermediate Result IT Information Technology KII Key Informant Interview LOKAL+ Limiyè ak Òganizasyon pou Kolektivite yo Ale Lwen MEF Ministére de l’Economie et des Finances MICT Ministère de l'Intérieur et des Collectivités Territoriales MMS Managers of Municipal Services MP Member of Parliament MPCE Ministère de la Panification et de la Cooperation Externe OMRH Office of Management and Human Resources PACTE Programme Appui aux Collectivités Territoriales PIC Communal Investment Plan PoC Point of Contact PSP Parliament Support Project QSIP Quality Service Improvement SI Social Impact, Inc. ToC Theory of Change ULCC Unité Pour la Lutte Contre la Corruption USAID United States Agency for International Development USG United States Government WASS Women’s Associations YASS Youth Associations vi | FINAL EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY ACTIVITY SUMMARY To support local government in Haiti, the United States Agency for International Development (USAID) financed the Limyè ak Òganizasyon pou Kolektivite yo Ale Lwen (LOKAL+) Activity, a third-generation Activity worth $19,749,345. USAID awarded LOKAL+ to Tetra Tech Associates in Rural Development (ARD) on January 25, 2013. LOKAL+ was designed to support the Government of Haiti’s (GoH) efforts to improve service delivery at the local level. It’s theory of change (ToC) posits that if the selected communes are provided with capacity building to increase tax collection and improve provision of quality and deconcentrated public services, then constituency satisfaction will be improved, and communal government legitimacy will be enhanced. EVALUATION PURPOSE AND QUESTIONS The main purpose of the evaluation was to inform possible future USAID/Haiti programming in decentralization and communal government strengthening; identify LOKAL+’s strengths and weaknesses; and assess the validity of the activity’s development hypothesis. The evaluation addressed the following evaluation questions (EQ): 1. To what extent was the original project design relevant to the concerns and needs of the GoH, including the parliament? 2. To what extent was the project effective, especially in terms of improving local accountability, transparency, and service delivery? This question will specifically examine the project’s effectiveness in terms of: a) Increasing tax revenues; b) Improving access to central government funding; c) Improving mayor and staff skills and knowledge of governance; d) Improving the legal framework for decentralization; and e) Improving constituency satisfaction. 3. To what extent will the local governments assisted by LOKAL+ be able to continue improved governance, especially in terms of transparency, accountability, and service delivery? EVALUATION DESIGN, METHODS, AND CHALLENGES The evaluation employed a qualitative design including a document review, 16 key informant interviews (KII), 13 group interviews (GI), 12 structured direct observation site visits, and 35 focus group discussions (FGD). Altogether, the evaluation team (ET) obtained data from 380 individuals (170 females and 210 males). Evaluation respondents included key partners (LOKAL+ and its two sub-partners, USAID, and Ministère de l’Interieur et des Collictivités Territoriales [MICT] officials); local governance partners (Members of Parliament (MPs); current/past mayors and communal officials); and citizens (entrepreneurs, Consultative Citizens Committees (CCC), women’s associations, and youth associations). Due to the evaluation design, limitations were expected. To mitigate them, the ET applied a variety of methods to allow for data triangulation and increase the reliability of findings, conclusions, and recommendations. USAID.GOV FINAL EVALUATION REPORT | vii FINDINGS AND CONCLUSIONS FNDINGS FOR EQ1 The main GoH local partner, MICT, was involved in the development and implementation of LOKAL+’s objective of deconcentration and decentralization starting at the request for proposal stage. According to all key partners, USAID’s facilitation was critical to establishing a good working relationship between LOKAL+ and the GoH. Activity documents also indicated that LOKAL+’s five intermediate results (IR) were built upon the MICT’s seven GoH action areas (AA) in its “Plan d’Action Pour les Collectivité Territorialles” (Communal Action Plan). The Activity was also relevant to the GoH’s desire for greater gender inclusion. In keeping with GoH gender inclusion policy, LOKAL+ staff consulted with the Ministry of Women’s Affairs and engaged a Gender Inclusion Specialist to develop a gender inclusion plan. Despite working closely with MICT, Activity documents and KIIs indicated that LOKAL+ had challenges engaging with Parliament. Although they aimed to engage Parliament at the Activity outset, documentation and the Tetra Tech ARD GI indicated that internal political issues within Parliament impacted LOKAL+ efforts to develop a decentralization policy. All key partner GIs and KIIs indicated that MPs’ expectations were a chief obstacle that may have impeded these early efforts to work with Parliament. CONCLUSIONS FOR EQ1 LOKAL+ was relevant to MICT, but not to MPs. LOKAL+ worked closely with the GoH/MICT in designing its activities and implementing its work plan. The Activity reflected GoH/MICT’s Communal Action Plan regarding capacity reinforcement, revenue generation, and accountability, and was responsive to GoH’s gender inclusion policy. However, as LOKAL+ was not designed to support Parliament, it did not meet MPs’ concerns and needs. FINDINGS FOR EQ2 Increasing Tax Revenues Activity documentation, and approximately half the current/past mayors and key partners interviewed, noted that LOKAL+ support for registering built properties, mobilizing and collecting property tax, and using CIVITAX resulted in significant increases in revenue generation. 1 KII and GI respondents also identified audiences publiques as an effective intervention for mobilizing taxes. Yet, current/past mayors admitted that much of their communes’ increased revenues is spent on paying staff salaries rather than improved services, local activities, and upkeep of important communal assets, for which they seek donor assistance or central government funding. Improving Access to Central Government Funding Activity documents and KII respondents suggested that LOKAL+ had a positive effect on communal investment plan (PIC) development and use. Activity documents showed that LOKAL+ successfully helped communes prepare their PICs by providing venues for and facilitating workshops with the CCC, 1 See Annex K for data showing the increases in tax revenue during 2013-2018. viii | FINAL EVALUATION REPORT USAID.GOV feasibility studies for some PIC activities, and funding to facilitate advocacy in support of PICs. However, some FGD respondents did not believe that women were included in PIC formulation, or that communal governments consulted women’s associations on gender matters. Many current/past mayors said that their access to central GoH funding had not really improved. Some evaluation respondents believed that MPs and mayors do not have good relationships. Current/past mayors explained that in the cases where there was not a good relationship, disagreements were due to actual or potential political rivalries. Some current/past mayors stated that MP support would facilitate accessing central government funding. The MICT claimed that to avoid cash flow interruption, the central government now disburses funds gathered from the Contribution Foncière des Propriétés Bâties and Patentes to the communes before they submit their monthly financial reports. Despite this, current/past mayors reported consistent delays in central government funding. Improving Mayor and Staff Skills and Knowledge of Governance GIs with key and local governance partners, and Activity documents all stated that LOKAL+ successfully transferred a wide range of capacities to communal staff. Almost all respondents claimed that the Functional Organizational Capacity Assessment is an effective organizational assessment tool. FGDs, KIIs, and GIs also indicate that LOKAL+ interventions led to increased communal service quality. Findings were mixed, however, regarding the quality of public sanitation. Few respondents stated that women’s participation in communal government meets the GoH’s 30 percent quota. None of the local governance partners stated that they were aware of the LOKAL+ gender inclusion plan. None of the KIIs with mayors reported that their communes have a Gender Affairs point of contact (PoC). Finally, FGD respondents reported that communal agents violated human rights 2 by “ municipal officers use violence against women who sell on the streets.” Improving the Legal Framework for Decentralization The desk review revealed that Parliament has not yet enacted the legal framework for decentralization. During the KIIs with MPs, none stated they were aware of the legal framework status. Improving Constituency Satisfaction Citizen satisfaction is defined as awareness of the services provided by the commune and having a positive image of the communes. More than half the FGDs claimed that citizens know about the services provided by their communes, whereas a third claimed that they do not. Many FGDs noted that citizens think positively about the services provided by their communes. In contrast, the majority of FGDs, as well as GIs with two local governance partners, and one GI with key partners reported that the communes do not have a positive image. Improving Transparency & Accountability LOKAL+ successfully implemented audience publiques meetings. However, a problematic factor is that not all audience publiques are open to all citizens who may be interested in attending. KIIs revealed that LOKAL+ is paying for venues and refreshments. Consequently, audience publiques participation is limited by the budget, and mayors decide who to invite. The desk review, FGDs, and KIIs or GIs with key partners and local governance partners, all indicate that LOKAL+’s use of audience publiques successfully created a space for citizens to communicate with mayors. Yet, there was also evidence that LOKAL+ was not as effective at generating accountability because the audience publiques content were not widely 2 See Annex I “Definition of Terms” USAID.GOV FINAL EVALUATION REPORT | ix distributed. KIIs, GIs, and FGDs also revealed other limitations to accountability and transparency. 3 For example, few communes publicly posted fees or other budget information. A third of FGDs reported that communes do not communicate at all with the citizenry. CONCLUSIONS FOR EQ2 LOKAL+ effectively increased tax revenues and improved mayor and staff skills and knowledge of governance. LOKAL+ support increased tax revenues via registering built properties, mobilizing and collecting property tax, and using CIVITAX. 4 However, mayors and senior communal staff do not prioritize increasing revenue generation when making budget decisions. To improve mayor and staff skills and knowledge of governance, LOKAL+ successfully provided training and equipment, which contributed to improving service delivery in tax collection and public sanitation. At the same time, women’s participation in communal governments remains elusive as communes have not met the 30 percent quota for women’s inclusion or have a Gender Affairs PoC, and communal employees have been cited for committing violence against women street vendors. LOKAL+ was partially successful in helping communes improve central government funding, transparency and accountability, and constituency satisfaction . The Activity assisted the communes to develop their PICs, which they used to advocate for central government funding, though it is notable that PIC development did not sufficiently include women or women’s associations. LOKAL+ improved transparency and accountability by facilitating audiences publiques, yet these are still limited since they only benefit respondents, and overall communication between communes and their citizenry remains poor. LOKAL+ helped its nine target communes improve public services, as reflected in citizens’ positive attitudes about communal service provision. However, many citizens still hold a negative image of their communes. LOKAL+ was not effective in improving the legal framework for decentralization as Parliament has yet to enact the legal framework for decentralization. FINDINGS FOR EQ3 Continuing Improved Transparency The majority of informants and FGD respondents did not believe that the audiences publiques model would continue after LOKAL+ ended because communal administrations have neither the management capability nor the resources to follow the audiences publiques model on their own. At the same time, most FGDs reported that citizens believe expanding the audiences publiques would improve local governance by enabling more citizens to access mayors regarding community affairs. Additionally, some FGDs and most key partner GIs believed that improved local governance could be achieved through capacity building for community-based organizations/civil society organizations. Continuing Improved Accountability Very few respondents stated that the PIC or CCCs would continue beyond LOKAL+ due to tensions between communal officials and CCCs. The desk review revealed that the CCC is not an elected body and that mayors can view it as a political rival. The desk review also indicated that some CCCs expected direct support from LOKAL+ which, if unmet, interfered with their ability to play an oversight role. 3 See Annex I “Definition of Terms” 4 See Annex K x | FINAL EVALUATION REPORT USAID.GOV Continuing Improved Service Delivery Few respondents and FGDs thought fiscal mobilization, as carried out by LOKAL+, will continue, or that communes can mobilize tax revenues as LOKAL+ did. Desk review and KIIs/GIs indicated that LOKAL+ hired personnel and sound trucks and paid for banners for tax mobilization, yet the current/past mayors interviewed claimed they lack the budget, personnel, and training to undertake these expenses and activities without LOKAL+. Many current/past mayors stated that they expect tax revenues to fall this year due to lack of service investments over the past year. Furthermore, almost half of current/past mayors conceded that they make limited service investments, which other key partners and some FGDs verified. Some respondents said that the built property census will not continue after LOKAL+, as communal officials did not have the necessary tools (computer tablets, smartphones, global positioning system devices, etc.) and were not able to enter new data. All current/past mayors stated that lack of internet connectivity was an impediment to CIVITAX continued use. Several stated that communal officials lack a collaborative relationship with the Direction Générale de Impots (DGI), which is needed for tax collection. CONCLUSIONS FOR EQ3 TRANSPARENCY : LOKAL+ developed the audiences publiques model of public discourse to increase transparency, and a majority of FGDs would like them to continue. However, this is unlikely as the communes do not have the financial or personnel resources to sustain them . ACCOUNTABILITY : As LOKAL+ created CCCs to serve as an interlocutor between mayors and citizens, they are also unlikely to continue beyond the Activity . Due to its non-elected status and political rivalries, the CCC’s effectiveness is limited. SERVICE DELIVERY : LOKAL+’s hands-on tax mobilization activities contributed to communes’ increased tax revenues, but did not increase their abilities to independently raise taxes . In addition to challenges with raising taxes, revenue increases have generally not been invested in improved service delivery . A key example of this is the incomplete CIVITAX software system and the poor collaboration between communes and DGI, which may impede CIVITAX sustainability . RECOMMENDATIONS USAID should direct future local governance programming to: • Provide technical and legal information to support the Parliamentary Commission’s work on decentralization and other GoH partners such as the Office of Management and Human Resources and MICT; • Help communes develop their own models of expanded audiences publiques; • Assist each commune to prepare a strategic development plan; • Tailor support to communes based upon the execution of their strategic development plans; • Involve DGI in the completion and expansion of CIVITAX; • Provide urban planning and management training to the appropriate communal government officials; • Develop a unique sustainability plan for each commune; • Support expanding establishing permanent core positions within communal administrations; and • Train mayors and senior communal staff in making budget decisions that prioritize revenue streams. USAID.GOV FINAL EVALUATION REPORT| 1 LOKAL+ BACKGROUND ACTIVITY SUMMARY Figure 1: Overview of the LOKAL+ Activity To support local government in Haiti, the United States Agency for International Development (USAID) financed the Limyè ak Òganizasyon pou Kolektivite yo Ale Lwen (LOKAL+) Activity, a third-generation Activity worth $19,749,345. LOKAL+ was awarded on January 25, 2013 to Tetra Tech Associates in Rural Development (ARD). Tetra Tech ARD also implemented both previous local governance activities, Programme Appui aux Collectivités Territoriales (PACTE) from 1995-2000, and the first LOKAL from 2007-2011. A natural progression of its predecessor activities, LOKAL+ was designed to support the Government of Haiti’s (GoH) efforts to improve service delivery at the local level. To do this, LOKAL+ collaborated with the Ministère de l’Interieur et des Collectivité Territoriales (MICT), utilizing individually tailored work plans prepared with selected communes to improve planning and budgeting, and increase revenue mobilization to improve local services. 5 According to Title V of the Haitian Constitution, communes are a territorial administrative entity with administrative and financial autonomy. Each commune is administered by a Municipal Council composed of three elected members. The President of the Council is called Mayor and is assisted by two Deputy Mayors. 6 As the communes are responsible for providing a wide range of services to the population, their autonomous administrative and financial structure depends on taxes levied on citizens. A 2007 review undertaken by the Unité de Lutte Contre la Corruption (ULCC), a Haitian governmental entity created by presidential decree, ranks communes as among the worst of Haiti’s public agencies/organizations in terms of service quality, earning the lowest levels of confidence in the ability to meet citizens’ needs. 7 The ULCC review stated that most local governments struggle to pay operating costs and are largely dependent on the national government to fund public services or infrastructure improvements. 8 While Haitian law establishes a legal basis for local taxation, many communes only collect a fraction of the possible revenue. Moreover, due to a scarcity of trained staff and antiquated management systems, most communes – where 60 percent of Haitians now live – lack the capacity to transform their few resources into quality public services that would encourage greater taxpayer 5 LOKAL Annual Work Plan- 2013 6 Constitution of the Republic of Haiti, March 29, 1987- http://ufdc.ufl.edu/AA00000457/00001/1j (Articles 66 and 66.1) 7 Corruption and Governance, Unité pour la Lutte Contre la Corruption (ULCC), 2007 8 Haitian Cities: Actions for Today with an Eye on Tomorrow World Bank 2017). Activity Name: Limyè ak Òganizasyon pou Kolektivite yo Ale Lwen (LOKAL+) Implementer: Tetra Tech / ARD Cooperative Agreement #: AID-521-C-13-00004 Total Estimated Ceiling: $19,749,345 Life of Activity: 01/2013 – 10/2017 Development Objective: To increase communal government legitimacy through improved governance, and to raise the quality of service delivery developing technical and management capacity 2 | FINAL EVALUATION REPORT USAID.GOV compliance. Adding to this problem, Haiti’s national government is highly centralized in the capital, where almost half of Haiti’s entire urban population lives. 9 There are, however, opportunities for improvement. Since the 2010 earthquake, there has been increased demand on communal governments to provide services, including support for internally displaced persons 10 in communes distant from earthquake-affected areas (e.g. schooling for displaced children and housing). 11 Improving and expanding local public service delivery requires that Haiti’s 140 rapidly growing communes build their administrative capacities to generate revenue and deliver services. This includes improving their capacities in areas such as strategic planning, budgeting, tax collection, financial and program management, service prioritization, citizen outreach, and significantly raising the level of expertise among both elected and administrative officials. THEORY OF CHANGE The LOKAL+ Theory of Change (ToC) posits that if the selected communes are provided the appropriate support for capacity building to increase tax collection and raise their capability to provide quality and deconcentrated public services, then constituency satisfaction will be improved, and communal government legitimacy will be further enhanced. Figure 2: LOKAL+ Theory of Change LOKAL+ operationalized this ToC through five Intermediate Results (IR) (see Annex N), termed “preferred outcomes:” 1. strengthen the capability of communal governments to provide services 2. generate a sustainable increase in local revenues for local services 3. leverage greater access to central government funding and services 9 Corruption and Governance, Unité pour la Lutte Contre la Corruption (ULCC), 2007 10 Amnesty International facts and figures document AI Index: AMR 36/003/2014 09 January 2014- https://reliefweb.int/sites/reliefweb.int/files/resources/Displaced percent20people percent20still percent20leave percent20in percent20despair percent20four percent20years percent20after percent20devastating percent20earthquake.pdf 11 FY 2010 Haiti supplemental budget justification- https://www.state.gov/documents/organization/141243.pdf USAID.GOV FINAL EVALUATION REPORT | 3 4. improve national-level policy and legislation in support of decentralization and deconcentrated services 5. increase transparency, oversight, and accountability of local governments and deconcentrated service providers As part of its effort to improve public services in its nine targeted communes (Acul du Nord, Cap- Haitien, Caracol, Carrefour, Delmas, Kenscoff, Limonade, Ouanaminthe, and Saint Marc), LOKAL+ designed a strategic planning exercise in which each communal government formally consulted with its citizenry to discuss public service priorities and budgeting. This exercise informed LOKAL+ about which public services needed targeted assistance. To facilitate this process, LOKAL+ provided training and technical assistance to the nine communes about how to obtain the desired feedback from citizens, the private sector, and other stakeholders. EVALUATION BACKGROUND EVALUATION PURPOSE AND AUDIENCE The main purpose of the evaluation was to inform possible future USAID/Haiti programming in decentralization and communal government strengthening. In addition, it identified LOKAL+’s strengths and weaknesses and assessed the validity of the Activity’s development hypothesis: “if public services are improved, then government legitimacy could be further enhanced.” The intended audience for this evaluation is USAID/Haiti, specifically the Office of Democracy and Governance, Tetra Tech ARD, and key GoH institutions including MICT, municipal officials in the nine target communes, and Members of Parliament (MP) interested in promoting decentralization. Findings may also be useful for other donors and entities seeking to facilitate decentralization and deconcentration in Haiti. EVALUATION QUESTIONS This evaluation answered the following three evaluation questions (EQ): 1. To what extent was the original project design relevant to the concerns and needs of the GoH, including the parliament? 2. To what extent was the project effective, especially in terms of improving local accountability, transparency, and service delivery? This question specifically examines the project’s effectiveness in terms of: a. increasing tax revenues b. improving access to central government funding c. improving mayor and staff skills and knowledge of governance d. improving the legal framework for decentralization e. improving constituency satisfaction 3. To what extent will the local governments assisted by LOKAL+ be able to continue improved governance, especially in terms of transparency, accountability, and service delivery? For EQ 1, the evaluation examined the extent to which the GoH had ownership in the original design and how the Activity’s goals were relevant to lawmakers. The evaluation also assessed the extent to 4 | FINAL EVALUATION REPORT USAID.GOV which the Activity built upon best practices in government decentralization and communal government strengthening. In the absence of baseline data, the evaluation team (ET) responded to EQ3 by assessing the opinions of constituents, current/past mayors, and communal staff regarding which interventions they believed are likely to continue at the end of LOKAL+ and on their appropriation of such activities. The ET also assessed what they believe was needed to strengthen local governance. EVALUATION TEAM AND SUPPORT STRUCTURE Social Impact’s (SI) Haiti Evaluation and Survey Services (ESS) fielded a four-person ET possessing a complementary mix of evaluation, local governance, and capacity building expertise, as well as methodological knowledge. The Team Leader, Bertrand Laurent, is an expert in civil society and local governance, food security and rural development, and capacity building. Since 1986, Mr. Laurent has been involved with USAID evaluations and assessments in countries such as Senegal, Madagascar, Haiti, and other Caribbean countries. The Local Government Specialist, Peterly Riche, worked in Haiti for the past 13 years on various USAID funded activities that addressed local capacity and local governance challenges. Most recently, he worked as a Client Relations Manager for USAID’s Konesans E Konpetans Teknik (KONEKTE) program, where he managed data collection and program design procedures. The Team Leader and Local Governance Specialist were supported by two notetakers to increase the efficiency of data collection and analysis. ESS sub-contracted CHASE, a Haitian research firm, to support additional qualitative data collection on constituent satisfaction. The firm is composed of development professionals with more than 18 years of experience in the design, and monitoring and evaluation of programs in various sectors (including education; agriculture; nutrition; food security; water, sanitation, and hygiene (WASH); natural resource management; and governance). Working closely with the ET, CHASE conducted 35 focus group discussions (FGD) across all nine communes with local government constituents. CHASE focused on EQ2 (“To what extent was the project effective, specifically in terms improving local accountability, transparency, and service delivery”), and more specifically, the sub-element that called for “examining the project’s effects on improving constituency satisfaction.” The ET was remotely supported by SI headquarters-based staff, including Project Director Erica A. Holzaepfel, Project Manager Marissa Germain, and Project Assistant Nora Chamma. During fieldwork in Haiti, the ET was supported by Mr. Wesner Antione, the Haiti ESS Senior Evaluation Specialist. EVALUATION METHODS AND CHALLENGES The evaluation employed multiple qualitative data collection methods: desk review, key informant interviews (KII), group interviews (GI), FGDs, and direct observations (DO). By collecting data from a wide range of sources, this methodology, of which each technique is described in detail below, increased the reliability and validity of the evaluation’s findings. The evaluation process comprised three phases: (1) planning, (2) data collection, and (3) analysis, reporting, and presentations. USAID.GOV FINAL EVALUATION REPORT | 5 Figure 3: Evaluation Phases 1) Planning 2) Data Collection 3) Analysis, Reporting, and Presentation Desk Review Team Planning Meeting Data Analysis Design Report Inbrief Report Writing Inception Report Primary Data Collection Presentation to USAID/Haiti Travel Mobilization Preliminary Analysis PHASE ONE: DESK REVIEW The ET conducted a desk review of internal LOKAL+ reports and external publications to gain an understanding of the context in which the Activity was implemented. The ET summarized the findings from the desk review in an inception report and used the background knowledge as the basis for designing the evaluation. The ET relied upon the Activity documents to identify individuals and groups to be interviewed, as well as sites at which to conduct direct observation. The ET made every effort to interview all the individuals specifically identified in the reports and documented the cases where an informant could not be reached, located, or declined to participate. PHASE TWO: FIELD DATA COLLECTION Primary data were collected in Haiti Figure 4: Data Collection Methods Used and Gender Breakdown Methods used: Gender breakdown: 14 Men 2 Women 16 Key Informant Interviews 16 Informants 28 Men 6 Women 13 Group Informant Interviews 34 Informants 168 Men 162 Women 35 Focus Group Discussions 330 Participants 12 DO Visits from March 19 to April 7, 2018. Overall, the ET conducted 16 KIIs and 13 GIs involving 34 informants, 35 FGDs involving 330 respondents (162 of whom were women), and 12 DO site visits. KEY INFORMANT INTERVIEWS The ET conducted 16 in-person, KIIs with targeted individuals deemed best placed to reflect knowledgeably on the LOKAL+ design and implementation progress to date. The ET conducted the KIIs using semi- structured interview guides (i.e. containing both structured and unstructured questions – see Annexes E and F). Data collected through KIIs helped inform findings for each of the EQs and was triangulated against data collected from the document review, GIs, FGDs, and DOs. KII Sampling The ET purposively sampled key info included: rmants based on a review of LOKAL+ reports. Key informants 6 | FINAL EVALUATION REPORT USAID.GOV • Current/Past Mayors : current/past mayors are key actors responsible for service delivery in the communes. The mayors are the main LOKAL+ partners for IRs 1, 2, 3, and 5. The evaluation included current/past mayors from all nine LOKAL+ communes. Interviewing past mayors was critical because LOKAL+ was implemented when many staffing changes occurred within local governments as a result of elections held for mayors and other local officials in 2015. • Senior Municipal Civil Servants : As key personnel in the communes, the Senior Municipal Civil Servants are involved in all administrative management for city hall. They received capacity and institutional building training. The ET triangulated data collected from them with data from current/past mayors. • Central GoH Representatives from MICT : LOKAL+ main partners for IR 4. • Members of Parliament : The sampled Members of Parliament (MP) were champions of decentralization and collaborated to varying degrees with LOKAL+, thus relevant to IR 4. • Former LOKAL+ Staff and Activity Subcontractors: Bureau de Recherche en Informatique et en Développement Économiques et Social (BRIDES) and Institut de Formation du Sud (IFOS). GROUP INTERVIEWS In addition to interviewing individual informants, the ET conducted 13 GIs with small groups of key informants who were unable to be interviewed individually due to availability constraints. The ET used the same interview guides and implemented the same protocols for interview documentation and analysis for GIs as it used for KIIs. Data collected through GIs contributed to all three EQs and was triangulated against data collected from the document review, KIIs, FGDs, and DOs. GI Sampling As with the KIIs, the ET purposively sampled GI respondents through a review of LOKAL+ reports. Informants included current/past mayors, senior municipal civil servants, current LOKAL+ staff who worked directly with communes, and USAID staff. The latter included staff from USAID/Haiti’s Office of Democracy, Rights, and Governance who were involved in LOKAL+ technical management. LOKAL+ had several Contracting Officer’s Representatives (COR) over its period of performance. To ensure a complete perspective of the Activity’s implementation, the ET met with all available CORs at once. FOCUS GROUP DISCUSSIONS Through its subcontractor, CHASE, the evaluation conducted 35 FGDs, including four FGDs per targeted commune, except for Cap Haitian, where CHASE conducted only three FGDs. The evaluation used FGDs to collect data at the citizen level and assess tax payers’ satisfaction with service delivery. As most of the communal taxes raised derive from property taxes (according to LOKAL+ Activity reports), it was, therefore, fitting for the evaluation to specifically collect information from tax payers. Collecting data from representatives of civic associations established to promote citizen interests (e.g., market vendors, Chamber of Commerce, and youth associations [YASS]) was an efficient way to broadly assess constituency satisfaction across social and economic interests. FGDs included an average of 10 respondents per group. Data collected through FGDs mostly contributed towards (but were not limited to) EQ2, namely “to what extent was the project effective, especially in terms of improving local accountability, transparency, and service delivery.” FGDs also helped gather information on respondents’ perceptions of the USAID.GOV FINAL EVALUATION REPORT | 7 communes’ revenue, and their satisfaction with communal services. FGD data provided some insight into EQ3, specifically citizens’ perspectives on communes’ capacity to sustain changes introduced through the Activity. FGD Sampling Unlike KIIs and GIs, FGD respondents were composed of representatives from specific interest groups affected by LOKAL+, rather than key actors or decision-makers involved in Activity implementation and/or oversight. To identify FGD respondents, the ET obtained lists of LOKAL+ respondents from Tetra Tech ARD. The ET did not invite Mayors to nominate FGD respondents to avoid the possible domination of the FGD by any individual deemed by other respondents to be a “representative” of the mayor. Besides ensuring that women were fully included among respondents in all FGDs, the evaluation also conducted FGDs specifically with women’s associations (WASS) to address their interests around inclusion. The ET organized FGDs to be as homogeneous as possible, with respondents sharing the same social rank and/or sex. FGDs employed homogeneity to minimize the effect of speech monopolization by an alpha respondent to the detriment of other respondents who might be intimidated or not eager to discuss and defend differing points of view. Specifically, participants recommended by the mayor were not invited to avoid any possibility of in the presence of a politically connected individual hindering everyone’s full participation. FGD respondents included representatives from: • Entrepreneurs: As citizens paying for services such as patentes and other taxes, entrepreneurs provided information to triangulate with current/past mayors and senior municipal civil servants. • Key CSO Representatives: CSOs are advocacy groups that represent citizen positions, play a watchdog role in governance, and advocate for change. CSOs received training from LOKAL+ and were important LOKAL+ partners for IR 5. • Consultative Citizens Committee Members: Comprising citizens groups reflecting a range of sectors at the communal level, Consultative Citizens Committee (CCC) members were important LOKAL+ partners for IR 5 (see Annex I “Definition of Terms”). • WASS Representatives: Important to ensure gender inclusion, women’s groups were key LOKAL+ partners for IR 5. • YASS Representatives: As a key communal demographic, it was important to assess LOKAL+’s success including youth groups in capacity building activities. DIRECT OBSERVATION To complement the KIIs, GIs, and FGDs, the ET conducted 12 DOs at locations or facilities representing key LOKAL+ related activities relevant to the EQs. DOs gave direct visible confirmation of reported communal activities. In addition, they provided some insight into the relative success of these activities, or lack thereof. The DO Site Observation Diagnostic Sheet contained in Annex G shows how each of the questions/observation items relate to the EQs. Data collected through DOs contributed to EQ2 regarding effectiveness, as well as to EQ3. For example, as LOKAL+ provided training in maintaining communes’ computers, networks, and troubleshooting the information technology (IT) systems to support CIVITAX, the ET observed how communes used the software and equipment and assessed the extent to which they were being used and maintained as expected. Similarly, DOs provided insight into the level of organization and competency for service delivery, which is relevant to EQ2. As LOKAL+ provided training on transparency and oversight, the DOs allowed the ET to evaluate citizen access to communal data. Notes 8 | FINAL EVALUATION REPORT USAID.GOV from the DOs were transcribed and coded with reference to the EQs and triangulated with data from other sources. DO Site Sampling Given the importance of revenue generation and CIVITAX in LOKAL+’s work, most of the sites visited were in the nine targeted communes’ city halls where LOKAL+ supported IT centers and offices, and the Fiscalité and Urbanism services. The complete list of sites visited is presented in Annex M. DO site visits included: • City Halls : Carrefour, Kenscoff, Delmas, Saint Marc, Cap Haitian, Caracol, Limonade, Ouanaminth, and Acul du Nord; • Communal Project Sites : Dumpsite at Limonade (Ti Mouchette), Fish Market at Caracol, and Public Market at Caracol. PHASE THREE: DATA PROCESSING AND ANALYSIS The ET documented the KIIs and GIs in three ways: (1) “take-away message” notes (taken by the Team Leader and/or the Local Governance Specialist), (2) detailed summary notes (taken by the notetaker), and (3) (unless an interviewee objected) an audio recording. During the data collection and analysis processes, the ET took care to capture nuances in interviewee responses, triangulate the respondent data with published data wherever possible, and triangulate statements among KIIs and GIs by asking all respondents a uniform set of questions in addition to open-ended probing questions. The Team Leader and Local Governance Specialist accomplished this by interviewing KIIs together whenever possible, so that they could compare notes and perceptions and agree on the key “take-away messages” from each interview. In addition, a notetaker was present to take notes and audio-record each interview. The recordings enabled the ET to make sure that the detailed summary notes reflected all the key information. The ET assured respondents that their participation was completely voluntary, affirmed that their information would remain confidential, and obtained their consent to record the discussions. To protect respondent confidentiality, the ET did not attribute findings to any single respondent, nor were individuals mentioned by name in the evaluation report. When all interviews were complete, the ET systematically coded the notes and tallied the responses for analysis. First, the ET identified all the recurring themes (open codes) that emerged in the notes to develop the coding framework. This framework, called a tally sheet, listed the codes in the first column (organized by EQ) and each data source (KIIs, GIs, FGDs, and DOs) listed across the top (one data source per column). The ET then coded the data into the tally sheet by carefully reviewing each set of notes and indicating the presence of a specific theme in each set with a “1” in the corresponding box in the tally sheet. They then compared the coding across team members to address any differences. The ET noted all present themes, even if they were contradictory. Tallies were then calculated for all relevant disaggregations (gender, geography, stakeholder group, etc.), and the completed tally sheet was used to analyze the data by identifying key findings and any relevant patterns in the data. Generally, the most important findings were those that occurred in many interviews, although sometimes the absence of a theme (or if an issue was only raised by one or two people who hold critical positions) was also important. Through the tally sheet process, the ET grouped stakeholders according to their position relative to the Activity. In part, this was done to ensure the anonymity and confidentiality the ET guaranteed all USAID.GOV FINAL EVALUATION REPORT | 9 respondents. Thus, in the findings and conclusions sections below, the various respondents are categorized according to the following groups: • Six KIIs/GIs – Key Partners : LOKAL+ and its two sub-partners (4), USAID (1), and MICT officials (1); • Twenty-three KIIs/GIs – Local Governance Partners : MPs (3), current/past mayors (10), and senior municipal civil servants (10); • Thirty-five FGDs – Citizens : Entrepreneurs (9), CCC (8), WASS (9), YASS (9) Total Notes: 64 CHALLENGES AND MITIGATION STRATEGIES “INTERVIEWER EFFECT” An interview is a social interaction, and many elements of an interview, such as the interviewer’s appearance and behavior, can influence a respondents’ answers. Mitigation: As the evaluation engaged a wide variety of respondents in terms of profession, social class, gender, and political party, the ET sought to mitigate biased responses by carefully formulating questions, employing a qualitative design, comparing responses to both structured and unstructured questions, and by triangulating data (comparing responses from various respondents across different methods). TIME AND GEOGRAPHICAL CHALLENGES As LOKAL+ was implemented in nine communes during a period of time characterized by political and economic change affecting Activity stakeholders, the ET anticipated a variety of limitations to KIIs, FGDs, and even documentation. Time and distance can sometimes contribute to partial or total loss of documents or information because of improper stor