Evalyasyon Final Pèfòmans Sistèm Entegre Jesyon Finansye (IFMS)

EVALUATION REPORT FINAL PERFORMANCE EVALUATION OF THE INTEGRATED FINANCIAL MANAGEMENT SYSTEM (IFMS) ACTIVITY Haiti Evaluation and Survey Services Project MARCH 5, 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently for USAID by Social Impact. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DIRECTION GENERALE DES IMPOTS CENTRAL NETWORK OPERATIONS CENTER (NOC) (LEFT), MINISTERE DE L'INTERIEUR ET DES COLLECTIVITES TERRITORIALES KENSKOFF NOC (TOP RIGHT), OFFICE DE MANAGEMENT ET DES RESSOURCES HUMAINES’S NOC (BOTTOM RIGHT). PHOTO CREDIT: WESLY JEAN. ABSTRACT The United States Agency for International Development (USAID)/Haiti Integrated Financial Management System (IFMS) activity supported the Government of Haiti (GOH) in developing a system for transparent management of public revenues and expenditures. This final performance evaluation focused on Phase V of the activity, implemented by Chemonics International from September 2014 to March 2018. To answer three evaluation questions (EQs) regarding activity effectiveness (EQ 1), utilization (EQ 2), and sustainability (EQ 3), the evaluation team conducted a document review, 20 key informant interviews, 8 group interviews, and site observations including 12 diagnostics. For EQ 1, the evaluation concluded that infrastructure and systems implementation remain ongoing. Because the infrastructure is incomplete, aspects of the systems are not being used by intended beneficiaries. The activity conducted several trainings, but participants have not had opportunities to apply acquired skills. Connectivity and electricity issues hindered IFMS, as did lack of buy-in from GOH leadership. For EQ 2, GOH is using IFMS hardware and infrastructure, but not using some of the software applications. Financial reporting within the GOH remains the same as before this phase, real-time data exchange is not occurring, and gains to improve fiscal transparency were minimal. For EQ 3, the IFMS intranet and Network Operating Center were the activity’s most sustainable aspects, whereas non-infrastructure investments are currently less sustainable. More training and coaching on the new systems are required. The activity did not have a sustainability plan or exit strategy in place at the time of the evaluation. TABLE OF CONTENTS ACKNOWLEDGEMENTS ............................................................................................................. I ACRONYMS .................................................................................................................................... II EXECUTIVE SUMMARY ............................................................................................................. I 1. INTRODUCTION ................................................................................................................. 1 2. EVALUATION METHODS AND LIMITATIONS ........................................................7 3. FINDINGS AND CONCLUSIONS ................................................................................. 11 4. RECOMMENDATIONS ................................................................................................... 34 ANNEXES ANNEX A: EVALUATION STATEMENT OF WORK ......................................................... 37 ANNEX B: IFMS PHASE IMPLEMENTATION ..................................................................... 43 ANNEX C: EVALUATION DESIGN MATRIX ...................................................................... 44 ANNEX D: IFMS LOGIC FRAMEWORK ................................................................................ 46 ANNEX E: DATA COLLECTION PROTOCOLS ................................................................... 48 ANNEX E.1: SEMI-STRUCTURED INTERVIEW QUESTIONS ............................49 ANNEX E.2: SITE OBSERVATION/HARDWARE DIAGNOSTIC SHEET .........54 ANNEX E.3: SITE OBERSERVATION/SOFTWARE DIAGNOSTIC SHEET ......62 ANNEX F: INDICATOR & OUTPUT-LEVEL ANALYSIS TABLE ...................................70 ANNEX G: ANALYSIS OF PROVIDED EQUIPMENT ........................................................ 76 ANNEX H: TRAINING MATERIALS AND IFMS POLICY DOCUMENTS ....................77 ANNEX I: FULL LISTING OF REFERENCES AND REPORTS UTILIZED ....................78 ANNEX J: EVALUATION DISSEMINATION PLAN ........................................................... 81 TABLES TABLE 1. REQUIREMENTS FOR THE NOC UPGRADE ........................................................... 13 TABLE 2. TRAININGS PROVIDED BY THE ACTIVITY ........................................................... 23 FIGURES FIGURE 1. OVERVIEW OF THE IFMS ACTIVITY ....................................................................... 1 FIGURE 2. IFMS THEORY OF CHANGE REVISITED ................................................................. 3 FIGURE 3. IFMS ACTIVITY TASKS .............................................................................................. 4 FIGURE 4. EVALUATION PHASES ............................................................................................... 7 FIGURE 5. EVALUATION METHODS STATISTICS .................................................................... 8 FIGURE 6. DGI CENTRAL NOC ................................................................................................... 11 FIGURE 7. MICT KENSKOFF NOC .............................................................................................. 15 FIGURE 8. OMRH'S NOC ............................................................................................................... 16 FIGURE 9. EFFECTIVENESS CONCLUSION’S SUMMARY AND OVERVIEW ..................... 20 FIGURE 10. IMPACT CONCLUSION’S SUMMARY AND OVERVIEW .................................. 28 FIGURE 11: SUSTAINABILITY CONCLUSION’S SUMMARY AND OVERVIEW ................ 33 i | IFMS EVALUATION REPORT USAID.GOV i | IFMS EVALUATION REPORT USAID.GOV ACKNOWLEDGEMENTS The evaluation team would like to thank the representatives from the Government of Haiti who made time to engage with the team for this evaluation, particularly staff at the Ministère de l’Economie et des Finances (MEF), Direction Générale des Impôts ( DGI), Administration Générale des Douanes (AGD), Treasury, Ministère de l'Intérieur et des Collectivités Territoriales (MICT), Office de Management et des Ressources Humaines ( OMRH), Unité de Lutte Contre la Corruption ( ULCC), Cour Supérieure des Comptes et du Contentieux Administratif ( CSCCA), Ministère des Travaux Publics, Transports et Communications (MTPTC), and many others. The team thanks Chemonics for its support during the Integrated Financial Management System (IFMS) evaluation. We also thank representatives from Canadian International Development Agency, European Union, World Bank, and InterAmerican Development Bank (IDB) for participating in interviews and sharing information, as well as the Softengi representative who made time to meet with the team and share resources. The team must also thank the United States Agency for International Development (USAID) Democracy and Governance representative for his assistance and guidance during the preparation and execution of the evaluation. ACRONYMS AGD Administration Générale des Douanes ATL Assistant Team Leader ATS Automatic Transfer System BG Back-up Generator COP Chief of Party CP Commercial Power CDCS Country Development Cooperation Strategy CSS Cascading Style Sheets CSCCA Cour Supérieure des Comptes et du Contentieux Administratif DGI Direction Générale des Impôts DSI Direction des Services Informatiques EQ Evaluation Question ESS Evaluation and Survey Services GI Group Interview GOH Government of Haiti HR Human Resources HTML Hypertext Markup Language ICT Information and Communication Technologies IDB InterAmerican Development Bank IFMS Integrated Financial Management System IP Implementing Partner IT Information Technology KII Key Informant Interview MAN Metropolitan Area Network MAN/WAN Metropolitan Area Network/Wide Area Network MEF Ministère de l’Economie et des Finances MENFP Ministère De L’Education Nationale Et De La Formation Professionnelle MICT Ministère de l'Intérieur et des Collectivités Territoriales MOU Memorandum of Understanding MPCE Ministère de Planifications et Coopération Externe MSPP Ministère de la Planification et de la Coopération Externe MTPTC Ministère des Travaux Publics, Transports et Communications NOC Network Operations Center OAVCT Office d'Assurance Vehicules Contre Tiers OMRH Office de Management et des Ressources Humaines PFM Public Financial Management SI Social Impact SIGRH Système Interministériel de Gestion des Ressources Humaines SOW Scope of Work USAID.GOV IFMS EVALUATION REPORT | ii iii | IFMS EVALUATION REPORT USAID.GOV SYSDEP Système de Gestion des Dépenses TL Team Leader ULCC Unité de Lutte Contre la Corruption UPS Uninterruptible Power Supply USAID United States Agency for International Development UXP Universal eXchange Platform VESDA Very Early Smoke Detection Apparatus EXECUTIVE SUMMARY PROJECT BACKGROUND The United States Agency for International Development (USAID)/Haiti Integrated Financial Management System (IFMS) activity—valued at $10,455,379 and implemented by Chemonics International from September 2014 to March 2018 1 —aimed to support the Government of Haiti (GOH) in the development of a full and complete financial management system that ensures transparent management of public revenues, expenditures, and other resources throughout the executive branch at the national, regional, and municipal levels. 2 Implemented in five phases, the activity supported the GOH in the restoration, operation, and expansion of the IFMS network infrastructure and financial management systems to improve financial governance and combat corruption. The IFMS activity Phase V 3 design highlights four major activity components that, in concert, seek to reinforce and consolidate the GOH’s IFMS, permitting it to manage financial governance, human resources (HR), as well as auditing, reporting, and financial transactions at the national and municipal levels. The four components are (1) hardware, (2) software, (3) capacity building, and (4) technical support. The four components were addressed through seven tasks. For more details on the project background, see page 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS This final performance evaluation assessed Phase V of the IFMS activity’s overall effectiveness. The evaluation findings will inform possible future programming in public finance and economic governance in Haiti and other fragile states, serve as a strategic document for the development of USAID/Haiti’s Country Development Cooperation Strategy (CDCS), and identify lessons learned and best practices to inform future governance and information and communication technologies (ICT) programing. The audience for this evaluation is USAID/Washington, USAID/Haiti, Chemonics International, and key GOH institutions. See Annex A for the Evaluation Scope of Work (SOW) and Annex C for the Evaluation Design Matrix. EVALUATION QUESTIONS This evaluation sought to answer the following overarching evaluation questions (EQs) as determined by USAID/Haiti. See the accompanying sub-questions on page 5. 1. To what extent was the program effectively implemented? 2. To what extent is the GOH generally, and the Ministère de l’Economie et des Finances (MEF) specifically, at both the national and municipal levels, using the IFMS? 3. How sustainable is the IFMS? 1 The original activity was set to end in September 2017, but the activity received a six-month extension. 2 SOW Final Performance Evaluation of Integrated Financial Management System (IFMS). USAID, 2017. 3 The IFMS activity is now in its fifth phase. Phases I and II were completed by 2006, and Phases III and IV ran from 2007 until implementation was interrupted by the 2010 earthquake. i | IFMS EVALUATION REPORT USAID.GOV EVALUATION DESIGN, METHODS, AND LIMITATIONS This evaluation employed a mixed-methods evaluation design that included a combination of qualitative data collection methods to answer the key questions listed above. Between October 25 and November 21, 2017, a four-member team collected data throughout Port-au-Prince, Haiti, with many stakeholders from USAID and Chemonics International and representatives from across the GOH. The evaluation team conducted a document review, 20 key informant interviews (KIIs), 8 group interviews (GIs), and structured site observations that included 12 diagnostics for the hardware and software components of the IFMS activity. 4 Altogether, the team collected data with over 57 individuals (48 males and 9 females) during the KIIs, GIs, and Diagnostics. See Annex G for the data collection protocols. LIMITATIONS AND MITIGATIONS STRATEGIES The evaluation team faced certain challenges and limitations in conducting this evaluation. Due to the evaluation’s primarily qualitative design, respondent biases such as recall, and selection bias may have occurred. Given the level of formality required to schedule meetings with GOH representatives and the lack of availability of GOH ministry leadership and non-information technology (IT) staff, the team was unable to meet with MEF leadership (with the exception of leadership from the IT department) and with Office d’Assurance Vehicules Contre Tiers (OAVCT) representatives. Moreover, a comprehensive list of project stakeholders and contacts was not available until the second week in country, which made the fieldwork preparation challenging, especially as related to defining the sample. Finally, the evaluation team experienced two changes before and during fieldwork. The team made every effort to mitigate the above limitations by triangulating different methods and believes that the robustness of the evaluation’s mixed-methods design has resulted in evidence-based findings, conclusions, and recommendations. For more detail on limitations and mitigation strategies, see page 10. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1: TO WHAT EXTENT WAS THE PROGRAM EFFECTIVELY IMPLEMENTED? 1. IFMS initiated activities under all seven tasks, and IFMS infrastructure and systems implementation remains ongoing. Substantial procurement and ICT investments have occurred that have updated, upgraded, and expanded the IFMS infrastructure. Diagnostics tools used during the team’s four site visits to the Network Operations Center (NOC) at MEF confirmed that Chemonics performed several improvements to the NOC infrastructure as requested by USAID in the project solicitation. Data shows that 254 pieces of equipment were replaced across existing NOCs; Chemonics procured and delivered 35 pieces of equipment to support the CIVITAX activities within several municipalities and under Ministère de l'Intérieur et des Collectivités Territoriales (MICT) oversight; and invested in generators, power inverters, batteries for the inverters, and almost all systems provided were coupled with an uninterruptible power supply (UPS). However, some of the infrastructure work that has been completed has bugs or show signs of poor maintenance (e.g. hardware not properly configured, dust accumulation in the equipment, broken air 4 The evaluation team conducted limited software diagnostics because most of the procured software, including SIGRH, UXP, and Business Intelligence, were only recently procured and introduced by the activity or are still in a test and installation phase. USAID.GOV IFMS EVALUATION REPORT | ii conditioning, inadequate cabling, etc.). Because the infrastructure is incomplete, major aspects of the systems are not being used by intended beneficiaries. 2. IFMS provided a significant amount of trainings on all the systems developed or delivered and developed policy and procedural documents on managing aspects of IFMS. However, the effectivenes of these trainings is greatly reduced, as the ongoing installation of associated infrastructure and software has not allowed GOH employees to apply skills learned. This conclusion is discussed under EQ 2. 3. IFMS helped GOH address connectivity and electricity issues in many instances, yet these issues remain a major problem for the activity because systems such as the Universal eXchange Platform (UXP) require dedicated internet connections for which GOH agencies often do not have budgets. Moreover, electricity remains unstable throughout the country, and many GOH agencies do not have stable generators or alternative power sources. 5 The support material provided by IFMS has not yet reached all the GOH agencies. 4. IFMS focused significant attention on working with GOH IT staff because these individuals will be the system’s caretakers. At the same time, IFMS did not successfully build buy-in from across the highest levels of GOH leadership. For example, a trip to Estonia to build support and buy-in for the UXP only included IT staff and not GOH leadership. This was a missed opportunity to standardize understanding of IFMS among GOH leadership, and identify champions to own and push forward the IFMS activity’s objectives. Chemonics also experienced communication and planning issues with GOH counterparts. GOH stakeholders and observers from the donor community suggested that the level of coordination and communication from Chemonics and USAID leadership was limited and thus insufficient to obtain buy-in from the relevant stakeholders. However, it should be noted that the activity was conducted in a very challenging context as Haiti went through a few transitions throughout implementation, from electoral transitions to political instability, and turnover of activity stakeholders both from Chemonics and GOH. EVALUATION QUESTION 2: TO WHAT EXTENT IS THE GOH USING THE IFMS? 1. The hardware and infrastructure elements of the project are the most successful aspects of the IFMS activity, but they also are still being completed. The activity struggled to implement the software and capacity building components. Many systems are still being rolled out and are not yet functional, and thus are not currently being used. 2. GOH agencies are currently using the Metropolitan Area Network/Wide Area Network (MAN/WAN), NOC, and other ICT equipment, such as computers, that were purchased. However, GOH is currently not using other systems or applications either because they are recent installations (e.g. Système Interministériel de Gestion des Ressources Humaines (SIGRH) – no interviewees reported using this software), are still being configured and are thus unavailable (e.g. VXRail and the Backup), or have yet to be purcahsed (e.g. Audit software). For some of these applications, their 5 According to interviews with OMRH representatives, their office does not have a backup generator to power its system in case of a commercial power outage. iii | IFMS EVALUATION REPORT USAID.GOV relevance to GOH remains uncertain due to other donors’ upcoming Public Financial Management (PFM) efforts. 3. The improved capabilities of MAN/WAN have supported continued functionality between GOH agencies and MEF. However MAN/WAN still faces challenges that inhibit its performance across many of the 61 sites. The network is working, but slowdowns are occurring that are affecting the use of software applications such as Système de Gestion des Dépenses (SYSDEP). Network service failures are quite common, and loss of service can last a long time, sometimes forcing employees to move sites to work. Lastly, UXP is not functional, so real-time financial data exchange on expenditures and revenue has not yet emerged outside of agencies that use the SYS application. 4. Financial reporting within the GOH remained the same as it was before this phase of IFMS. Although some GOH entities are still using the SYS software suite, no new data is being shared in real-time as the UXP is not yet in use for data exchange, and no data sources have been dedicated for the Business Intelligence software, rolled out in August 2017. Audit performance remained unchanged, as software had yet to be purchased at the time of the evaluation. 5. Although the activity established policies and procedures for the management of the MEF NOC and network monitoring (as noted in EQ 1), the team did not find any concrete evidence of buy-in and engagement across GOH agencies regarding managing the IFMS-developed systems, policies, and procedures. This was due, in part, to limited IT leadership at the MEF in the early stages of this phase of the IFMS activity. Another possible factor was the mixed understanding of the IFMS activity and its intended results among senior level, non-IT GOH stakeholders who were not targeted or included in some of the capacity building efforts. 6. Chemonics conducted a few efforts to promote transparency, but the full potential of these efforts is currently limited. Chemonics conducted an audit assessment and decided on an audit software, but the software has yet to be purchased and rolled out. The installed UXP, if functional, could help improve transparency by allowing the exchange of financial data among government agencies in real time. However, the platform is not functional and much remains to be done to make it functional across government agencies. Outside of these efforts, there was little evidence that Chemonics did much to promote IFMS among non-IT staff. 7. The Direction Générale des Impôts ( DGI) was the happiest and most satisfied IFMS beneficiary. DGI central office was happy to share their success with the team, and several DGI offices visited clearly associated their increased revenues with the equipment received. However, many other GOH partners seemed unhappy with the project management and implementation approach. There is also a fundamental difference in stakeholders’ understanding of their own needs and what the activity promised or delivered. Some agencies expressed needs that were not, in their opinion, considered in activity implementation. Others critiqued the quality of the collaboration and inputs they received from IFMS. EVALUATION QUESTION 3: HOW SUSTAINABLE IS THE IFMS? 1. The IFMS MAN/WAN and NOC upgrades are the most sustainable aspects of the activity. GOH is currently using them to support financial transactions and offer a certain level of corruption USAID.GOV IFMS EVALUATION REPORT | iv deterrence because of these systems’ ability to document transaction data that is captured and housed on the MEF servers. They are critical to GOH economic governance and financial management oversight, and the World Bank plans on using these aspects of the IFMS activity as the foundation of its Integrated Financial Management Information Systems (IFMIS) project. 2. Outside of the infrastructure investments, other elements of IFMS are less sustainable. UXP and software such as Business Intelligence, SIGRH, and Audit are in a fragile introductory stage; their integration into and utility for the relevant GOH institutions remain unproven. Moreover, more training and coaching are required to build staff confidence and work out any issues with these new systems. 3. Chemonics and USAID have yet to initiate exit and post-IFMS transition planning with the GOH. Additionally, the future and continuity of IFMS investments currently rely heavily on continued donor community funding and support, rather than a GOH-led budget. RECOMMENDATIONS 1. USAID should ensure that its activities are more integrated with and have stronger buy-in from the GOH. This could be achieved through the creation of a GOH-wide IFMS governance structure with representation from across all participating GOH entities, including signed Memorandum of Understanding (MOUs) and pledged cash or in-kind contributions for participation from each entity. 2. Local partners need early involvement in the needs identification and funded activity design. The evaluation showed that there were disconnects between Chemonics and some GOH entities over the technical decisions made, such as which applications to purchase. This stemmed from the fact that those agencies felt they had no input into the decisions made. As such, future activities should actively seek to increase the involvement and contribution of participating GOH entities—including technical, political, and IT representatives—early in the activity conception phase to promote acceptance of and support for the activity. 3. Future Fiscal Governance and Transparency activities should include non-IT staff and be conditioned on specific performance goals . Learning from the challenges raised by positioning IFMS as an IT project, future activity designs should target both IT and non-IT staff in efforts to promote fiscal governance and transparency. Incorporating realistic targets and benchmarks for improved fiscal governance and transparency in activity monitoring, evaluation, and learning plans can also help in this regard. 4. Given that some, but not all aspects of IFMS are sustainable, USAID should collaborate with other donors in the public financial reform sector for the remainder of activity implementation to strengthen IFMS activity gains. The evaluation showed that other donor-funded PFM projects are ongoing or in the planning stages and are looking to leverage IFMS gains. To avoid duplication and irrelevance, USAID should collaborate with the World Bank and other donors and pool efforts around consolidating and improving Haiti’s IFMS system. Recommended next steps are: (1) complete and strengthen Task 1 (IFMS technological infrastructure update, upgrade, and extension), and (2) establish a strong knowledge base among GOH staff to support the delivered systems. The latter v | IFMS EVALUATION REPORT USAID.GOV should include reviewing the skills and capabilities needed to support the NOC and the MAN/WAN, align training with existing and installed systems, and provide training and mentoring until a reasonable standard of integration, use, and functionality are achieved. 5. USAID should develop and implement an exit strategy in collaboration with GOH and other partners, well before the end of the project. The evaluation showed the extent of the work done by Chemonics over the last three years in upgrading sites, repairing the NOC and establishing systems (SIGRH, Business Intelligence, etc.) to support proper GOH fiscal governance activities. The evaluation also showed the limited involvement of the GOH overall, and the lack of ownership exerted towards these systems. As the activity nears its end, USAID should collaborate more closely with GOH to ensure the ownership of the activity’s results, the assets procured, and the systems installed to ensure the survival of the systems implemented. USAID.GOV IFMS EVALUATION REPORT | vi 1 | IFMS EVALUATION REPORT USAID.GOV 1. INTRODUCTION PROJECT BACKGROUND In November 2004, the United States Agency for International Development (USAID) signed an innovative agreement with the Government of Haiti (GOH) to support public finance reform and develop an Integrated Financial Management System (IFMS) with the Ministère de l’Economie et des Finances (MEF) at its core. USAID developed the IFMS activity to address a need for increased efficiency and transparency in GOH operations. There were many databases in use across the GOH, which limited its capacity to use information on public financial management (PFM) in a timely manner. 6 Among this agreement’s stated goals were to develop the MEF’s ability to exercise control over its financial transfers to line ministries and other government agencies and municipalities, prevent corruption, and ensure transparency. 7 The initial activity phases (Phases I and II) were completed by 2006 and most notably produced a network linking 40 GOH entities in the capital region, a remotely accessible expenditure tracking system [Système de Gestion des Dépenses (SYSDEP)], and strengthened the government Independent Audit entity and Anti-Corruption Unit. 8 The initial work to establish better control of financial transactions and fight corruption led to the creation of a Metropolitan Area Network (MAN) of 20 GOH sites. Phases III and IV started in 2007, but were interrupted by the 2010 earthquake. The initiative suffered a significant setback as the GOH suffered significant loss of life (technicians working on the IFMS activity) and materials, notably, the MAN infrastructure. To address ongoing business needs, GOH stitched together a MAN built mostly of surviving network equipment housed in a building donated by USAID and using radio towers to connect to the different ministries relocated throughout Port-au-Prince. The technology used was based on Cisco 1310 and 1410 radios in the 2.4 GHz (802.11B/G) and 5.7Ghz (802.11A) unlicensed bands configured as point-to-point and point-to-multipoint. After the emergency resulting from the earthquake began to subside, USAID hired an independent provider, Phase One, to assess the state of the IFMS activity. 9 Following the firm’s comprehensive report, USAID issued a new solicitation to continue the IFMS activity, address identified weaknesses and issues, and implement the recommendations. 6 Taken from notes from a phone meeting with Chemonics on September 13, 2017. 7 USAID Combatting Corruption through IFMIS (Brochure). 8 Integrated Financial Management Systems (IFMS) (Brochure). TDY Report Interim Government of Haiti—Integrated Financial Management Systems: USAID/M/PMO Assessment Team— November 29–December 3, 2004. 9 Haiti IFMS Assessment (Final Report) February 15, 2013—Report developed by Phase One. Activity Name: Integrated Financial Management System (IFMS) Implementer: Chemonics International Cooperative Agreement #: AID-OAA-I-12-00035 Total Estimated Ceiling: $10,455,379 Life of Activity: 09/26/2014 – 09/25/2017 Development Objective: Representative and Effective Governance FIGURE 1. OVERVIEW OF THE IFMS ACTIVITY The new IFMS activity (Phase V)—valued at $10,455,379 and implemented by Chemonics International from September 2014 to March 2018 (following a 6-month extension from the original September 2017 end date)—aimed to support GOH in developing a full and complete financial management system that ensures transparent management of public revenues, expenditures, and other resources throughout the executive branch at the national, regional, and municipal levels. 10 The activity provided support to GOH for the restoration, operation, and expansion of the IFMS network infrastructure and financial management systems that contribute to improved financial governance and combating corruption. In addition to improving existing functionality, access, security, and accountability, the new IFMS activity sought to develop and implement business continuity and contingency plans, as well as facilitate a process optimization exercise over all functions related to the IFMS before applications were updated to ensure a better alignment with GOH’s business processes and technology. This result was expected given the following key outcomes: • IFMS network infrastructure updated and improved • GOH capacity to utilize the IFMS for financial reporting and auditing processes expanded • The capacity of the network to support both operational support and financial auditing built • The expenditure and revenue systems integrated, and financial management improved • MEF information technology (IT) governance and management capabilities to properly document and implement governance policies and procedures for each of the stakeholder GOH ministries and municipalities enhanced Theory of Change The Phase V IFMS activity’s theory of change, illustrated in Figure 2, posits that if GOH and selected municipalities are provided the appropriate support for the restoration, operation, and expansion of both their IFMS-related network infrastructure and financial management systems, then they would be able to manage public revenues, expenditures, and other resources more effectively and transparently. The IFMS Phase V design highlighted four major activity components that, in concert, sought to reinforce and consolidate the GOH’s existing IFMS, permitting it to manage financial governance, human resources (HR), as well as auditing, reporting, and financial transactions at the national and municipal levels. The four components can be summarized as follows: • Component 1: Hardware , which focused on building both the physical and information and communication technologies (ICT) infrastructure at the national and municipal levels that will serve as the platform for Haiti’s IFMS. • Component 2: Software, which focused on developing, updating, and installing the requisite software and programs that serve as the user interface that GOH entities will use to securely conduct financial, HR, reporting, and auditing functions. • Components 3 and 4 : Capacity building and technical support , which focused on promoting the use, maintenance, and sustainability of GOH’s updated and expanded IFMS, including: − capacity building of key staff at provincial and national levels, promoting dialogue with other donors and strategic planning for maintaining and extending the IFMS; and 10 SOW Final Performance Evaluation of Integrated Financial Management System (IFMS). USAID, 2017. USAID.GOV IFMS EVALUATION REPORT | 2 3 | IFMS EVALUATION REPORT USAID.GOV − the creation of system protocols, procedures, and operational manuals for the maintenance and utilization of the IFMS. The IFMS activity was organized into seven tasks (see Figure 3 ) that address these four components. Annex E outlines the IFMS logical framework for the detailed tasks and subtasks. Annex B provides detailed information on the equipment utilized during each phase of IFMS. FIGURE 2 . IFMS THEORY OF CHA NGE REVISITED GOH has complete and functional IFMS Skill building, capacity building , and appropiation for IFMS (including reporting and auditing) Technical Support for the creation of systems protocols, procedures, and operations manuals IFMS administration software and interface development and security protocol Hardware and infrastucture development, expansion, and maintenance USAID.GOV IFMS EVALUATION REPORT | 4 Task 1 •Replace IFMS Information Technology to strengthen the existing network infrastructure. Task 2 •Build the capacity of MEF IFMS technical support team to ensure transparency and accountability for the IFMS. Task 3 •Integrate IFMS software to support GOH PFM policies including the design and installation of an IT interface. Task 4 •Expand the GOH capacity to utilize the IFMS for financial reporting and auditing processes. Task 5 •Extend the IFMS Network Infrasturcture to new GOH sites. Task 6 •Coordinate USAID IFMS contract activities with other donors in PFM. Task 7 •Assess and implement a government-wide HR management application with OMRH. Goal: The Government of Haiti to possess a full and complete IFMS that ensures transparent management of public revenues, expenditures, and other resources throughout the executive branch and the regional and municipal levels. FIGURE 3. IFMS ACTIVITY TASKS 5 | IFMS EVALUATION REPORT USAID.GOV Operating Environment Phase V of the IFMS activity navigated a challenging political environment in the one-year period following the 2016 presidential elections during which Haiti did not have a formal government, as well as during the new administration’s installation. This involved working with only willing government counterparts who did not have authority to sign Memoranda of Understanding or make data exchange agreements, which were critical steps to ensuring GOH ownership and buy-in. Additionally, turnover at the highest levels of GOH leadership remained a challenge throughout the life of the activity. New Ministers, Director Generals, and Managers were routinely appointed, which required the activity to continually begin anew its awareness raising and counterpart engagement work. 11 EVALUATION PURPOSE AND AUDIENCE This purpose of this final performance evaluation was to assess the overall effectiveness of the current IFMS activity, which started in 2014. The evaluation findings will inform future programming in public finance and economic governance in Haiti and other fragile states, serve as a strategic document for the development of USAID/Haiti’s Country Development Cooperation Strategy (CDCS), and identify lessons learned and best practices to inform future programming related to governance and ICT. The intended audience for this evaluation is USAID/Washington, USAID/Haiti (Office of Democracy and Governance), Chemonics International, and key GOH institutions including MEF, Direction Générale des Impôts (DGI), Administration Générale des Douanes (AGD), Cour Supérieure des Comptes et du Contentieux Administratif (CSCCA), Ministère de la Planification et de la Coopération Externe (MPCE), the Office de Management et des Ressources Humaines (OMRH), Unité de Lutte contre la Corruption ( ULCC), and all members of Haiti’s Public Finance Reform Commission. Findings may also be useful for other donors and entities seeking to develop new public finance reform projects in Haiti. EVALUATION QUESTIONS This evaluation sought to answer the following evaluation questions (EQs) and accompanying sub- questions as determined by USAID/Haiti: 1. To what extent was the program effectively implemented? 12 1.1. Is the IFMS infrastructure complete? 1.2. Are the new IFMS infrastructure and systems updated and improved, relative to the previous systems? 1.3. Have staff been sufficiently trained and coached in using th IFMS systems and interfaces? 1.4. Are the IFMS interfaces and ICT systems fully functional and without major bugs? e 1.5. To what extent do internet connectivity and electricity challenges inhibit the system’s use? 1.6. How effectively has the implementing partner (IP) negotiated and adapted to changing political circumstances in the implementation of the PFM system? 11 Input from Chemonics International into the Final Performance Evaluation of the Integrated Financial Management System (IFMS) activity (2018). 12 Definitions on the textboxes come from the USAID/Office of U.S. Foreign Disaster Assistance’s Evaluation Guidelines for Foreign Assistance - http://pdf.usaid.gov/pdf_docs/Pnado818.pdf . Effectiveness “Effectiveness refers to the extent to which an intervention’s objectives are achieved or are likely to be achieved.” USAID.GOV IFMS EVALUATION REPORT | 6 2. To what extent is the Haitian Government, generally, and the Ministry of Economy and Finance, specifically, at both the national and municipal levels, using the IFMS? 2.1. To what extent has IFMS contributed to improving GOH capacity to utilize IFMS for financial reporting and auditing processes? 2.2. To what extent has the program facilitated staff to connect the revenue and expenditure streams of the GOH financial chain? 2.3. How is IFMS enhancing MEF IT governance and management capabilities to properly document and implement governance policies and procedures? 2.4. What transparency measures have been implemented, and how are the GOH agencies tasked with combating corruption utilizing the IFMS to better do their jobs? 2.5. Are the beneficiary entities satisfied by the results of the program? Do they correspond to their real needs? 3. How sustainable is the IFMS? 3.1. Are there plans and processes in place to continue providing the training necessary to effectively use the IFMS? 3.2. Is the necessary budget in place to maintain the IFMS infrastructure and continue ongoing training needs? 3.3. To what extent do internet connectivity and electricity challenges inhibit the system’s use? TEAM COMPOSITION The evaluation team was comprised of four core team members and one supplementary technical expert. Evaluation Team Leader (TL), Dr. Brandy Jones , brought more than 10 years of experience leading mixed-methods performance evaluations for USAID and other donors, especially with regards to governance and decentralization project evaluations in French-speaking nations. Assistant Team Leader (ATL), Rudolph P. St. Jean , is a Haitian information technology professional with expertise in monitoring and evaluating advanced information management and financial solutions. PFM Technical Expert, Stephen Leeds , brought over 30 years of experience conducting evaluations of USAID and other donors’ PFM programming. IT Technical Expert, Carl Lacrête , is a local Information System Engineer with technical skills and training in information system management and database implementation. Mr. Lacrête served as the supplementary technical expert on the team. Notetaker, Wesly Jean , had seven years of research experience working throughout Haiti collecting both quantitative and qualitative data and managing research teams of various sizes. The evaluation team was supported by an SI management team composed of Evaluation and Survey Services (ESS) Chief of Party (COP), Dr. Jennifer Mandel ; Project Director, Jean-Camille Kollmorgen ; Project Manager, Mariana Davila ; and Project Assistant, Lisa Anderberg . This team provided technical and managerial oversight, quality assurance, and logistical support throughout the evaluation. Impact “In evaluation parlance, impacts refer to results or effects that are caused by, or are attributable to, a project or program or policy.” Sustainability “Sustainability refers to the continuation of an intervention’s services and benefits after foreign assistance ends. Three dimensions of sustainability— financial, institutional and environmental—can be examined in evaluations.” 7 | IFMS EVALUATION REPORT USAID.GOV 2. EVALUATION METHODS AND LIMITATIONS This evaluation employed a mixed-methods evaluation design, including a combination of qualitative data collection methods. Between October 25 and November 21, 2017, the four-member team collected data throughout Port-au-Prince, Haiti, with stakeholders from USAID, Chemonics International, and across the GOH. See Annex C for the evaluation design matrix. The evaluation included a document review, key informant interviews (KIIs), group interviews (GIs), and structured site observations that included IFMS hardware and software component diagnostics. 13 The evaluation process consisted of three phases: (1) planning; (2) data collection; and (3) analysis, reporting, and presentations. FIGU RE 4. EVALUATION PHASES 3) Analysis, Reporting, 1) Planning 2) Data Collection and Presentation Desk Review Team Planning Meeting Data Analysis Design Report Inbrief Report Writing Inception Report Primary Data Collection Presentation to USAID/Haiti Travel Mobilization Preliminary Analysis PHASE ONE: PLANNING From August 2017 until the team’s arrival in Haiti on October 25, 2017, Haiti ESS and the evaluation team solicited all relevant documents for the desk review and conducted teleconferences with USAID and Chemonics to clarify the IFMS activity scope as it related to design, geographic reach, and sampling. After these meetings and initial sharing of project documents, the team began the desk review, which included program-specific documents that USAID and Chemonics provided, as well as secondary literature regarding the PFM sector more broadly. See Annex K for a list of documents reviewed. The team summarized the findings from the desk review in an Inception Report and used the background knowledge as the basis for the designing the evaluation. PHASE TWO: DATA COLLECTION Upon arrival of the TL in Port-au-Prince, the team held a meeting with Chemonics on October 25, 2017 and an in-brief with USAID on October 27, 2017. Following these meetings, the team finalized fieldwork preparation and began data collection. 13 Limited software diagnostics were conducted because most of the procured software—including SIGRH, UXP, and Business Intelligence—were only recently procured and introduced by the activity or were still in a test and installation phase. KEY INFORMANT AND GROUP INTERVIEWS The evaluation team conducted 20 KIIs and 8 GIs 14 representing the following organizations: • USAID • Chemonics International; • International Donors: World Bank, Embassy of Canada, and InterAmerican Development Bank (IDB) • GOH: MEF, DGI, AGD, ULCC, CSCCA, Ministère des Travaux Publics, Transports et Communications (MTPTC), Ministère de l'Intérieur et des Collectivités Territoriales (MICT), Ministère De L’Education Nationale Et De La Formation Professionnelle (MENFP), and OMRH • IFMS Service Providers KII and GI protocols were tailored to each informant’s role, the nature and extent of their involvement in the IFMS activity, and the time available for interviewing. All data was collected in French. KII and GIs protocols are in Annex G . IFMS HARDWARE AND SOFTWARE DIAGNOSTICS The evaluation team also conducted diagnostics using an observation checklist (see Annex G.3 ) to verify the quality and content of materials, software, and infrastructure that were installed and provided at the sampled sites. While hardware diagnostics were conducted at almost all visited sites, the software diagnostics were not conducted at all sites because many of the software and applications were either still being installed or developed (Network Monitoring Software, Virtualization Software, Backup System, Business Intelligence Software, Human Resource Application), were in an initial test phase (Universal eXchange Platform [UXP]), or had yet to be purchased (Audit Software). The evaluation team conducted 12 diagnostics overall at the following sites: • MEF • DGI headquarters and 4 Sub-Offices (Delmas, Carrefour, Kenscoff, and Pétionville) • MICT sub-offices • CSCCA headquarters • ULCC headquarters • MENFP headquarters Altogether, the evaluation team collected data with 57 individuals (48 males and 9 females) during the KIIs, GIs, and Diagnostics. 14 GIs are interviews where more than one person is present to participate in the key informant interview. USAID.GOV IFMS EVALUATION REPORT | 8 The methods we used: How many people we interviewed: 2 Women 18 Men 20 Key Informants 7 Women 18 Men 25 Informants Key Informant Interviews 20 12 Men 12 Informants Diagnostics 1 Group Interviews 8 FIGURE 5 . EVALUATION METHODS STATISTICS SAMPLING The evaluation employed a purposive sampling approach, inviting individuals to participate in KIIs and GIs based on (1) their association with the seven IFMS tasks and (2) their representation across national, municipal, and local level stakeholders. Annex F provides an overview of the sampling frame. PHASE THREE: ANALYSIS, REPORTING, PRESENTATION DATA CLEANING AND ANALYSIS During fieldwork, the team captured all notes in electronic form. Data were organized and checked for quality weekly. When necessary, the team conducted follow-up meetings and email correspondence with interviewees to ensure that data were as complete and comprehensive as possible. The team shared the notes with the SI management team weekly who reviewed them for quality and clarity. Following fieldwork, the evaluation team organized and prepared the KIIs/GIs, document review, structured site observations, and hardware and software diagnostics for analysis. The team assembled in the Haiti ESS office to assess the data collected and prepare for data analysis. The team developed a debriefing document, organizing the data collected by the IFMS activity’s planned tasks. The team developed a coding sheet using Microsoft Excel, then coded two interviews together to test the coding approach and clarify any discrepancies. The team agreed on coding responsibilities for each team member and created identification codes for each evaluation respondent to ensure that all data were confidential. After departure from the field, the team coded all interview notes, and used spot checks across team members to verify coding consistency. In addition to coding, the team also conducted content and trend analysis and data triangulation to confirm findings. The triangulation approach included comparing data from different interviewees, document review, and diagnostics to confirm validity and accuracy. Finally, the team used the data from all sources (site observations, diagnostics, KII/GI, and document review) to complete an output and indicator level analysis sheet. LIMITATIONS AND MITIGATION STRATEGIES The team identified the following challenges and solutions during the evaluation: QUALITATIVE APPROACH: The primary approach for this evaluation was qualitative data collection. The full capture and analysis of qualitative data is complex, and the opinions of informants are, by their nature, subjective. The team members may have failed to record or correctly transcribe important data due to issues with translation or the use of technical jargon. The evalua