Evalyasyon Pèfòmans Pwogram Ogmantasyon Mendèv KONEKTE

DISCLAIMER- The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Photo Credit: PERFORMANCE EVALUATION OF WORKFORCE AUGMENTATION PROGRAM KONEKTE EVALUATION REPORT NOVEMBER 2018 NATIONAL PALACE OF HAITI PHOTO CREDIT: DRGULAS, 2009 PERFORMANCE EVALUATION OF WORKFORCE AUGMENTATION KONesans E Konpetans TEknik (“KONEKTE”) EVALUATION REPORT November 2018 Evaluation Mechanism Number: AID-521-C-17-00002 Haiti Evaluation and Survey Services for USAID/Haiti ABSTRACT Social Impact’s Haiti Evaluation and Survey Services project conducted an independent final evaluation of the United States Agency for International Development (USAID)/Haiti’s Workforce Augmentation Program KONesans E Konpetans TEknik (KONEKTE) Activity implemented by Management Systems International. KONEKTE was designed to fill workforce vacancies and provide Technical Advisors (TAs) to select Government of Haiti (GOH) counterparts to support the United States Government’s post- earthquake strategy. The evaluation report assesses whether KONEKTE achieved intended results, identifies lessons learned, and provides guidance for future USAID/Haiti programs. KONEKTE placed 45 TAs with 11 GOH counterparts. However, recruitment was slower than planned. Changes in USAID, Implementing Partner and GOH counterpart leadership, challenges aligning multiple key stakeholders’ vision and needs and external social, political and economic factors affected efficiency. Although USAID largely determined priority intervention areas, KONEKTE’s GOH counterparts chose and managed their own TAs, which made the Activity relevant to counterpart concerns and needs. GOH officials indicated that KONEKTE strengthened their voice in day-to-day decision-making within USAID programs. KONEKTE technical expertise and tools contributed to on-going improvements in GOH recruitment and human resource (HR) management systems. However, KONEKTE did not make significant progress building GOH capacity to assess HR needs systematically. In addition, KONEKTE was not very effective at transferring knowhow to GOH counterparts due to their limited absorptive capacity, TAs’ focus on doing versus transferring skills, and weak oversight to promote TA deliverables’ use. Some KONEKTE GOH counterparts are sustaining KONEKTE TAs work, while many others are not. Likely sustainable KONEKTE interventions include introducing new laws supporting on-going reforms and creating tools and guidelines to support offices and institutions’ core activities. TABLE OF CONTENTS ABSTRACT II TABLE OF CONTENTS III LIST OF TABLES AND FIGURES IV ACRONYMS V EXECUTIVE SUMMARY I I. INTRODUCTION 5 Project Background 5 Theory of Change 6 Activities/Components 6 Stages/Activities Completed 7 Evaluation Purpose and Audience 8 Evaluation Questions 9 II. EVALUATION DESIGN 9 Sampling 9 Data Collection Methods 10 Analysis 11 Limitations and Mitigation Strategies 12 III. FINDINGS AND CONCLUSIONS 13 EQ1: To what extent was the original project design relevant to the GoH concerns and needs? 13 EQ2: How effectively was the activity implemented and managed? 16 EQ2A. The extent to which activity interventions actively built GOH capacity to identify and hire appropriate technical assistance 19 EQ2B. The extent to which technical assistance provided through the activity transferred knowledge and capacity to GOH officials and offices 24 EQ3: How sustainable are the activity’s interventions? 27 IV. RECOMMENDATIONS FOR USAID 31 ANNEX A: EVALUATION STATEMENT OF WORK 32 ANNEX B: REFERENCES AND ENDNOTES 38 ANNEX C: EVALUATION TEAM MEMBERS 40 ANNEX D: EVALUATION DESIGN MATRIX 42 ANNEX E: LIST OF PERSONS INTERVIEWED 45 ANNEX F: INFORMED CONSENT SCRIPTS & DATA COLLECTION PROTOCOLS 48 ANNEX G: DISCLOSURE OF CONFLICT OF INTEREST 63 LIST OF TABLES AND FIGURES Table 1: KONEKTE Sub-purposes, Outputs, and Activities 7 Table 2: Key Informant Interviews by Stakeholder Group and Gender 11 Table 3: Focus Group Discussions by Stakeholder Group and Gender 11 Table 4: KONEKTE: Timing of Changes in Key Stakeholder Institutions’ Leadership 18 Figure 1: Four Pillars of the USG’s Post-Earthquake Strategy for Renewal and Economic Opportunity in Haiti 6 Figure 2: Number of TAs by Institution 8 Figure 3: Steps in KONEKTE Recruitment Process 20 ACRONYMS CFET CG COP Centre de Formation et d’Encadrement Technique Coordinator General Chief of Party COR CRM D&G DIS Contracting Officer’s Representative Client Relations Manager Democracy and Governance Department of Information Systems DRH Directeurs et Directrices des Ressources Humaines EPAP Public Administration Performance Evaluation ESS Evaluation and Survey Services ET Evaluation Team EQ FGD Evaluation Question Focus Group Discussion FY Fiscal Year GOH HICD Government of Haiti Human and Institutional Capacity Development HR HRD HRMIS HQ IFMS Human Resources Human Resources Department Human Resources Management Information System Headquarters Integrated Financial Management System IP Implementing Partner/Counterpart IT KII Information Technology Key Informant Interview KONEKTE KONesans E Konpetans TEknik M&E MARNDR MAST MENFP Monitoring and Evaluation Ministry of Agriculture, Natural Resources and Rural Development Ministry of Social Affairs and Labor Ministry of National Education and Vocational Training MOEF Ministry of Economy and Finance MSI Management Systems International NGO Nongovernmental Organization OMRH Office of Management and Human Resources SI Social Impact, Inc. SOW Scope of Work TA TL Technical Advisor Team Leader USAID United States Agency for International Development USG United States Government i | KONEKTE PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) project design and conduct an independent final evaluation of the Workforce Augmentation Program KONesans E Konpetans TEknik (KONEKTE) implemented by Management System International (MSI) under Contract No. AID-521-C-13-00001. The purpose of this report is to present the evaluation team’s (ET) main findings, conclusions, and recommendations. EVALUATION PURPOSE AND EVALUATION QUESTIONS The objective of the KONEKTE final evaluation is to assess the extent to which the Activity achieved intended results/outcomes, identify any lessons learned, and recommend possible follow-up actions, as well as provide guidance for similar USAID programs in Haiti and elsewhere. Its primary audience is USAID/Haiti, which may use the evaluation findings and recommendations to inform the Mission’s future programming in the areas of governance and institutional strengthening. The secondary audiences are MSI, MSI’s local subcontractor, Centre de Formation et d’Encadrement Technique (CFET), and KONEKTE’s main Government of Haiti (GOH) counterpart, the Office of Management and Human Resources (OMRH), which may also consider evaluation findings and recommendations in their current programs and possible future collaboration with USAID/Haiti. This final evaluation will answer the evaluation questions (EQs) below. When answering these EQs, the ET will consider stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent was the original project design relevant to the GOH concerns and needs? 2. How effectively was the Activity implemented and managed? a. In answering this question, the evaluation assessed two fundamental Activity elements: b. The extent to which Activity interventions actively build GOH capacity to identify and hire appropriate technical assistance. c. The extent to which technical assistance provided through the Activity transferred knowledge and capacity to GOH officials and offices. 3. How sustainable are the Activity’s interventions? PROJECT BACKGROUND On October 30, 2012, USAID awarded the Haiti “Workforce Augmentation Program” contract, known as “KONEKTE” to MSI and its local counterpart, CFET, with a ceiling price of $21,795,809. KONEKTE was designed to fill workforce vacancies and provide technical expertise to select USAID GOH counterparts in support of the post-earthquake United States Government (USG) Haiti strategy. The Activity was intended to contribute to achieving USAID objectives under the strategy's four pillars: Infrastructure and Energy, Food and Economic Security, Health and Other Basic Services, and Governance and Rule of Law. Beginning in November 2015, KONEKTE expanded its strategy to include interventions to strengthen OMRH institutional capacity to lead human resource (HR) management-related reforms within the Haitian government’s civil service. EVALUATION DESIGN, METHODS, AND LIMITATIONS The ET used a qualitative evaluation design, consisting of document review, key informant interviews (KIIs), focus group discussions (FGDs) and direct observation. During three weeks of in-country data collection, the ET interviewed 52 individuals (18 women and 34 men) from various stakeholder groups. The ET also organized one FGD with GOH Human Resource Department (HRD) personnel with 12 ii | KONEKTE PERFORMANCE EVALUATION USAID.GOV participants (eight female, four male). The team used purposeful sampling to select informants who were well-informed about the Activity and represented diverse stakeholder groups including representatives of USAID, MSI and CFET, OMRH and other GOH counterparts, technical advisors (TAs), other donors, and other USAID Activities. The ET was aware of potential recall, selection, response, and gender bias risks in data collection and analysis, and developed mitigation strategies to overcome these. To overcome bias and increase the validity of evaluation findings, the team asked similar questions to all informants and triangulated responses. The team also prefaced all interviews and group interviews with a protocol highlighting the evaluation purpose and the confidentiality of interviews. Although some key stakeholders were no longer in Haiti, the ET was able to interview most by telephone or Skype. The ET obtained interviews with individuals from all stakeholder categories, but interviewed fewer USAID technical office and OMRH officials than planned. FINDINGS AND CONCLUSIONS EQ1: TO WHAT EXTENT WAS THE ORIGINAL PROJECT DESIGN RELEVANT TO THE GOH CONCERNS AND NEEDS? Based on Activity reports and KIIs, KONEKTE’s design enabled GOH officials and offices to choose and manage their own technical assistants. This made KONEKTE relevant to its main GOH counterparts’ concerns and needs. The Activity administered client satisfaction surveys where high marks were received and feedback from GOH officials the ET interviewed indicate overall high levels of GOH counterpart satisfaction with KONEKTE-supported TAs. KONEKTE filled gaps in both strategic and day-to-day needs of GOH counterparts. In theory, KONEKTE assistance started with GOH counterpart requests. In practice, USAID technical offices oriented GOH institutions’ requests to intervention areas that aligned with its priorities, limiting the GOH counterparts’ opportunity to set their own priorities for KONEKTE assistance. USAID’s Democracy and Governance office allowed the 2012 – 2016 OMRH leadership a greater role in setting KONEKTE’s agenda within this office than did other USAID technical offices, which appeared to use KONEKTE to reinforce existing programs, narrowing KONEKTE’s scope to pre-determined intervention areas. Based on GOH officials’ accounts, KONEKTE strengthened GOH counterparts’ voice in day-to-day decision-making within USAID programs. Embedding senior TAs in GOH offices strengthened GOH dialogue with USAID, and in some cases, GOH oversight of USAID implementor interventions. This was not an explicit objective of KONEKTE, but is a highly positive, unintended consequence of placing TAs in USAID GOH counterpart offices. EQ2: HOW EFFECTIVELY WAS THE ACTIVITY IMPLEMENTED AND MANAGED? KONEKTE's implementation was not efficient during its first 12 to 18 months, and again at the Activity’s end. Although, after a slow start, it recruited approximately the targeted number of TAs, KONEKTE was significantly less efficient than planned in terms of the number of days required to field a TA after receiving a validated scope of work (SOW) from USAID. The causes of inefficiency were related to variety factors, some within the control of USAID and the IP and others not. USAID was slow issuing written approvals of implementing partner (IP) work and monitoring and evaluation (M&E) plans as well as modifying the contract to reflect changes in the Activity’s scope, which resulted in over-dependence on oral agreements and created management uncertainty at times. Frequent changes among decision- makers within USAID, GOH counterparts, and the IP, also negatively affected KONEKTE's overall effectiveness. During its five-year implementation period, KONEKTE had three Chiefs of Party (COPs), four USAID Contracting Office Representatives (CORs), three OMRH Coordinator Generals (CGs), and experienced several changes in government. USAID and IP internal management factors such as lack iii | KONEKTE PERFORMANCE EVALUATION USAID.GOV of shared vision and cohesive management, too many decision-makers, poorly-defined and communicated roles and responsibilities, as well as unclear conditions for receiving KONEKTE assistance in the early implementation stages (USAID and IP) likewise also diminished implementation efficiency. EQ2A. THE EXTENT TO WHICH ACTIVITY INTERVENTIONS ACTIVELY BUILD GOH CAPACITY TO IDENTIFY AND HIRE APPROPRIATE TECHNICAL ASSISTANCE KONEKTE contributed to building GOH counterparts’ capacity to identify and hire appropriate technical assistance. KONEKTE used an open and transparent recruitment process in which GOH counterparts were actively engaged, potentially contributing to the GOH personnel’s awareness and hands-on experience applying good recruitment practices. MSI and USAID showed vision and adaptive capacity in re-framing KONEKTE for OMRH institutional capacity building, the GOH entity mandated to oversee public sector HR reforms. Although its contributions ended up being more modest than planned, KONEKTE supplied specialized technical assistance and recruitment and HR management tools that shaped OMRH intervention strategies and affirmed OMRH leadership of ongoing improvements in GOH recruitment and other HR management practices. KONEKTE did not make significant progress building GOH counterparts’ capacity to use systematic approaches to identify HR needs such as the Human and Institutional Capacity Development (HICD) framework, which was cancelled by USAID before planned implementation was complete. A specific focus on gender equity was not explicitly required in KONEKTE’s SOW. IP team members indicated that the Activity made efforts to ensure woman candidates for TA positions were proposed when possible, but did not require GOH counterparts, who were responsible for selecting TAs, to have a policy of “affirmative action.” According to KONEKTE Activity reports, out of a total of 45 TAs, seven were female (16 percent). Capacity-building for HRD personnel remains a significant need in this area as well as other HR management areas. EQ2B. THE EXTENT TO WHICH TECHNICAL ASSISTANCE PROVIDED THROUGH THE ACTIVITY TRANSFERRED KNOWLEDGE AND CAPACITY TO GOH OFFICIALS AND OFFICES Overall, KONEKTE was not very effective at transferring capacity and knowledge to GOH counterparts. KONEKTE TAs carried out a wide variety of activities and produced most planned deliverables to the satisfaction of GOH counterparts, especially during its first three years of implementation. By their nature, many of these deliverables (guides, manuals, plans, training modules, etc.), could have been used to transfer capacity and knowledge, but due to weak GOH ownership of some TA work, these outcomes were limited. Getting OMRH buy-in for KONEKTE work done in collaboration with the original CG was especially problematic after his departure. There were, however, exceptions including KONEKTE’s work with the Ministry of National Education and Vocational Training (MENFP) on improving the teaching of reading, and with OMRH and HRDs on recruitment and HR management (as highlighted in EQ2a findings). KONEKTE TAs faced significant external obstacles to transferring capacity and knowledge to its GOH counterparts, but also made tactical errors. The limited absorptive capacity of GOH counterparts was a significant factor preventing KONEKTE assistance from contributing to large-scale institutional capacity improvements. However, insufficient support for TAs on assessing needs and coaching, and weak oversight by IP and USAID managers to promote the use of TAs’ deliverables also limited KONEKTE’s effectiveness at transferring knowledge and capacity. EQ3: HOW SUSTAINABLE ARE THE ACTIVITY’S INTERVENTIONS? Some KONEKTE GOH counterpart institutions are currently able to sustain some work started by iv | KONEKTE PERFORMANCE EVALUATION USAID.GOV KONEKTE TAs while many others are not. Institutional restructuring/strengthening work within OMRH, MENFP, and Ministry of Agriculture, Natural Resources, and Rural Development’s (MARNRD) Department of Innovation contributed to organizing the institutions’ work and attracting government and other donor resources to support their activities. The positive outcomes of TA work in most other GOH institutions appears to have been short-term and did not lead to any lasting improvement in the host institutions’ capacity. Frequent changes in Haiti’s political landscape, overall weak governance capacity, and severely under-resourced government offices were significant obstacles to sustainability. KONEKTE TA interventions that are likely to have sustainable outcomes include the changes introduced in the legal framework supporting on-going reforms, and creating tools and guidelines to be used by offices and institutions for their core activities. KONEKTE’s experiment recruiting OMRH TAs on the government salary scale to fill middle management vacancies conditioned by their recruitment into the civil service after one year was a mixed success. OMRH retained two of four TAs recruited in this way, but none have stayed with OMRH post-Activity. KONEKTE TAs who integrated into the civil service were moved to other host institutions and expressed disappointment in the opportunities afforded them to contribute reforms outside the context of KONEKTE. There appears to be political will within the GOH to continue KONEKTE-supported HR reforms, especially with current OMRH leadership. All GOH counterparts expressed willingness to collaborate with USAID in future programs. RECOMMENDATIONS FOR USAID 1. USAID should not use a stand-alone, HR out-sourcing mechanism like KONEKTE to fill gaps in its counterparts’ capacity in Haiti in the future. KONEKTE showed that such a mechanism is not a useful way to streamline assistance to fill critical HR vacancies in an emergency recovery situation or more generally. The mechanism did not appear to help to overcome administrative and management burdens within USAID nor to save time. 2. USAID should continue to involve GOH counterparts in the implementation of its future programs. Ways to do this may include seconding TAs in GOH counterpart offices through existing technical Activities (versus through a stand-alone mechanism), as well as other institutional capacity building interventions. KONEKTE demonstrated that it is possible to involve GOH counterparts as implementers in USAID programs, in ways that enhance interventions’ sustainability. Even though KONEKTE showed there are significant obstacles to sustainably increasing GOH offices and officers’ capacity, not involving the GOH only reinforces weak governance. 3. USAID should consider additional support for OMRH on civil service administrative reforms. Among other topics, future programs should identify interventions to promote nondiscrimination in hiring. Future programs should explore work on policy and regulatory issues that affect the GOH ability to attract qualified candidates such as wage and professional advancement programs. 4. USAID should ensure that the future program agendas that cross-cut USAID technical offices are led/monitored by senior USAID staff, if not the Mission Director, to ensure coherence and buy-in from mission technical offices. To strengthen oversight and decision- making by technical offices, USAID should avoid using basket funding mechanisms except in situations where there is strong demand and similar objectives and needs across technical offices. 5. USAID should approve workplans, M&E frameworks, and process contract modifications in a timely manner to guide consistent decision-making by implementers, and avoid implementation preceding formal, written approvals. 5 | KONEKTE PERFORMANCE EVALUATION USAID.GOV I. INTRODUCTION The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) project design and conduct an independent final evaluation of the Workforce Augmentation Program, KONesans E Konpetans TEknik (KONEKTE), implemented by Management System International (MSI) (see Annex A for the evaluation Statement of Work). 1 Prior to conducting the fieldwork for this evaluation, the evaluation team (ET) presented its initial findings based on a document review (see Annex B for list of documents consulted and Annex C for profiles of the ET members) in an inception report. The ET presented its evaluation design, a detailed report approved by USAID/Haiti prior to starting fieldwork (see summary of evaluation design in the Evaluation Design Matrix found in Annex D ). The purpose of this report is to present the ET’s main findings, conclusions, and recommendations. PROJECT BACKGROUND The 2010 earthquake had disastrous effects on people’s lives and national infrastructure, and exacerbated the pre-earthquake development challenges of poor governance, weak institutions, and widespread corruption. United States Government (USG) assistance for re-building was guided by the Government of Haiti’s (GOH) Action Plan for National Recovery and Development and its subsequent Strategic Development Plan. 2 On October 30, 2012, USAID awarded the Haiti “Workforce Augmentation Program” contract, known as “KONEKTE” to MSI and its local counterpart Centre de Formation et d’Encadrement Technique (CFET), with a ceiling price of $21,795,809. KONEKTE was designed to fill workforce vacancies and provide technical expertise to select USAID GOH counterparts in support of the post-earthquake USG Haiti strategy. 3 The Activity was intended to contribute to achieving USAID objectives under the strategy's four pillars: 1 Throughout the report, “KONEKTE” is used interchangeably with “the Activity.” For example, a “KONEKTE technical advisor (TA)” is an advisor recruited through the KONEKTE Activity. “KONEKTE was effective…” Refers to the outcome of the Activity which includes the outcomes of both external and internal management factors. Specific references to the actions and outcomes attributed to implementing partners (IP) or USAID are clearly labeled. 2 Post-Earthquake USG Haiti Strategy: Toward Renewal and Economic Opportunity, https://reliefweb.int/report/haiti/post-earthquake-usg-haiti-strategy- toward-renewal-and-economic-opportunity accessed 8/29/18. 3 The Activity scope of work included the possibility of hiring TAs for private sector and nongovernmental organizations (NGOs) as well as GOH partners, but USAID decided early on to limit KONEKTE’s focus to GOH partners. Box 1. Program Overview Workforce Augmentation Program, KONEKTE (KONesans E Konpetans TEknik) Program Objective: To fill human resource gaps and provide technical expertise to select Haiti government partners in support of the “Post- Earthquake USG Haiti Strategy: Toward Renewal and Economic Opportunity.” Period of Performance: November 2012 – October 2017 Award Number: AID-521-C-13-00001 Total Funding Amount: $21,795,809 Implementing Partners: Management Systems International (MSI) and the Centre de Formation et d'Encadrement Technique (CFET) Key Partner: Government of Haiti’s Office de Management et des Ressources Humaines (OMRH), Geographic Presence: Port-au-Prince, Haiti 6 | KONEKTE PERFORMANCE EVALUATION USAID.GOV Figure 1: Four Pillars of the USG’s Post-Earthquake Strategy for Renewal and Economic Opportunity in Haiti 4 THEORY OF CHANGE KONEKTE’s main development hypothesis 5 was “if highly qualified technical advisors (TAs) are seconded to select GOH ministries and agencies, then GOH capacity to support and sustain the positive outcomes of the USG’s post-earthquake assistance will be enhanced. Beginning in November 2015, KONEKTE expanded its strategy to include interventions to strengthen the institutional capacity of the Office for Human Resource Management (OMRH) to lead Human Resource (HR) management-related reforms within the Haitian government’s civil service. This part of KONEKTE’s intervention strategy was based on the assumptions that if GOH institutions possess the capacity to carry-out fair, transparent, and competitive recruitment processes, and develop and apply better HR policies to manage civil servants, the civil service performance will improve and contribute to the GOH reaching its development and reform objectives. ACTIVITIES/COMPONENTS KONEKTE’s overall purpose was to fill gaps in both the skills and numbers of workers in selected Haitian government institutions so that these GOH counterparts had more capacity to carry out their mandated activities. This was to be accomplished by hiring TAs and seconding them to USAID GOH counterpart institutions. KONEKTE was also charged with supplying the essential tools TAs needed to do their job (computers, internet connections, and logistics support). Starting in September 2015, KONEKTE established four sub-purposes (one of which USAID cancelled in March 2016), each with specific outputs and sets of activities as summarized in Table 1 below: 4 USAID/Haiti. KONEKTE Program Year 5 Annual Progress Report. N.p.: USAID, 2017. Print. 5 Project documents do not explicitly state KONEKTE’s theory of change except in the draft 2016 AMELP, which was not approved by USAID. Figure l: FourPillarsof the USG'sPost-Earthquake Strategyfor Renewaland EconomicOpportunity in Haiti4 Infrastructure and Energy: Intermediate Result I : "Improved Infrastructure that Supports Community and Commercial Development" PILLAR I Health and Other Basic Services: Intermediate Result 3: "Improved Health Status and Learning Environment" PILLAR 3 THEORY OF CHANGE Food and Economic Security: Intermediate Result 2: "Increased Food and Economic Security" PILLAR 2 Governance and Rule of Law: Intermediate Result 4: "More Responsive Governance and Improved Rule Of Law" PILLAR 4 KONEKTE's main development hypothesis5 was "if highly qualified technical advisors (TAs) are seconded to select GOH ministries and agencies, then GOH capacity to support and sustain the positive outcomes of the USG's post-earthquake assistance will be enhanced. Beginning in November 20 15, KONEKTE expanded its strategy to include interventions to strengthen the institutional capacity of the Office for Human Resource Management (OMRH) to lead Human Resource (HR) management-related reforms within the Haitian government's civil service. This part of KONEKTE's intervention strategy was based on the assumptions that if GOH institutions possess the capacity to carry-out fair, transparent, and competitive recruitment processes, and develop and apply better HR policies to manage civil servants, the civil service performance will improve and contribute to the GOH reaching its development and reform objectives. ACTIVITIES/COMPONENTS KONEKTE's overall purpose was to fill gaps in both the skills and numbers of workers in selected Haitian government institutions so that these GOH counterparts had more capacity to carry out their mandated activities. This was to be accomplished by hiring TAs and seconding them to USAID GOH counterpart institutions. KONEKTE was also charged with supplying the essential tools TAs needed to do their job (computers, internet connections, and logistics support). Starting in September 201 5, KONEKTE established four sub-purposes (one of which USAID cancelled in March 20 16), each with specific outputs and sets of activities as summarized in Table I below: 4 USAID/Haiti. KONEKTE Program Year 5 Annual Progress Report. N.p.: USAID, 2017. Print. 5 Project documents do not explicitly state KONEKTE's theory of change except in the draft 2016 AMELP, which was not approved by USAID. 61 KONEKTE PERFORMANCE EVALUATION USAID.GOV 7 | KONEKTE PERFORMANCE EVALUATION USAID.GOV Table 1: KONEKTE Sub-purposes, Outputs, and Activities SUB - PURPOSES AND ASSOCIATED OUTPUTS AND ACTIVITIES SUB - PURPOSE OUTPUTS MAIN ACTIVITIES Activity Sub - Purpose 1: TAs p laced in select GOH entities efficiently Timely recruitment of appropriate, qualified TAs. • Establish and implement a competitive and transparent recruitment process to hire TAs and place them in requesting GOH counterpart institutions; • Support TA work place integration and monitor their work; • Manage TA remuneration, annual leave, and other benefits; • Report on TA activities to USAID. Activity Sub-Purpose 2: HR Management in OMRH and selected GOH ministries improved OMRH approved TA deliverables. • Provide qualified long and short-term TAs to OMRH (as above); • Procure complementary expertise, materials, and logistic support for OMRH capacity building activities; • Implement “Job Portal” module of the Inter-Ministry System for HR Management. Activity Sub-Purpose 3: Contract management skills in OMRH improved Removed from scope as per Contract Modification 4. Activity Sub-Purpose 4: Local service provider capacity in managing USAID/GOH projects improved Plans, manuals, systems, procedures and tools related to CFET institutional capacity development plan. • Assess needs and procure technical assistance, training, complementary software packages, and other logistic support to build CFET’s capacity based on the needs outlined its institutional capacity development plan. STAGES/ACTIVITIES COMPLETED From 2012 – 2017, the IP received and processed USAID-validated GOH requests to hire 92 TAs. According to its final report, over five years, KONEKTE placed 45 TAs (38 men and seven women) with GOH counterparts. KONEKTE seconded TAs to 11 GOH counterparts including the presidency, the Prime Minister’s office, ministries, secretaries of state, and autonomous organizations. As part of the recruitment process, the IP helped to refine the scope of work (SOW) and candidate profiles that were drafted by the GOH institution, advertised the positions, screened applicants, supported GOH counterparts to evaluate finalists and select the final candidate, negotiated salaries, and issued a contract. After hiring TAs, the IP also, procured basic equipment and logistical support and monitored TA progress against the assignment SOW. 8 | KONEKTE PERFORMANCE EVALUATION USAID.GOV Figure 2: Number of TAs by Institution 6 From September 2015 through October 2017, KONEKTE added interventions to strengthen OMRH’s institutional capacity to promote improvements in public sector HR management systems. In this phase, KONEKTE planned to strengthen OMRH’s capacity to conduct public administration performance assessments, monitor and evaluate State reform initiatives, update and deploy an existing computerized HR management system, and create an online public sector job portal. KONEKTE carried out activities in all planned intervention areas. USAID cut short work on the public administration performance assessment in 2016. KONEKTE did not complete planned activities related to the HR management system or finish the job portal before the Activity closed. In the context of USAID Forward, 7 KONEKTE’s second phase, which started in 2015, also set out a strategy and implemented activities to develop OMRH capacity to manage government-to- government contracts. USAID later cancelled the planned interventions for developing OMRH’s contract management capacity in the Activity’s final contract amendment in March 2016. In addition, phase two planned and delivered activities to strengthen the institutional capacity of MSI’s local counterpart, CFET, with the aim of preparing the organization to counterpart directly with USAID and/or to provide recruitment services to the GOH following KONEKTE’s completion. CFET is a Haitian consulting company that offers HR outsourcing and capacity building services to national and international organizations and businesses with activities in Haiti. KONEKTE concluded activities on October 31, 2017. EVALUATION PURPOSE AND AUDIENCE The objective of the KONEKTE final evaluation was to assess the extent to which the Activity achieved intended results/outcomes, identify any lessons learned, and recommend possible follow-up actions as well as provide guidance for similar USAID programs in Haiti and elsewhere. Its primary audience is USAID/Haiti, which may use the evaluation findings and recommendations to inform the Mission’s future programming in the areas of governance and institutional strengthening. The secondary audiences are MSI, MSI’s local subcontractor, CFET, and KONEKTE’s main GOH counterpart, the OMRH, which may also consider 6 USAID/Haiti. KONEKTE Program Year 5 Annual Progress Report. N.p.: USAID, 2017. Print. 7 The USAID Forward reform initiative ran from 2010-2016. One of its pillars which guided the KONEKTE strategy was to “Promote sustainable development through high-impact partnerships and local solution.” In particular, phase two of the Activity sought to build the capacity of OMRH of CFID, a local organization, to obtain and manage contracts directly with the USG. Under the direct authority of the Prime Minister’s office, OMRH is mandated to oversee public sector reforms to modernize and standardize HR systems and practices across the GOH. It is likewise changed with overseeing government devolution and decentralization initiatives. 0 1 1 1 1 2 3 5 6 7 18 Bureau du Ministre à la Sécurité Energétique Ministère de la Santé Publique et de la Population Bureau du Secrétaire d'Etat à l'intégration des Personnes Handicapées Primature Presidency Ministry of Agriculture, Natural Resources and Rural Development Ministry of Social Affairs and Labor Direction de la Protection Civile Ministère de l'Economie et des Finances Ministry of National Education and Vocational Training Office of Management and Human Resources 9 | KONEKTE PERFORMANCE EVALUATION USAID.GOV evaluation findings and recommendations in their current programs and possible future collaboration with USAID/Haiti. EVALUATION QUESTIONS The final evaluation answers the questions below. When answering these evaluation questions (EQs), the ET considered stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent was the original project design relevant to the GOH concerns and needs? In answering this question, the evaluation focused particularly on the extent to which the Activity design facilitated empowering GOH officials and offices to project their own HR needs and identify their own technical experts. The evaluation examined these issues comparing GOH, implementing partner/counterpart (IP), and USAID perspectives, and how they evolved throughout the life of the Activity. 2. How effectively was the Activity implemented and managed? In answering this question, the evaluation assessed two fundamental Activity elements: a. The extent to which Activity interventions actively build GOH capacity to identify and hire appropriate technical assistance. b. The extent to which technical assistance provided through the Activity transferred knowledge and capacity to GOH officials and offices. To answer this question, the evaluation examined the interaction between USAID technical offices, the Contracting Officer’s Representative (COR), and KONEKTE. Specifically, the ET assessed how effectively the Activity managed GOH requests for technical assistance to ensure that it was appropriate and effectively used, including the extent to which the GOH was involved in the identification of the HR needs and the selection of appropriate technical experts to fill identified gaps. 3. How sustainable are the Activity’s interventions? To answer to this question, the evaluation specifically addressed the implications of USAID’s decision not to extend the Activity beyond five years, the impact of this decision on USAID’s relationship with the GOH, and the GOH’s willingness and ability to continue the Activity’s interventions in the absence of USAID support. II. EVALUATION DESIGN SAMPLING The ET collected data on the perspectives and experiences of key stakeholders involved in the KONEKTE Activity using a purposeful sampling technique (see list of persons consulted in Annex E ). The team sampled representatives from all key stakeholder groups: USAID, MSI and CFET, TAs, GOH counterparts, and other donors or USAID Activity implementor representatives who collaborated with KONEKTE. Within each stakeholder group, the ET selected stakeholders from different subcategories: • USAID COR, as well as technical office representatives 8 , • IP personnel from (a) the prime and subcontractor organizations, (b) headquarters (HQ) and Haiti-based personnel, and (c) Chiefs of Party (COP) and other program staff, • TAs who had (a) served for various lengths of time, (b) had varying degrees of seniority, (c) were men and women, and (d) who had worked with a variety of GOH counterparts, 10 | KONEKTE PERFORMANCE EVALUATION USAID.GOV • GOH counterparts from among the various government offices that collaborated with KONEKTE, both senior leaders and TA technical interfaces, • Different donor organizations. In cases where several different individuals held the same position, the ET endeavored to interview as many individuals as possible who held a given position to collect as many viewpoints as possible and limit bias. For example, the ET interviewed two out of three MSI COPs, three out of five USAID CORs, five MSI HQ based backstopping personnel, 17 out of 38 technical advisors, etc. Purposeful sampling was supplemented by snowball sampling to enable the ET to identify additional key informants and fill gaps in its initial list of key stakeholders, which was provided by MSI. For example, the initial list did not include a list of all USAID CORs, a complete list MSI technical and administrative backstopping personnel, USAID technical office representatives, or CFET client relations managers. During initial interviews, MSI, CFET and USAID provided addition names and contact information for missing individuals. Overall, the final informant sampling allowed the ET to compare the perspectives of different key stakeholder groups and cross-check and fill gaps in KONEKTE reporting. Although more men than women participated in key informant interviews (KIIs), the proportion was a fair reflection of the roles played by men and women in KONEKTE. DATA COLLECTION METHODS The ET used a qualitative design, consisting of several different data collection methodologies, including document review, analysis of Activity monitoring and evaluation data, KIIs, focus group discussions (FGDs) and direct observation of KONEKTE-supported activities. The ET reviewed Activity documents and produced an inception report in June 2018. The team conducted interviews, FGDs, and direct observation during the evaluation fieldwork period, from July 31, 2018 – August 29, 2018. The ET developed data collection protocols to guide fieldwork, which are found in Annex F . The purpose of the protocols was to: (1) ensure all key issues are covered during data collection, (2) elicit rich, sometimes unanticipated, information from respondents, (3) help organize information in a form that could be usefully and efficiently analyzed, and (4) ensure that sex-disaggregated data were collected. The protocols consisted of questions that address and derive from the EQs, as well as from the ET’s document review, its initial discussions with USAID, and its evaluation design knowledge. DOCUMENT REVIEW The ET reviewed KONEKTE documents to understand the Activity design and implementation, extract findings relevant to the EQs, and inform data collection protocol development so that instruments appropriately supplemented background documents information. Categories of documents are listed below and detailed in Annex B : • KONEKTE annual reports • KONEKTE annual work plans • KONEKTE monitoring and evaluation (M&E) plans • Activity descriptions and modifications • Journals and other publications KEY INFORMANT INTERVIEWS The ET interviewed 52 individuals (18 women and 34 men) from various stakeholder groups. The team interviewed nearly all informants individually, except for one group interview of three informants. Findings from KIIs contributed to the ET’s responses to all EQs. The ET conducted face-to-face interviews in the Haitian capital, Port-au-Prince, or via phone, or Skype. Table 2 shows the distribution of interviews by key stakeholder group and gender. 11 | KONEKTE PERFORMANCE EVALUATION USAID.GOV Each KII was guided by an interview protocol adjusted for different types of respondents (See Annex F for protocols), and was conducted in French or English. Table 2: Key Informant Interviews by Stakeholder Group and Gender FOCUS GROUP DISCUSSIONS The ET organized one FGD with HR Directors from various GOH institutions. The ET held the discussion at the end of a regular meeting organized by OMRH. The discussion was brief, but allowed the team to initiate contact with members of the group. The ET followed up and organized KIIs with four group members to supplement FGD information on current practices and challenges facing GOH HR Managers. Table 3: Focus Group Discussions by Stakeholder Group and Gender STAKEHOLDER TOTAL FEMALE MALE GOH HR directors 12 8 4 DIRECT OBSERVATION The ET triangulated interview data with direct observation when possible and relevant. It observed whether or not the KONEKTE-supported biometric time and attendance keeping system was still functional , if the web applications developed for OMRH (the job portal, the website) were online, and participated in a meeting organized by OMRH with HR Directors from various ministries to observe how OMRH was following up on HR management capacity building activities carried out by KONEKTE TAs. ANALYSIS ET took detailed notes of all KIIs and FGDs, and cleaned and shared electronic summaries on a rolling basis throughout fieldwork. Team members conducted internal debriefs regularly throughout fieldwork to discuss progress and make adjustments that were needed in the evaluation schedule. The team also discussed evidence collected to help answer the EQs, as well as to identify discrepancies. During the debriefs the team also developed and updated a findings, conclusions and recommendations matrix, focused particularly on findings. This served as a basis for identifying key themes that emerged from the data collection. The team then developed a coding framework called a tally sheet, in which each set of notes, organized by stakeholder group, KI and sex, was coded using the identified set of themes. The tally sheet is a matrix that allows for identification of dominant themes. It also easily facilitates comparative analysis both within and across stakeholder groups and by sex. The ET employed several data analysis methods to identify key findings from the collected data, as well as to draw conclusions and make recommendations for Activity follow-up or future potential USAID programming like KONEKTE. 1. Content Analysis – The ET reviewed and coded KII and FGD data to identify and highlight notable examples of KONEKTE successes (or lack of success) that contributed to (or inhibited) achievement of its objectives. 2. Trend Analysis – The team examined KONEKTE M&E data on its indicators over time to identify trends in performance relative to its Activity targets. STAKEHOLDER INTERVIEWS FEMALES MALES Total 5 2 1 8 34 USAID 5 3 2 Implementing Counterparts 14 10 4 Technical Advisors 17 3 14 GOH Counterparts 14 2 12 Other Donors/USAID Activity 2 0 2 12 | KONEKTE PERFORMANCE EVALUATION USAID.GOV 3. Gap Analysis – The team analyzed where KONEKTE fell short of anticipated performance targets, and the likely factors contributing to these gaps. 4. Comparative Analysis – The ET compared different stakeholder groups perspectives to assess either convergence or divergence. 5. Gender Analysis – All data collected through KIIs and FGDs were disaggregated by sex and analyzed for effects on both male and female beneficiaries to show any significant differences. The Team Leader captured preliminary findings, conclusions, and recommendations in an Excel-based matrix that categorizes analysis by EQ. The matrix: a) ensures the ET prepares a systematic and thorough response to each EQ, b) verifies that preliminary analysis accounts for gender and social dimensions, c) identifies any gaps where additional clarification or analysis may be necessary, and d) serves as the basis for developing the evaluation report. LIMITATIONS AND MITIGATION STRATEGIES BIAS The ET was aware of several bias risks in data collection and analysis and developed mitigation strategies to overcome these: 1. Recall Bias : Recall bias is a common challenge in final evaluations, especially evaluations that take place months after an Activity’s close. KONEKTE recipients might not have remembered accurately or responded to the ET’s questions with answers that blended their experiences into a composite memory if, for example, they had received technical assistance from multiple sources. The ET compared stakeholder responses, for example what the TA recalled as compared to the GOH counterpart that hosted the TA, to increase the validity of the evaluation findings. It also probed respondents’ answers to facilitate accurate recollections. 2. Response Bias : Response bias is the risk that key informants may have been motivated to provide responses that would be considered socially desirable or influential in obtaining donor support or, alternatively, to discredit the activity or a given individual involved in the activity. For example, a GOH counterpart may have provided positive remarks about KONEKTE because s/he would like his/her institution to receive similar support in the future or because s/he wanted to please the interviewer. A KI may have provided negative feedback because s/he was dissatisfied with one or more persons involved in activity management. ET mitigated this risk by interviewing a wide variety of informants within and across stakeholder categories. In its analysis it prepared a tally sheet that allowed the ET to observe convergence of points by view and outliers by stakeholder group and question sub-finding. The ET also began each KII and FGD with a short explanation of the purpose of the evaluation, highlighting that the team hoped to learn about both successful and unsuccessful strategies, achievements and shortfalls, as relevant, to draw out lessons learned that could benefit future Activities. The ET also highlighted it would keep interview content confidential and omit personally identifiable information from the report (see informed consent protocol in Annex F). 3. Selection Bias : Selection bias is an inherent risk when implementers help to facilitate contact with Activity counterparts. There was a risk that the ET would be oriented to stakeholders with primarily positive views of the Activity. To mitigate the risk of selection bias, the ET identified the individuals to be interviewed using lists provided by MSI and supplemented using the snow ball method to fill in gaps (missing stakeholder categories, missing individuals who held a key position at some time during the evaluation period, missing contact information for some individuals in various stakeholder categories) with help from IP, USAID, and GOH KI. To mitigate the potential bias that KI would refer the ET to others sharing similar viewpoints, the ET accepted referrals from several different sources and used the strategies described above to mitigate for response bias. The ET arranged meetings directly without going through the implementer. 13 | KONEKTE PERFORMANCE EVALUATION USAID.GOV 4. Gender Bias : Gender bias is a risk because most individuals have a subconscious sense of appropriate roles and behavior for women and men. The ET reviewed gender-sensitive approaches during the inception period by taking and discussing USAID’s Gender 101 training, focusing dis