Evalyasyon Pèfòmans Pwogram Ogmantasyon Mendèv KONEKTE

DISCLAIMER- The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Photo Credit: PERFORMANCE EVALUATION OF WORKFORCE AUGMENTATION PROGRAM KONEKTE EVALUATION REPORT NOVEMBER 2018 NATIONAL PALACE OF HAITI PHOTO CREDIT: DRGULAS, 2009 PERFORMANCE EVALUATION OF WORKFORCE AUGMENTATION KONesans E Konpetans TEknik (“KONEKTE”) EVALUATION REPORT November 2018 Evaluation Mechanism Number: AID-521-C-17-00002 Haiti Evaluation and Survey Services for USAID/Haiti ABSTRACT Social Impact’s Haiti Evaluation and Survey Services project conduct ed an independent final evaluation of the United States Agency for International Development (USAID)/Haiti’s Workforce Augmentation Program KONesans E Konpetans TEknik (KONEKTE) Activity implemented by Management Systems International. KONEKTE was designed to fill workforce vacancies and provide Technical Advisors (TAs) to select G overnment of H aiti (GOH) counterparts to support the United States Government’s post- earthquake strategy. The evaluation report assesses whether KONEKTE achieved intended results, identifies lessons learned, and provides guidance for future USAID/Haiti programs. KONEKTE placed 45 TAs with 11 GOH counterparts. However, recruitment was slower than planned. Changes in USAID, Implementing Partner and GOH counterpart leadership, challenges aligning multiple key stakeholders’ vision and needs and external social, political and economic factors affected efficiency. Although USAID largely determined priority intervention areas, KONE KTE’s GOH counterparts chose and managed their own TAs, which made the Activity relevant to counterpart concerns and needs. GOH officials indicated that KONEKTE strengthened their voice in day- to-day decision-making within USAID programs. KONEKTE technical expertise and tools contributed to on- going improvements in GOH recruitment and human resource (HR) management systems. However, KONEKTE did not make significant progress building GOH capacity to assess HR needs systematically. In addition, KONEKTE was not very effective at transferring knowhow to GOH counterparts due to their limited absorptive capacity, TAs’ focus on doing versus transferring skills, and weak oversight to promote TA deliverables’ use. Some KONEKTE GOH counterparts are sustaining KONEKTE TAs work, while many others are not. Likely sustainable KONEKTE interventions include introducing new laws supporting on-going reforms and creating tools and guidelines to support offices and institutions’ core activities. TABLE OF CONTENTS ABSTRACT II TABLE OF CONTENTS III LIST OF TABLES AND FIGURES IV ACRONYMS V EXECUTIVE SUMMARY I I. INTRODUCTION 5 Project Background 5 Theory of Change 6 Activities/Components 6 Stages/Activities Completed 7 Evaluation Purpose and Audience 8 Evaluation Questions 9 II. EVALUATION DESIGN 9 Sampling 9 Data Collection Methods 10 Analysis 11 Limitations and Mitigation Strategies 12 III. FINDINGS AND CONCLUS IONS 13 EQ1: To what extent was the original project design relevant to the GoH concerns and needs? 13 EQ2: How effectively was the activity implemented and managed? 16 EQ2A. The extent to which activity interventions actively built GOH capacity to identify and hire appropriate technical assistance 19 EQ2B. The extent to which technical assistance provided through the activity transferred knowledge and capacity to GOH officials and offices 24 EQ3: How sustainable are the activity’s interventions? 27 IV. RECOMMENDATIONS FOR USAID 31 ANNEX A: EVALUATION STATEMENT OF WORK 32 ANNEX B: REFERENCES AND ENDNOTES 38 ANNEX C: EVALUATION TEAM MEMBERS 40 ANNEX D: EVALUATION DESIGN MATRIX 42 ANNEX E: LIST OF PERSONS INTERVIEWED 45 ANNEX F: INFORMED CO NSENT SCRIPTS & DATA COLLECTION PROTOCOL S 48 ANNEX G: DISCLOSURE OF CONFLICT OF INTER EST 63 LIST OF TABLES AND FIGURES Table 1: KONEKTE Sub-purposes, Outputs, and Activities 7 Table 2: Key Informant Interviews by Stakeholder Group and Gender 11 Table 3: Focus Group Discussions by Stakeholder Group and Gender 11 Table 4: KONEKTE: Timing of Changes in Key Stakeholder Institutions’ Leadership 18 Figure 1: Four Pillars of the USG’s Post-Earthquake Strategy for Renewal and Economic Opportunity in Haiti 6 Figure 2: Number of TAs by Institution 8 Figure 3: Steps in KONEKTE Recruitment Process 20 ACRONYMS CFET CG COP Centre de Formation et d’Encadrement Technique Coordinator General Chief of Party COR CRM D&G DIS Contracting Officer’s Representative Client Relations Manager Democracy and Governance Department of Information Systems DRH Directeurs et Directrices des Ressources Humaines EPAP Public Administration Performance Evaluation ESS Evaluation and Survey Services ET Evaluation Team EQ FGD Evaluation Question Focus Group Discussion FY Fiscal Year GOH HICD Government of Haiti Human and Institutional Capacity Development HR HRD HRMIS HQ IFMS Human Resources Human Resources Department Human Resources Management Information System Headquarters Integrated Financial Management System IP Implementing Partner/Counterpart IT KII Information Technology Key Informant Interview KONEKTE KONesans E Konpetans TEknik M&E MARNDR MAST MENFP Monitoring and Evaluation Ministry of Agriculture, Natural Resources and Rural Development Ministry of Social Affairs and Labor Ministry of National Education and Vocational Training MOEF Ministry of Economy and Finance MSI Management Systems International NGO Nongovernmental Organization OMRH Office of Management and Human Resources SI Social Impact, Inc. SOW Scope of Work TA TL Technical Advisor Team Leader USAID United States Agency for International Development USG United States Government i| KONEKTE PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) project design and conduct an independent final evaluation of the Workforce Augmentation Program KONesans E Konpetans TEknik (KONEKTE) implemented by Management System International (MSI) under Contract No. AID-521-C-13-00001. The purpose of this report is to present the e valuation team’s (ET) main findings, conclusions , and recommendations. EVALUATION PURPOSE A ND EVALUATION QUESTI ONS The objective of the KONEKTE final evaluation is to assess the extent to which the Activity achieved intended results/outcomes, identify any lessons learned, and recommend possible follow-up actions, as well as provide guidance for similar USAID programs in Haiti and elsewhere. Its primary audience is USAID/Haiti, which may use the evaluation findings and recommendations to inform the Mission’s future programming in the areas of governance and institutional strengthening. The secondary audiences are MSI, MSI’s local subcontractor, Centre de Formation et d’Encadrement Technique (CFET), and KONEKTE’s main Government of Haiti (GOH) counterpart, the Office of Management and Human Resources (OMRH), which may also consider evaluation findings and recommendations in their current programs and possible future collaboration with USAID/Haiti. This final evaluation will answer the evaluation questions (EQs) below. When answering these EQs, the ET will consider stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent was the original project design relevant to the GOH concerns and needs? 2. How effectively was the Activity implemented and managed? a. In answering this question, the evaluation assessed two fundamental Activity elements: b. The extent to which Activity interventions actively build GOH capacity to identify and hire appropriate technical assistance. c. The extent to which technical assistance provided through the Activity transferred knowledge and capacity to GOH officials and offices. 3. How sustainable are the Activity’s interventions? PROJECT BACKGROUND On October 30, 2012, USAID awarded the Haiti “Workforce Augmentation Program” contract, known as “KONEKTE” to MSI and its local counterpart , CFET, with a ceiling price of $21,795,809. KONEKTE was designed to fill workforce vacancies and provide technical expertise to select USAID GOH counterparts in support of the post-earthquake United States Government (USG) Haiti strategy. The Activity was intended to contribute to achieving USAID objectives under the strategy's four pillars: Infrastructure and Energy, Food and Economic Security, Health and Other Basic Services, and Governance and Rule of Law. Beginning in November 2015, KONEKTE expanded its strategy to include interventions to strengthen OMRH institutional capacity to lead human resource (HR) management-related reforms within the Haitian government’s civil service. EVALUATION DESIGN, METHODS, AND LIMITATIONS The ET used a qualitative evaluation design, consisting of document review, key informant interviews (KIIs), focus group discussions (FGDs) and direct observation. During three weeks of in-country data collection, the ET interviewed 52 individuals (18 women and 34 men) from various stakeholder groups. The ET also organized one FGD with GOH Human Resource Department (HRD) personnel with 12 ii| KONEKTE PERFORMANCE EVALUATION USAID.GOV participants (eight female, four male). The team used purposeful sampling to select informants who were well-informed about the Activity and represented diverse stakeholder groups including representatives of USAID, MSI and CFET , OMRH and other GOH counterparts, technical advisors (TAs) , other donors, and other USAID Activities. The ET was aware of potential recall, selection, response, and gender bias risks in data collection and analysis, and developed mitigation strategies to overcome these. To overcome bias and increase the validity of evaluation findings, the team asked similar questions to all informants and triangulated responses. The team also prefaced all interviews and group interviews with a protocol highlighting the evaluation purpose and the confidentiality of interviews. Although some key stakeholders were no longer in Haiti, the ET was able to interview most by telephone or Skype. The ET obtained interviews with individuals from all stakeholder categories , but interviewed fewer USAID technical office and OMRH officials than planned. FINDINGS AND CONCLUSIONS EQ1: TO WHAT EXTENT WAS THE ORIGINAL PROJECT DESIGN RELEVANT TO THE GOH CONCERNS AND NEEDS? Based on Activity reports and KIIs , KONEKTE’s design enabled GOH officials and offices to choose and manage their own technical assistants. This made KONEKTE relevant to its main GOH counterparts’ concerns and needs. The Activity administered client satisfaction surveys where high marks were received and feedback from GOH officials the ET interviewed indicate overall high levels of GOH counterpart satisfaction with KONEKTE-supported TAs. KONEKTE filled gaps in both strategic and day-to-day needs of GOH counterparts. In theory, KONEKTE assistance started with GOH counterpart requests. In practice, USAID technical offices oriented GOH institutions’ requests to intervention areas that aligned with its priorities, limiting the GOH counterpart s’ opportunity to set their own priorities for KONEKTE assistance. USAID’s Democracy and G overnance office allowed the 2012 – 2016 OMRH leadership a greater role in setting KONEKTE’s agenda within t his office than did other USAID technical offices, which appeared to use KONEKTE to reinforce existing programs, narrowing KONEKTE’s scope to pre-determined intervention areas. Based on GOH officials’ accounts, KONEKTE strengthened GOH counterpart s’ voice in day-to-day decision-making within USAID programs. Embedding senior TAs in GOH offices strengthened GOH dialogue with USAID, and in some cases, GOH oversight of USAID implementor interventions. This was not an explicit objective of KONEKTE, but is a highly positive, unintended consequence of placing TAs in USAID GOH counterpart offices. EQ2: HOW EFFECTIVELY WAS THE ACTIVITY IMPLEMENTED AND MANAGED? KONEKTE's implementation was not efficient during its first 12 to 18 months, and again at the Activity’s end. Although, after a slow start, it recruited approximately the targeted number of TAs, KONEKTE was significantly less efficient than planned in terms of the number of days required to field a TA after receiving a validated scope of work (SOW ) from USAID. The causes of inefficiency were related to variety factors, some within the control of USAID and the IP and others not. USAID was slow issuing written approvals of implementing partner (IP) work and monitoring and evaluation (M&E) plans as well as modifying the contract to reflect changes in the A ctivity’s scope, which resulted in over-dependence on oral agreements and created management uncertainty at times. Fre quent changes among decision- makers within USAID, GOH counterpart s, and the IP, also negatively affected KONEKTE's overall effectiveness. During its five-year implementation period, KONEKTE had three Chiefs of Party (COPs), four USAID Contracting Office Representatives (CORs), three OMRH Coordinator Generals (CGs), and experienced several changes in government. USAID and IP internal management factors such as lack iii| KONEKTE PERFORMANCE EVALUATION USAID.GOV of shared vision and cohesive management, too many decision-makers, poorly-defined and communicated roles and responsibilities, as well as unclear conditions for receiving KONEKTE assistance in the early implementation stages (USAID and IP) likewise also diminished implementation efficiency. EQ2A. THE EXTENT TO WHICH ACTIVITY INTERVENTIONS ACTIVELY BUILD GOH CAPACITY TO IDENTIFY AND HIRE APPROPRIATE TECHNICAL ASSISTANCE KONEKTE contributed to building GOH counterparts’ capacity to identify and hire appropriate technical assistance. KONEKTE used an open and transparent recruitment process in which GOH counterparts were actively engaged, potentially contributing to the GOH personnel’s awareness and hands-on experience applying good recruitment practices. MSI and USAID showed vision and adaptive capacity in re-framing KONEKTE for OMRH institutional capacity building, the GOH entity mandated to oversee public sector HR reforms. Although its contributions ended up being more modest than planned, KONEKTE supplied specialized technical assistance and recruitment and HR management tools that shaped OMRH intervention strategies and affirmed OMRH leadership of ongoing improvements in GOH recruitment and other HR management practices. KONEKTE did not make significant progress building GOH counterpart s’ capacity to use systematic approaches to identify HR needs such as the Human and Institutional Capacity Development (HICD) framework, which was cancelled by USAID before planned implementation was complete. A specific focus on gender equity was not explicitly required in KONEKTE’s SOW. IP team members indicated that the Activity made efforts to ensure woman candidates for TA positions were proposed when possible, but did not require GOH counterparts, who were responsible for selecting TAs, to have a policy of “affirmative action.” According to KONEKTE Activity reports, out of a total of 45 TAs, seven were female (16 percent). Capacity- building for HRD personnel remains a significant need in this area as well as other HR management areas. EQ2B. THE EXTENT TO WHICH TECHNICAL ASSISTANCE PROVIDED THROUGH THE ACTIVITY TRANSFERRED KNOWLEDGE AND CAPAC ITY TO GOH OFFICIALS AND OFFICES Overall, KONEKTE was not very effective at transferring capacity and knowledge to GOH counterparts. KONEKTE TAs carried out a wide variety of activities and produced most planned deliverables to the satisfaction of GOH counterparts, especially during its first three years of implementation. By their nature, many of these deliverables (guides, manuals, plans, training modules, etc.), could have been used to transfer capacity and knowledge, but due to weak GOH ownership of some TA work, these outcomes were limited. Getting OMRH buy-in for KONEKTE work done in collaboration with the original CG was especially problematic after his departure. There were, however, exceptions including KONEKTE’s work with the Ministry of National Education and Vocational Training (MENFP) on improving the teaching of reading, and with OMRH and HRD s on recruitment and HR management (as highlighted in EQ2a findings). KONEKTE TAs faced significant external obstacles to transferring capacity and knowledge to its GOH counterparts, but also made tactical errors. The limited absorptive capacity of GOH coun terparts was a significant factor preventing KONEKTE assistance from contributing to large-scale institutional capacity improvements. However, insufficient support for TAs on assessing needs and coaching, and weak oversight by IP and USAID managers to promote the use of TAs’ deliverables also limited KONEKTE’s effectiveness at transferring knowledge and capacity. EQ3: HOW SUSTAINABLE ARE THE ACTIVITY’S INTERVENTIONS? Some KONEKTE GOH counterpart institutions are currently able to sustain some work started by iv| KONEKTE PERFORMANCE EVALUATION USAID.GOV KONEKTE TAs while many others are not. Institutional restructuring/strengthening work within OMRH, MENFP, and Ministry of Agriculture, Natural Resources, and Rural Development’s (MARNRD) Department of Innovation contributed to organizing the institutions’ work and attracting government and other donor resources to support their activities. The positive outcomes of TA work in most other GOH institutions appears to have been short-term and did not lead to any lasting improvement in the host institutions’ capacity. Frequent changes in Haiti’s political landscape, overall weak governance capacity, and severely under-resourced government offices were significant obstacles to sustainability. KONEKTE TA interventions that are likely to have sustainable outcomes include the changes introduced in the legal framework supporting on-going reforms, and creating tools and guidelines to be used by offices and institutions for their core activities. KONEKTE’s experiment recruiting OMRH TAs on the government salary scale to fill middle management vacancies conditioned by their recruitment into the civil service after one year was a mixed success. OMRH retained two of four TAs recruited in this way, but none have stayed with OMRH post-Activity. KONEKTE TAs who integrated into the civil service were moved to other host institutions and expressed disappointment in the opportunities afforded them to contribute reforms outside the context of KONEKTE. There appears to be political will within the GOH to continue KONEKTE-supported HR reforms, especially with current OMRH leadership. All GOH counterpart s expressed willingness to collaborate with USAID in future programs. RECOMMENDATIONS FOR USAID 1. USAID should not use a stand-alone, HR out-sourcing mechanism like KONEKTE to fill gaps in its counterparts’ capacity in Haiti in the future. KONEKTE showed that such a mechanism is not a useful way to streamline assistance to fill critical HR vacancies in an emergency recovery situation or more generally. The mechanism did not appear to help to overcome administrative and management burdens within USAID nor to save time. 2. USAID should continue to involve GOH counterparts in the implementation of its future programs. Ways to do this may include seconding TAs in GOH counterpart offices through existing technical Activities (versus through a stand- alone mechanism), as well as other institutional capacity building interventions. KONEKTE demonstrated that it is possible to involve GOH counterparts as implementers in USAID programs, in ways that enhance interventions’ sustainability. Even though KONEKTE showed there are significant obstacles to sustainably increasing GOH offices and officers’ capacity, not involving the GOH only reinforces weak governance. 3. USAID should consider additional support for OMRH on civil service administrative reforms. Among other topics, future programs should identify interventions to promote nondiscrimination in hiring. Future programs should explore work on policy and regulatory issues that affect the GOH ability to attract qualified candidates such as wage and professional advancement programs. 4. USAID should ensure that the future program agendas that cross-cut USAID technical offices are led/monitored by senior USAID staff, if not the Mission Director, to ensure coherence and buy-in from mission technical offices. To strengthen oversight and decision- making by technical offices, USAID should avoid using basket funding mechanisms except in situations where there is strong demand and similar objectives and needs across technical offices. 5. USAID should approve workplans, M&E frameworks, and process contract modifications in a timely manner to guide consistent decision-making by implementers, and avoid implementation preceding formal, written approvals. 5| KONEKTE PERFORMANCE EVALUATION USAID.GOV I. INTRODUCTION The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) project design and conduct an independent final evaluation of the Workforce Augmentation Program, KONesans E Konpetans TEknik (KONEKTE), implemented by Management System International (MSI) (see Annex A for the evaluation Statement of W ork). 1 Prior to conducting the fieldwork for this evaluation, the e valuation team (ET) presented its initial findings based on a document review (see Annex B for list of documents consulted and Annex C for profiles of the ET members) in an inception report. The ET presented its evaluation design, a detailed report approved by USAID/Haiti prior to starting fieldwork (see summary of evaluation design in the Evaluation Design Matrix found in Annex D ). The purpose of this report is to present the ET’s main findings, conclusions, and recommendations. PROJECT BACKGROUND The 2010 earthquake had disastrous effects on people’s lives and national infrastructure, and exacerbated the pre-earthquake development challenges of poor governance, weak institutions, and widespread corruption. United States Government (USG) assistance for re-building was guided by the Government of Haiti’s (GOH) Action Plan for National Recovery and Development and its subsequent Strategic Development Plan. 2 On October 30, 2012, USAID awarded the Haiti “Workforce Augmentation Program” contract, known as “KONEKTE” to MSI and its local counterpart Centre de Formation et d’Encadrement Technique (CFET), with a ceiling price of $21,795,809. KONEKTE was designed to fill workforce vacancies and provide technical expertise to select USAID GOH counterparts in support of the post-earthquake USG Haiti s trategy. 3 The Activity was intended to contribute to achieving USAID objectives under the strategy's four pillars: 1 Throughout the report, “KONEKTE” is used interchangeably with “the Activity.” For example, a “KONEKTE technical advisor (TA)” is an advisor recruited through the KONEKTE Activity. “KONEKTE was effective…” Refers to the outcome of the Activity which includes the outcomes of both external and internal management factors. Specific references to the actions and outcomes attributed to implementing partners (IP) or USAID are clearly labeled. 2 Post-Earthquake USG Haiti Strategy: Toward Renewal and Economic Opportunity, https://reliefweb.int/report/haiti/post-earthquake-usg-haiti-strategy- toward-renewal-and-economic-opportunity accessed 8/29/18. 3 The Activity scope of work included the possibility of hiring TAs for private sector and nongovernmental organizations (NGOs) as well as GOH partners, but USAID decided early on to limit KONEKTE’s focus to GOH partners. Box 1. Program Overview Workforce Augmentation Program, KONEKTE (KONesans E Konpetans TEknik) Program Objective: To fill human resource gaps and provide technical expertise to select Haiti government partners in support of the “Post- Earthquake USG Haiti Strategy: Toward Renewal and Economic Opportunity.” Period of Performance: November 2012 – October 2017 Award Number: AID-521-C-13-00001 Total Funding Amount: $21,795,809 Implementing Partners: Management Systems International (MSI) and the Centre de Formation et d'Encadrement Technique (CFET) Key Partner: Government of Haiti’s Office de Management et des Ressources Humaines (OMRH), Geographic Presence: Port-au-Prince, Haiti Figurel:FourPillarsoftheUSG'sPost-EarthquakeStrategyforRenewalandEconomicOpportunityinHaiti4 InfrastructureandEnergy: Intermediate ResultI: "ImprovedInfrastructurethat SupportsCommunity and CommercialDevelopment" PILLAR I Health andOther BasicServices: Intermediate Result3: "ImprovedHealthStatusand LearningEnvironment" PILLAR 3 THEORY OFCHANGE FoodandEconomicSecurity: Intermediate Result2: "Increased FoodandEconomic Security" PILLAR 2 Governance andRuleofLaw: Intermediate Result4: "MoreResponsiveGovernance andImprovedRuleOfLaw" PILLAR 4 KONEKTE'smaindevelopmenthypothesis5was"ifhighlyqualifiedtechnicaladvisors(TAs)aresecondedto selectGOHministriesandagencies,thenGOHcapacitytosupportandsustainthepositiveoutcomesofthe USG'spost-earthquakeassistancewillbeenhanced. BeginninginNovember2015,KONEKTE expandeditsstrategytoincludeinterventionstostrengthenthe institutionalcapacityoftheOfficeforHumanResourceManagement(OMRH)toleadHumanResource(HR) management-relatedreformswithintheHaitiangovernment'scivilservice.ThispartofKONEKTE's interventionstrategywasbasedontheassumptionsthatifGOHinstitutionspossessthecapacitytocarry-out fair,transparent,andcompetitiverecruitmentprocesses,anddevelopandapplybetterHRpoliciestomanage civilservants,thecivilserviceperformancewillimproveandcontributetotheGOHreachingitsdevelopment andreformobjectives. ACTIVITIES/COMPONENTS KONEKTE'soverallpurposewastofillgapsinboththeskillsandnumbersofworkersinselectedHaitian governmentinstitutionssothattheseGOHcounterpartshadmorecapacitytocarryouttheirmandated activities.ThiswastobeaccomplishedbyhiringTAsandsecondingthemtoUSAIDGOHcounterpart institutions.KONEKTE wasalsochargedwithsupplyingtheessentialtoolsTAsneededtodotheirjob (computers,internetconnections,andlogisticssupport). StartinginSeptember2015,KONEKTE establishedfoursub-purposes(oneofwhichUSAIDcancelledinMarch 2016),eachwithspecificoutputsandsetsofactivitiesassummarizedinTableIbelow: 4USAID/Haiti.KONEKTE ProgramYear5AnnualProgressReport.N.p.:USAID,2017.Print. 5ProjectdocumentsdonotexplicitlystateKONEKTE'stheoryofchangeexceptinthedraft2016AMELP,whichwasnotapprovedbyUSAID. 61KONEKTE PERFORMANCE EVALUATION USAID.GOV 7| KONEKTE PERFORMANCE EVALUATION USAID.GOV Table 1: KONEKTE Sub- purposes, Outputs, and Activities SUB-PURPOSES AND ASSOCIATED OUTPUTS AND ACTIVITIES SUB-PURPOSE OUTPUTS MAIN ACTIVITIES Activity Sub-Purpose 1: TAs placed in select GOH entities efficiently Timely recruitment of appropriate, qualified TAs. • Establish and implement a competitive and transparent recruitment process to hire TAs and place them in requesting GOH counterpart institutions; • Support TA work place integration and monitor their work; • Manage TA remuneration, annual leave, and other benefits ; • Report on TA activities to USAID. Activity Sub-Purpose 2: HR Management in OMRH and selected GOH ministries improved OMRH approved TA deliverables. • Provide qualified long and short-term TAs to OMRH (as above); • Procure complementary expertise, materials, and logistic support for OMRH capacity building activities; • Implement “Job Portal” module of the Inter-Ministry System for HR Management. Activity Sub-Purpose 3: Contract management skills in OMRH improved Removed from scope as per Contract Modification 4. Activity Sub-Purpose 4: Local service provider capacity in managing USAID/GOH projects improved Plans, manuals, systems, procedures and tools related to CFET institutional capacity development plan. • Assess needs and procure technical assistance, training, complementary software packages, and other logistic support to build CFET’s capacity based on the needs outlined its institutional capacity development plan. STAGES/ACTIVITIES CO MPLETED From 2012 – 2017, the IP received and processed USAID-validated GOH requests to hire 92 TAs. According to its final report, over five years, KONEKTE placed 45 TAs (38 men and seven women) with GOH counterparts. KONEKTE seconded TAs to 11 GOH counterpart s including the presidency, the Prime Minister’s office, ministries, secretaries of state, and autonomous organizations. As part of the recruitment process, the IP helped to refine the scope of work (SOW) and candidate profiles that were drafted by the GOH institution, advertised the positions, screened applicants, supported GOH counterparts to evaluate finalists and select the final candidate, negotiated salaries, and issued a contract. After hiring TAs, the IP also, procured basic equipment and logistical support and monitored TA progress against the assignment SOW. 8| KONEKTE PERFORMANCE EVALUATION USAID.GOV Figure 2: Number of TAs by Institution 6 From September 2015 through October 2017, KONEKTE added interventions to strengthen OMRH’s institutional capacity to promote improvements in public sector HR management systems. In this phase, KONEKTE planned to strengthen OMRH’s capacity to conduct public administration performance assessments, monitor and evaluate State reform initiatives , update and deploy an existing computerized HR management system, and create an online public sector job portal. KONEKTE carried out activities in all planned intervention areas. USAID cut short work on the public administration performanc e assessment in 2016. KONEKTE did not complete planned activities related to the HR management system or finish the job portal before the Activity closed. In the context of USAID Forward, 7 KONEKTE’s second phase, which started in 2015, also set out a strategy and implemented activities to develop OMRH capacity to manage government-to- government contracts. USAID later cancelled the planned interventions for developing OMRH’s contract management capacity in the Activity’s final contract amendment in March 2016. In addition, phase two planned and delivered activities to strengthen the institutional capacity of MSI’s local counterpart, CFET, with the aim of preparing the organization to counterpart directly with USAID and/or to provide recruitment services to the GOH following KONEKTE’s completion. CFET is a Haitian consulting company that offers HR outsourcing and capacity building services to national and international organizations and businesses with activities in Haiti. KONEKTE concluded activities on October 31, 2017. EVALUATION PURPOSE AND AUDIENCE The objective of the KONEKTE final evaluation was to assess the extent to which the Activity achieved intended results/outcomes, identify any lessons learned, and recommend possible follow-up actions as well as provide guidance for similar USAID programs in Haiti and elsewhere. Its primary audience is USAID/Haiti, which may use the evaluation findings and recommendations to inform the Mission’s future programming in the areas of governance and institutional strengthening. The secondary audiences are MSI, MSI’s local subcontractor, CFET, and KONEKTE’s main GOH counterpart, the OMRH, which may also consider 6 USAID/Haiti. KONEKTE Program Year 5 Annual Progress Report. N.p.: USAID, 2017. Print. 7 The USAID Forward reform initiative ran from 2010-2016. One of its pillars which guided the KONEKTE strategy was to “Promote sustainable development through high-impact partnerships and local solution.” In particular, phase two of the Activity sought to build the capacity of OMRH of CFID, a local organization, to obtain and manage contracts directly with the USG. Under the direct authority of the Prime Minister’s office, OMRH is mandated to oversee public sector reforms to modernize and standardize HR systems and practices across the GOH. It is likewise changed with overseeing government devolution and decentralization initiatives. 0 1 1 1 1 2 3 5 6 7 18Bureau du Ministre à la Sécurité Energétique Ministère de la Santé Publique et de la Population Bureau du Secrétaire d'Etat à l'intégration des Personnes Handicapées Primature Presidency Ministry of Agriculture, Natural Resources and Rural Development Ministry of Social Affairs and Labor Direction de la Protection Civile Ministère de l'Economie et des Finances Ministry of National Education and Vocational Training Office of Management and Human Resources 9| KONEKTE PERFORMANCE EVALUATION USAID.GOV evaluation findings and recommendations in their current programs and possible future collaboration with USAID/Haiti. EVALUATION QUESTIONS The final evaluation answers the questions below. When answering these evaluation questions ( EQs), the ET considered stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent was the original project design relevant to the GOH concerns and needs? In answering this question, the evaluation focused particularly on the extent to which the Activity design facilitated empowering GOH officials and offices to project their own HR needs and identify their own technical experts. The evaluation examined these issues comparing GOH , implementing partner/counterpart (IP), and USAID perspectives, and how they evolved throughout the life of the Activity. 2. How effectively was the Activity implemented and managed? In answering this question, the evaluation assessed two fundamental Activity elements: a. The extent to which Activity interventions actively build GOH capacity to identify and hire appropriate technical assistance. b. The extent to which technical assistance provided through the Activity transferred knowledge and capacity to GOH officials and offices. To answer this question, the evaluation examined the interaction between USAID technical offices, the Contracting Officer’s Representative (COR), and KONEKTE. Specifically, the ET a ssessed how effectively the Activity managed GOH requests for technical assistance to ensure that it was appropriate and effectively used, including the extent to which the GOH was involved in the identification of the HR needs and the selection of appropriate technical experts to fill identified gaps. 3. How sustainable are the Activity’s interventions? To answer to this question, the evaluation specifically addressed the implications of USAID’s decision not to extend the Activity beyond five years, the impact of this decision on USAID’s relationship with the GOH, and the GOH’s willingness and ability to continue the Activity’s interventions in the absence of USAID support. II. EVALUATION DESIGN SAMPLING The ET collected data on the perspectives and experiences of key stakeholders involved in the KONEKTE Activity using a purposeful sampling technique (see list of persons consulted in Annex E ). The team sampled representatives from all key stakeholder groups: USAID, MSI and CFET, TAs, GOH counterparts, and other donors or USAID Activity implementor representatives who collaborated with KONEKTE. Within each stakeholder group, the ET selected stakeholders from different subcategories: • USAID COR, as well as technical office representatives 8, • IP personnel from (a) the prime and subcontractor organizations, (b) h eadquarters (HQ) and Haiti-based personnel, and (c) Chiefs of Party (COP) and other program staff, • TAs who had (a) served for various lengths of time, (b) had varying degrees of seniority, (c) were men and women, and (d) who had worked with a variety of GOH counterparts, 10| KONEKTE PERFORMANCE EVALUATION USAID.GOV • GOH counterparts from among the various government offices that collaborated with KONEKTE, both senior leaders and TA technical interfaces, • Different donor organizations. In cases where several different individuals held the same position, the ET endeavored to interview as many individuals as possible who held a given position to collect as many viewpoints as possible and limit bias. For example, the ET interviewed two out of three MSI COPs, three out of five USAID CORs, five MSI HQ based backstopping personnel, 17 out of 38 technical advisors, etc. Purposeful sampling was supplemented by snowball sampling to enable the ET to identify additional key informants and fill gaps in its initial list of key stakeholders, which was provided by MSI. For example, the initial list did not include a list of all USAID CORs, a complete list MSI technical and administrative backstopping personnel, USAID technical office representatives, or CFET client relations managers. During initial interviews, MSI, CFET and USAID provided addition names and contact information for missing individuals. Overall, the final in formant sampling allowed the ET to compare the perspectives of different key stakeholder groups and cross-check and fill gaps in KONEKTE reporting. Although more men than women participated in key informant interviews (KIIs), the proportion was a fair reflection of the roles played by men and women in KONEKTE. DATA COLLECTION METHODS The ET used a qualitative design, consisting of several different data collection methodologies, including document review, analysis of Activity monitoring and evaluation d ata, KIIs, focus group discussions (FGDs) and direct observation of KONEKTE- supported activities. The ET reviewed Activity documents and produced an inception report in June 2018. The team conducted interviews, FGDs, and direct observation during the evaluation fieldwork period, from July 31, 2018 – August 29, 2018. The ET developed data collection protocols to guide fieldwork, which are found in Annex F . The purpose of the protocols was to : (1) ensure all key issues are covered during data collection, (2) elicit rich, sometimes unanticipated, information from respondents , (3) help organize information in a form that could be usefully and efficiently analyzed, and (4) ensure that sex- disaggregated data w ere collected. The protocols consisted of questions that address and derive from the EQs, as well as from the ET’s document review, its initial discussions with USAID, and its evaluation design knowledge. DOCUMENT REVIEW The ET reviewed KONEKTE documents to understand the Activity design and implementation, extract findings relevant to the EQs, and inform data collection protocol development so that instruments appropriately supplemented background documents information. Categories of documents are listed below and detailed in Annex B: • KONEKTE annual reports • KONEKTE annual work plans • KONEKTE monitoring and evaluation (M&E) plans • Activity descriptions and modifications • Journals and other publications KEY INFORMANT INTERVIEWS The ET interviewed 52 individuals (18 women and 34 men) from various stakeholder groups. The team interviewed nearly all informants individually, except for one group interview of three informants. Findings from KIIs contributed to the ET’s responses to all EQs. The ET conducted face-to-face interviews in the Haitian capital, Port-au-Prince, or via phone, or Skype. Table 2 shows the distribution of interviews by key stakeholder group and gender. 11| KONEKTE PERFORMANCE EVALUATION USAID.GOV Each KII was guided by an interview protocol adjusted for different types of respondents (See Annex F for protocols), and was conducted in French or English. Table 2: Key Informant Interviews by Stakeholder Group and Gender FOCUS GROUP DISCUSSIONS The ET organized one FGD with HR Directors from various GOH institutions. The ET held the discussion at the end of a regular meeting organized by OMRH. The discussion was brief, but allowed the team to initiate contact with members of the group. The ET followed up and organized KIIs with four group members to supplement FGD information on current practices and challenges facing GOH HR Managers. Table 3: Focus Group Discussions by Stakeholder Group and Gender STAKEHOLDER TOTAL FEMALE MALE GOH HR directors 12 8 4 DIRECT OBSERVATION The ET triangulated interview data with direct observation when possible and relevant. It observed whether or not the KONEKTE -supported biometric time and attendance keeping system was still functional , if the web applications developed for OMRH (the job portal, the website) were online, and participated in a meeting organized by OMRH with HR Directors from various ministries to observe how OMRH was following up on HR management capacity building activities carried out by KONEKTE TAs. ANALYSIS ET took detailed notes of all KIIs and FGDs, and cleaned and shared electronic summaries on a rolling basis throughout fieldwork. Team members conducted internal debriefs regularly throughout fieldwork to discuss progress and make adjustments that were needed in the evaluation schedule. The team also discussed evidence collected to help answer the EQs, as well as to identify discrepancies. During the debriefs the team also developed and updated a findings, conclusions and recommendations matrix, focused particularly on findings. This ser ved as a basis for identifying key themes that emerged from the data collection. The team then developed a coding framework called a tally sheet, in which each set of notes, organized by stakeholder group, KI and sex, was coded using the identified set of themes. The tally sheet is a matrix that allows for identification of dominant themes. It also easily facilitates comparative analysis both within and across stakeholder groups and by sex. The ET employed several data analysis methods to identify key findings from the collected data, as well as to draw conclusions and make recommendations for Activity follow-up or future potential USAID programming like KONEKTE. 1. Content Analysis – The ET reviewed and coded KII and FGD data to identify and highlight notable examples of KONEKTE successes (or lack of success) that contributed to (or inhibited) achievement of its objectives. 2. Trend Analysis – The team examined KONEKTE M&E data on its indicators over time to identify trends in performance relative to its Activity targets. STAKEHOLDER INTERVIEWS FEMALES MALES Total 52 18 34 USAID 5 3 2 Implementing Counterparts 14 10 4 Technical Advisors 17 3 14 GOH Counterparts 14 2 12 Other Donors/USAID Activity 2 0 2 12| KONEKTE PERFORMANCE EVALUATION USAID.GOV 3. Gap Analysis – The team analyzed where KONEKTE fell short of anticipated performance targets, and the likely factors contributing to these gaps. 4. Comparative Analysis – The ET compared different stakeholder groups perspectives to assess either convergence or divergence. 5. Gender Analysis – All data collected through KIIs and FGDs were disaggregated by sex and analyzed for effects on both male and female beneficiaries to show any significant differences. The Team Leader captured preliminary findings, conclusions, and recommendations in an Excel-based matrix that categorizes analysis by EQ. The matrix: a) ensures the ET prepares a systematic and thorough response to each EQ, b) verifies that preliminary analysis accounts for gender and social dimensions, c) identifies any gaps where additional clarification or analysis may be necessary, and d) serves as the basis for developing the evaluation report. LIMITATIONS AND MITIGATION S TRATEGIES BIAS The ET was aware of several bias risks in data collection and analysis and developed mitigation strategies to overcome these: 1. Recall Bias: Recall bias is a common challenge in final evaluations, especially evaluations that take place months after an Activity’s close. KONEKTE recipients might not have remembered accurately or responded to the ET’s questions with answers that blended their experiences into a composite memory if, for example, they had received technical assistance from multiple sources. The ET compared stakeholder responses, for example what the TA recalled as compared to the GOH counterpart that hosted the TA, to increase the validity of the evaluation findings. It also probed respondents’ answers to facilitate accurate recollections. 2. Response Bias: Response bias is the risk that key informants may have been motivated to provide responses that would be considered socially desirable or influential in obtaining donor support or, alternatively, to discredit the activity or a given individual involved in the activity. For example, a GOH counterpart may have provided positive remarks about KONEKTE because s/he would like his/her institution to receive similar support in the future or because s/he wanted to please the interviewer. A KI may have provided negative feedback because s/he was dissatisfied with one or more persons involved in activity management. ET mitigated this risk by interviewing a wide variety of informants within and across stakeholder categories. In its analysis it prepared a tally sheet that allowed the ET to observe convergence of points by view and outliers by stakeholder group and question sub-finding. The ET also beg an each KII and FGD with a short explanation of the purpose of the evaluation, highlighting that the team hoped to learn about both successful and unsuccessful strategies, achievements and shortfalls, as relevant, to draw out lessons learned that could benefit future Activities. The ET also highlighted it would keep interview content confidential and omit personally identifiable information from the report (see informed consent protocol in Annex F). 3. Selection Bias: Selection bias is an inherent risk when implementers help to facilitate contact with Activity counterparts. There was a risk that the ET would be oriented to stakeholders with primarily positive views of the Activity. To mitigate the risk of selection bias, the ET identified the individuals to be interviewed using lists provided by MSI and supplemented using the snow ball method to fill in gaps (missing stakeholder categories, missing individuals who held a key position at some time during the evaluation period, missing contact information for some individuals in various stakeholder categories) with help from IP, USAID, and GOH KI. To mitigate the potential bias that KI would refer the ET to others sharing similar viewpoints, the ET accepted referrals from several different sources and used the strategies described above to mitigate for response bias. The ET arranged meetings directly without going through the implementer. 13| KONEKTE PERFORMANCE EVALUATION USAID.GOV 4. Gender Bias: Gender bias is a risk because most individuals have a subconscious sense of appropriate roles and behavior for women and men. The ET reviewed gender-sensitive approaches during the inception period by taking and discussing USAID’s Gender 101 training, focusing discussion on what possible gender preconceptions might come into play during this evaluation, and reviewing how to minimize these during data collection and analysis. The ET also actively searched for female respondents. OTHER LIMITATIONS AND MITIGATION STRATEGIES Limited availability of key informants: Because KONEKTE was implemented over a five- year period and was closed for approximately nine months at the time of the evaluation fieldwork, the ET anticipated potential challenges locating and obtaining meetings with some key stakeholders who changed positions, joined new organizations, or left the country. To mitigate this risk, the ET contacted stakeholders using their personal emails, drew upon the national evaluation team member’s personal network to locate missing contact information, and requested support from MSI and USAID/Haiti to obtain former personnel contact information. The ET conducted many interviews by telephone or Skype, which allowed it to interview stakeholders outside Port-au-Prince. The ET’s biggest challenge was obtaining meetings with the current leadership of OMRH and relevant representatives of USAID technical offices. Through USAID support and perseverance, the team succeeded in obtaining meetings with representatives of both stakeholder sub-groups before the conclusion of fieldwork, but not in the numbers planned. 9 Fewer than planned OMRH representatives’ interviews was balanced by interviewing a than planned number of “other” GOH representatives . Similarly, the fewer than planned USAID technical office representative interviews was balanced by interviewing a greater than planned number of USAID COR. III. FINDINGS AND CONCLUSI ONS This section synthesizes the evaluation’s main findings and conclusions, which are presented by EQ. EQ1: TO WHAT EXTENT WAS T HE ORIGINAL PROJECT DESIGN RELEVANT TO T HE GOH CONCERNS AND NEEDS? In answering this question, the ET first analyzed what USAID designed KONEKTE to do as well as the evolution of how various USAID technical offices , in fact, used the Activity in its implementation phase. This section also reviewed the roles and perspectives of KONEKTE GOH counterparts including the extent to which KONEKTE empowered GOH officials and offices to identify their own technical assistance needs, as well as to choose and manage TAs. The ET collected GOH counterpart feedback on KONEKTE services as one means to gauge its relevance to GOH concerns and needs. Finally, the ET examined stakeholder perspectives on GOH counterparts’ capacity to project strategic HR needs. FINDINGS Evolution of KONEKTE’s purpose and USAID technical office buy-in over time According to the IP and USAID KII s, KONEKTE was designed by USAID, following the earthquake, to hire personnel to supplement USG in- country human resources. It was also conceived as a mechanism to respond quickly and relatively easily to ad hoc GOH requests to fill post-earthquake vacancies in government institutions, which were numerous considering earthquake-related fatalities and migration. 9 The ET planned to meet with 13 GOH representatives (5 OMRH and 8 other) and 4 USAID technical officers (1 COR, 3 technical office). It succeeded to meet with 14 GOH representatives (3 OMRH representatives and 9 other) and 5 USAID technical offices representative, 3 COR and 2 technical offices (Education and Food and Economic security). 14| KONEKTE PERFORMANCE EVALUATION USAID.GOV By the time KONEKTE became operational two years after the earthquake, many of the needs the Activity was designed to meet had changed or had been met in other ways. USAID /Haiti was no longer receiving urgent requests from various USG Washington offices to hire additional personnel to handle post-earthquake recovery work. Likewise, GOH counterpart personnel needs, while still great, did not have the same urgency as was the case in the months immediately following the earthquake. When implementation started, KONEKTE’s main mandate was to provide technical expertise to select GOH counterparts in support of USAID programs falling under its four strategic pillars. KONEKTE was a basket mechanism to which USAID technical offices contributed resources. Based on IP and USAID key informants, not all USAID technical offices bought into KONEKTE, and some reduced their support during the implementation period. For example, after hiring one TA through KONEKTE, the Health office decided to drop KONEKTE and withdraw funding. The Education office mainly used KONEKTE to bridge its Activities on improving early grade reading in collaboration with the Ministry of National Education and Vocational Training (MENFP). The Agriculture and Food Security office mainly used KONEKTE to assist GOH counterpart offices in the Ministries of Social Affairs and Labor (MAST) and Agriculture, Natural Resources and Rural Development (MARNDR) on food security Activities, but reduced support in later years . In contrast, the Democracy and Governance (D&G) technical office increased its support for KONEKTE over time. For the D&G office , the KONEKTE recruitment process, which strengthened OMRH leadership within the government on HR matters, as well as the actual placement of TAs to work within the OMRH, were effective means to build the capacity of a key GOH counterpart. In addition to providing TAs to OMRH, the D&G office also funded KONEKTE advisors for the Ministry of Finance, the Ministry of Interior and the Anti- Corruption Unit, among others. Evolving USAID and IP perceptions of KONEKTE’s efficiency as hiring mechanism According to three USAID informants, some USAID technical office managers wanted or needed technical support faster than KONEKTE was able to provide while the USAID contracting office insisted that recruitment should follow an open and competitive process, even if that slowed down recruitment. According to one USAID technical office representative, as KONEKTE developed its procedures, s/he found the mechanism increasingly rigid, which diminished its usefulness. Eventually, some technical offices preferred to use alternative contract mechanisms t o hire technical experts rather than go through KONEKTE. Extent KONEKTE empowered GOH counterparts to identify needs, choose, and manage TAs In theory, KONEKTE assistance started with GOH counterpart requests. In practice, USAID technical offices oriented GOH institutions’ requests to intervention areas that aligned with its priorities. Based on KII s with IP, USAID, and GOH stakeholders, the latitude allowed to GOH counterparts to negotiate KONEKTE support varied according to the USAID technical office. In all cases, USAID, rather than KONEKTE personnel, were responsible for approving GOH TA requests. USAID’s D&G office offered OMRH leadership significant latitude to determine its own needs and choose its own technical resources throughout KONEKTE. Other USAID technical offices appeared to use KONEKTE to reinforce existing programs, narrowing KONEKTE’s scope to pre-determined intervention areas. KONEKTE work with the MENFP, MAST, Ministry of Economy and Finance (MOEF), and MARNDR, was related to other ongoing USAID programs and Activities (Early Grade Reading program, Koré Lavi, Food Security, Integrated Financial Management Systems). All key stakeholder groups affirmed that GOH counterparts were primarily responsible for choosing KONEKTE TAs. GOH representatives sat on TA selection committees (KONEKTE and USAID had observer status only) to choose advisors from among available candidates. KONEKTE personnel managed administrative and contractual aspects of the recruitment process and ensured that the c andidates selected by GOH counterparts aligned with the qualifications determined in the TA SOW. “It was the OMRH that developed TA SOWs and decided what their priorities should be.” - OMRH official 15| KONEKTE PERFORMANCE EVALUATION USAID.GOV According to the procedures established by KONEKTE and GOH counterpart and TA accounts, once TAs were hired, GOH personnel were responsible for their day-to-day technical supervision, which included approving monthly and annual reports and specific deliverables. TAs reported helping with many unplanned tasks based on their host GOH institution’s daily needs. GOH counterparts were also responsible for decisions about if/how to use TAs’ work. Annual reviews allowed GOH to discontinue TA services that did not respond to their needs, a right they occasionally exercised. Extent KONEKTE strengthened GOH counterpart s’ voice in day-to-day decision-making within USAID programs Both GOH and USAID technical office stakeholders reported that TAs strengthened their host GOH office ’s capacity to participate in USAID Activity implementation and in some cases, enabled them to oversee USAID implementers interventions. Two KONEKTE TAs reported that their efforts and KONEKTE logistic support was critical to carry out consultations to gather and share information with MENFP personnel outside the capital. A MENFP official reported working closely with KONEKTE TAs to update learning materials for teaching reading. Because of his involvement in the materials development, the Ministry official was well- informed about what had been accomplished. Later, a new USAID Activity that followed KONEKTE developed new learning materials, which he considered to be of lesser quality, and attempted to introduce them into the Ministry without extensive consultations. He reported raising the issue at the highest levels of both the Ministry and USAID and eventually succeeded in establishing a joint Ministry/Activity process for revising the material, taking the best of both products. A USAID official reported that the KONEKTE TA seconded to the MAST strengthened communication between USAID, the IP (Care) and the GOH on implementation and policy-related issues affecting the social safety net program, Koré Lavi. The KONEKTE TA and a MAST official reported that KONEKTE expertise and logistic support enabled the MAST M&E unit to be involved in Koré Lavi monitoring, giving them a role in identifying and resolving implementation problems. Extent GOH counterparts were satisfied with KONEKTE TA services Overall, GOH counterparts interviewed during the evaluation fieldwork affirmed they were very satisfied wit h KONEKTE assistance, corroborating the results of KONEKTE IP-conducted client satisfaction surveys. According to these surveys, over four years (fiscal year [FY]2014 – FY2017), on average, 89 percent of GOH clients were very satisfied or satisfied with technical assistance provided by KONEKTE TAs. OMRH’s satisfaction for KONEKTE services, a more specific indicator of the A ctivity’s institutional capacity building activities, was slightly lower overall - 82 percent. Based on GOH KIIs, aspects of KONEKTE services that GOH counterparts appreciated most were: • TAs functioned as government employees rather than consultants (they responded to evolving priorities, had to work regular hours in the office); • TAs offered specialized expertise that the GOH could not afford via their regular budget; • TAs supplemented limited human resources; and • KONEKTE logistic support to TAs facilitated their mission in ways not possible within the GOH budget. Aspects that GOH counterparts thought could have been improved were: • KONEKTE assistance could have been provided for longer period 10; • the number of TAs increased, and • some TAs could have been better qualified. 10 Based on KONEKTE’s final report, 13 KONEKTE TAs served for two- to three-year terms, 18 served one-year terms, and the remainder served for less than one year. 16| KONEKTE PERFORMANCE EVALUATION USAID.GOV According to USAID informants, the number of TAs offered to GOH counterparts and the duration of their assignments were based on USAID priorities, available budget, and time remaining in KONEKTE’s implementation period. Further, the Agency had originally thought it would be providing primarily senior- level TAs; this turned out not to be the case mainly due to challenges finding appropriately qualified, French -speaking experts willing to work in Haiti. Also, starting in 2015, KONEKTE in collaboration with OMRH, decided to recruit mainly mid-level TAs and pay them at GOH salary scales with the understanding that the OMRH would recruit them as regular staff after one year. Stakeholder perspectives on GOH counterparts’ capacity to project strategic human resource needs Eleven KIIs (two IP, one USAID, two TA, six GOH) cited weak capacity within GOH counterpart institutions to do strategic needs assessment to determine its needs for and use of technical assistance . Gaps identified included: insufficient consultations by GOH decision-makers to assess needs in a participative and/or systematic way before determining the TA SOW, the absence of a clear strategic framework to set institutional priorities, too much or too little consideration of GOH political issues, and low capacity within GOH human resource departments (HRDs) to identify hiring priorities and defend the activities against personal and political interests. KONEKTE delivered technical assistance to OMRH to address these weaknesses. KONEKTE TAs and consultants implemented activities to build the capacity of HRDs and introduced an organizational development and change management tool to assess and propose actions to improve the performance of GOH institutions. The extent to which KONEKTE interventions were effective building GOH counterparts’ capacity in these areas is discussed under EQ2a. CONCLUSIONS KONEKTE’s design empowered GOH officials and offices to choose and manage their own TAs . This made KONEKTE relevant to its main GOH counterpart s’ concerns and needs. KONEKTE filled gaps in both strategic and day- to-day needs of GOH counterpart s. USAID technical offices determined the scope of KONEKTE assistance to GOH offices to fit with its priorities, limiting the GOH counterparts’ opportunity to set their own priorities for KONEKTE assistance. USAID’s D&G office allowed the 2012 – 2016 OMRH leadership a greater role in setting KONEKTE’s agenda within his office than did other USAID technical offices. KONEKTE strengthened GOH counterparts’ voice in day- to-day decision-making within USAID programs. Embedding senior TAs in GOH offices strengthened GOH dialogue with USAID and in some cases, GOH oversight of USAID implementor interventions. This was not an explicit objective of KONEKTE, but is a highly positive, unintended consequence of placing TAs in USAID GOH counterpart offices. EQ2: HOW EFFECTIVELY WAS THE ACTIVITY IMPLEMENTED AND MANAGED? To answer the first part of EQ2, the ET examined KONEKTE achievements meeting its main performance indicator targets. The team also reviewed key stakeholder perceptions of KONEKTE efficiency as hiring mechanism, internal management issues as well as political, social, and economic challenges affecting progress. FINDINGS Extent KONEKTE met its indicator targets “I don’t think KONEKTE was effective in knowing what the needs were and responding to them. GOH entities were not able to articulate what they needed.” - USAID informant 17| KONEKTE PERFORMANCE EVALUATION USAID.GOV Based on the Activity’s M&E data, KONEKTE’s performance achieving its performance indicator targets was mixed. On average, over four years, KONEKTE was slightly under its targets for the numbers of TAs placed and the percentage of TA deliverables that met or exceeded SOW requirements. Performance on the latter indicator deteriorated in the final two years of implementation. KONEKTE was significantly less efficient than planned in terms of the number of days required to field a TA after receiving a validated SOW from USAID. Performance on this indicator likewise declined in the final two years. The number of days required by KONEKTE to process a GOH request from USAID validation to placement at the receiving entity or organization averaged 123 days over four years, while the target was 84 days. TAs reported their recruitment processes took anywhere between three months to over one year. In FY 2016-2017, the average number of days to process a placement was 24 1. KONEKTE IP progress reports indicated that factors beyond its control such as slow USAID and GOH counterpart response times, absence of appropriate candidates, unsuccessful salary negotiations, and broader environmental factors such as deteriorating political, social, and economic conditions in Haiti, negatively affected its efficiency fielding TAs. Deteriorating performance meeting targets on the number of days required to field a TA in the final two years was likewise affected by leadership changes within OMRH and the IP (see below). Efforts to create a common understanding of KONEKTE’s purpose and operational norms IP and USAID informants (five total, two USAID, three IP) described the involvement of too many decision- makers within USAID and the GOH and lack of common understanding of KONEKTE’s purpose and operational norms during its first 12 to18 months and again during KONEKTE’s final months. Measures were gradually put into place that improved procedures and communication: • Initially, conditions for receiving KONEKTE assistance were not clearly defined by USAID and communicated to potential GOH counterparts. This resulted in unacceptable requests from some GOH entities and low uptake by others. After some initial cases, when USAID refused certain GOH requests, USAID technical offices negotiated GOH requests before they became official to avoid having to turn down requests. The IP also organized meetings with GOH counterparts to explain the Activity and its procedures. • In the early months of implementation, the KONEKTE COP interfaced directly with GOH entities to determine needs without coordinating with USAID, which generated GOH expectations that did not align with USAID plans and strategies and ended in reversals in commitments vis à vis counterpart s. By 2014, according to USAID and IP managers, KONEKTE’s role had evolved into a mainly administrative one. OMRH became the immediate interface between USAID and GOH entities to shield USAID offices from stakeholder pressure, which was positive for KONEKTE and contributed to strengthening OMRH’s position within the GOH on HR reforms. However, in late 2016, when a new COP took over and shortly afterwards, a new Coordinator General (CG) was appointed at OMRH, previous misunderstandings and disagreements about roles and responsibilities between the IP, USAID, and OMRH returned. “It took more than a year to finalize the recruitment because of internal issues at USAID and KONEKTE. There was a list of those who had been selected and I expected to have them quickly at my disposal. We had to wait a very, very long time.” - GOH counterpart 18| KONEKTE PERFORMANCE EVALUATION USAID.GOV Leadership changes negatively affected progress During its five -year implementation period, KONEKTE had three COPs, four USAID CORs, three OMRH CGs and experienced several changes in government. Table 4, below, summarizes these changes. Informants from all key stakeholder groups reported that frequent IP, USAID, and GOH counterpart leadership turno ver, as well as instability at the Prime Minister’s level were sources of uncertainty, slowing decision-making and shifting priorities. Table 4: KONEKTE: Timing of Changes in Key Stakeholder Institutions’ Leadership FY 12/13 FY 13/14 FY 14/15 15/16 FY16/17 USAID COR1 COR2 COR3/COR4 COR4 COR4 OMRH CG1 CG1 CG1 CG1/CG2 CG3 IP COP COP1 COP2 COP2 COP2 COP3 GOH/Prime Minister PM1 PM1/PM2 PM3 PM4 PM5 Change in original OMRH leadership was a major setback Evaluation stakeholders in all categories agreed that changes in OMRH leadership negatively affected KONEKTE progress supporting OMRH capacity building. According to KONEKTE progress reports corroborated by interviews with TAs and GOH officials, the departure of the original OMRH CG, and his replacement by an interim leader , halted or slowed the work of many OMRH TAs. The same stakeholders reported that when the new CG was appointed near the end of the Activity, many institutional priorities shifted, leaving the newly appointed KONEKTE COP little time to reposition its support to earn the buy-in of the new OMRH leadership. Lack of cohesion within KONEKTE IP management team affected morale and efficiency According to the majority of IP informants (ten out of 14 consulted) at various times throughout the life of KONEKTE, internal conflict within the KONEKTE IP management team (between program managers, between the prime and the subcontractor, and between Haiti- based managers and HQ) led to negative working environments, significant disagreements about how the project should be managed , and COP and other key position turn-over, which, among other factors already cited contributed to the Activity’s declining success meeting some of it performance indicator targets. There was no formal handover process between COPs, which did not facilitate a smooth transition between them. One member of the IP team remarked to the ET, “What I can say is there was a lack of cohesion within the project management team… At certain moments there was even a lot hostility.” According to two IP informants involved in trying to remediate conflict, although the IP managers made efforts, some of them successful, to address the various issues causing disagreement, conflict was a reoccurring theme throughout the Activity implementation period. USAID slow issuing written approvals, modifying contract to reflect changes Activity scope Because GOH requests for assistance and TA nomination required USAID technical office review and approval and sign- off by USAID’s C ontracts office, USAID was more involved in the day-to-day implementation of KONEKTE than was typical of other USAID contracts . Weaknesses in USAID management cited by the IP included frequent change over in USAID personnel interfacing with KONEKTE, shifting and sometimes “Changes within government and within KONEKTE did not help. When there are changes in leadership, it is difficult to make progress. At times the program was paralyzed.” - KONEKTE IP team member “The disadvantage from the KONEKTE IP point of view was we were not just working with all the Ministries, but also with all the various USAID technical offices. This was challenging. The visions were different.” - IP key informant 19| KONEKTE PERFORMANCE EVALUATION USAID.GOV conflicting views within the Agency on how to capitalize on the Activity , and slow decision-making. IP representatives and KONEKTE documents indicated USAID contract modifications, formal workplan approvals and validation of M&E frameworks were likewise slow or absent, creating over-reliance on oral agreements between USAID and the IP that could be misunderstood or later reversed, and cause management uncertainty. Political, social, and economic challenges also affected progress According to KONEKTE progress reports, throughout the life of the Activity, implementation was affected by political, social, and economic factors which disrupted implementation and contributed to delays. These included delayed elections, contested elections, social unrest, rising levels of insecurity, natural disaster, and a rising cost of living. One informant highlighted that political changes and anticipated changes at the highest levels of government trickled down to lower-level managers, who often preferred to take a wait-and-see approach to decision-making. Violent street protests linked to the political and economic climate erupted at various times during the implementation period, lim iting movements around Port-au-Prince. CONCLUSIONS KONEKTE's implementation was not efficient during its first 12 to 18 months, and again at the Activity’s end. The causes of inefficiency were related to variety factors, some within the control of USAID and the IP and others not. USAID and IP internal management factors such as lack of shared vision and cohesive management, too many decision-makers and slow and/or poorly documented decision-making (USAID), poorly-defined and communicated roles and responsibilities, and lack of clarity on the conditions for receiving KONEKTE assistance in the early implementation stages (USAID and IP) diminished implementation effectiveness. Frequent changes among decision-makers within USAID, GOH counterparts, and the IP also negatively affected KONEKTE’s overall effectiveness, especially changes in OMRH top leadership. Haiti’s political, social, and economic instability during most of the Activity implementation period likewise affected KONEKTE implementation. EQ2A. THE EXTENT TO WHICH ACTIVITY INTERVENTIO NS ACTIVELY BUILT GOH CAPACITY TO IDENTIF Y AND HIRE APPROPRIA TE TECHNICAL ASSISTANCE To answer EQ2a, the ET looked first at KONEKTE’s model for TA recruitment and management, and the extent to which it influenced OMRH practices and strategies. Related to the former, the ET examined the extent that KONEKTE’s model integrated USAID cross-cutting concerns on gender, employment of persons with disabilities, and USAID Forward principles. Because the answer to EQ2a is closely related to KONEKTE’s assistance to strengthen OMRH-led GOH reforms on recruitment and HR management, this section also analyzed what was done in th ese areas, and what GOH stakeholders reported as changes during , and immediately after KONEKTE. FINDINGS ReframingKONEKTE tobuildOMRH capacitytoleadGOH HR reforms ActivitydocumentsshowtheMinistryofPlanningwasoriginallyKONEKTE's GOHcounterpartinchargeofoverallcoordination.Thisroledidnotalignwell withitsmission,accordingUSAIDandIPkeyinformants.OMRH,theGOH institutionresponsibleforleadingGOHcivilservicereforms,andwhichwasled byadynamicCG,tookoverastheleadgovernmentcounterpartwithinthefirst 12monthsofimplementation.USAIDandIPmanagerscharacterizedtheir decisiontogiveOMRH asignificantroleasanimportantturningpointfor KONEKTE,leadingtoitseventualreframingasaninstitutionalcapacitybuilding endeavor. Beginningin2014,USAIDandtheKONEKTE COPworkedcloselywiththe OMRHCGtonegotiateOMRHcoordinationoftheoverallTAselectionprocess aswellastoallocateKONEKTETAtoOMRHforinstitutionalcapacitybuilding. GOH,USAIDandIPinformantshighlightedthattheKONEKTE COPandthe USAIDCORhadaneffectiveworkingrelationshipwiththeOMRHCG,which contributedtoreorientingKONEKTE atacriticaljuncture. DevelopinganopenandcompetitiverecruitmentmodelwithOMRH USAIDandIPinformantsreportedthatafter18to24months(until2014)of Figure3:stepsinKONEKTE Recruitment Process Requestreceived, preparationofrequest andapproval Publication ofposition CVscreening, interviewsand selection ofcandidates Verificationofreferences andsalaryhistory Approvalofthe contractbyUSAID Finalizationandsigning ofthecontract experimentation,KONEKTE,establishedaclearprocessforreceivingGOHrequestsforassistance, formulatingTASOVVsandrecruitingTAsinsignificantnumbers.KONEKTE'sfinalreportshowsthenumberof TAcontractssignedjumpedfromfiveinFY2012—2013to22inFY2013—2014. Beforetheprocesswasestablished,accordingtoUSAIDandIPinformants,therewereproblems.Oneissue wastheroleoftheIPinidentifyingGOHneeds.USAIDwasunhappywhenIPCOPsnegotiatedKONEKTE assistancewithGOHcounterpartswithoutitsinvolvement,whichwasdoneinKONEKTE'sfirstandfinalyear. OMRH wasmadetheinterfacebetweenGOH institutionsandUSAID,whichshieldedUSAIDandKONEKTE managersfromdirectpoliticalpressure.AnotherissuewastheprocessforselectingTAS.InKONEKTE'searly months,someGOHcounterpartsidentifiedthecandidatesthemselvesandaskedUSAIDtorecruitthem throughKONEKTE.USAIDcanceledatleastonelargerecruitmenteffortofthiskindatthelastminute becausetheprocessusedbytheGOHtoidentifythecandidatesdidnotalignwithUSAIDprocurement regulationsrequiringopencompetition. USAIDandtheKONEKTE IPgraduallyputintoplaceeffectivemeasurestolimitpoliticalandpressures influencingtheTArecruitmentprocess.Informantsfromacrossallkeystakeholdergroups(17informants,six IP,oneUSAID,sixTA,fourGOH)agreedthatKONEKTE TAswererecruitedthroughanopen,competitive processrespectingUSAIDnorms.USAIDandGOHcounterpartinstitutionsdeterminedTASOVVsandsaton theTAselectioncommittees.StepsintherecruitmentprocessthatKONEKTE personnelmanagedwere developingatailoredjobdescriptionandcandidateprofile,advertisingtheposition,initialscreeningof candidates'CVs,negotiatingsalary,andcontracting,orienting,andmonitoringTAS. AccordingtotheIPlocalcounterpart,CFET,KONEKTEdocumenteditsproceduresinamanual.Thismanual washandedovertoOMRHatitsrequest.BasedonKONEKTE reportscorroboratedbyinterviewswithGOH officialsandTAsworkinginOMRH,OMRHadaptedthemanualforitsownuse,andshareditwithHR directorates atvariousministries. Cross-cuttingissues:gender,employment ofpersonswithdisabilities,andUSAIDForward principlesintheKONEKTE recruitmentmodel 201KONEKTE PERFORMANCE EVALUATION USAID.GOV 21| KONEKTE PERFORMANCE EVALUATION USAID.GOV A specific focus on g ender equity was not explicitly required in KONEKTE’s SOW. Members of the IP team indicated that the Activity made efforts to ensure woman candidates for TA positions were proposed when possible, but did not have a policy of “affirmative action.” According to KONEKTE Activity reports, out of a total of 45 TAs, seven were female (16 percent). IP and GOH stakeholders indicated that KONEKTE TA selection committees selected candidates based on their qualifications only. Based on evaluation interviews, there were few other examples of KONEKTE integrating gender considerations into its recruitment/HR management interventions. A notable exception, two senior KONEKTE TAs formed an HR working group on gender and persons with disabilities within OMRH to discuss how OMRH could address nondiscrimination dimensions of HR management. The working group planned to organize an OMRH -sponsored forum to discuss how to improve the implementation of existing policies on gender and employment of persons with disabilities. Unfortunately, the planned event was cancelled by KONEKTE management due to insufficient resources near the end of the Activity. The USG’s Haiti strategy and USAID Forward set out principles to strengthen Haitian leadership of USAID-supported development programs. KONEKTE largely aligned with this principle by favoring the recruitment of Haitian nationals to fill TA positions and working to strengthen the management of CFET to take on USAID awards directly. The Activity’s first preference was to hire Haitians living in Haiti; second, members of the Haitians diaspora; and lastly, if appropriately qualified Haitians could not be found, the Activity hired expatriate experts. There were only three non-Haitian TAs out of the 45 TAs supported by KONEKTE. IP managers, TAs, and GOH officials expressed differing views of the effectiveness of this policy. One GOH informant saw advantages in hiring Haitian consultants in Haiti because they knew the country and its norms well . Based on KIIs with OMRH officials, the current OMRH leadership prefers, when resources are available, to hire senior TAs with specialized skills, and is not concerned by issues of nationality. KONEKTE’s assistance to strengthen OMRH-led GOH reforms on recruitment and HR management KONEKTE supported OMRH with 18 TAs, the largest number of advisors allocated to any KONEKTE GOH counterpart according to KONEKTE’s final report. The TAs included senior and mid-level experts with expertise on HR management systems, law, communication, information technology (IT), monitoring and evaluation and decentralization and devolution. Many OMRH TAs served multi-year terms. Based on Activity progress reports and TA and GOH officials’ descriptions, KONEKTE’s main activities to improve GOH recruitment and HR management practices were: • developing training modules for GOH HRD personnel, • drafting HR legal frameworks (examples include circulars on employment conditions and professional development within the civil service, selection of candidates and management of overseas academic scholarships recipients, and on HR dimensions related to decentralization and devolution), • developing manuals and procedural guidelines on HR recruitment, management, pe rformance evaluation, and communication, • supporting improvements of OMRH’s Human Resource Management Information System (HRMIS) , and • organizing GOH HRD forums to share good practices (three forums attended by a total of 137 people). “In Haiti, the question of gender can be tricky. It goes back to decisions made in families. In my class, there were only five women agricultural engineers out of 80. Obviously, this makes hiring and enabling women’s access to employment tricky when you need specific qualifications.” - IP informant “I believe that the hiring Haitian nationals diluted the spirit of KONEKTE. It was created to get in the diaspora and international experts to give strong support to the GOH. People hired locally do not challenge the way things are done.” - TA informant 22| KONEKTE PERFORMANCE EVALUATION USAID.GOV KONEKTE TAs proposed to implement a robust HRD capacity building program, but reported being turned- down by OMRH leadership. They reported conducting activities with five pilot HRDs from 2015 to 2016; the interim CG halted activities in the second half of 2016. Planned HRD forums were likewise scaled back during the same period. KONEKTE assistance to improve GOH HR needs assessment and strategic planning To address gaps in GOH capacity to use systematic approaches to identify HR needs (see EQ1 findings identifying weaknesses in these areas), KONEKTE, in collaboration with the OMRH, developed a public administration performance evaluation (EPAP) methodology by adapting USAID’s Human and Institutional Capacity Development (HICD) approach. 11 The purpose of the EPAP was to help OMRH to assess the performance of ministries, to set performance improvement objectives, and establish an action plan to achieve their objectives. KONEKTE work on the EPAP started in January 2016. Initially, OMRH proposed to collaborate with KONEKTE to pilot the methodology on itself and with a small number of other ministries. Later it decided that the political environment was not favorable 12 to pilot the methodology outside OMRH. Although KONEKTE organized one training workshop, it was not able to pilot EPAP within OMRH as planned. USAID stopped KONEKTE from continuing EPAP- related work in November 2016. According to its final progress report, KONEKTE, in collaboration with OMRH, developed a step-by-step EPAP implementation guide and training manual that was handed over to OMRH in December 2016. According to one USAID informant, USAID cancelled the EPAP effort because of lack of progress; s/he expressed disappointment that the IP did not send a more senior HICD expert to lead the intervention. IP informants did not agree that their advisor’s qualifications were an issue affecting progress; they cited the change in OMRH leadership and subsequent lack of institutional buy-in for the intervention, and internal conflict 13 on the implementation team as the primary causes of limited progress. One IP informant reported that OMRH, under its current CG, is planning to reviv e work on EPAP. KONEKTE assistance to computerize HR management systems Among its interventions to strengthen OMRH’s role in leading HR reforms, KONEKTE planned to help OMRH to update and complete a previously initiated, donor-funded effort to computerize HR management systems across the GOH known as the HRMIS. The idea of the system was to computerize HR management from recruitment to retirement. KONEKTE supported the project by placing four IT experts as TAs in 2016 and by funding improvements in OMRH’s IT network and related infrastructure. One of the key objectives of the system was to link government worker attendance information with pay. Among other interventions related to the system, KONEKTE installed a bio-metric time and attendance keeping system at OMRH. To link attendance with salary payments, the HRMIS needed to integrate with another IT system supported by USAID known as the Integrated Financial Management System (IFMS). Key informants involved in the intervention ( three IP, one TA, one other USAID project representative) described the collaboration between KONEKTE and USAID Haiti’s IFMS activity as ineffective. KONEKTE TAs and IFMS managers had conflicting views on their respective roles and responsibilities and what should be done, which blocked progress. As a result, KON EKTE ended without an operational HRMIS system. One key informant reported that another donor is taking over the project and plans to introduce a completely different software package. KONEKTE also support ed the development of a public sector job portal, which also was not completed and put online by OMRH. According to the KONEKTE TA who oversaw implementation, OMRH 11 Human and Institutional Capacity Development Handbook, A USAID Model for Sustainable Performance Improvement, USAID, August 2011. Https://usaidlearninglab.org/sites/default/files/resource/files/HICD%20Handbook%202011%20-%2008.pdf, accessed on 9/12/18 12 In 2016, the lack of a legislature and protracted political stalemates over elections hindered the Haitian government’s ability to govern. 13 Various stakeholders within KONEKTE’s management team described conflict between the IP COP and the HICD expert. The conflict and how it was handled within the IP eventually led to the COP’s resignation and USAID asking the IP to replace the HICD expert. 23| KONEKTE PERFORMANCE EVALUATION USAID.GOV significantly revised it portal specifications after the first prototype was already developed. During the evaluation, the ET observed that the bio-metric time keeping system is still used to track employee presence; other ministries have installed similar systems. One OMRH official indicated that KONEKTE- supported IT interventions were over-ambitious for the time and resources available. Reported changes and developments in GOH HRD during and immediately after KONEKTE HRD directors reported that their offices’ capacity had improved during the last five years (2013 – 2018) and attributed some of this progress to the efforts of OMRH. 14 They indicated that the GOH HR functions had been elevated from a unit to a full-fledged department in several ministries with additional personnel allocated. HR directors also reported increased use of competitive recruitment processes, as directed by the 2013 OMRH circular on competitive recruitment. They likewise described using the OMRH Human Resources Management Procedure Manual and performance evaluation guidelines (both produced by OMRH with KONEKTE TA assistance) to create job descriptions and organizational charts, as well as to assess the performance of government employees. Obstacles to competitive recruitment and improved HR management practices cited by HRD officials during evaluation interviews and the FGD included: lack of political will among Ministry hierarchy to adopt new practices, absence of updated personnel classifications and salary grid, non-participation of the HRD in the preparation of the budget, competency deficits among HR managers, and corruption (political patronage in hiring practices) within the civil service. Stakeholder suggestions on how KONEKTE OMRH capacity building interventions could have been more effective Informants from all KONEKTE stakeholder groups had suggestions for how KONEKTE’s OMRH capacity building interventions could have been improved and its outcomes promoting changes in recruitment and HR management practices enhanced . These included: • stronger coordination and follow- up by USAID and IP managers with OMRH leadership to ensure OMRH personnel were engaged in developing and/or reviewing KONEKTE TAs’ work, • stronger advocacy by USAID and IP managers at the political level in favor of HR reforms, • use of more participative methods to assess needs and promote changes among OMRH and other ministry HRD personnel by KONEK TE TAs, and • use of a longer-term intervention strategy. CONCLUSIONS KONEKTE contributed positively to building GOH counterparts’ capacity to identify and hire appropriate technical assistance. KONEKTE used an open and transparent recruitment process in which GOH counterparts were actively engaged, potentially contributing to the GOH personnel’s awareness and hands-on experience applying good recruitment practices. Reported changes and developments in GOH HRD during and immediately after KONEKTE indicate that the Activity likely contributed positively to ongoing reforms in civil service HR practices 14 FGD participants explained that they were not aware of the role of KONEKTE within OMRH. For them, all training or forums were organized by OMRH. A KONEKTE IP confirmed that KONEKTE agreed that its activities would be OMRH-branded in an effort to strengthen the OMRH’s role in promoting reforms. “After the training received at the OMRH, the HRD conducts performance evaluations. This contributes to the advancement of HRD. The performance evaluation is done in all departments of the Ministry.” - Focus group participant 24| KONEKTE PERFORMANCE EVALUATION USAID.GOV The IP & USAID showed vision and adaptive capacity in re-framing KONEKTE to strengthen OMRH’s leadership on civil service HR management reforms. Its contributions ended up being more modest than planned largely due to many factors beyond its control such as weak political will for HR reforms, HRD capacity deficits as well as operational challenges coordinating multiple USAID IP IT interventions. Despite these obstacles, KONEKTE supplied specialized technical assistance and tools that shaped OMRH intervention strategies and affirmed OMRH leadership of ongoing improvements in GOH recruitment and other HR management practices. KONEKTE did not make significant progress building GOH counterparts’ capacity to use systematic approaches to identify HR needs such as the HICD framework. Capacity- building for HRD personnel remains a significant need in this area as well as other HR management areas. EQ2B. THE EXTENT TO WHICH TECHNICAL ASSISTANCE PROVIDED THROUGH THE ACTIVITY TRANSFE RRED KNOWLEDGE AND C APACITY TO GOH OFFICIALS AND OFFICE S To answer EQ2b, the ET takes a closer look at KONEKTE’s intervention model to identify its strength s and weaknesses transferring knowledge and capacity to GOH officials and offices. T he ET presents what various stakeholders, mainly KONEKTE TAs and GOH counterparts, thought was effective in building capacity of GOH stakeholders, what were the limitations, and what might have been done better. FINDINGS KONEKTE TAs carried out a wide variety of tasks designed to fill gaps in GOH counterpart capacity From 2012 – 2017, KONEKTE placed 45 TAs within 11 GOH institutions. KONEKTE TAs reported carrying out a wide variety of tasks designed to meet both the long and short-term needs of GOH counterparts. These included: • developing training modules, guidelines, and tools, • identifying needs and hiring additional staff for some departments, • drafting legal frameworks, • developing strategic plans, • introducing new technologies and improving existing IT systems, • delivering training, and • collecting data and carrying out studies to support on-going activities. According to KONEKTE M&E data, on average, 77 percent of TA deliverables met or exceeded SOW requirements over four years, slightly under target of 80 percent. Performance was initially better. The Activity recorded 100 percent rates in FY 2014 and FY 2015, falling to 44 percent in FY 2016, a year with leadership turnover within both KONEKTE and OMRH, and then rising somewhat in FY 2017 to 65 percent. KONEKTE mainly measured it effectiveness in hiring and managing TA contracts not capacity building Based on ET analysis of KONEKTE M&E frameworks, in the first three years, Activity indicators were designed primarily to measure its efficiency hiring and administering advisor contracts. The main indicators were the time required to field an advisor, whether the recruitment was competitive, GOH counterpart satisfaction with TA services, and the percentage of planned deliverables satisfactorily delivered. In FY 2015 – 2016, t he IP introduced a revised M&E framework which included new indicators to measure outputs and outcomes related to its OMRH capacity building strategy. These included indicators on HRD training programs, EPAP training, the creation of an M&E and Contracts department within OMRH as well as interventions to upgrade the computerized HR management system. To measure OMRH capacity- building outcomes overall, the IP developed an institutional capacity index which measured capacity changes in work 25| KONEKTE PERFORMANCE EVALUATION USAID.GOV areas supported by KONEKTE. According to one IP informant, OMRH did not agree with the indicators used in the index and did not participate in calculating results from the baseline to the follow-up assessment. In its year four annual report (October 2016) , KONEKTE reported OMRH achieved 67 percent of the performance benchmarks expected in the previous 12 months. Following KONEKTE’s final contract revision which scaled back planned capacity building activities, the IP dropped most of the indicators that it introduced in 2016 including the OMRH capacity index. One of two remaining capacity indicators, the IP tracked the number of policy reforms supported by OMRH with KONEKTE support. The indicator classified TA deliverables according to strategic objectives in GOH reform plan. If the deliverable was completed and submitted to OMRH management, then KONEKTE counted it as having “contributed.” KONEKTE reported it contributed to 47 OMRH-supported policy reforms, exceeding its target of 26. The other institutional capacity indicator was the scale of achievement of the Job Portal . KONEKTE planned, but failed, to launch the portal by Activity’s end. GOH and IP views on the extent TA deliverables contributed to OMRH capacity According to IP and GOH informants, in the final year of project implementation, following the arrival of the final KONEKTE COP and the current OMRH CG, both parties investigated the status of the KONEKTE TA work products. The new OMRH leadership indicated it did not know “where” most of the nearly 60 documents KONEKTE TAs produced were (guides, policy documents, training modules). The IP compiled the deliverables and engaged the OMRH in a process to review and “validate” about ten. According to an IP informant, OMRH indicated it would follow- up on the other deliverables later. At the time of the evaluation, current GOH OMRH informants stated that many KONEKTE TA work products had not become official OMRH documents, including some that had been printed with the OMRH logo and distributed by the previous CG. OMRH and KONEKTE informants interpreted this situation differently: some IP team members said it was a politically-motivated maneuver to undercut the work of the previous CG; GOH officials said it was evidence that KONEKTE TAs had not been effective getting broad institutional buy-in for their work (below and above the level of the CG), while other IP managers said it was a sign of USAID and IP management ineffectiveness holding OMRH leadership accountable to provide appropriate oversight to TAs. TA perspectives on USAID and IP support and effectiveness getting institutional buy-in for their work Many KONEKTE TA (8 out of 17 interviewed) indicated that the work they did was strategically important and that they performed many useful planned and unplanned tasks. Many (eight out of 14 interviewed) also said they would have liked more support from USAID and/ or IP managers to encourage GOH institutional and political leaders to engage with and follow- up on their work. One TA said, “KONEKTE was not involved in the day to day details of what I did. I reported to the Ministry Leadership. KONEKTE was responsible for the administrative details. It was a body shop.” Five TAs said they felt over -burdened with a large number of planned deliverables as well as many unplanned and strategically unimportant tasks given them by the GOH supervisors. KONEKTE TA opinions on the effectiveness of IP personnel was somewhat mixed. Six TAs praised KONEKTE CRM and other managers while four indicated they felt the CRMs did not have adequate technical and contextual knowledge to be effective supporting their work within the GOH counterparts. To promote information sharing between TAs, as well as to deal with administrative issues, the IP organized quarterly TA meetings. Generally, TAs expressed appreciation for these meetings. However, several (at least 3 TAs) thought that the gatherings could have been used more strategically. Rather than reporting what they had “We placed people who produced stuff, but it was not integrated into the institution. This negated the institutional capacity objectives of the project.” - IP informant “Sometimes we had the impression that we were considered a machine to produce deliverables and this made me angry. I produced a lot of documents, but I did not feel that they were used.” - TA informant 26| KONEKTE PERFORMANCE EVALUATION USAID.GOV done, these TAs thought the meeting should have focused on helping them to clarify objectives, coordinate their efforts, and work more effectively to ensure their efforts contributed to tangible outcomes within the GOH counterpart organization. GOH counterpart’s “absorptive capacity” limited capacity transfer The majority of KONEKTE TAs indicated they faced significant obstacles when it came to transferring knowledge and capacity to KONEKTE GOH counterpart s: • Although having a counterpart in the GOH counterpart office was a condition to receive KONEKTE assistance, in some cases, KONEKTE TAs had no meaningful counterpart, • In several cases, counterparts to whom TAs transferred knowledge and capacity were transferred or quit the GOH office during or just after the end of KONEKTE, • Regular government employees assigned to work with TAs lacked motivation due to low and/or late pay, weak oversight and management and insufficient qualifications and training relative to their assigned tasks, • GOH counterpart offices had inadequate material resources to carry- out office and officer duties, and • In some GOH offices, office and/or party politics affected the degree to which their TA efforts were capitalized by co-workers. Stakeholder views on extent TA methods facilitate d transfer of capacity and knowledge Most informants from OMRH said the methods used by some TAs were not sufficiently participative, which limited the transfer of knowledge and capacity. Two noted many TAs reported directly to the previous CG, who did not consult with his team before determining priorities. In addition to the GOH informants other KIIs (1 USAID, 2 IP, 1TA) likewise noted that TAs needed to be more effective at assessing needs, coaching co-workers, and generally promoting change. In 2016, IP informants reported that “coaching” training for TAs was introduced. The IP manager who led the initiative indicated that s/he did not believe his/her efforts had been very successful because it was too late in KONEKTE’s implementation to change the orientation of the Activity. Based on TA and GOH reports, there were significant differences in individual TA’s approach regarding capacity and knowledge transferal. Based on both TA and GOH informant accounts, KONEKTE TAs’ activities implemented under USAID’s education pillar integrated good practices and succeeded in transferring capacity to the MENFP. The senior TA reported, “I insisted that my team work with Ministry counterparts to enable them to do the work rather than doing the work in their place.” S/he also reported making extensive efforts to get institutional buy-in at both the national and local levels. USAID and GOH informants likewise reported that GOH personnel at various levels within the Ministry were actively involved in implementation. CONCLUSIONS Overall, KONEKTE TAs were not very effective transferring capacity and knowledge to GOH counterparts. KONEKTE TAs carried out a wide variety of activities within the GOH counterpart offices and produced most planned deliverables to the satisfaction of GOH counterparts, especially during its first three years of implementation. By their nature, many of these deliverables (guides, manuals, plans, training modules, etc.), held potential to contribute to transferring capacity and knowledge, but because of weak GOH ownership of TA work (often affected by political factors), especially within the OMRH following the departure of the original CG, the actual transfer of capacity and knowledge appears limited. There were, however, exceptions including KONEKTE TA work with the MENFP on improving the teaching of reading, and with OMRH and HRDs on recruitment and HR management (as highlighted in EQ2a findings). “One of the questions I asked TAs was did you produce this yourself or did you consult? Many of the documents that were produced were not produced with the proper authorities.” - IP informant 27| KONEKTE PERFORMANCE EVALUATION USAID.GOV KONEKTE TAs faced significant external obstacles to transferring capacity and knowledge to its GOH counterparts, but also made tactical errors. The limited absorptive capacity of GOH counterparts was a significant factor preventing KONEKTE assistance from contributing to large -scale institutional capacity improvements. However, insufficient support for TAs on assessing needs and coaching and weak oversight by the IP and USAID to promote the use of TAs’ deliverables also limited KONEKTE’s effectiveness at transferring knowledge and capacity. EQ3: HOW SUSTAINABLE ARE THE ACTIVITY’S INTER VENTIONS? To answer this question, the ET highlight ed the strategies used by KONEKTE to favor sustainability and describes the degree these strategies contributed to tangible outcomes that remain post Activity. The ET also lists examples of interventions that were not sustained and explanations for these outcomes. Finally, the ET examined whether there is evidence of political will to continue work initiated by KONEKTE, especially on civil service HR related reforms, as well as GOH counterpart willingness to collaborate with USAID. FINDINGS Extent that KONEKTE favored sustainability in its intervention strategies Key informants from all stakeholder groups identified some successful KONEKTE strategies favoring sustainability. These included: • Strengthening laws and regulations on public sector HR management to provide a framework for reforms, • Assisting GOH offices and units to define and implement new institutional mandates and attract/diversify funding for related activities, • Developing and testing manuals and procedural guidelines to direct GOH employees’ activities, and • Piloting an effort to recruit some TAs at the GOH salary scale with an explicit agreement that they would be recruited by OMRH. KONEKTE contributions to expanded legal frameworks KONEKTE TAs contributed to two draft laws regulating Haiti’s civil service administration. The first decree covers the procedures to recruit government employees and regulate employment conditions in the civil service. The second established a multi-stakeholder council to oversee civil service reforms. According to TA interviews and evidence in Haiti’s legal register, the first decre e was published during KONEKTE’s life and the other in the months following its closure. KONEKTE TAs also contributed to draft laws on devolution and decentralization which an OMRH official reported to be in the process of validation. One draft regulatory change contributed by a KONEKTE TA proposed limitations on scope of political appointments allowed by locally elected officials in local governance units. The TA who worked on the draft law, also developed and took part delivering a training program on decentralization for local government. KONEKTE contributions to better organized and funded GOH institutions 28| KONEKTE PERFORMANCE EVALUATION USAID.GOV Stakeholders in all categories (10 KI, four IP, two TA, three GOH, one other donor) indicated that OMRH is a stronger organization than when KONEKTE started. An OMRH official agreed, “KONEKTE helped the institution to gain prominence.” Based on direct observation and KIIs , OMRH is actively working with ministry HRD and is viewed by them as a credible counterpart . One HR director remarked, “OMRH is the government institution that deals with Human Resources Management, and when it comes to recruitment, we need to involve the OMRH.” According to the various informants cited above, OMRH currently benefits from significant donor support and is involved in managing numerous priority GOH administrative reforms. A KONEKTE TA worked with MARNDR to produce a five- year strategic plan for its recently created Department of Innovation. A MARNDR official reported using the document to restructure his Department as well as to attract funding from existing and new donor counterpart s. S/he likewise credited having a strategic document outlining planned activities, roles and responsibilities with the Department’s success attracting new national budget allocations. Under the education pillar, KONEKTE TAs contributed to establishing a unit for promoting better learning outcomes in reading, writing, and math, and trained unit personnel. During KONEKTE’s life, the unit was connected directly to the Prime Minister’s office and was associated with what was reported to be a very visible national campaign promoting reading in which the Haitian President participated. The unit also developed or updated guidelines and resource materials for public schools and other GOH institutions involved in teaching reading. Currently, the unit is relocated within the MENFP and is carrying out a high volume of activities with support from USAID and several other donors. KONEKTE manuals and guidelines contributions to new practices KONEKTE TAs working with OMRH developed manuals and guidelines on recruitment, restructuring HR departments, and performance evaluation. As outlined in EQ2a findings, HRD directors reported using these OMRH manuals and guidelines produced with KONEKTE support to guide recruitment procedures, the development of job descriptions, and performance evaluation processes. A MENFP official likewise reported that many of the instructor manuals and other resources developed by the unit for promoting better learning outcomes in reading, writing, and mathematics, which were produced with the KONEKTE team, are currently being used in public schools and other education programs in Haiti. KONEKTE efforts to recruit and train future civil service employees GOH officials and TAs highlighted that large differences in TA and regular GOH employees’ salaries discouraged TAs from staying on as regular GOH personnel after KONEKTE. Several TAs also reported that having a significantly superior salary to their co-workers created friction and limited collaboration during their tenure as advisors. Only one of 18 OMRH TAs is currently employed by OMRH even though several TAs reported being offered positions. To address salary imbalances between regular GOH personnel and TAs, starting in 2015, KONEKTE piloted an innovative effort to recruit OMRH TAs at the GOH salary scale and place them under a more senior KONEKTE TA to be trained. OMRH committed to recruit the individuals after the end of the first year’s contract. The purpose of the experiment was to fill vacancies in OMRH middle management positions with motivated and qualified individuals ready to accept Haitian civil service salary and working conditions. “There is a problem with the sustainability of salaries covered by donors. The problem is on the Haitian side. The government needs to offer more competitive salaries. I understand that OMRH is working on this. We can’t keep good people unless it is through a project but that does not contribute to sustainability.” - GOH official “When I started, OMRH had four to five employees. The Coordinator General was not even on the payroll – he was considered a consultant. They were housed in UNDP. By the time we left, the OMRH had its own office and many government employees. I won’t say we were the only one that contributed to this achievement, but we were a major contributor.” - IP informant 29| KONEKTE PERFORMANCE EVALUATION USAID.GOV The pilot initiative was a limited success. Of the four that participated in the experiment, two joined the Haitian civil service, but did not stay on long term with OMRH . One TA who remained in the civil service indicated that improving working conditions as well as salary is needed to retain qualified government employees. During KONEKTE, s/he indicated his/her work mainly followed a plan and a strategy and was results-driven while post- KONEKTE, his/her activities were largely driven by day-to-day needs. The former TA likewise was frustrated with his/her working environment, indicating personal initiative and team work were discouraged and it was hard “to make a difference.” During KONEKTE, the TA was trained on HR management by a senior OMRH KONEKTE TA, but is no longer working on HR matters in his/her new assignment. Many KONEKTE TA activities had short-term benefits that were not sustained GOH officials and TAs reported many short-term changes in GOH counterparts’ offices contributed by KONEKTE TAs that were not sustained post-Activity due to the end of KONEKTE TA contracts and other financial and technical constraints. Examples include the following: • Temporarily filling technical personnel gaps in the MOEF and the OMRH Departments of Information Systems (DIS). In the former case, the technical personnel were able to stay on as consultants after KONEKTE through support from the World Bank. In the latter case (OMRH), the capacity of the DIS increased to four to five technical staff during KONEKTE, but at the time of the evaluation, only one person remained. • Enabling the MAST personnel to monitor implementation of the USAID- funded Koré Lavi Activity. According to one MAST official, “The absence of a TA has created a great void, we cannot really have a concrete relationship with our donor-funded NGO counterparts anymore. They are doing everything now.” • Collecting data and reporting on the status of GOH efforts to comply with the United Nations Convention of the Rights of Persons with Disabilities for the State Secretary for the Integration of Persons with Handicap. With the departure of the KONEKTE advisor, the GOH was unable to respond to follow-up questions or present its record in an international meeting in Geneva. • Assisting the GOH Unit to Fight Corruption to establish a whistle-blower hotline to report cases of corruption. The hotline is no longer operational due to budget and technical constraints. • Collecting rainfall data from the Regional Observatories on Food Security that the Coordination of Food Security used to produce regular bulletins. The Coordination reports it has not been able to maintain the same data collection frequency since the departure of the KONEKTE advisor. Effect of KONEKTE’s end on OMRH’s will and ability to lead HR reforms Stakeholder views on how KONEKTE’s conclusion affected OMRH diverged. One OMRH official indicated that long term technical assistance creates dependence; for him/her, the end came at the right time. Even so, he confessed, “Honestly, we were not ready for the end of KONEKTE. We are trying to manage the end of donor support through strategic planning.” An IP manager also thought long term assistance in the conditions that prevailed within OMRH was not sustainable, “I think the whole strategy dreamed up by the GOH to place counterparts to our TAs was a pipe dream. Our TAs became the staff of OMRH. When the project shut down, OMRH lost staff.” However, an OMRH official indicated that the end of KONEKTE did not have a negative effect on OMRH because many TAs were not working on high priority objectives: “OMRH should have felt the effect of the end. But it did not. For me, the fact that we were able to continue, and we did not have to replace them (KONEKTE TAs), means that there was something wrong with the program.” He was referring to the many TA outputs that were never integrated into OMRH’s regular work. 30| KONEKTE PERFORMANCE EVALUATION USAID.GOV According to the latter official, KONEKTE would have been more effective had more effort been made to align TAs’ work with politically-driven priorities at the Prime Minister and/or President level. S/ he indicated that during the time of KONEKTE , “The political leaders did not take ownership of the reform and that had repercussions on our work.” S/he added, “USAID might have been able to be successful if it had used political pressure as well as technical pressure.” One former TA agreed “If, in the future, there is another KONEKTE project, there is the potential to produce results but only if the leaders have a clear vision of what they want to do, and OMRH officials share the same vision.” OMRH informants indicate there is currently political will within the current government to continue work on some KONEKTE- supported HR reforms with potentially better results because the current OMRH CG benefits from the President’s support. Effect of end of KONEKTE on USAID-GOH relationship Several GOH officials reported that the phase-out of KONEKTE was handled poorly. They explained that KONEKTE IP managers believed an extension was likely and led some GOH counterparts to believe that at least some TA contracts would be renewed for another year. IP man agers confirmed that they had received informal indications from USAID that the Activity would be extended and were surprised when it was not. USAID turned down the extension with little time remaining for implementation, which stakeholders (IP, GOH and TA) felt to be an “abrupt” end to KONEKTE support. Since the end of KONEKTE, the distance between USAID and some of the GOH partners has increased. One USAID technical office representative said, “I don’t think that the TA’s departure had any kind of immedi ate impact on the office s/he worked with. I do think that it had a negative impact on communication. We are missing that at the end of the program now.” A GOH representative underlined the importance of maintaining dialogue with USAID: “We can have a lot of resources without the dialogue we do not move forward. We have to sit down and talk frankly.” Despite this, most KONEKTE GOH counterparts indicated gratitude for USAID assistance and understood, like most forms of donor support, KONEKTE support would not go on forever. An OMRH official affirmed , “There is still a good relationship between USAID and OMRH.” CONCLUSIONS Some KONEKTE GOH count erpart institutions are currently able to sustain work started by KONEKTE TAs while many others are not. Institutional restructuring/strengthening work within OMRH, MENFP, and MARNRD contributed to organizing the institutions’ work and attracting government and other donor resources to support their activities. The positive outcomes of TA work in most other GOH institutions appears to have been short-term and did not lead to any lasting improvement in the host institutions’ capacity. Frequent changes in Haiti’s political landscape, overall weak governance capacity, and severely under-resourced government offices were significant obstacles to sustainability. KONEKTE interventions that are likely to have sustainable outcomes include introducing changes in the legal framework in support of on-going reforms, and creating tools and guidelines to be used by offices and institutions for their core activities. KONEKTE’s experiment recruiting middle managers on the government salary scale demonstrated that it is possible for GOH institutions to retain personnel recruited in this way, but does not provide much evidence of the value of doing so. KONEKTE TAs who integrated into the civil service were moved to other host institutions and expressed disappointment in the opportunities afforded them to contribute reforms outside the context of KONEKTE. There appears to be political will within the GOH to continue KONEKTE supported HR reforms. There appears to be political will and support for OMRH leadership within the current government to continue work in areas that were supported by KONEKTE and willingness by all GOH counterparts to collaborate with USAID in future programs. “After the end of KONEKTE, we continued to work with the DRH (Directeurs et Directrices des Ressources Humaines). We are now trying to develop the expertise of human resource managers. Now in the OMRH, we have strengthened our role in auditing the way hiring is done. We also review contracts.” - OMRH official 31| KONEKTE PERFORMANCE EVALUATION USAID.GOV IV. RECOMMENDATIONS FOR USAID 1. USAID should not use a stand-alone, HR out-sourcing mechanism like KONEKTE to fill gaps in its counterparts’ capacity in Haiti in the future. KONEKTE showed that such a mechanism is not a useful way to streamline assistance to fill critical HR vacancies in an emergency recovery situation or more generally. The mechanism did not appear to help to overcome administrative and management burdens within USAID, nor to save time. 2. USAID should continue to involve GOH counterpart s in the implementation of its future programs. Ways to do this may include seconding TAs in GOH counterpart offices through existing technical Activities (versus through a stand-alone mechanism), as well as other institutional capacity building interventions. KONEKTE demonstrated that it is possible to involve GOH counterpart s as implementers in USAID programs, in ways that enhance interventions’ sustainability. Even though KONEKTE showed there are significant obstacles to sustainably increasing GOH offices and officers’ capacity, not involving the GOH only reinforces weak governance. 3. USAID should consider additional support for OMRH on civil service administrative reforms. Among other topics, future programs should identify interventions to promote nondiscrimination in hiring. Future programs should explore work on policy and regulatory issues that affect the GOH ability to attract qualified candidates such as wage and professional advancement programs. 4. USAID should ensure that the future program agendas that cross-cut USAID technical offices are led/ monitored by senior USAID staff, if not the Mi ssion Director, to ensure coherence and buy-in from mission technical offices. To strengthen oversight and decision-making by technical offices, USAID should avoid using basket funding mechanisms except in situations where there is strong demand and similar objectives and needs across technical offices. 5. USAID should approve workplans, M&E frameworks, and process contract modifications in a timely manner to guide consistent decision- making by implementers, and avoid implementation preceding formal, written approvals. 32| KONEKTE PERFORMANCE EVALUATION USAID.GOV ANNEX A: EVALUATION STATEMENT OF WORK STATEMENT OF WORK Performance Evaluation of Workforce Augmentation Program “KONEKTE” I. PURPOSE OF THE EVALUATION The purpose of the evaluation is to inform USAID/Haiti’s and the Agency’s possible future similar workforce programming in fragile states. Lessons learned, and recommendations provided through the final report at the end of the evaluation process should guide future programming in the areas of governance and workforce development in Haiti or elsewhere. The primary stakeholders for this evaluation include USAID/Haiti, Management System International (MSI), Office de Management et des Resources Humaines (OMRH) and CFET. II. SUMMARY INFORMATION Strategy/Project/Activity Name Workforce Augmentation Program Implementer Management System International (MSI) Cooperative Agreement/Contract # AID – 521 – C – 13 - 00001 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $21,795,894.00 Life of Strategy, Project, or Activity 10/30/2012 to12/31/2017 Active Geographic Regions N/A Development Objective(s) (DOs) [Insert number and name of the DO that this evaluation relates to] USAID Office Governance Office III. BACKGROUND A. Description of the Problem, Development Hypothesis(es), and Theory of Change In the aftermath of the 2010 earthquake, the U.S. Government Strategy in Haiti was to follow the Government of Haiti’s Action Plan for National Recovery and Development and subsequent Plan Strategique de Developpementd’Haiti, and to complement the assistance of other donors. Challenges to rebuilding included pre-earthquake realities of poor governance, weak institutions and a culture of corruption. USG strategy aimed at engaging the private sector in the reconstruction process, stimulating the economic Activity, and enhancing the delivery of basic services in designated USG development corridors while promoting stronger democratic governance and rule of law. In implementing this strategy, the USG sought at the same time to build the capacity of state and local institutions to respond to local constituencies as a prerequisite to overall program sustainability. In this context, USG Project activities encompassed technical assistance, training and equipment and supported specific developmental objectives of GOH line ministries and local government entities targeting improved public service delivery and individual well- being. This included support to the Ministries of Health, Agriculture, Education, Finance, Commerce, Public Works, Justice, and the Haitian National Police. At the same time, NGOs and community-based cooperatives and associations were being strengthened and supported in the provision of services which might otherwise be delivered by government. The workforce development program was designed to be a complementary and important source of added expertise for accomplishment of the institutional strengthening objectives of the USG strategy. The program 33| KONEKTE PERFORMANCE EVALUATION USAID.GOV was viewed as a specialist advisory technical Assistance Program, which is in accordance with the USAID Forward objective of strengthening counterpart country capacity to improve aid effectiveness and sustainability. B. Summary Strategy/Project/Activity/Intervention to be evaluated The Contractor was to provide management, administrative, and technical services to alleviate current capacity constraints in select governmental, non-governmental and other private organizations of key importance to the implementation of the USG‟s Post-Earthquake Strategy for Renewal and Economic Opportunity in Haiti. The recruitment and placement of individual subject matter experts was expected to provide the planning, managerial and other subject matter skills needed for GOH follow- through on the Action Plan for National Recovery and Development and contribute to the implementation and achievement of mutually agreed objectives for sustained economic growth and development. The recruitment and placement of specialists was supposed to be demand-driven in the collaborative undertaking of joint USG and GOH development programming. The skill and qualification requirements, purpose, objective and expected deliverables, time frame and duration of each Specialist was supposed to be determined prior to recruitment and documented in an agreed scope of work for each specialist. In- country, locally- recruited Haitian expertise was to be the preferred source for of identified skill requirements. Only upon determination of lack of in-country availability, was out-sourcing to US and other third country nationals expected. Country-knowledge and French and Kreyol language skills may suggest the recruitment of Haitian diaspora when out-of-country sourcing was necessary. Entities receiving Specialists was supposed to be primarily those of the GOH and to a lesser extent local nongovernmental organizations (NGOs) and the USG Mission to Haiti. USAID/Port-au-Prince intended to procure short-term, intermittent technical assistance through the proposed contract for overall Strategy and program development and performance monitoring purposes. Similarly, although assistance to USAID‟s NGO counterparts was typically provided through ongoing development programs, there is evident need for added operational and managerial assistance. In no event should Specialists encumber GOH civil service positions. Specialists placed with USG entities shall not during the term of this contract perform any inherently governmental functions. USAID recognizes, however, that institutional capacity building as an expected result is not within the manageable capacity of this Workforce Augmentation Program. The provision of technical assistance to achieve the institutional development objective shall be necessary when applied, but not in itself sufficient. The expected result was that GOH and other IP capacity was strengthened for follow-through on the agreed objectives of the USG Strategy for Haiti. The project was amended in FY 2015 and the capacity building component of the project framework modified to include: 1) provision of software and equipment to the Office of Management and Human Resources (OMRH) and possibly other selected government entities toward fulfilling their mandate; 2) capacity development efforts for Haitian organization working in the area of management in HR and placement of technical experts within the Haitian Government so that at least one organization would be eligible to receive a direct award from USAID or other donors in the future. C. Summary of the Project/Activity Monitoring, Evaluation, and Learning (MEL) Plan The following documents and reports will be made available upon request: the project description and subsequent amendments, annual monitoring and evaluation plans, annual work plans, monthly and annual reports. IV. EVALUATION QUESTIONS In evaluating the Workforce Augmentation Program, the evaluators shall address the following evaluation questions: 1. To what extent was the original project design relevant to GOH concerns and needs? In answering this question, the evaluation should focus particularly on the extent to which the Activity design facilitated 34| KONEKTE PERFORMANCE EVALUATION USAID.GOV empowering GOH officials and offices to project their own HR needs and identify their own technical assistants. The evaluation should also explicitly examine these issues comparing GOH , IP and USAID perspectives and how they evolved throughout the life of the project. 2. How effectively was the Activity implemented and managed? To answer this question, the evaluation should examine the interaction between USAID technical offices, the COR and KONEKTE, how effectively the Activity managed GOH requests for technical assistance to ensure that it was appropriate and effectively used, including the extent to which the GOH was involved in the identification of the HR needs and selection of appropriate technical assistance to fill any gaps. In addition, it should specifically assess the two fundamental Activity elements: a. The extent to which Activity interventions actively build GOH capacity to identify and hire appropriate technical assistance. b. The extent to which technical assistance provided through the Activity transferred knowledge and capacity to GOH officials and offices. 3. How sustainable are the Activity’s interventions? The answer to this question should specifically address the implications of USAID’s decision not to extend the Activity beyond five years and the impact of this decision on USAID relationship with the GOH, and the GOH ’s willingness and ability to continue the Activity’s interventions in the absence of USAID support. V. EVALUATION DESIGN AND METHODOLOGY It is expected that the evaluation team proposes an overall research design to address the evaluation questions and a plan for collecting and analyzing the data. Nevertheless, given the nature of the evaluation questions and limited time available to plan and implement the evaluation, the evaluation design should be based solely on a combination of qualitative techniques to address the evaluation questions. Key informant interviews, focus group interviews, field observation, and in-depth review of projects reports are among the techniques that the evaluation team should consider when addressing the evaluation questions. VI. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work plan: Upon receipt of this Activity Request, Social Impact (SI) shall submit within two weeks a draft work plan to the Contracting Officer’s Representative (COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; (2) a list of the members of the evaluation team, delineated by roles and responsibilities with their level of effort;(3) the identification of other required personnel and relevant local subcontractors, their LOE, roles and responsibilities and qualifications; and (4) the deliverable schedule. 2. Inception Report: The evaluation team will have two weeks to review the project documents and produce an inception report or background report that addresses what the evaluation team has learned based on program documents provided to them. The inception report should inform the design of the evaluation. Therefore, during this phase period, USAID/Haiti should consider the possibility of revising evaluation questions based on evaluation team input. Any revisions to the questions in the SOW should be documented in writing in the evaluation report. 3. Evaluation Design: Within four weeks of approval of the work plan, SI must submit to Contracting Officer’s Representative (COR) an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited; (4) known limitations to the evaluation design; and (5) a dissemination plan. USAID/Haiti will take up to 10 business days to review and consolidate comments through the COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 business days. 35| KONEKTE PERFORMANCE EVALUATION USAID.GOV 4. In-briefing: Prior undertaking field work, the evaluation team will have an in-briefing with the USAID/Governance Team and the Evaluation and Survey Services (ESS) COR to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and to clarify any questions or logistic needs. 5. Evaluation Briefing/Presentation: The evaluation team is expected to hold a final presentation in person to discuss the summary of findings and recommendations to USAID within 20 business days after the conclusion of fieldwork. 6. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format . The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, USAID/Haiti will have 0 working business days in which to review and comment on the initial draft, after which point the ESS COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit revised final draft report 5 business days hence, and again USAID/Haiti will review and send comments on this final draft report within 5 business days of its submission. 7. Final Evaluation Report: The evaluation team will be asked to take no more than 15 business days to respond/incorporate the final comments from the Governance Office. The evaluation team leader will then submit the final report to the COR. All project data and records (FGD and KII summary reports) shall be submitted in full and should be in electronic form in easily readable format, organized, and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. VII. EVALUATION TEAM COMPOSITION The Evaluation Team shall be comprised of two Key Personnel positions: (i) a Team Leader, and ii) a specialist in Governance. The Team Leader (TL) is ultimately responsible for the overall management of the evaluation team, coordinating the implementation of the evaluation, assigning evaluation responsibilities and tasks, and authoring the final evaluation report in conformity with this Statement of Work. The TL must be an experienced evaluation expert, with a documented track record of 10 years of experience in the field of evaluation. S/he should have a strong background in the governance and/or public administration field. S/he should be fluent in French and English. S/he should have at least a Master’s Degree in Economics, Statistics, Political Science, or Public Administration. The Governance Specialist will help the TL in the overall management of the evaluation team and the final products, in conformity with this Statement of Work. S/he should be familiar with the Haitian administration. S/he must possess excellent writing and interpersonal skills and must be familiar with USAID programs, objectives, and reporting requirements. S/he should have experience in designing and implementing FGDs. Fluency in French is required. English and Haitian Creole are highly desirable, as is significant prior work experience in Haiti. A Bachelor's degree in Political Science, Public Administration, or a related field is required to ensure that all areas of technical expertise required for the evaluation are effectively covered. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. 36| KONEKTE PERFORMANCE EVALUATION USAID.GOV VIII. EVALUATION SCHEDULE Timing (Anticipate d Weeks or Duration) Scheduled activities 6 Weeks Preparation of the work plan, inception report and evaluation design 3 Weeks USAID review of the work plan, inception report and evaluation design 1 Week Submission of the revised evaluation design, inception report and work plan; in-briefing 3 Weeks Data collection 2 Weeks Data analysis and Evaluation Briefing 2 Weeks Draft Report writing 2 Weeks USAID review of Draft Report 2 Weeks Incorporate USAID comments and prepare Final Report IX. FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2– 5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information properly identified and listed; and • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. X. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 1 • Evaluation reports should represent a thoughtful, well-researched, and well- organized effort to 37| KONEKTE PERFORMANCE EVALUATION USAID.GOV objectively evaluate the strategy, project, or Activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person- level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. 38| KONEKTE PERFORMANCE EVALUATION USAID.GOV ANNEX B: REFERENCES AND ENDNOTES Annual Reports, KONEKTE Background Documents USAID/Haiti. KONEKTE Program Year I Annual Progress Report. N.p.: USAID, 2013. Print. USAID/Haiti. KONEKTE Program Year 2 Annual Progress Report. N.p.: USAID, 2014. Print. USAID/Haiti. KONEKTE Program Year 3 Annual Progress Report. N.p.: USAID, 2015. Print. USAID/Haiti. KONEKTE Program Year 4 Annual Progress Report. N.p.: USAID, 2016. Print. USAID/Haiti. KONEKTE Program Year 5 Annual Progress Report. N.p.: USAID, 2017. Print. Annual Work Plans, KONEKTE Background Documents USAID/Haiti. Draft Annual Workplan for the KONEKTE Program. N.p.: USAID, 2012. Print. USAID/Haiti. Haiti Workforce Augmentation Program Year 2 Annual Work Plan. N.p.: USAID, 2013. Print. USAID/Haiti. Haiti Workforce Augmentation Program Year Two Annual Work Plan. N.p.: USAID, 2015. Print. USAID/Haiti. Haiti Workforce Augmentation Program Work Plan Fiscal Year 2016. N.p.: USAID, 2016. Print. USAID/Haiti. Haiti Workforce Augmentation Program Work Plan Fiscal Year 2017. N.p.: USAID, 2016. Print. Monitoring & Evaluation Plans, KONEKTE Background Documents USAID/Haiti. Haiti Workforce Augmentation Program Project Monitoring Plan. N.p.: USAID, 2013. Print USAID/Haiti. KONEKTE Program Fiscal Year 2014 Draft Project Monitoring and Evaluation Plan. N.p.: USAID, 2014. Print. USAID/Haiti. KONEKTE Activity Monitoring & Evaluation Plan (AMEP) Draft for Discussion. N.p.: USAID, 2016. Print. USAID/Haiti. KONEKTE Activity Monitoring & Evaluation Plan (AMEP). N.p.: USAID, 2016. Print. USAID/Haiti. KONEKTE Fiscal Year 2017 Activity, Monitoring, Learning and Evaluation Plan. N.p.: USAID, 2017. Print. Project Description & Modifications, KONEKTE Background Documents USAID/Haiti. AID-521-C-13-00001 Section C - Statement of Work. N.p.: USAID, n.d. Print. USAID/Haiti. AID-521-C-13-00001 Modification 4. N.p.: USAID, n.d. Print. Journals and Publications Brière, S., Jobert, S., & Poulin, Y. (2010). Enhancing public governance in fragile states: Support for Haiti. International Journal, 65(3), 653-667. Retrievedfromhttp://www.jstor.org/stable/25762023 Gouvernement De La République d’Haïti. Plan Stratégique d e Développement d’Haïti. May 2012. Retrieved from: http://www.ht.undp.org/content/dam/haiti/docs/Gouvernance%20d%C3%A9mocratique%20et% 20etat%20de%20droit/UNDP_HT_PLAN%20STRAT%C3%89GIQUE%20de%20developpement %20Haiti_tome1.pdf 39| KONEKTE PERFORMANCE EVALUATION USAID.GOV United States Government Accountability Office. GAO-15-517, Haiti Reconstruction: USAID Has Achieved Mixed Results and Should Enhance Sustainability Planning.June 2015. Retrieved from: https://www.gao.gov/assets/680/670616.pdf USAID/Haiti. Post-Earthquake USG Haiti Strategy - Programs/Activities List. Retrieved from: https://www.state.gov/documents/organization/251427.pdf Weisenfeld, Paul E.Successes and Challenges of the Haiti Earthquake Response: The Experience of USAID. Emory International Law Review. Retrieved from http://law.emory.edu/eilr/content/volume-25/issue-3/symposium/haiti-earthquake-experience- usaid.html 40| KONEKTE PERFORMANCE EVALUATION USAID.GOV ANNEX C: EVALUATION TEAM MEMBERS Team Leader: Ms. Sandra Wark is an evaluation specialist with extensive experience in democracy and governance, capacity building, institutional strengthening and human development (including specialized work on international labor rights and child labor programming). Ms. Wark has 25 years of experience designing, implementing and evaluating programs that use integrated development approaches and include components on national and local government counterpart capacity building. Ms. Wark has served as a team leader for USAID and other donor-funded evaluations in Africa, Asia and the Caribbean. She recently completed two USAID-funded, mid-term performance evaluations: the USAID Career Center Activity in Morocco (March-May 2018) and the Workers’ Empowerment Program in Bangladesh (October-December 2017), for which she used mainly qualitative methods to assess project outcomes and effectiveness. In January 2018, Ms. Wark was principle evaluator of a US Department of Labor (USDOL) funded program to combat child labor in agriculture in northern Haiti. She previously evaluated another USDOL-funded, International Labor Organization funded child labor program in Haiti that was mainly focused on building the capacity of the Ministry of Social Affairs and Labor to provide social protection services to vulnerable children. Ms. Wark is a graduate of Georgetown University with a Master of Science in Foreign Service, as well as a graduate of Whitworth College with a Bachelor’s in International Studies. She is a US citizen but resides in Morocco. Ms. Wark is fluent in English and French. Assistant Team Leader: Mr. Gary Paul has over 24 years as a professional Technical Assistant experience in the fields of project development, capacity building (especially for MARNDR, MSPP, MPCE, MEF), public politics development, Food and Nutritional Security (SAN in French), support to SAN governance in all of its components (health, nutrition and agricultural production). He has provided support to the Haitian government in the subject of reform/strategy interventions to the administrative, territorial and financial governance. He has more than 10 years of experience providing technical assistance in EU programs (conception, management, monitoring and Evaluation, services tenders at different levels) in the food and nutritional security/agriculture sector, urban planning and regional development, capacity building, environment. Mr. Gary has extensive knowledge of the various ministries among which the MARNDR, the MSPP, the MCI, the MEF and the MPCE. He also has good knowledge in local development experiences and decentralization, fight against poverty, urban planning, economy of natural resources and the environment. Position Responsibilities Sandy Wark Team Leader/Evaluation Specialist • Lead the evaluation, including desk review, data collection and analysis, and report writing. • Manage the team, including delegating responsibilities, training and guiding team members, monitoring progress, and providing feedback on inputs. • Serve as the primary liaison with USAID/Haiti’s Democracy and Governance Office, facilitate briefings, and lead presentations. Gary Paul Assistant Team Leader • Leverage sectoral and contextual expertise in the design of the evaluation and data collection instruments. • Participate in desk review, data collection, and analysis. • Lead a sub-team during fieldwork. 41| KONEKTE PERFORMANCE EVALUATION USAID.GOV Position Responsibilities • Contribute to the production of high-quality deliverables 42| KONEKTE PERFORMANCE EVALUATION USAID.GOV ANNEX D: EVALUATION DESIGN MATRIX Analysis Plan Sample Questions Data Sources Data Collection Method EQ • Content analysis • Trend analysis • Gap analysis • Comparative analysis • Gender analysis To what extent did the services provided by KONEKTE meet GOH institution’s expectations/needs? What were the factors that contributed to and/or limited KONEKTE from meeting GOH expectations? To what extent did KONEKTE consult and engage GOH counterparts at various stages of Activity implementation? To what extent did KONEKTE adjust its strategies in response to stakeholder feedback? To what extent did the scope of the program meet GOH needs and expectations (the types of work that TAs could be assigned to perform, types of experts that could be employed, length of assistance, complementary institutional capacity building support offered, etc.) What, if anything, might have been done to improve the program design? • USAID KONEKTE COR and Technical Office personnel • Management Systems International KONEKTE personnel • CFET KONEKTE personnel • KONEKTE Technical Advisors • OMRH • Other GOH TA recipients • Other donors, international organizations or INGOs • Activity documentation • USAID strategy/other program documents • GOH strategy documents Document Review KIIs FGD EQ1: To what extent was the original project design relevant to the Government of Haiti (GOH) concerns and needs? • Content analysis • Trend analysis • Gap analysis What were management strengths? Weaknesses? What, if anything, might have been done better? Were there significant contextual factors (political, economic, social, and institutional) that affected KONEKTE implementation? • USAID KONEKTE COR and Technical Office personnel • Management Systems International KONEKTE personnel Document Review KIIs (up to 1.5 hours) EQ2: How effectively was the Activity implemented and managed? 43| KONEKTE PERFORMANCE EVALUATION USAID.GOV Analysis Plan Sample Questions Data Sources Data Collection Method EQ • Comparative analysis • Gender analysis To what extent were the roles and responsibilities of different stakeholders involved in KONEKTE recruitment process well defined and clear to those involved? To what extent were adjustments made to improve management efficiency during the life of the Activity? a.) The extent to which Activity interventions actively built GOH capacity to identify and hire appropriate technical assistance. What were the main changes associated KONEKTE activities in GOH recipient procedures, practices, and working environment/infrastructure? What, if anything, changed concerning the role played by OMRH in improving/standardizing HR practices across government? What were the strengths and weaknesses of the various activities/approaches? What, if anything, might have been done differently or what else might have been done? b.) Extent to which technical assistance provided through the Activity transferred knowledge and capacity to GOH officials and offices. To what extent were the TAs able to achieve the outputs in their SOW? To what extent were the outcomes of TA assistance what was expected? To what extent did GOH recipients capitalize on the technical advisory services provided by KONEKTE? What factors helped or hindered the transfer of knowledge and capacity to GOH officials and offices? • CFET KONEKTE personnel • KONEKTE Technical Advisors • OMRH • Other GOH TA recipients • GOH participants in Human Resource Forums • KONEKTE reports and M & E data FGDs (up to 2 hours) Direct Observation 44| KONEKTE PERFORMANCE EVALUATION USAID.GOV Analysis Plan Sample Questions Data Sources Data Collection Method EQ • Content analysis • Trend analysis • Gap analysis • Comparative analysis • Gender analysis To what extent have GOH institutions carried on/completed/developed KONEKTE supported initiatives or practices? To what extent and how did KONEKTE build the capacity of its GOH to sustain activities beyond the conclusion of KONEKTE? To what extent did it build the capacity of its local counterpart (CFET) to be a viable GOH and/or donor direct contractor following the conclusion of KONEKTE? What are the most significant obstacles/constraints affecting sustainability? To what extent do the initiatives supported by KONEKTE remain a priority of the GOH institutions that received support? How has the end of KONEKTE affected GOH counterparts? Has it affected GOH perceptions of USAID/Haiti? • USAID KONEKTE COR and Technical Office personnel • Management Systems International KONEKTE personnel • CFET KONEKTE personnel • KONEKTE Technical Advisors • OMRH • Other GOH TA recipients • Other donors, international organizations or INGOs • Activity reports • GOH documentation Document Review KIIs (up to 1.5 hours) FGDs (up to 2 hours) Direct Observation EQ3: How sustainable are the Activity’s interventions? 45| KONEKTE PERFORMANCE EVALUATION USAID.GOV ANNEX E: LIST OF PERSONS INTERVIEW ED USAID Contract Officer’s Representative • Alphonse Nkunzimana, USAID COR, Democracy & Governance Office, male • Rebecca Jones USAID COR, Program Office, female • Hubert Sylney USAID COR, Program Office, male USAID Technical Office Representatives • Dana Stinson, Food for Peace Office Director, USAID, Female • Fabiola Lopez, USAID Education Officer MSI • Genevieve Bonny, MSI Program Manager, female • Marceau Jude Edouard, Jr., MSI Chief of Party, male • Annemiek van Rooyen, MSI Chief of Party, female • Kathy Shelton MSI Director of Operations, female • Suzanne Bond MSI Technical Director, female • Jacques Berard, MSI Technical Director, male • Brian Calhoon, MSI, Home office technical support, male CFET • Marlene Gay, CFET Executive Director, female • Elsie Depeignes, CFET Client Relations Manager, female • Germain Sherley, CFET, (during life of Activity) HICD expert, female • Myrta Eutache, CFET, Director of Program Implementation, female • Peterly Riche, CFET Client Relations Manager, male • Nadine Boniface CFET Communication Manager, female Government of Haiti • Harmel Cazeau, Coordinator, National Coordination on Food Security, Ministry of Agriculture, male • Antoine Verdier, OMRH Deputy Coordinator (General Coordinator, 2016- 2017), male • Henri Boucicaut, Coordinator of Civil Service, OMRH (former technical advisor) male 46| KONEKTE PERFORMANCE EVALUATION USAID.GOV • Uder Antoine, OMRH General Coordinator (2012- 2016), male • Donald Marcelin, Technical Advisor and Information Systems Director, Ministry of Economy, male • Garry Augustin, Direction of Innovation, Ministry of Agriculture, male • Marie Ange Pierre, HRD Director, Ministry of Economy, female • Jean Wilnor Pierre, Coordinator UDCLEM/Ministry of Education, male • Jean Odney Ricot, M&E unit director, Ministry of Social Affairs and Labor, male • Jose Joseph, Human Resources Director, Ministry of Education and Vocational Training • Maurice Aurelus, Human Resource Director, Ministry of Environment, male • Dr Michaelle E. Laforest, Human Resources Director, Ministry of Public Health, female • Elie Jean Philippe, OMRH Coordinator of Public Administration, male KONEKTE Technical Advisors • Lemane Delva, Technical Advisor, Department of Innovation, Ministry of Agriculture, male • Christel Gaboriau, Technical Advisor, OMRH, male • Sandra Jean Gilles, Technical Advisor, Ministry of Social Affairs and Labor, female • Odnel Eleazard, Technical Advisor, Office of the State Secretary for the Integration of Persons with Handicaps, male • Thalès Fleur-Aimé, Technical Advisor, OMRH, male • Yverose Francois, Technical Advisor, OMRH, female • Nahum Lafleur Technical Advisor, Ministry of Education and Vocational Training, male • Pierre Joseph, Technical Advisor, Ministry of Economy, m ale • Carly Bajat, Technical Advisor, Ministry of Economy, male • Pierre Bob Charlemagne, Technical Advisor, Ministry of Economy, m ale • Luc Derival, Technical Advisor, Information Systems Department/Ministry of Economy, male • Dener Francois, OMRH Technical Advisor, male • Nicolas Pierre Maxime, OMRH Technical Advisor, male • Gerry Delphan Technical Advisor, National Coordinator for Food Security, Ministry of Agriculture, male • Narcisse Fievre Technical Advisor, Ministry of Education and Vocational Training, male • Jean Robert Brutus, Technical Advisor, Ministry of Social Affairs and Labor, male • Daniel Silva, Technical Advisor, Anti- corruption Unit, Ministry of Economy, male • Chantal Roques, Technical Advisor, Ministry of Education and Vocation Training, female 47| KONEKTE PERFORMANCE EVALUATION USAID.GOV Other USAID Activity and Donor Representatives • Jerome Pennec, Cooperation Officer, Embassy of France, male • Philippe Canal, former Chief of Party USAID Haiti Integrated Financial Management Systems (IFMS), Senior Expert EU state building, male 48| KONEKTE PERFORMANCE EVALUATION USAID.GOV ANNEX F: INFORMED CONSENT SCRIPTS & DATA COLLECTION PROTOCOLS Informed Consent Agreement - KII Purpose: Thank you for taking the time to meet with us today. My name is [NAME ]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Haiti to conduct a study about the work of a USAID/Haiti project known as the Workforce Augmentation Program or KONEKTE for short. Implemented between 2012 and 2017 by Management Systems International (MSI) and its local counterpart Centre de Formation et d’Encadrement Technique (CFET), KONEKTE recruited and placed advisers in 11 GOH institutions. KONEKTE also provided institutional capacity building assistance to the government of Haiti’s (GOH’s) Office of Management and Human Resources (OMRH) and other GOH Human Resource (HR) departments to strengthen human resource management practices and support on-going reforms of the Haitian civil service. You have been asked to participate today so that we can learn more about the support your institution received from KONEKTE/your involvement in the implementation of the KONEKTE program/your experiences as a technical advisor. We are speaking with about 50 individuals who participated in the program either as implementers or recipients of program services. We would like your honest impressions, opinions and thoughts about various issues related to this Activity’s implementation and outcomes. We are independent consultants who have no affiliation with those who implemented KONEKTE nor do we represent the GOH . Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities and services implemented under the KONEKTE program. The interview will take about 1 hour of your time. Although we will publish our findings in a public report, all of your answers will be kept confidential. Nothing you tell us will be attributed to any individual person. Rather the report will include only a composite of all of the answers received by all of the individuals we interview. Although we may use quotes, none of the individuals interviewed will be named in the report. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future institutional capacity building programs for government institutions in Haiti. Voluntary Participation: Participation in this interview is completely volun tary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the interview as outlined above and understand that I can withdraw from the interview at any time and for any reason. I agree to participate in the interview (Evaluator records). Yes  No Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ 49| KONEKTE PERFORMANCE EVALUATION USAID.GOV Informed Consent Agreement - FGD Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Haiti to conduct a study about the work of a USAID/Haiti project known as the Workforce Augmentation Program or KONEKTE for short. Implemented between 2012 and 2017 by Management Systems International (MSI) and its local counterpart Centre de Formation et d’Encadrement Technique (CFET), KONEKTE recruited and placed advisers in 11 GOH institutions. KONEKTE also provided institutional capacity building assistance to the government of Haiti’s (GOH’s) Office of Management and Human Resources (OMRH) and other GOH Human Resource (HR) departments to strengthen human resource management practices and support on-going reforms of the Haitian civil service.You have been invited to participate in this discussion because you participated in at least one KONEKTE Activity for GOH personnel involved in Human Resource (HR) management. We would like your opinions and thoughts about the KONEKTE Activity or activities that you attended. We are independent consultants who have no affiliation with KONEKTE or the GOH. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities and services implemented under the KONEKTE program. The FGD will take up to 2 hours of your time. Although we will publish our findings in a public report, all your answers will be kept confidential. Nothing you tell us will be attributed to any individual person. Although we may use quotes, none of the individuals interviewed will be named in the report. However, as this is a group setting, to preserve confidentiality, we ask you not to share anything we discuss here today with anyone outside of this group. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future institutional capacity building programs for government institutions in Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to leave the FGD at any time or to decline to answer any question which you do not wish to answer. If you decline to participate, no one will be informed of this. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the interview as outlined above and understand that I can withdraw from the interview at any time and for any reason. I agree to participate in the interview (Evaluator records). Yes  No Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ 50| KONEKTE PERFORMANCE EVALUATION USAID.GOV Protocol for KIIs with GOH Recipients of Technical Advisors A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title: 7. Respondent Institution: 8. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. KII Protocol for GOH Recipients of Technical Advisors (TAs) 1. Please provide an overview of the support you received from KONEKTE. (EQ1, EQ2) 2. To what extent did the services provided by KONEKTE meet your institution’s expectations/needs? What were the factors that contributed to and/or limited KONEKTE from meeting your expectations? (EQ1, EQ2) Probe: Were you satisfied with KONEKTE responsiveness to your requests for TA? Were you satisfied with the work performed by KONEKTE TAs? Was the support provided by KONEKTE for TA orientation and monitoring sufficient? 3. What processes were followed by KONEKTE for determining and elaborating technical advisor’s scope of work and skills profile, identifying and screening candidates, and making the final selection? Were you satisfied with the outcomes of the recruitment process? (EQ1, EQ2) Probe: What were the strengths and weaknesses of the process for formulating and validating GOH requests? To what extent was the GOH involved in determining its own human resources needs? To what extent did the process result in requests that were relevant to GOH needs and concerns and met USAID eligibility criteria? What, if anything, might have been done better? What were the strengths and weaknesses of the recruitment process in terms of identifying and matching TA profiles and skills with GOH needs? What, if anything, might have been done better? 51| KONEKTE PERFORMANCE EVALUATION USAID.GOV 4. What were the most significant outcomes/achievements of KONEKTE for your institution? In which areas, if any, were results less than desired? Were there any unintended outcomes? (EQ2, EQ3) Probe: To what extent was your institution able to capitalize on the work performed by TAs? Why or why not? Were some technical advisors better suited than others to carry out their assignments? What were the factors affecting KONEKTE’s ability to match advisors to GOH needs? Were some assignments better designed than other? What were important characteristics of a good advisory service SOW? 5. In terms of the overall design of KONEKTE assistance, to what extent did the scope of the program meet your needs and expectations (the types of work that TAs could be assigned to perform, types of experts that could be employed, length of assistance, complementary institutional capacity building support offered, etc.) What, if anything, might have been done to improve the program design? (EQ1) Probe: Was your institution consulted in the design stage of KONEKTE? Were any adjustments made to the scope of the program during the life of the program based on stakeholder feedback? What was the effect of these adjustments? 6. To what extent was the assistance offered through KONEKTE similar or different to the types of assistance offered by other donors/development counterparts to GOH counterpart institutions during the life of the Activity? (EQ1, EQ2) To what extent did the Activity fill gaps and/or complement what other GOH counterparts’ assistance programs? 7. To what extent has assistance provided by KONEKTE to your Institution contributed to your capacity to carry out effective and fair recruitment process? To what extent has KONEKTE assistance changed or influenced your institutions HR management policies and practices? What factors have enabled/constrained progress in this area? (EQ1, EQ3) Probe: Has KONEKTE affected your ability to assess short-term or long-term personnel needs and requirements? How? Identify qualified candidates? How? 8. To what extent has your counterpart ship with KONEKTE contributed to building your awareness and capacity to integrate gender/nondiscrimination considerations into your HR policies? To what extent were gender considerations (addressing the particular needs/concerns of women, offering equitable access to opportunities) integrated in KONEKTE services or implementation? What, if anything, could have been done better? (Cross Cutting) 9. Do you think that KONEKTE has been effectively managed? Do you have any suggestions for how management could be improved? (EQ2) Probe: Was the level of communication between your institution and KONEKTE personnel adequate ? Were the roles and responsibilities for project implementation clearly defined? Were activities carried 52| KONEKTE PERFORMANCE EVALUATION USAID.GOV out in a timely manner ? To what extent were Activity managers able to adapt and find solutions to unexpected challenges? 10. To what extent do you think that the results of KONEKTE assistance will be sustained now that the program is closed? What are factors that contribute to and/or limit sustainability? (EQ3) Probe: To what extent do you think measures taken by KONEKTE to foster sustainability were adequate? Was there adequate capacity building of permanent staff? Consideration of maintenance needs and costs of interventions? Sufficient documentation of efforts left behind? What more, if anything, could have been done to ensure the sustainability of KONEKTE results? 11. How has the end of KONEKTE assistance affected your institution? Has it/how has it affected your perception of USAID/Haiti? (EQ3) Protocol for KIIs with KONEKTE Technical Advisors A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title (during life of Activity): 7. Respondent Institution (during life of Activity): 8. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ B. KII Protocol for KONEKTE Technical Advisors (TAs) 1. Please describe your role in KONEKTE. Please provide an overview of your scope of work within KONEKTE? (EQ2) 2. To what extent do you believe the services you provided as a KONEKTE TA met your host institution’s expectations/needs? What were the factors that contributed to and/or limited your ability to meet their expectations? (EQ1, EQ2) 53| KONEKTE PERFORMANCE EVALUATION USAID.GOV Probe: Was the scope of work that you were provided realistic? Did it match with what your host institution expected from you? Was the support provided by KONEKTE for TA orientation sufficient? Were there contextual factors within the institution or the country that affected your capacity to carry out your assignment? What were they? To what extent was the support provided by KONEKTE to identify and resolve issues affecting your capacity to carry out your assignment adequate? What more, if anything, could have been done to facilitate your work by KONEKTE? 3. Please describe the process that KONEKTE followed for your recruitment? How was it different or similar to previous recruitment experiences that you may have had in Haiti or elsewhere? (EQ1, EQ2) Probe: What, if anything, might have been done better? 4. What do you think were your most significant achievements as a KONEKTE TA? In which areas, if any, were results less than desired? Were there any unintended outcomes?(EQ2, EQ3) Probe: To what extent do you think your institution has been able to capitalize on the work you performed? To what extent was the GOH involved in your recruitment? What were the factors that contributed to and/or limited your host institution’s capacity to capitalize on your work? 5. In terms of the overall design of KONEKTE assistance, to what extent do you think the scope of the program was adequate to contribute meaningfully to bridging GOH knowledge and skill gaps? What, if anything, might have been done to improve the program design? (EQ1) Probe: Were any adjustments made to the scope of the program during the implementation period based on stakeholder feedback? What was the effect of these adjustments? 6. To what extent was the assistance offered through KONEKTE similar or different to the types of assistance offered by other donors/development counterpart s to GOH counterpart institutions during the life of the Activity? Probe: To what extent did the Activity fill gaps and/or complement other GOH counterparts’ assistance programs? 7. To what extent do you think that assistance provided by KONEKTE contributed to your host institution’s capacity to carry out an effective and fair recruitment process? To manage short and long-term personnel effectively? What factors have enabled/constrained progress in this area? (EQ1, EQ3) 8. To what extent do you think KONEKTE contributed to building awareness and capacity to integrate gender/nondiscrimination considerations into GOH HR policies? To what extent were gender considerations (addressing the particular needs/concerns of women, offering equitable access to opportunities) integrated in KONEKTE services or implementation? What, if anything, could have been done better? (Cross Cutting) 54| KONEKTE PERFORMANCE EVALUATION USAID.GOV Probe: Did KONEKTE contribute to increasing the number of female technical advisors? Greater attention being paid to HR issues affecting women such as discrimination, sexual harassment, access to maternity leave, child care, etc. 9. Do you think that KONEKTE has been effectively managed? Do you have any suggestions for how management could be improved? (EQ2) Probe: Was the level of communication with you adequate? Between your institution and KONEKTE? Were the roles and responsibilities for project implementation clearly defined? Were activities carried out in a timely manner? To what extent were Activity managers able to adapt and find solutions to unexpected challenges? 10. To what extent do you think that the results of KONEKTE assistance will be sustained by your host institution now that the program is closed? What are factors that contribute to and/or limit sustainability? (EQ3) Probe: To what extent do you think measures taken by KONEKTE to foster sustainability were adequate? Was there adequate capacity building of permanent staff? Consideration of maintenance needs and costs of interventions ? Sufficient documentation of efforts left behind? What more, if anything, could have been done to ensure the sustainability of KONEKTE results? 11. How do you think the end of KONEKTE assistance affected your institution? (EQ3) Protocol for KIIs with representatives of other Donors/International Organizations/INGOs Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title (during life of Activity): 7. Respondent Institution (during life of Activity): 8. Sex of respondent: A. KII Protocol for representatives of other Donors/International Organizations/INGOs 1. Please describe the role/mission of your organization and your main activities in Haiti. 55| KONEKTE PERFORMANCE EVALUATION USAID.GOV 2. Please describe your involvement (if any) with KONEKTE. Please characterize how informed you are about KONEKTE activities. Probe: Have you participated in any of the activities i.e. training, meeting, workshop, or events organized by KONEKTE? Did you interact with KONEKTE personnel and/or its main GOH counterparts during your work in Haiti? 3. What is your perception of the work done by KONEKTE? (EQ2) Probe: Do you think the type of work performed by TA is relevant to the needs of GOH institutions)? Do you think it has changed the institution’s ability to fulfill it mandated functions? 4. To what extent did the work of KONEKTE coordinate with or complement the work of your organization? (EQ1, EQ2) 5. What were the main factors that enabled/constrained KONEKTE’s efforts to improve GOH institutional capacity in your areas of work? (EQ2) Protocol for KII with USAID COR and Technical Office Representatives A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title (during life of Activity): 7. Respondent Institution (during life of Activity): 8. Sex of respondent: B. KII questions for USAID 1. To what extent was KONEKTE an effective mechanism for responding to skills and knowledge gaps within GOH counterpart institutions? What, if anything, might have been done to improve the program design? (EQ1, EQ2) 2. To what extent did the Activity respond to GOH counterparts’ priority concerns and needs? What were the main strategies used by KONEKTE to consult and engage GOH counterparts at various stages of Activity implementation (design, phase one technical advisory services, phase two institutional capacity building)? (EQ1, EQ2) 56| KONEKTE PERFORMANCE EVALUATION USAID.GOV Probe: Were GOH counterparts consulted in the design stage of KONEKTE? Were any adjustments made to the scope of the program during the life of the program based on stakeholder feedback? What was the effect of these adjustments? What more, if anything, might have been done to ensure the relevance of KONEKTE to GOH needs and concerns? 3. Which KONEKTE activities do you consider particularly successful/unsuccessful? Why? (EQ2) Probe: To what extent was the process that KONEKTE put in place to recruit TAs effective in helping GOH identify their own technical assistance needs? In matching TA profiles and skills to the identified needs? To what extent was KONEKTE effective in orienting and then monitoring/supporting TAs once they were on the job? Assessing institutional capacity needs? In building the capacity of OMRH and other GOH HR departments on HR management? 4. What unintended outcomes did you see, if any? (Cross Cutting) 5. What factors (external or internal) best facilitated KONEKTE’s achievement of the Activity goals and objectives? Which factors (external or internal) most constrained KONEKTE’s achievement of the Activity goals and objectives? (EQ1, EQ2) Probe: To what extent did external factors within counterpart institutions or in the general political and economic environment affect KONEKTE? To what extent did internal factors such as the distribution of roles and responsibilities between USAID, the implementor and GOH affect KONEKTE? 6. To what extent do you think that KONEKTE was effectively managed? What were management strengths? Weaknesses? What, if anything, might have been done better? (EQ2) Probe: Was the level of communication with you/your office adequate? Were the roles and responsibilities for project implementation clearly defined? Were activities carried out in a timely manner? To what extent were KONEKTE managers able to adapt and find solutions to unexpected challenges? 7. To what extent has KONEKTE integrated cross cutting concerns related to gender into the Activity? Please provide examples. If not, what more might be done? (CC Gender) 8. To what extent and how did KONEKTE build the capacity of its GOH counterparts’ and local counterpart s’ (CFET) capacity to sustain activities beyond the conclusion of KONEKTE? What are the most significant obstacles/constraints affecting sustainability? (EQ3) Probe: To what extent do you think measures taken by KONEKTE to foster sustainability were adequate? Was there adequate capacity building of permanent staff? Consideration of maintenance needs and costs of interventions? Sufficient documentation of efforts left behind? What more, if anything, could have been done to ensure the sustainability of KONEKTE results? 57| KONEKTE PERFORMANCE EVALUATION USAID.GOV 9. To what extent, if at all, has the conclusion of KONEKTE assistance affected USAID/Haiti’s work and relationships with its main GOH counterparts? (EQ3) 10. What examples are there (if any) of KONEKTE coordinating with other USAID or US Government funded programs? With other donor-funded programs? (EQ2) Probe: To what extent did the Activity fill gaps and/or complement other assistance programs? 11. What lessons were learned in the implementation of the KONEKTE? (Cross Cutting Lessons learned) 12. What (if any) were the good practices used by KONEKTE that you think might be useful to other organizations working to strengthen the capacity of GOH counterparts in Haiti? (Cross Cutting) Protocol for KII with Implementing Organizations A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title (during life of Activity): 7. Respondent Institution (during life of Activity): 8. Sex of respondent: B. KII questions for Implementing Organizations 1. In which areas were KONEKTE’s greatest achievements (please provide examples)? Please highlight factors that have contributed to these achievements. (EQ2)What are examples of KONEKTE strategies and interventions that were less successful? Please highlight factors that have hindered progress. (EQ1, 2) Probe: To what extent was the process that KONEKTE put in place to recruit TAs effective in helping GOH counterparts to identify their own technical assistance needs and develop relevant TA SOWs? In matching TA profiles and skills to the identified needs? To what extent was KONEKTE effective in orienting and then monitoring/supporting TAs once they were on the job? Assessing institutional capacity needs? In building the capacity of OMRH and other GOH HR departments on HR management? 58| KONEKTE PERFORMANCE EVALUATION USAID.GOV 2. What are the most significant obstacles/constraints in KONEKTE’s ecosystem or implementing environment that hindered progress? How did KONEKTE adapt its strategies to deal with unforeseen obstacles/constraints? (EQ2) Probe: To what extent did external factors within counterpart institutions or in the general political and economic environment affect KONEKTE? To what extent did internal factors such as the distribution of roles and responsibilities between USAID, the implementor and GOH affect KONEKTE? 3. To what extent was KONEKTE an effective mechanism for responding to skills and knowledge gaps within GOH counterpart institutions? What, if anything, might have been done to improve the program design? (EQ1, EQ2) Probe: To what extent was the scope of KONEKTE services adequate to meet GOH concerns and needs? 4. To what extent and how did KONEKTE build the capacity of its GOH counterparts’ capacity to sustain activities beyond the conclusion of KONEKTE? To what extent did it build the capacity of its local counterpart (CFET) to sustain activities beyond the conclusion of KONEKTE? What are the most significant obstacles/constraints affecting sustainability? (EQ3) Probe: To what extent do you think measures taken by KONEKTE to foster sustainability were adequate? Was there adequate capacity building of permanent staff? Consideration of maintenance needs and costs of interventions? Sufficient documentation of efforts left behind? What more, if anything, could have been done to ensure the sustainability of KONEKTE results? 5. Do you think that cross-cutting concerns related to gender were adequately integrated into KONEKTE? Please provide examples. If not, what more might be done? (Cross Cutting Gender) 6. What examples are there (if any) of KONEKTE coordinating with other USAID or US Government funded programs? With other donor-funded programs? (EQ2) Probe: To what extent did the Activity fill gaps and/or complement other assistance programs? 7. What lessons were learned in the implementation of the KONEKTE? (Cross Cutting Lessons learned) 8. What (if any) were the good practices used by KONEKTE that you think might be useful to other organizations working to strengthen the capacity of GOH counterparts in Haiti? (Cross Cutting) 59| KONEKTE PERFORMANCE EVALUATION USAID.GOV Protocol for FGDs with Human Resource forum participants (Use the informed consent procedures) A. Demographic Info: 1. Date: 2. FGD location: 3. FGD leader: 4. Primary Notetaker Name: Name Institution Sex 1. 2. 3. 4. 5. 6. B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask organizer(s) if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ 1. FGD questions for Worker Organizers We will ask each participant to introduce herself/himself and describe if/how s/he was involved in KONEKTE activities. 2. What are your main objectives and activities in Human Resource (HR) management? (EQ1, EQ2, EQ3) 3. What have been the most significant challenges/obstacles you have faced in carrying out your work? (EQ1) 4. What was the purpose of KONEKTE assistance? What types of changes in HR practices did KONEKTE training and knowledge sharing activities promote? 60| KONEKTE PERFORMANCE EVALUATION USAID.GOV 5. To what extent did KONEKTE address/promote nondiscrimination in employment practices and/or address the role of HR in addressing the particular workplace concerns of women? What are some examples? What are the main challenges? (CC) 6. In what ways, if any, has KONEKTE support changed how you work? (EQ2, EQ3) Probe: In which ways have you been able to apply the skills you learned from KONEKTE in your day- to-day work? What are some examples? What types of resistance have you encountered? From whom? 7. Are there ways that KONEKTE capacity building activities might have been improved or adapted to be more effective? (EQ1) Probe: Which, in any, KONEKTE efforts are not fully realized? What further resources or efforts would it take to ensure their sustainability? 8. How, if at all, did the conclusion of KONEKTE affect your work? (EQ3) 9. What additional efforts are needed to ensure that you are able to improve your practices as RH managers? (EQ3) 61| KONEKTE PERFORMANCE EVALUATION USAID.GOV Protocol for Direct Observation of KONEKTE Interventions 1. Date 2. Primary Observer: 3. Description of the Activity intervention being observed: 4. GoH Institution: 5. Location: Questions to be answered through stakeholder interview or document review: 1. What were the topics/issues addressed by the intervention? 2. To what extent might this intervention contribute toward one of the KONEKTE objectives? Which one? How? 3. To what extent was the intervention fully implemented? 4. What are constraints to full exploitation of the intervention? 5. What, if any, changes within the institution are attributable to the intervention? (examples: recipient institution staff trained? user training and support offered?) 6. To what extent were sustainability requirements adequately addressed? (Existence of maintenance/support contract; training and support systems in place to support use). Observation of KONEKTE provided IT equipment and software packages Ask stakeholder to demonstrate if/how the equipment/software package is being used. Request to see evidence of features that are used – for example reports produced. Ask for the person/persons in charge of system management, providing user support, or performing data entry to be identified. Ask about training they received. Ask to see maintenance contract and/or maintenance logs if these exist. 1. To what extent is the equipment/software package still functional? (rate: not functional, partially functional, fully functional) 2. If still functional, to what extent are the features of the equipment/software being exploited? (rate: not at all exploited, partially exploited, fully exploited) 3. To what extent was the software adapted/customized for local needs? Is the interface in a local language? Have forms been customized to match forms in use by entity? (rate: not adapted/partially adapted/fully adapted) 4. Were user manuals provided? Are they in a language that is easily understood? (rate no user manual, user manual not in local language, user manual in local languages) 5. To what extent is the system being properly maintained? Is there evidence of maintenance occurring? (no evidence, some evidence, extensive evidence) Observation of KONEKTE supported-websites 62| KONEKTE PERFORMANCE EVALUATION USAID.GOV 1. Is the website still online? If offline now, according to stakeholder was it ever online? (rate: never online, currently offline, currently online) 2. To what extent has the website been fully populated with content? (rate: no content, partially populated, fully populated) 3. To what extent is the website being regularly updated (rate: never updated since creation, periodically updated but not recently, periodically updated with recent content; regularly updated) Observation of KONEKTE supported tools, checklists, manuals 1. Are copies available in the recipient offices? (yes/no) 2. Are they in an accessible language? (yes/no) 3. Are filled in versions of checklists/forms found in relevant files? (yes/no) 4. Have any updates been made to the tools by recipients based on use? (yes, no) Notes by Observer 63| KONEKTE PERFORMANCE EVALUATION USAID.GOV ANNEX G: DISCLOSURE OF CONF LICT OF INTEREST 64| KONEKTE PERFORMANCE EVALUATION USAID.GOV | KONEKTE PERFORMANCE EVALUATION USAID.GOV U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523