LEARNING, EVALUATION, AND ANALYSIS PROJECT-II (LEAP-II) FINAL REPORT

DISCLAIMER This publication was produced for review by the United States Agency for International Development. It was prepared by International Development Group, LLC. LEARNING, EVALUATION, AND ANALYSIS PROJECT-II (LEAP-II) FINAL REPORT PHOTO CREDIT: MELISSA STONE LEARNING, EVALUATION, AND ANALYSIS PROJECT-II (LEAP-II) Final Report Contract No. AID-OAA-I-12-00042, Task Order No. AID-OAA-TO-14-00046 Cover photo: Entrepreneur in Sierra Leone DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States government. CONTENTS EXECUTIVE SUMMARY 1 INTRODUCTION 1 ACTIVITIES 1 PERFORMANCE EVALUATIONS 1 #1 – LIBERIA MUNICIPAL WATER PROJECT (LMWP) MID-TERM EVALUATION 1 #2 – PARTNERSHIP FOR GROWTH (PFG) MID-TERM EVALUATION 3 #3 – FINAL PERFORMANCE EVALUATION OF THE LIBERIA ENERGY SECTOR SUPPORT PROGRAM (LESSP) 4 #5.A EVALUATION OF THE BEYOND ADVOCACY FUND (BAF) IN SOUTH AFRICA 5 #7 – FINAL EVALUATION OF THE USAID ME/TS IMPROVING WATER AND SANITATION SERVICES IN THE MIDDLE EAST AND NORTH AFRICA REGION (IWSMR) ACTIVITY 7 #10 – MID-TERM PERFORMANCE EVALUATION: LIBERIA INVESTING FOR BUSINESS EXPANSION PROJECT (IBEX) AND SUSTAINABLE MARKETS INITIATIVE – LIBERIA (SMI-L) 8 #17 – FINAL PERFORMANCE EVALUATION OF THE “TACKLING YOUTH EMPLOYMENT IN TUNISIA” PROJECT (MASHROU3I) 9 #28 – MID-TERM PERFORMANCE EVALUATION: USAID/INDIA COUNTRY DEVELOPMENT AND COOPERATION STRATEGY DEVELOPMENT OBJECTIVE 4 11 #29 – FINAL PERFORMANCE EVALUATION OF THE FURTHER ADVANCING THE BLUE REVOLUTION INITIATIVE (FABRI) 19 #30 – MID-TERM PERFORMANCE EVALUATION OF THE BIZ+ PROJECT 20 IMPACT EVALUATIONS/LARGE HOUSEHOLD SURVEYS 23 #4 AND #14– SENEGAL YAAJEENDE MIDTERM EVALUATION AND KOLDA REGION BASELINE IMPACT EVALUATION 24 #11 – LIBERIA FEED THE FUTURE ZONE OF INFLUENCE INTERIM ASSESSMENT 27 #15 SENEGAL FEED THE FUTURE ZONE OF INFLUENCE INTERIM ASSESSMENT 29 #25 LIBERIA FOOD AND ENTERPRISE DEVELOPMENT (FED) IMPACT SURVEY 31 ECONOMIC ANALYSES 35 #0.A – CARGO PREFERENCE ANALYSIS 35 #5.A – ANALYSIS TO DEVELOP A TRADE INTERVENTION AGENDA: PRIORITIZING COUNTRIES AND ACTIVITIES IN SOUTHERN AFRICA 35 #25.B – LIBERIA CROSS BORDER TRADE ASSESSMENT 37 #27.A – TIMOR-LESTE CUSTOMS ASSESSMENT AND DESIGN 39 #27.B – TIMOR-LESTE IMPORT AND EXPORT REGULATIONS ADMINISTERED BY GOVERNMENT AGENCIES 41 #13 AND #16 – AGRICULTURAL COMMERCIAL, LEGAL, AND INSTITUTIONAL REFORM (AGCLIR) DIAGNOSTICS IN SIERRA LEONE AND GUINEA 43 #33 – ANALYSIS OF THE USAID-CILSS INTERREGIONAL TRADE MONITORING DATA 49 COST-BENEFIT ANALYSIS (CBA) 53 CBA METHODOLOGY 53 #18 – COST-BENEFIT ANALYSIS OF USAID/SENEGAL’S RICE VALUE CHAIN 55 #19 – COST BENEFIT ANALYSIS OF USAID/RWANDA’S DAIRY VALUE CHAIN INTERVENTION 56 #20 – COST BENEFIT ANALYSIS OF USAID/LIBERIA’S RICE AND GOAT VALUE CHAIN INTERVENTIONS 58 #21 – COST BENEFIT ANALYSIS OF USAID/MALI’S SORGHUM AND MILLET VALUE CHAIN INTERVENTIONS 60 #23 – FEED THE FUTURE COST-BENEFIT ANALYSIS SYNTHESIS REPORT 63 #26 – INTEGRATING GENDER IN COST-BENEFIT AND COST-EFFECTIVENESS ANALYSIS 67 #6 – HAITI: INCREASING THE EFFICIENCY FOR PUBLIC INVESTMENT PROJECT 70 #12 – KENYA PROGRAM ON INTEGRATED INVESTMENT APPRAISAL AND RISK ANALYSIS71 #31 – COST-BENEFIT ANALYSIS-COST-EFFECTIVENESS ANALYSIS OF USAID ACTIVITIES IN NIGERIA AND ZIMBABWE 73 #32 – CBA OF USAID RESILIENCE IN THE SAHEL ENHANCED 84 OTHER ACTIVITIES 90 #0.B – LOGISTICAL SUPPORT TO THE SECOND ECONOMISTS REGIONAL WORKSHOP 90 #8 AND #24 – ADVISORY SERVICES TO THE POWER AFRICA COORDINATOR’S OFFICE 90 LIST OF FIGURES FIGURE 1: ACCELERATING INNOVATION, TRANSFER, AND DIFFUSION IN DEVELOPING COUNTRIES..................................................................................................................................................................... 14 FIGURE 2: INNOVATION PROXIMITY ................................................................................................................... 19 FIGURE 3: ECONOMIC AND FINANCIAL ANALYSIS ....................................................................................... 53 FIGURE 4: MAP OF FTF PROJECTS FOR CBA SYNTHESIS (2015 VC ONLY)............................................. 63 FIGURE 5: INCREMENTAL ECONOMIC NET PRESENT VALUE (ENPV) ..................................................... 64 FIGURE 6: INCREMENTAL ECONOMIC NET PRESENT VALUE (ENPV) (USAID PERSPECTIVE) BY VALUE CHAIN ................................................................................................................................................................ 65 FIGURE 7: POVERTY ALLEVIATION OF REGIS-ER’S AND REGIS-AG’S INTERVENTIONS ................... 87 LIST OF TABLES TABLE 1: LEAP-II ACTIVITIES BY TYPE ..................................................................................................................... 1 TABLE 2: COUNTRIES, MARKETS, AND COMMODITIES COVERED BY THE USAID-CILSS TRADE DATA. ................................................................................................................................................................................ 50 TABLE 3: PCE CBA IMPACT FIGURES ..................................................................................................................... 55 TABLE 4: RDCP II CBA IMPACT FIGURES ............................................................................................................. 57 TABLE 5: INCREMENTAL ANALYSIS OF FED PROJECT (USAID PERSPECTIVE) ...................................... 58 TABLE 6: ARDT_SMS CBA IMPACT FIGURES ...................................................................................................... 62 TABLE 7: CBA SYNTHESIS HIGH-LEVEL FIGURES .............................................................................................. 64 TABLE 8: INCREMENTAL ECONOMIC NET PRESENT VALUE (USD) ......................................................... 74 TABLE 9: INCREMENTAL ECONOMIC NET PRESENT VALUE (USD) ......................................................... 82 LIST OF ACRONYMS ACWUA Arab Countries Water Utility Association AEO Authorized Economic Operators AgBEE Agriculture business enabling environment AgCLIR Agricultural Commercial, Legal, And Institutional Reform AGUIPEX Agence Guinéenne de Promotion des Exportations AI Artificial insemination ASYCUDA Automated System for Customs Data ATE Average treatment effect BAF Beyond Advocacy Fund BCR Benefit-cost ratios BFS Bureau for Food Security BLSA Business Leadership South Africa BSO Business services organization CA Constraints analysis CBA Cost-benefit analysis CADS Cluster Agricultural Development Services CAPI Computer Assisted Personal Interviewing CAQDAS Computer assisted qualitative data analysis CBL Central Bank of Liberia CBSPs Community-Based Solution Providers CBT Cross-border trade CD Customs Directorate CDCS Country Development Cooperation Strategy CEA Cost-effective analysis CF Conservation Farming CIEPEX Centre International d’Echanges et de Promotion des Exportations CILSS Permanent Interstate Committee for Drought Control in the Sahel CIP Capital improvement project COR Contracting Officer Representative CRDH Centre de Recherche pour le Developpement Humain CRI Cambridge Resources International, Inc. CRP Customs Reform Project CTPEA Centre de Techniques de Planification et d'Économie Appliquée CWG Citizen Work Group DALY Disability adjusted life year DCA Development Credit Authority DD Difference-in-difference DEC Development Experience Clearinghouse DO Development Objective E3 Economic Growth, Education and Environment ECOWAS Economic Community of West African States ENAF Ecole Nationale d’Administration Financiere ENPV Economic net present value EP Office of Economic Policy ERR Economic Rate of Return EVD Ebola Virus Disease FABRI Further Advancing the Blue Revolution Initiative FCT Federal Capital Territory FED Food and Enterprise Development FGDs Focus group discussions FNPV Financial Net Present Value FRC Fiscal Reform Commission FTCC Fast Track Commercial Court FTF Feed the Future FTFZ-CD Feed the Future Zimbabwe Crop Development FTFZ-LD Feed the Future Zimbabwe Livestock Development GAHP Good Animal Husbandry Practices GAP Good Agricultural Practices GDA Global Development Alliance GOH Government of Haiti GOI Government of India GOG Government of Ghana GOK Government of Kenya GOM Government of Mali GON Government of Nigeria GOS Government of Senegal GOSL Government of Sri Lanka GOT Government of Tanzania IBEX Investing for Business Expansion Project ICBT Informal cross-border trade ICER Interventions Combined Incremental Cost-Effectiveness Ratios IDG International Development Group LLC (IDG) IRB Institutional Review Board IRR Internal Rate of Return JCAP Joint Country Action Plan HH Household HP Hewlett Packard IWSMR Improving Water and Sanitation Services in the Middle East and North Africa Region KII Key informant interview KSG Kenya School of Government LEAD Linkages for Economic Advancement of the Disadvantaged LEAP-II Learning Evaluation and Analysis Project-II LESSP Liberia Energy Sector Support Project LGA Local Government Areas LMWP Liberia Municipal Water Project LSA Liberia Strategic Analysis M&E Monitoring and evaluation MARAD US Maritime Administration MCC Milk Collection Centers ME/TS Middle East Bureau/Technical Services MENA-NWC Middle East and North Africa Network of Water Centers of Excellence MIE Midterm impact evaluation MOF Ministry of Finance MPEC Ministry of Planning and External Cooperation MRU Mano River Union MUAC Mid-upper arm circumference NRVCC Nutrient-rich value chain commodity NRW National Single Window NUPAS Non-US Organization Pre-Award Survey O&M Operation and maintenance OAA Office of Acquisition and Assistance ODK Open Data Kit OGAs Other Government Agencies OHADA Organisation pour l'Harmonisation en Afrique du Droit des Affaires PBS Population-based surveys PCE Economic Growth Project PFG Partnership for Growth PICS Purdue Improved Crop Storage PIO Public International Organization PIP Public Investment Program PPA Public private alliance PPP Public-private partnership PSM Propensity score matching QED Quasi-experimental designs QPI Qualitative Personal Interviewing RAF Risk assessment framework RD Regression discontinuity RDCP Rwanda Dairy Competitiveness Program II REGIS-AG Resilience and Economic Growth in the Sahel – Accelerated Growth REGIS-ER Resilience and Economic Growth in the Sahel - Enhanced Resilience RISE Resilience in the Sahel Enhanced ROCTP Regional Operator Certification and Training Program SBS Business support organizations SEforALL Sustainable Energy for All SILC Savings and Internal Lending Communities SME Small and medium enterprise SMI-L Sustainable Markets Initiative – Liberia TCLP Total Consumption Poverty Line TFA WTO Trade Facilitation Agreement TLTIP Timor-Leste Trade Information Portal TOT Training of trainers TVET Technical Vocational and Education Training UEMOA West African Economic and Monetary Union UNIDO United Nations Industrial Development Organization USAID United States Agency for International Development VC Value chain VE/VA Value engineering/value analysis WASH Water, sanitation and hygiene WCO World Customs Organization WEAI Women’s Empowerment in Agriculture Index WCO World Customs Organization WCS Warrantage credit scheme ZIMSTAT Zimbabwe National Statistics Agency ZOI Zone of Influence 1 | LEAP-II FINAL REPORT USAID.GOV EXECUTIVE SUMMARY PROJECT OVERVIEW The Learning, Evaluation, and Analysis Project-II (LEAP-II) provided economic analysis and monitoring and evaluation support services to satisfy the overall operational objectives of USAID’s Bureau for Economic Growth, Education and Environment (E3). LEAP-II was a $14m project with a period of performance of October 2014-September 2018. The objective of the project was to provide a mechanism for the Office of Economic Policy in E3 (E3/EP) and for Missions and Bureaus to increase access to rigorous, independent technical evaluation and analysis within and beyond USAID. LEAP-II provided a vehicle for USAID Washington and USAID Missions to contract for analysis services to be executed for projects within their economic growth or other portfolios. LEAP-II was structured as a contract mechanism implemented through buy-ins.1 Services under LEAP-II’s scope of work included:  Cost-benefit/cost-effectiveness analysis  Macroeconomic and financial analysis  Single- and multi-country evaluations  Randomized controlled trials (RCTs)  Policy Analysis Matrix/Domestic Resource Cost Analysis  Constraints Analysis/Inclusive Growth Diagnostics  Other Analytical and Project Support  Integrating Gender Equality and Female Empowerment In addition to the activities listed above, LEAP-II provided for the presentation and dissemination of analysis findings and lessons to help guide the design, implementation, and subsequent evaluation of economic growth and other sectors’ activities. LEAP-II also provided a vehicle for USAID Missions to support capacity building of partner government and civil society counterparts. Mission and Bureaus interested in engaging LEAP-II services submitted a SOW to the Contracting Officer Representative (COR). The COR determined if the proposed activities were within the SOW of 1 A buy-in is defined as a funds-transfer to LEAP-II during a fiscal year where the buy-in objectives are consistent with LEAP-II objectives and Mission use of the mechanism will advance common strategic objectives. USAID.GOV LEAP-II FINAL REPORT | 2 the contract, and if so, provided the SOW to IDG. IDG would then prepare and submit to the designated Activity Manager at the Mission or Bureau and COR a business proposal including a brief technical approach, work plan, budget, and the CVs for the proposed team. Business proposals were submitted to the Activity Manager, COR, and CO for approval. Once the approvals were in place, IDG would quickly mobilize the team and launch the activity, often within just two weeks of receiving approval. PROJECT RESULTS LEAP-II was immensely popular and by year 3 could no longer accommodate new buy-ins. Thirty-six activities in 30 countries were implemented over the four years of the project including 15 performance and impact evaluations and 21 economic analyses covering a wide range of technical areas: water, energy, trade policy and trade facilitation, agriculture and agribusiness, nutrition, access to finance, SME development, youth, gender, infrastructure, workforce development, education, and health. (See table 1 below for LEAP-II work orders by activity time.) LEAP-II’s key achievement was generating evidence for development. LEAP-II activities contributed to improving the effectiveness of USAID programming through generating evidence and actionable recommendations for its use to inform policy and programming decisions for maximum impact. Learning was at center of LEAP-II activities and the majority included learning events such as presentations and workshops so results yielded change. LEAP-II services helped Bureaus and Missions apply USAID Program Cycle Principles and increased compliance with the USAID Evaluation Policy (January 2011, updated in October 2016) and ADS 201 (revised in September 2016). LEAP-II also helped Bureaus and Mission to manage resources effectively by providing the data and information required to align resources against country strategies, make difficult trade-offs, and re-allocate resources toward activities that are demonstrating meaningful results. LEAP-II deliverables presented findings, conclusions, and recommendations that are based on facts, evidence, and data, and supported by quantitative and qualitative information that is reliable, valid and generalizable. Deliverables included: an introduction; a full description of the methodology used; findings and results; limitations; conclusions and inferences; and options or recommendations, as appropriate. Recommendations were specific, action-oriented, and scalable and with defined responsibility for the action allowing USAID to integrate key findings into strategic plans and program management decisions. LEAP-II deliverables can be found on the Development Experience Clearinghouse (DEC) under contract no. AID-OAA-I-12-00042/AID-OAA-TO-14-00046. 1 | LEAP-II FINAL REPORT USAID.GOV TABLE 1: LEAP-II ACTIVITIES BY TYPE PERFORMANCE EVALUATIONS IMPACT EVALUATIONS/ LARGE HOUSEHOLD SURVEYS ECONOMIC AND TRADE ANALYSES COST-BENEFIT ANALYSIS OTHER ACTIVITIES #1 – Liberia Municipal Water Project (LMWP) Mid-term Performance Evaluation #4 – Senegal Yaajeende Midterm Impact Evaluation #0.b – Cargo Preference Analysis #6 – Haiti: Increasing the Efficiency for Public Investment Project #0.a – Logistical Support to The Second Economists Regional Workshop #2 – Partnership for Growth (PFG) Mid- term Evaluations #11 – Liberia Feed the Future Zone of Influence Interim Assessment #5 – Analysis to Develop a Trade Invention Agenda in Southern Africa #12 – Kenya Program on Integrated Investment Appraisal and Risk Analysis #8 – Advisory Services to the Power Africa Coordinator’s Office #3 – Final Performance Evaluation of the Liberia Energy Sector Support Program (LESSP) #14 – Senegal Yaajeende Kolda Impact Evaluation Baseline #13 – Agricultural Commercial, Legal, And Institutional Reform (AgCLIR) Diagnostics in Sierra Leone #18 – Cost-Benefit Analysis of USAID/Senegal’s Rice Value Chain Intervention #24 – Advisory Services to the Power Africa Coordinator’s Office II #5.a – South Africa – Evaluation of the Beyond Advocacy Fund (BAF) #15 – Senegal Feed the Future Zone of Influence Interim Assessment #16 – Agricultural Commercial, Legal, And Institutional Reform (AgCLIR) Diagnostics in Guinea #19 – Cost-Benefit Analysis of USAID/Rwanda’s Dairy Value Chain Intervention #7 – Final Evaluation of the USAID ME/TS Improving Water and Sanitation Services in the Middle East and North Africa Region (IWSMR) Activity #25 – Liberia Food and Enterprise Development (FED) Impact Survey #25.a – Liberia Cross-Border Trade Assessment #20 – Cost-Benefit Analysis of USAID/Liberia’s Rice and Goat Value Chain Interventions #10 – Mid-term Performance Evaluation: Liberia Investing for Business Expansion Project (IBEX) and Sustainable Markets Initiative – Liberia (SMI-L) #27.a – Timor-Leste Customs Assessment and Design #21 – Cost-Benefit Analysis of USAID/Mali’s Sorghum and Millet Value Chain Interventions #17 – Final Performance Evaluation of “Tackling Youth Employment in Tunisia” #27.b – Timor-Leste Import and Export Regulations Administered by Government Agencies #23 – Feed the Future Cost-Benefit Analysis Synthesis Report #28 – USAID/India Country Development and Cooperation Strategy Development Objective 4 Mid-term Performance Evaluation #33 – Analysis of the USAID- CILSS Inter-Regional Trade Monitoring Data #26 – Integrating Gender in Cost-Benefit and Cost- Effectiveness Analysis #29 – Final Performance Evaluation of the Further Advancing the Blue Revolution Initiative (FABRI) #31 – Cost-Benefit Analysis-Cost-Effectiveness Analysis of USAID Activities in Nigeria and Zimbabwe USAID.GOV LEAP-II FINAL REPORT | 2 TABLE 1: LEAP-II ACTIVITIES BY TYPE PERFORMANCE EVALUATIONS IMPACT EVALUATIONS/ LARGE HOUSEHOLD SURVEYS ECONOMIC AND TRADE ANALYSES COST-BENEFIT ANALYSIS OTHER ACTIVITIES #30 – Mid-term Performance Evaluation of the BIZ+ Project #32 – Cost-Benefit Analysis of USAID Resilience in the Sahel Imitative #9 – Sierra Leone and Guinea Cost Benefit Analysis Activity was canceled. #22 – Cost-Benefit Analysis of USAID/Malawi’s Value Chain Interventions was canceled. 1 | LEAP-II FINAL REPORT USAID.GOV INTRODUCTION This report presents summaries of the 36 activities implemented under LEAP-II. The report is divided into the following sections: Performance Evaluations, Impact Evaluations, Economic and Trade Analyses, Cost-benefit Analysis, and Other Activities. Each section begins with a brief description of LEAP-II’s approach. Each activity summary includes the following sub-sections: Introduction, Methodology, Key Results, and Learning. ACTIVITIES PERFORMANCE EVALUATIONS LEAP-II conducted 10 performance evaluations oriented toward methodological rigor, utility to USAID, and cost effectiveness. LEAP-II applied a set of six quality standards – utilizing a set of checklists and rating/review tools – to ensure close attention to quality at each step of an evaluation: 1) ensuring a clear statement of purpose regarding a given evaluation; 2) defining or refining evaluation questions directly linked to the evaluation purpose; 3) developing an evaluation design matrix that ensures evaluation questions are effectively addressed; 4) developing high-quality data collection instruments that are linked to outcomes and indicators of interest; 5) employing a range of approaches to data analysis and data visualization to allow for the identification of important trends and patterns in the collected data, thereby ensuring that evaluation findings drive conclusions and conclusions inform recommendations; 6) completing all evaluation products – reports, debriefs, recommendation work plans – with a special focus on quality and utilization. The majority of performance evaluations utilized mixed methods research with data collection involving a desk review, key informant interviews (KIIs), focus group discussions (FGDs), online surveys, and site visits. #1 – LIBERIA MUNICIPAL WATER PROJECT (LMWP) MID-TERM EVALUATION INTRODUCTION The primary objective of LMWP was to support the design, tendering, execution and operation of water supply infrastructure improvements in the three Liberian county capitals – Robertsport (Grand Cape Mount County), Sanniquellie (Nimba County), and Voinjama (Lofa County). The LMWP implementing partner, Tetra Tech, was tasked with assisting local and national authorities in developing plans for water supply and sanitation improvements, overseeing construction of this capital improvement project (CIP), supporting initial operations of water supply infrastructure improvements, and establishing local capability to sustainably operate and maintain constructed water supply systems. LEAP-II conducted a mid-term evaluation to determine whether LMWP was meeting its objectives, and if adjustments were needed as USAID advanced plans to begin capital works in line with the plans developed by LMWP for the three cities. The evaluation addressed the following three key areas: (1) Effectiveness of infrastructure planning/construction oversight approach, (2) Effectiveness of institutional framework and capacity building, and (3) Overall project positioning and strategy for phase-out of USAID assistance. METHODOLOGY LEAP-II’s approach to the mid-term evaluation included a desk review, key informant interviews, and a field visit to Liberia from November 24 to December 15, 2014. The evaluation team reviewed LMWP’s deliverables and cross referenced them against the expected deliverables as outlined USAID.GOV LEAP-II FINAL REPORT | 2 in the Objectives and Scope of Work. The team conducted over 20 key informant interviews and consulted over 15 organizational stakeholders. Field activities included visits to the cities of Robertsport, and Kakata. The evaluation was conducted during the Ebola outbreak in 2014 and required proper training for the team and extensive operational support from the IDG home office. LEAP-II utilized a risk assessment framework (RAF) to evaluate potential risks and impacts of recommended CIP improvements including analysis of the operation and maintenance (O&M) cost recovery and sustainability, capacity building and institutional strengthening. The RAF was used as a strategy for prioritizing and sharing information about the risks to infrastructure investment. The evaluation team analyzed the risks and rewards of a decision using data collected during the evaluation. KEY FINDINGS The evaluation team found that while LMWP employed key tools and concepts in designing the CIP, it did not apply a value engineering/value analysis (VE/VA) for the design (which is suggested by international best practice). As a result, the O&M cost recovery period of one to three years which is projected by LMWP is not realistic. The evaluation team conducted a sensitivity analysis and found that the O&M cost recovery period was closer to seven years. Another key finding was that the institutional framework and capacity building provided by LMWP was effective and one of the project’s key achievements. LEARNING The final report presented the RAF and detailed findings supported by evidence and data to the evaluation questions. The report included three levels of priority recommendations to reflect the degrees and levels (1-3) of importance (high-low) of the proposed recommendations to allow timely and concurrent, yet priority-driven improvements to the project. Priority recommendations included conducting VE/VA prior to and during construction; re-evaluating the economic and financial analysis including identifying trade -offs through sensitivity analysis; and utilizing performance-based contracting with provisions for incentives; and providing support to the government to overcome external and internal institutional challenges. To facilitate learning, the Key informant interview in Robertsport, Liberia PHOTO CREDIT: SAM KODUAH 3 | LEAP-II FINAL REPORT USAID.GOV evaluation team presented the results to USAID/Liberia and discussed in detail the recommendations and how to implement them. #2 – PARTNERSHIP FOR GROWTH (PFG) MID-TERM EVALUATION INTRODUCTION The Partnership for Growth (PFG) initiative aims to achieve accelerated, sustained, and broad-based economic growth in selected partner countries through bilateral agreements between the USG and the partnering countries’ national governments. PFG requires rigorous, joint analyses of countries’ individual constraints to growth, joint action plans to address the most pressing of these constraints, and high-level mutual accountability for the goals and activities selected to alleviate them. In February 2011, the Governments of Ghana and Tanzania committed to work with the USG to accelerate and sustain broad-based and inclusive growth in Ghana and Tanzania through the PFG initiative. This included a commitment to jointly prepare a Constraints Analysis (CA) using the Growth Constraints approach of Hausmann, Rodrik, and Velasco (2006). The CA’s for Ghana and Tanzania were finalized in August and September 2011 respectively. These were used as the basis for the development of the five-year Joint Country Action Plans (JCAP) for Tanzania 2012-2016, issued in April 2012 and for Ghana which was finalized in March 2013 with the five-year program running to February 2018. LEAP-II carried out the mid-term evaluations in Ghana and Tanzania included examining PFG’s effectiveness in addressing binding constraints identified in each country (energy and access to finance in Ghana, and energy and rural roads in Tanzania). The mid-term evaluations sought to determine whether the PFG process represent an improvement over the pre-PFG assistance approach. The evaluations also examined whether PFG analyses and activities are sufficient for addressing the identified constraints, realizing the desired outcome and attributing the impact of PFG interventions on reducing the constraints. METHODOLOGY The evaluation team used three data collection methods: 1) A desk review based on the program documentation received primarily from USAID for the PFG initiatives as a whole, as well as documents that refer exclusively to the core focal areas addressed under the PFG in each country. 2) Semi-structured interviews with key PFG stakeholders, including high-level leadership, leadership, architects, goal leads, implementers, and independent experts in the US, Ghana, and Tanzania. The team spoke with 83 people in Ghana and 129 people in Tanzania. 3) A web-based survey of officials and technical specialists. KEY FINDINGS Key findings from the evaluation include:  At the midterm, PFG is regarded as a successful initiative by GOG, GOT and USG leaders, designers and goal leads.  All parties involved (interviewed) stated their belief that the PFG process had contributed to improved dialogue as partners in a shared commitment to development.  The processes with the CA, JCAP, log frames and a developing a commitment to M&E advanced understanding and encouraged deeper thinking on causal linkages. USAID.GOV LEAP-II FINAL REPORT | 4  The JCAP in both Ghana and Tanzania were ambitious and optimistic taking into account the difficult development challenges.  PFG approach adopted and owned in the power sector, but more challenges were experienced in access to finance in Ghana and rural roads in Tanzania.  Many of the PFG interventions are on target and substantial progress has been made on a range of challenging policy and regulatory issues in a relatively short period of time.  PFG is partially on track in Ghana and Tanzania for the power sector, which has received additional financial resources.  There has been limited progress in Ghana on the access to finance constraint, and modest progress in Tanzania or rural roads connectivity. LEARNING LEAP-II produced three evaluation reports: Ghana, Tanzania, and a crosscutting report. The reports addressed country specific questions for Ghana and Tanzania, and crosscutting questions. They included detailed recommendations on the PFG process and targeted sectors. The evaluation team also produced detailed case studies of the power sector in each country that addressed: public-private partnerships, policy, regulatory and institutional challenges; energy mix and power generation sources; and power usage and transmission and distribution system coverage as well as case studies on the roads sector in Tanzania and access to finance in Ghana. The team presented the results and recommendations to USAID/Ghana, USAID/Tanzania, and USAID/Washington. #3 – FINAL PERFORMANCE EVALUATION OF THE LIBERIA ENERGY SECTOR SUPPORT PROGRAM (LESSP) INTRODUCTION LESSP (implemented by Winrock International) aimed to achieve the following objectives: 1) Increased, sustainable access and affordability of electricity within selected urban and rural poor communities; 2) Improved performance of local governments, civil society and the private sector in monitoring, regulating and managing the use of renewable energy; 3) An increase in the percentage of households and businesses utilizing clean energy, and a corresponding increase in economic activity; and 4) Policy changes that improve the investment climate for the energy sector. In 2014, the effects of the Ebola epidemic began to impact project performance. Winrock International requested a no-cost extension through March 2015. However, given the uncertainties with the project performance, contract compliance, and expected construction end dates, USAID did not grant an extension to the contract, and let the project end in October 2014 as originally scheduled. The construction of the pilot projects at Sorlumba, Kwendin, and Gbarnway were incomplete at project close out. The primary purpose of the evaluation was to determine whether the assistance provided by LESSP met the stated development objectives, and identify lessons learned. The evaluation provided a detailed picture of the major accomplishments and weaknesses of the project since inception, including management issues, and indicating the recommended changes in implementation approach to future energy programs, and USAID’s project design/programming approach to assure successful completion of the project objectives. The evaluation questions were organized into three categories: 1) Effectiveness of infrastructure planning / construction oversight approach; 2) Effectiveness of institutional framework and capacity building; and 3) Overall project positioning and strategy for phase-out of USAID assistance. 5 | LEAP-II FINAL REPORT USAID.GOV METHODOLOGY Data collection methods included a comprehensive desk review of relevant documents and reports and KIIs of key actors and stakeholders both in the U.S. and in Liberia (the field work took place April 17 – May 12, 2015). The evaluation team conducted KIIs with 27 key stakeholder organizations and approximately 60 participants in the US and Liberia. In-country interviews were supplemented by visits to the three LESSP pilot sites located in Gbarnway (solar in Lofa County), Sorlumba (biomass in Lofa County) and Kwendin (biomass in Nimba County) as well as two visits to the Booker Washington Institute to assess the situation on-the-ground with respect to sustainability of activities. KEY FINDINGS 1) The program scope was too broad with too many initiatives each of which required time and resources. The lack of focus created implementation challenges. Ultimately by overreaching, the program was unable to accomplish its objectives. 2) The program design was not strategic and stakeholder driven making it difficult for stakeholders to claim ownership to facilitate sustainability. 3) At the time of the evaluation, there was little evidence that LESSP built any meaningful relationship within the GOL. Most parties outside of GOL had positive interactions with LESSP but had high expectations of what would still be delivered based on their interactions with LESSP. 4) Capacity building did not emphasize organizational performance improvement sufficiently and did not have a rigorous evaluation mechanism to determine long term impacts. 5) The pilots initiated by LESSP did not achieve their original purpose – the demonstration of sustainable mechanisms to provide modern energy service in rural poor communities. Putting pilots in remote areas (in a post-conflict country) where there are technological and logistical challenges requires more resources to facilitate effective implementation which should have been identified during the due diligence. LEARNING The evaluation report included specific, action-oriented recommendations including improving and expanding stakeholder partnerships, aligning initiatives with national goals and objectives, strengthening M&E, and conducting a sensitivity analysis. The report also identified priority areas that should be the focus of possible future programming in the Liberian energy sector, including the renewable energy market and rural development strategies. The evaluation team presented the results of the evaluation to USAID/Liberia. #5.A EVALUATION OF THE BEYOND ADVOCACY FUND (BAF) IN SOUTH AFRICA INTRODUCTION Beyond Advocacy Fund (BAF) was established through a Memorandum of Collaboration signed between USAID and Business Leadership South Africa (BLSA) in November 2013. BAF, which was co-financed by matching commitments of US$1.5m each from USAID (in cash) and BLSA (in cash or in-kind contributions), was scheduled to operate for an initial period of three years, which was extendable depending on performance and the availability of funds. BAF was conceived as a tool to help address South Africa’s major development challenges by catalyzing BLSA’s members (private sector businesses) to work with the Government of South Africa on collaborative projects that address SA’s most pressing development challenges. The Fund aims to promote partnership-based approaches between government and business to identify, test and replicate innovative and lasting solutions to major development challenges in South Africa. USAID/South Africa requested that LEAP-II conduct an evaluation to determine how effective and efficient BAF is in achieving its goals. It also assessed BAF’s efforts to promote better government- private sector relations through funding small projects. The findings were designed to inform adjustments and improvements to current activities and a second three-year funding phase. USAID.GOV LEAP-II FINAL REPORT | 6 METHODOLOGY Data collection methods included:  Desk review of project documentation. The evaluation team reviewed documents relating to BLSA and BAF strategy and management, quarterly and annual reports, selection criteria, project proposals, and background reports.  Key informant interviews (KIIs) with key project stakeholders and grant recipients.  Field observations. For two projects the evaluators visited sites where the projects are active in order to better understand project issues and assess potential project effects. For other projects, there were no physical locations to visit or work had not yet begun. KEY RESULTS Selection process. BAF selection procedures, driven by informal connections and discussions, can be characterized as relationship-driven. By not advertising widely or marketing BAF, the program does not attract a large pool of applicants. Projects. The evaluation team’s review of the projects found that each indeed focuses on facilitating systemic change, freeing blockages, and acting as a catalyst for further growth and development. In this respect, they align clearly with a core BAF objective. The team also found that the selection process and focus of BAF has evolved over time, with later projects showing a clearer and stronger case for private- public sector collaboration. Closer engagement between government and private sector. While the facilitator role BAF plays is clearly important, it is not a guarantor of ultimate success. The question over the longer-term is whether there is enough support to hold the various initiatives together and keep them moving forward – especially once/if the facilitator leaves. The grantees are usually the facilitators, but in some projects their long-term engagement is not assured. Evidence of program outcomes. BAF funded projects rate highly in terms of relevance. Although the focus of projects has evolved, most have demonstrated, or show the potential to demonstrate, ways in which partnering between the public and private sector has addressed complex barriers which have to date curtailed South Africa’s development. Private sector interest and awareness in BAF. BAF has been able to show that there are private sector players willing to engage in a new approach to South Africa’s development challenges. They realize their contribution cannot simply be a donation and photo opportunity arranged by the marketing department. To date, a number of partners have also been willing to put substantial amounts of money and effort behind this. LEARNING The evaluation team produced a comprehensive evaluation report providing evidence and data to address the evaluation questions. The report included a summary of BAF’s projects’ key features and performance, and case studies of six projects that examined in detail the elements that contributed to their successes or lost opportunities. The team presented the results to USAID/South Africa. The results were used by USAID/South Africa to inform the extension of BAF and to improve the operational efficiency and the management of the activity. 7 | LEAP-II FINAL REPORT USAID.GOV #7 – FINAL EVALUATION OF THE USAID ME/TS IMPROVING WATER AND SANITATION SERVICES IN THE MIDDLE EAST AND NORTH AFRICA REGION (IWSMR) ACTIVITY INTRODUCTION IWSMR was a two-year, $1.9 million water activity, managed by USAID’s Middle East Bureau/Technical Services (ME/TS), and implemented by Chemonics International, Inc. IWSMR was designed to work with Arab Countries Water Utility Association (ACWUA) and members from USAID-eligible countries to build capacity (both that of ACWUA’s and utility company members), and to develop training and certification programs. ACWUA is a non-governmental organization that operates as a local, regional, and international platform offering opportunities to exchange knowledge, best practices in water and wastewater management, and experiences among stakeholders. As of June 2015, ACWUA had 330 members in 18 countries in the Middle East North Africa (MENA). Of these, the USAID Bureau for the Middle East supports 10 countries: Algeria, Egypt, Iraq, Jordan, Lebanon, Libya, Morocco, Tunisia, West Bank/Gaza, and Yemen. PURPOSE The purpose of the evaluation was to generate insights into IWSMR’s success in building ACWUA’s capacity to sustainably carry out its mandate to promote certification and accreditation, capacity development and information exchange among water and wastewater utilities and professionals during and beyond project completion. The evaluation also assesses ACWUA’s ability to sustain itself as a regional entity and manage funds given its future role in the region and its capacity at end of activity. METHODOLOGY The evaluation team used the following methods to conduct the evaluation:  Document review  KIIs (55 key informants, of which 48 were in person, six were by telephone, and one was by email)  Focus group discussions (FGDs) in Jordan and Egypt (two groups with six persons each)  Online survey of the Regional Operator Certification and Training Program (ROCTP) participants (13 responses out of 40 participants contacted) Key informants were selected from a broad range of stakeholders (representing nine different groups) based on the desk review and recommendations from USAID, ACWUA, and Chemonics/IWSMR staff, in order to obtain diverse institutional perspectives. The evaluation team spent seven days in Jordan and five days in Egypt and traveled to seven cities for interviews with utility company management (Amman, Aqaba, Karak, Salt, and Tafila in Jordan; Alexandria and Cairo in Egypt). KEY RESULTS  ACWUA management viewed the Pre-award Survey as an effective tool to develop an understanding of needed changes, including the tools to ensure ACWUA’s operational efficiency and effectiveness. By adopting a more transparent accounting system, in line with the Pre-award Survey recommendations which incorporates a breakdown of funding sources, ACWUA should be able to better manage member funds.  ACWUA’s leadership demonstrates an understanding of the importance of accountability, transparency and sustainability in its management and financial practices. IWSMR has, despite leadership challenges, equipped ACWUA with financial and managerial tools to strengthen a foundation for ongoing reforms and improvements. There is reason to believe the organization will USAID.GOV LEAP-II FINAL REPORT | 8 continue to grow and show prudent financial performance after the IWSMR activity ends. ACWUA’s ability to continue acting as a service provider will most likely depend on the following factors: i) the quality of the services it provides; ii) the resources (staff, organizational capacity, etc.) it has to organize and provide services; iii) its reputation; and iv) demand for services.  The positive feedback from both trainees and utility companies on the training organized by ACWUA has strengthened the organization’s reputation among members who have benefited from it. LEARNING LEAP-II produced an evaluation organized by theme that analyzed in detail ACUWA’s mission and objectives, accomplishments to date, and sustainability and growth post-IWSMR assistance. The report also examined in detail the results of ACWUA’s Non-US Organization Pre-Award Survey (NUPAS) and the status of implementation of the recommendations. Key recommendations included: conducting a follow-up pre-award survey of ACWUA; introducing internal M&E; implementing a financial audit mechanism, follow-on technical assistance to ACWUA (developing management capacity, building staff capacity, improving communications with membership, reducing the cost of training, expanding individual training programs, and benchmarking performance indicators appropriately); updates to ACWUA’s business plan; and increasing the number and capacity of trainers. The annexes to the report included a table with a summary of findings, conclusions, and recommendations that was designed to facilitate learning. #10 – MID-TERM PERFORMANCE EVALUATION: LIBERIA INVESTING FOR BUSINESS EXPANSION PROJECT (IBEX) AND SUSTAINABLE MARKETS INITIATIVE – LIBERIA (SMI-L) INTRODUCTION The purpose of the mid-term evaluations of the IBEX and SMI-L projects was to assess whether the programs were on course to meet their overall goals and objectives, to determine the effectiveness, efficiency, sustainability, and added value of these programs in Liberia, and to recommend specific opportunities to enhance programmatic effectiveness and impact. The three main objectives of the evaluation were:  Map efforts of private sector strengthening in Liberia and determine where USAID/Liberia should focus its efforts for future programming;  Identify factors that help or hinder projects’ achievements of expected outcomes and determine specific opportunities to enhance programmatic effectiveness and long-term sustainability;  Evaluate the existing Development Credit Authority (DCA) credit guarantee programs with Ecobank and International Bank (IB) Liberia. METHODOLOGY The evaluation drew on three types of data sources: 1) secondary data and information contained in activities and non-activities documents (both qualitative and quantitative), 2) KIIs, and 3) a survey of IBEX and SMI-L-assisted SMEs. The evaluation took place October to December 2015. The evaluation team conducted 35 semi-structured interviews with key stakeholders, including USAID, program staff, beneficiaries (SMEs and banks), donors, and government officials. The evaluation team analyzed qualitative data collected from KIIs for patterns that address the relevant questions and speak to effectiveness, efficiencies, relevance, potential sustainability, impeding/facilitating factors, and cause and effect. 9 | LEAP-II FINAL REPORT USAID.GOV The evaluation team also implemented a survey of IBEX and SMI-L client SMEs using Survey Monkey software. The survey was designed to collect data on the quality of services provided by IBEX and SMI-L, the ease at which SMEs can obtain financing, and general obstacles to doing business in Liberia. The survey was conducted primarily over the phone; several were completed in-person. A total of 121 surveys were completed (117 were conducted over the phone or in-person by the evaluation team and four were completed online by SMEs). Of the 200 SMEs selected for the survey, only 121 were completed given the time and budget constraints. KEY FINDINGS Overall the IBEX and SMI-L programs are performing as or better than expected at generating new loans for SMEs that are enhanced by the DCA guarantee (IBEX) and in creating value chain linkages between lead firms and SMEs (SMI-L). A review of program performance against targets shows that each program has achieved or was on its way to completing or exceeding its goal performance, and the SME Survey rates services from each program as good to excellent. SMI ‐ L’s services and training are well ‐ regarded by SMEs. However, the sustainability of loan facilitation is a major concern. Both IBEX and the banks view IBEX’s role as that of a gatekeeper rather than a loan facilitator. Overall the DCA facilities in Liberia have been successful at increasing access to credit for particular businesses but change of behavior among the banks is expected to be more limited. The banks indicate that the Central Bank of Liberia (CBL) expects for loans to be fully secured. By contrast CBL stated that the banks establish their own collateral requirements. The banks in the DCA program prefer land and buildings as collateral but are also broadening collateral categories to include contracts and assignment of proceeds. Banks and their loan officers could benefit from establishment of a national credit bureau, improved enabling environment, longer-term funding for loans, and training on collections and on serving women-owned SMEs. LEARNING IDG produced a comprehensive evaluation report with specific, actionable recommendations on how to expand the use of local BDS providers to deliver training and loan facilitation services to SMEs, suggested improvements to the projects’ M&E systems, how to systematically collaboration with the IFC and Central Bank on the DCA program, and making DCA guarantees for sourcing longer-term financing, training for commercial banks. The report also discussed in detail adjustments/areas of improvement to ensure IBEX and SMI-L achieve their objectives. At the USAID/Liberia’s request, the report included a section on private sector developing programing which covered the provision of broader access-to-finance assistance, and for placing access-to-finance programming within broader private-sector development competitiveness programming so that SMEs are being assisted not only with access-to-finance but also in a range of areas where help is needed. To facilitate the application of the recommendations, the report included annexes with descriptions of successful DCA programs and illustrative scopes of work for competit