Evalyasyon Final Pèfòmans Pwojè “CHABON BOUL” la

DISCLAIMER THIS PUBLICATION WAS PRODUCED AT THE REQUEST OF THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMEN T. IT WAS PREPARED INDEPENDENTLY BY PHANOL PHILIPPE WITH THE AS SISTANCE OF OTHER “CHASE CONSULTING” PERSONNEL. FINAL PERFORMANCE EVALUATION OF THE “CHABON BOUL” PROJECT PHOTO: A BIGCHAR PYROLYSIS RETORT AT THE “CHABON BOUL” FACTOR Y Photo: a BigChar pyrolysis retort at the “Chabon Boul” factory USAID/HAITI “CHABON BOUL” PROJECT IMPLEMENTED BY CARBON ROOTS HAITI S.A. Final Performance Evaluation Report Contract Number: AID-521-0-00-17-00039-00 Period of performance: 07/21/2017 to 12/13/2017 CHASE Consulting 10, Rue Acacia, Port -au-Prince, Haïti November 13, 2017 DISCLAIMER THIS PUBLICATION WAS PRODUCED AT THE REQUEST OF THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMEN T. IT WAS PREPARED INDEPENDENTLY BY PHANOL PHILIPPE WITH THE AS SISTANCE OF OTHER “CHASE CONSULTING” PERSONNEL. CONTENTS ACRONYMS 1 SUMMARY 2 1 EVALUATION PURPOSE AND EVALUATION QUESTI ONS 7 1.1. EVALUATION PURPOSE 7 1.2. EVALUATION QUESTIONS 7 2 PROJECT BACKGROUND 7 2.1 PROJECT DESCRIPTION 8 2.2 THEORY OF CHANGE 10 3 EVALUATION METHODS A ND LIMITATIONS 10 3.1. EVALUATION METHODS 10 3.1.1 OVERVIEW 10 3.1.2 DATA COLLECTION METHODS 11 A. IN-DEPTH PROJECT DOCUMENT REVIEW AND COST-BENEFIT ANALYSIS 11 B. FOCUS GROUP DISCUSSIONS (FGDS) AND WORKSHOP 12 C. SEMI-STRUCTURED INTERVIEWS WITH KEY INFORMANT 13 3.1.3 DATA SYNTHESIS AND ANALYSIS 14 3.2 LIMITATIONS 14 4 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 15 4.1 FINDINGS 15 4.1.1 NET REVENUE FROM THE GREEN CHARCOA L ACTIVITY 15 4.1.2 FUNDING, TECHNOLOGY AND SKILLS 15 4.1.3 PRODUCTION 17 4.1.4 RAW MATERIALS AVAILABILITY 19 4.1.5 QUALITY AND MARKETING 20 4 .2 CONCLUSIONS 21 4.3 RECOMMENDATIONS 21 4.4 SYNTHESIS OF LEARNING FROM THE IMPLEMENTATION OF THE “CHABON BOUL” ACTIVITY 21 ANNEXES 1 A1. SCOPE OF WORK 1 A2. DATA COLLECTION TOOLS 10 A3. DATA SOURCES 19 A4. SUMMARY OF PROJECT HISTORY 22 I. 1 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV ACRONYMS APS Annual Program Statement BoP Base of the Pyramid CBA Cost-Benefit Analysis CHASE Collectif Haïtien pour l’Avancement du Suivi et de l’Evaluation CRH Carbon Roots Haiti CRI Carbon Roots International DIV Development Innovations Ventures FGD Focus Group Discussions KII Key Informant Interview NPV Net Present Value ROI Return on Investment USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 2 SUMMARY Background Context and Objectives In July 2017, USAID/Haiti awarded CHASE 1 a contract to conduct the performance evaluation of the “Chabon Boul” project implemented in the Northern Department over the last two years. The purpose of the evaluation is to assess whether the Activity was able to achieve its intended outcome to produce a viable, affordable and cost-effective product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. This evaluation shall provide pertinent information to USAID and its partners on the overall impact of the Activity and serve as a basis for considering whether this activity can be replicated on a larger scale. The “Chabon Boul” Project hopes to achieve three major and intertwined strategic objectives: 1) increase briquette production, 2) expand the retail distribution network, and 3) increase sales of green charcoal briquettes. Green charcoal creates financial sustainability through jobs, income, and value throughout the local value chain while achieving a reduction in deforestation. The following questions were provided by USAID to be addressed during the evaluation process: 1. To what extent the Green Charcoal for Haiti Activity was able to achieve a positive net revenue? If yes, what were the most important factors contributing to this achievement? If not, what were the main constraints leading to this non-achievement? 2. What lessons learned can be derived from the Green Charcoal for Haiti business model to reduce the dependency on firewood and traditional charcoal? Evaluation Methodology Only qualitative methods were used for this evaluation. The evaluation team used the following methods: o Secondary data review, including all available project documentation: periodic reports, project design document, documents on the various modifications (See appendices 1), studies carried out within the framework of this project, annual plans, follow-up plans, M&E plans, etc. o 23 Key Informant Interviews (KII) with key stakeholders, including Mission staff, CRH personnel, beneficiaries, local rum distillers and micro-entrepreneur agricultural processors and other market actors; o Interactive workshop with CRH employees to discuss lessons learned from the project implementation; o Cost Benefit Analysis Findings, Conclusion, Recommendations and Lessons Learned 1 CHASE: “Collectif Haïtien pour l’Avancement du Suivi et de l’Évaluation” I. 3 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV A. Net revenue from the green charcoal activity (Question 1) The production of “Chabon Boul” in its present form is not a viable business. CRH has not been able to generate positive net revenue, despite USAID support. According to the project managers, the income from the sale of green charcoal, which represents the only source of revenue for the company, is always less than the sum of the expenses incurred by the activity. The company is now closed due to lack of financial means. The goals for the USAID-awarded 24-month grant period were to: • Grow revenue to achieve positive cash flow at the north Haiti production facility • Upgrade production equipment to commercialize all processing, from pyrolysis to briquette and drying. • Raise private sector financing for capital expenditures on production and transport equipment Key aspects that are considered in the analysis of the company’s operation are funding, production, key inputs, marketing and product quality. REVENUE. A large part of the amount granted was used, as planned, for the acquisition of modern equipment. In early 2017, Carbon Roots Haiti (CRH) installed and commissioned a commercial pyrolysis unit that convert sugarcane bagasse to charcoal. According to the members of the production team, these orientation sessions gave them the necessary capacity to use the machines. When planning the BigChar purchase, it was hoped that the machine could both convert bagasse into green charcoal and provide clean high heat to dry bagasse and green charcoal briquettes coming off the production line. This assumption was incorrect. The current process relies on solar energy over several days (three to five) to dry both the bagasse and the freshly released charcoal from the commercial briquette pressing machine. According to document review and various meetings in the field, the income from the sale of coal still decreased with the financing. According to the project managers, to reach the break -even point, the company must have at least five equipment units (BigChar) like the one bought with USAID funds. Today the company is looking for a grant of about 2.5 million USD to accomplish this benchmark. According to the project proposal, the project designers acknowledged that additional equipment units were needed for the activity to be profitable. As expected, and according to project personnel, the acquisition of the new equipment (the BigChar unit) was more like a demonstration of what the company could do with a more appropriate technology. The acquisition of the new equipment also puts the company in a mode of operation where it is more easily scalable. CRH reported that increasing the price of the ‘Chabon Boul’ product to increase revenue is not a viable option. Currently, an equal volume of the two brands of charcoals (traditional charcoal and Chabon Boul) are sold at the same average price. Monthly sales prior to the current shutdown are far below the projection of a $60,000 monthly sale limit to achieve profitability. With the acquisition of the new equipment, the company has reduced its staff. Therefore, the costs of the operations were reduced. However, the company still operates with more than twenty employees in addition to other expenses such as carrying bagasse (about 350 USD / month), taxes, fuel, repairs, etc. It is difficult for the evaluation team to provide strong conclusions and recommendations without access to information about the company’s monthly expenditures. There is no way to determine how increasing production will affect the future profitability of CRH. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 4 PRODUCTION . According to CRH managers, the green charcoal production has decreased with the USAID grant. Prior the BigChar installation, the company produced about 5 to 6 tons of charcoal per day; now the charcoal production is 1 to 1.5 tons per working day. This is related to the capacity of the machine. As production has decreased, less green charcoal is sold with support received from USAID. In addition to the reduction in production, the various stakeholders in the field also report lack of regularity in production. In fact, repeated breakdowns prevented the company from producing continuously with detrimental consequences on marketing and sales. Conversely, it is encouraging for the company that it usually sells everything it produces. Production varies with seasons as well since solar energy is required to dry the final products. The company tried an alternative means to dry the briquettes without much success. The company’s large distribution network was reduced with the decrease in production and sales RAW MATERIALS . Sugarcane bagasse is used to produce the “Chabon Boul”. According to several informants, the bagasse available can readily support several green charcoal factories the size of CRH. With dozens of distilleries in the northern plain, production has suffered just one slowdown due to a scarcity of bagasse since the launch of the activity. A disadvantage in the use of bagasse lies in the fact that it must be dried for use. In the rainy season, it is difficult to dry bagasse collected because the drying done under the sun. The project has been successful in promoting bagasse as a byproduct of distilleries. As an effect of CRH activities, bagasse now has a price. Distillery owners complained as well that the price paid by CRH for the cart of bagasse is not the same everywhere. At the beginning of operations, kits were provided to bagasse suppliers who were engaged in preparing the char themselves. It was soon realized that the char prepared in the 55- gallon metal drums was of poor quality which would have a direct impact on the quality of the finished product. Drums of char were bought according to their weight and some suppliers fraudulently placed stones in the drums to increase the weight of the char to gain more money. The decision was then made to centralize the management of bagasse combustion (char production) at the company level which increased the quality of the product and efficacy of the resources. Starch is another essential input in the production of green charcoal. According to information collected, locally produced starch made from cassava, especially in “cassaveries, 2” was used initially. Again, the company would face some suppliers who mixed starch with other products (flours) to increase their earnings. The mixture reduces the quality of the starch and consequently the quality of the resulting green charcoal. The decision was then made to source corn starch from major US-based industries, a fruitful initiative according to CRH managers. 2 A place where cassava or other cassava products are made. I. 5 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV MARKETING AND PRODUCT QUALITY . Immediately after the launch of the project, several types of marketing were used to reach potential consumers. Visits with potential buyers proved to be the most beneficial. According to stakeholders, consumers are generally satisfied with the price and the quality of the product. The quality of “Chabon Boul” has increased gradually with several key decisions made by the company's management. According to the consumers consulted, Chabon Boul is easy to use. They estimated that the Chabon Boul intensified the heat produced resulting in a shorter cooking time. Also reported was the Chabon Boul does not produce smoke that darkens cooking utensils. Finally, unlike traditional, bagged wood charcoal, the Chabon Boul briquettes remain intact in the bag. However, consumers visited also claim that Chabon Boul produces a lot of ash. According to the information gathered, if you do not pay attention, the large amount of ash produced is susceptible to blowing around, dirtying the kitchen and beyond. B. Synthesis of learning from the project implementation (Question 2) The “Lessons Learned” workshop offered an opportunity for reflection after the completed project. It was valuable to reflect on what worked well with the project and what could be improved upon from the perspective of representatives from the project team. Project documents were consulted as well in this sense. The main lessons are the following: Leadership. One of the main lessons learned revolves around decisions made by the management team. One such decision was the rotation of the production team for better results and better personnel management. The multiple tasks performed by the production team vary with some being far more demanding than others. There is the handling of the machine that refines dried bagasse, the preparation of the starch, the mixture of char, starch and other materials needed, monitoring the temperature of the BigChar to make sure everything is working fine. Since the company decided to give the same salary to everyone, it initially created frustration. The managers chose to rotate the 14-member production team and better distribute the workload. This improved the dynamics of the team and reduced their frustration. Research. Consistent conversations with consumers lead to ongoing research on how to improve the product to meet their requirements and achieve client satisfaction. Resulting improvements in char production include smokeless green charcoal and improved materials in its composition. Marketing. As mentioned above, several marketing methods have been used by the company to promote the "Chabon Boul." The marketing method of visiting potential consumers of the product and providing a sample amount to test at home yielded the best results; people like to test the product before making the decision to buy. After having tested it, the purchase decision followed almost automatically. Insufficient product supply was a barrier to marketing efforts. Sales strategy. Imitating the traditional charcoal marketing circuit directly in competition with the "Chabon Boul," sales managers sought to understand and then facilitated a rapid integration of the market. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 6 Quality control. Performing quality control throughout the entire production chain increases the quality of the final product. This quality control cannot be trusted to the many actors in the production chain, but by a company manager with control mechanisms implemented at all stages. Repeated stoppages of production. Repeatedly stopping production is detrimental to the range of customers the industry would like to have. Stoppages were most often due to equipment breakdown or problems related to lack of funds. Conclusion and recommendations According to all the information shared with the evaluation team and in line with projections, the “Chabon Boul” business did not break even with the grant they received from USAID. Already in the fourth modification, it is clearly stipulated that the project did not intend to achieve profitability with this grant and that it would need an additional funding. It said that the funds from this second grant from USAID should be used to install one new pyrolysis retort, which will serve as a demonstration of technology and scale to investors, who would then lend and invest another $1 million to bring the production factory to a sufficient scale to reach financial sustainability. As a result of decreased production and sales, projections have been further modified or adjusted as the company now needs a grant of approximately $ 2.5 million to achieve profitability. According to managers, in its current status, the company will still need external support to survive. Despite the repeated work stoppages, the company has a relatively faithful client base. While there is the chance that CRH may lose some potential customers and market share to propane, the main raw material, sugarcane bagasse, is available in a quantity largely sufficient to operate the current enterprise and after an increase in size. Two major drawbacks with bagasse are 1) in wet weather bagasse cannot be used due to lack of sunshine, and 2) in times of drought, there is shortage of bagasse. The quality of the product, according to the majority of the customers, is good and they are generally happy. The inconvenience of too much ash is offset by a competitive price, relatively shorter cooking time, and no smoke. The evaluation team recommends that the industry: • Has better expense appraisal to better estimate the break-even level and increase the level of transparency in the communication of financial information; • Regularizes the “Chabon Boul” production in order to increase customer loyalty (repeated shutdowns due to energy or other problems affect the market); • Regularizes the price of the bagasse cart to reduce frustrations; • Continues to improve the quality of the product (specifically the complaint of ashes during the burning process); Find better ways to dry bagasse and charcoals (Drying in open air is unreliable during raining seasons). I. 7 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV 1 EVALUATION PURPOSE AND EVALUATION QUESTI ONS 1.1. EVALUATION PURPOSE In July 2017, USAID/Haiti awarded CHASE (Collectif Haïtien pour l’Avancement du Suivi et d’Évaluation ) a contract to conduct the performance evaluation of the Green Charcoal for Haiti Activity (Chabon Boul Project). The purpose of the evaluation was to assess whether the Green Charcoal Activity was able to achieve its intended outcome to produce a viable, affordable and cost-effective product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. This evaluation shall provide pertinent information to USAID and its partners on the overall impact of the Green Charcoal Activity and serve as a basis for considering whether this activity can be replicated on a larger scale. The primary stakeholders for this evaluation include: USAID/Haiti, Carbon Roots International (CRI) and Carbon Roots Haiti (CRH). This document presents the final evaluation report including detailed methodology for the evaluation and describing key findings, conclusions and recommendations. 1.2. EVALUATION QUESTIONS 1. To what extent the Green Charcoal for Haiti Activity was able to achieve a positive net revenue? If yes, what were the most important factors contributing to this achievement? If not, what were the main constraints leading to this non-achievement? 2. What lessons learned can be derived from the Green Charcoal for Haiti business model to reduce the dependency on firewood and traditional charcoal? 2 PROJECT BACKGROUND The least well- off strata of the population, also called the communities at the Base of the Pyramid (BoP), are caught in a vicious circle. 3 This vicious circle consists of relying on charcoal as their main source of energy, triggering negative health outcomes, deforestation, increased greenhouse gas emissions, erosion, flooding, decreased arable land, declining crop yields, and depressed incomes forcing reliance on polluting, tree-consuming charcoal. According to the best available data 4, Charcoal is used by 90% of the households from Port-au-Prince and other major cities, and the charcoal chain from production to final consumption employs more than 150 thousand persons. Charcoal is made from trees, with a low conversion efficiency of about only 20% (5 kg of wood for 1 kg of charcoal) and its production contributes to degrade the local environment by adding pressure to the plant stand throughout the countryside which is already over 98% deforested. 3 Information from the “Chabon Boul” project document. 4 Bureau of Mines and Energy: Haiti Energy Sector Development Plan 2007 - 2017 USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 8 Viable alternative cooking technologies are not being adopted fast enough to stop or even slow deforestation, and forest decline has led to a rise in charcoal prices, in part due to the rapidly growing black market for illegal charcoal smuggled into Haiti from the Dominican Republic. In this broader context, the dependency on charcoal and firewood becomes a major factor in the environmental degradation of Haiti and poses a serious threat to the lives and livelihoods of Haitian citizens. The Green Charcoal for Haiti activity was designed partly to address this problem by demonstrating that cooking briquettes made from agricultural waste can be a viable, affordable, cost-effective alternative to the use of traditional charcoal, that will help reducing deforestation in Haiti and increasing rural incomes. Additionally, the use of green charcoal would reduce carbon emissions compared to the traditional charcoal, and it burns somewhat cleaner, lessening the adverse health effects on its users. 2.1 PROJECT DESCRIPTION The Green Charcoal for Haiti is a 24-month, $838,000 program implemented by CRH, a Haiti-based for-profit social business affiliated with a U.S.-based non-profit organization, Carbon Roots International (CRI). Since 2013, USAID/Haiti has supported Stage 1 and 2 grants (a total of $938,000) to CRI and CRH through the (DIV) Annual Program Statement (APS) to support their plan to replace the use of charcoal fuel in Haiti with a green alternative fuel. The overall goal of the activity is to create sustainable jobs and reduce deforestation, with an Intermediate Result of achieving financial sustainability at the briquette production factory. As presented in the project’s Monitoring and Evaluation (M&E) Plan, the “Chabon Boul” Project hopes to achieve three major and intertwined strategic objectives: 1) increase briquette production, 2) expand the retail distribution network, and 3) increase sales of green charcoal briquettes. Green charcoal creates financial sustainability through jobs, income, and value throughout the local value chain while achieving a reduction in deforestation. CRH’s business consists of four activities that comprise the green charcoal value chain: 1. Raw materials purchasing from local suppliers The value chain begins with a network of smallholder farmers and micro‐entrepreneur agricultural processors who earn added income by selling their agricultural waste biomass to CRH. In northern Haiti, the primary biomass feedstock used is sugarcane bagasse. Carbon Roots Haiti pays the suppliers for their waste, then transports it from the farm or processing facility to the production site. 2. Processing At a central production facility, CRH staff sorts and dries the feedstock, then pyrolyzes (i.e. carbonize) it into charcoal dust. Prior to February 2017, pyrolysis was achieved using handmade 55‐gallon batch kilns. Utilizing funding from USAID, CRH began upgrading the pyrolysis process to use continuous-feed retorts that also provide heat to streamline drying. After pyrolysis, CRH staff mixes the char dust with binding agents, resulting in a char slurry. The slurry is then pressed into dense briquettes using commercial roller briquette equipment, consisting of: I. 9 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV • A large batch mixer, • A conveyor belt, and • A commercial briquette pressing machine, which uses two opposing cylindrical molds to compress char slurry into pillow briquettes. After processing by the briquetting equipment, the briquettes are transferred to portable racks and dried in the sun for 3‐ 5 days. Upon upgrading to a new pyrolysis process, CRH should be able to dry briquettes in a matter of hours. After drying, the briquettes are packed in re‐useable sacks that are ready for transport and sale. 3. Marketing Most products available at local markets in Haiti are sold under brand names. However, due to the decentralized nature of wood charcoal production in Haiti, there are no recognized nationwide brands. This presents a competitive advantage for CRH by being able to produce and distribute green charcoal centrally, allowing the business to sell green charcoal under the “Chabon Boul” brand name. Marketing focuses on cost savings, and highlights that green charcoal is a locally‐ produced product that benefits the environment. CRH engages in many marketing media, including: • Television commercials and shows, • Radio commercials, • Banners and signs throughout the sales region, • Signage on public transportation, • Promotional events. Due to limited income in the BoP, potential customers may be cautious to purchase new products, the results of which they are unsure. To combat this tendency, CRH encourages consumption through special offers and free samples, which has proven successful at spreading awareness and increasing sales at new retail locations. For example, an individual distribution point selling green charcoal will drop the cost of a marmite (roughly one kilogram of charcoal) 40% for several hours, doubling new customer purchases during the sale. CRH continues to test different marketing strategies to determine the best ROI 5 and scaling up those effective tactics. By following up with prospective buyers, they can emphasize savings, appeal to widespread notions of civic duty, and tie CRH’s marketing to the Haitian Government’s reforestation initiatives. 4. Distribution CRH’s distribution strategy is three‐pronged: 5 ROI: Return on Investment USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 10 • Wholesale customers: CRH targets restaurants, informal roadside cooks, schools and organizations, selling in 30kg sacks and providing delivery service to the customer. • Women retailers: In Haiti, women traditionally sell charcoal at local markets. CRH’s second distribution strategy reinforces this inclusion of women in market activities by selling green charcoal briquettes to household users through a network of women retailers. These retailers are “based” out of one of CRH’s distribution points (depots/boutiques), where they purchase green charcoal in 30kg sacks, and then sell to household customers in smaller retail quantities. • Direct Sales: Carbon Roots Haiti has launched branded brick‐and‐mortar green charcoal boutiques, located in high‐traffic areas, and serve the household market directly. Generally, household customers purchase in “ marmite” units, often patronizing the boutique daily. CRH currently has boutiques in Port‐au‐Prince and Cap‐Haitian. Beneficiaries of the project include: • Local rum distillers and micro-agricultural entrepreneurs, who will earn significantly more than the national minimum wage, and several times more than the national average daily income. • Charcoal retailers, mostly women, who will sell green charcoal briquettes at a market discount and still earn roughly 15% profit on their sales. • End users, who will save between 25-40% of their income by using green charcoal. 2.2 THEORY OF CHANGE CRH’s Theory of Change postulates that deforestation, rural poverty and unemployment, energy demand, and low agricultural yields can be addressed through the deployment of a profitable social business engaged in the production of green charcoal. This enterprise benefits a wide range of stakeholders: • Farmers monetize their agricultural waste by converting it to char dust and selling it to CRH • Women retailers offer a highly- competitive product, enjoy higher profit margins, develop business skills, and build a broad customer base • Institutions such as schools and restaurants, and households have access to sustainable cooking fuel at a price point that is cheaper than traditional charcoal While the pathway to change is complex in a socioeconomic environment su ch as Haiti, a market-based enterprise that manufactures products locally, for local consumption, is likely the most impactful and sustainable solution. 3 EVALUATION METHODS A ND LIMITATIONS 3.1. EVALUATION METHODS 3.1.1 OVERVIEW The evaluation team had to determine whether the “Chabon Boul” Project was able to achieve its intended outcome, which is to produce a viable, affordable and cost-effectiveness product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. The team through analysis had to also help the mission to appreciate the overall impact of the project. To do this, the evaluation team considered using the following methods: I. 11 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV a) Secondary data review, including all available project documentation: periodic reports, project design document, documents on the various modifications (See appendices 1), studies carried out within the framework of this project, annual plans, follow-up plans, M&E plans, etc. b) Key Informant Interviews (KII) and Focus Group Discussions (FGDs) with key stakeholders, including Mission staff, CRH personnel, beneficiaries, local rum distillers and micro- entrepreneur agricultural processors and other market actors; c) Interactive workshop with CRH employees to discuss lessons learned from the project implementation; d) Cost Benefit Analysis. Above methods were used separately and the findings have not been integrated until after data analysis. Much of the document review took place first and findings from this step were used to inform the design and implementation of key informant interview and focus group methods. The same evaluation team was then involved in implementing key informant interview and focus group methods, and the actual data collection and analysis happened congruently over the same period. CHASE met first with the Carbon Roots Haiti personnel involved in monitoring the “Chabon Boul” Project. It was an opportunity to 1) know how the project was doing from the implementers’ perspective, 2) to record long term issues regarding the project and the evaluation, and 3) to identify other influential key stakeholders. This meeting also helped to determine specific geographical mapping of activities. Data collection was carefully planned in collaboration with Carbon Roots Haiti personnel. 3.1.2 DATA COLLECTION METH ODS A. IN-DEPTH PROJECT DOCUMENT REVIEW AND COST- BENEFIT ANALYSIS i. Literature review CHASE personnel conducted an extensive review of the literature. The documents review help in understanding both the theory of change behind the project and key project activities and results. The evaluation team drew on existing quantitative and qualitative data collected during the literature review to (1) refine sampling strategy and/or approach for collecting qualitative data, (2) design relevant qualitative data collection tools, and (3) seek compelling evidence to answer the evaluation questions. Relevant documents include, among others: the USAID/Haiti country strategy; “Chabon Boul” project reports such as work plans; PMP; M&E plans; quarterly progress reports and technical reports; the project proposal; and, documents relating to amendments. Ultimately, the literature review helped to gather relevant information to produce and enhance the comprehensive evaluation design. ii. Cost-Benefit Analysis Cost-benefit analysis (CBA) techniques were to be used to compare the total costs of the activity with its revenue, using only monetary units. This would enable the calculation of the net cost or benefit associated with the production of green charcoal. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 12 As a technique, CBA evaluates the overall impact of the activity in quantifiable and monetized terms by adding up the total costs of an activity and comparing it against its total revenue. For this evaluation, Net Present Value (NPV) of the project cost was to be compared with the net present value of its revenue. Decisions would be based on whether there was a net revenue or cost to the approach, i.e. total revenue less total costs. This comprehensive analysis takes a broad view of costs and benefits, including indirect and longer-term effects. This type of review typically begins with an exhaustive list of all the different costs that could arise, even if some will be later excluded: fixed costs included equipment partially mortgaged, rent, advertising, insurance, office supplies, etc. Variable costs included: personnel, training, vehicle (maintenance and repair), fuel, product inputs, etc. To find the business's profit, the team, with support from accounting and leadership teams, would add up all of the total sales of green charcoal the Activity has made in a set period of time (one month) and other revenues including (but not limited to) gain or loss on disposal. The team would then subtract any amount of cash refunded to customers for returns or disputes to find an accurate figure for the total income. Once the team found accurate values for the Activity’s total income and expenses, calculating the profit would simply be to subtract expenses from the income. The data would be calculated monthly on the last six months and averaged to get the monthly figure. It should be noted that the evaluation team was not provided the information necessary to achieve this Cost Benefit Analysis B. FOCUS GROUP DISCUSSIONS (FGDS) AND WORKSHOP i. Focus Group Discussions Focus Group Discussion was planned to be held during the data collection phase with relevant stakeholders to obtain viewpoints and evidence to answer the evaluation questions. Small groups of eight to 12 were set for a one- hour discussion around main topics identified in the design phase. Candidates for focus groups discussions were the charcoal retailers. Please note, the evaluation team was not able to conduct these focus groups. These discussions would have helped determine the status of the project implementation and assess whether the Green Charcoal Activity was able to achieve its intended outcome, to produce a viable, affordable and cost-effectiveness product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. These discussions would have highlighted the lessons learned. ii. Workshop with CRH staff to determine lessons learned A two-hour workshop was organized with Carbon Roots Haiti personnel to discuss about the implementation and to document lessons learned. The purpose of documenting lessons learned was to share and use knowledge derived from experience to promote the recurrence of desirable outcomes and preclude the recurrence of undesirable outcomes. This “Lessons Learned” session documented the cause of issues and the reasoning behind any corrective action already taken to address those issues. For the identified ““Lessons Learned”” where no action I. 13 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV had yet been taken, the team determined next steps and to identified who will be responsible for implementing those measures. Questions discussed during the session included the following: • What was learned about the project in general? Did the delivered Green Charcoal meet the specified requirements and goals of the project? • What was learned about project management? Did the project management methodology work? If not, why not? • What was learned about communication? What changes would assist in speeding up in the future while increasing communication? • What was learned about budgeting? Were costs budgets met? If not, why not? • What was learned about procurement? Was the schedule met? If not, why not? • What was learned about working with sponsors? • What was learned about working with customers? Was the customer satisfied with the green charcoal? If not, why not? • What was learned about what went well? • What was learned about what did not go well? • How will/was this incorporated into project? What could be done to improve the process? C. SEMI-STRUCTURED INTERVIEWS WITH KEY INFORMANT Interviews were conducted with key stakeholders and resources persons such as key CRH personnel, local distillers of rum, end users such as CRH targets restaurants, informal roadside cooks, schools and organizations, and branded brick‐and‐mortar green charcoal boutiques vendors. The main objective of these interviews was to assess whether the project was 1) able to achieve its intended outcome, which is to produce a viable, affordable and cost-effectiveness product, and 2) to discuss best practices and lessons learned, keeping focus on seeking evidence to answer the evaluation questions. Key stakeholders involved in the project activities and systems in place to boost green charcoal production were visited during data collection. Key Informant Interviews (KII) were conducted with: - CRH staffs: o To collect qualitative data about performance, get more insight about project implementation, challenges and constraints on the field, and document lessons learned (Question 1 and 2) o To collect information on the functioning and achievements of the “Chabon Boul” project from the implementer’s perspective (Question 1) o To have information on what facilitated achievement of performance or what prevented CRH from achieving the expected results (Question 1) o To collect information on effective measures/decisions taken to reduce production costs and increase the viability of the project (Question 1) - Local rum distillers and Micro-Entrepreneur agricultural processors: o Availability of sugarcane bagasse on a continuous basis (Question 1); o Their transformation of sugarcane bagasse into char dust (Question 1); o Their overall learning from the project (Question 2); o Their level of satisfaction with the production of the Green Charcoal (Questions 1 and 2) o How they compare this activity with the traditional charcoal production (Question 1). USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 14 - Green Charcoal end users (CRH targets restaurants, informal roadside cooks, schools and organizations) o To collect information on the ease of access to the product. (Question 1) o To get more insight about their level of satisfaction with the use of Green Charcoal (Question 1) o To have information on how they compare this product with the traditional charcoal (Question 1) o Collect some suggestions for improving the project in the future (Question 1) - Branded brick‐and‐mortar green charcoal boutique vendors o To collect information on performance, get more insight about project implementation, challenges and constraints in the field, and lessons learned (Questions 1 and 2); o Difference in product sales prices compared to traditional charcoal and what can motivate people to switch from traditional charcoal to Green Charcoal (Question 1); o Their level of satisfaction with the sale of the Green Charcoal and the type of feedback they have from customers (Questions 1 and 2). o Above described methods and techniques were used to collect data and provide answers for the two (2) evaluation questions. An exhaustive list of planned KII is presented in the summary table of data collection. 3.1.3 DATA SYNTHESIS AND ANALYSIS Parallel data analysis was conducted, weighting answers from various sources. Data from each source were analyzed independently using content analysis methodology, and the findings were integrated and triangulated into coherent statements about the evaluation questions. A synthesis framework was developed to coherently address the wealth of information yielded daily by the key informants’ interview reports. Evidence was clustered into findings that would lead to valid and relevant conclusions and recommendations about the evaluation questions. Triangulation of data from a variety of sources, including both quantitative and qualitative data from literature review, qualitative data collected on the field, was valued such that the evaluation produced evidence-based findings, conclusions, and recommendations. The contents of the discussion were examined to discover the meaning and its particular implications for the evaluation questions. Every effort to interpret focus group/key informant interview data occurred through analysis of content. 3.2 LIMITATIONS The focus groups scheduled with retailers could not take place because this group has been demobilized for several months and according to the managers contacted, it was not possible to have them for several semi-structured meetings as planned. The cost-benefit analysis was not performed as planned in the design document/evaluation plan. According to the director and other managers contacted, the nature of the contract with the donor and the type of business involved meant that the director was not obliged to share cost information incurred during the operations and chose not to. Therefore, the analysis considers only the limited information available in reports of milestones or collected during field interviews. No accounting documents were shared with the evaluation team. I. 15 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV 4 FINDINGS, CONCLUSION S AND RECOMMENDATION S 4.1 FINDINGS 4.1.1 NET REVENUE FROM THE GREEN CHARCOA L ACTIVITY The production of Chabon Boul in its present form is not a viable business. CRH, a small enterprise, producing green charcoal (the "Chabon Boul") in the northern department of Haiti, has not been generating positive net revenue, despite USAID support. According to the project managers, the income from the sale of green charcoal, which represents the only source of revenue for the company, is always less than the sum of the expenses incurred by the activity. The company is now closed due to lack of financial means. 4.1.2 FUNDING, TECHNOLOGY AND SKILLS The goals for the USAID- awarded 24-month grant period were to: • Grow revenue to achieve positive cash flow at the north Haiti production facility • Upgrade production equipment to commercialize all processing, from pyrolysis to briquette and drying. • Raise private sector financing for capital expenditures on production and transport equipment A large part of the amount granted was used, as planned, for the acquisition of modern equipment. In early 2017, Carbon Roots Haiti (CRH) installed and commissioned a commercial pyrolysis unit that convert sugarcane bagasse to charcoal, replacing the organization’s reliance on hand-made 55-gallon barrel kilns. The pyrolysis unit is a BigChar 1200 model made by Pyrocal Ltd in Australia. Several training sessions were organized on the use of equipment. According to the members of the production team, these orientation sessions gave them the necessary capacity to use the machines. The BigChar equipment was purchased with the expectation that it could convert bagasse into green charcoal, and provide clean, high heat to dry both bagasse and green charcoal briquettes coming off the production line. Unfortunately, it was discovered that the BigChar cannot reliably dry the bagasse or freshly- manufactured green charcoal. The project relies on three to five days of solar power to dry both the bagasse and the freshly released charcoal from the commercial briquette pressing machine. With no assurances on weather, an alternative for drying and storage must be found. According to document review and various meetings in the field, CRH received all the funds allocated to the project but the income from the sale of coal still decreased with the financing. According to the project managers, to reach the break-even point, the company must have at least five equipment units (BigChar) like the one bought with USAID funds. Today the company is looking for a grant of about 2.5 million USD to accomplish this benchmark. In the project design document, it was clearly stated that the activity would not be profitable with this second grant granted by USAID. The project designers acknowledged that additional equipment units were needed for the activity to be profitable. As expected, and according to project managers, the acquisition of the new equipment (the BigChar unit) was more like a demonstration of what the company could do with a more appropriate technology. The acquisition of the new equipment also puts the company in a mode of operation where it can become scalable. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 16 Figure 1. Monthly sales for 3 months (Source: CRH Milestone reports) The enterprise relies only on green charcoal sales for revenues. In the past, the char dust was given free to farmers but sold to some agricultural sector institutions as organic fertilizer. CRH reported that increasing the price of the ‘Chabon Boul’ product to increase revenue is not a viable option. Currently, an equal volume of the two brands of charcoals (traditional charcoal and Chabon Boul) are sold at the same average price. The 30-kg bag that looks like most of traditional charcoal bags sells for 500 gourdes 6. According to various stakeholders, any increase in the price of “Chabon Boul” higher than that of traditional charcoal will scare away the green charcoal potential buyers. As can be seen in Figure 1, monthly sales prior to the current shutdown are well below the projection of a $60,000 monthly sale limit to achieve profitability. With the acquisition of the new equipment, the company has reduced its staff. Therefore, the costs of the operations were reduced. However, the company still operates with more than twenty employees in addition to other expenses such as carrying bagasse (about 350 USD / month), taxes, fuel, repairs, etc. Since the company did not aim for profitability with this second grant, the numbers featured in Figure 1 do not come as a surprise. However, it is difficult for the evaluation team to provide strong conclusions 6 500 Gourdes is the equivalent of 7.91 USD (1:63.18) 209 331 508 33 1081 Apr-17 May-17 Jun-17 Jul-17 Total Monthly sale in US Dollar I. 17 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV and recommendations without access to information about the company’s monthly expenditures. There is no way to determine how increasing production will affect the future profitability of CRH. 4.1.3 PRODUCTION As mentioned earlier, this second grant enabled CRH to acquire new equipment and at the same time enabled the company to move from a traditional processing to a less labor-intensive industrial processing. Today, the production team is reduced from more than 34 employees’ pre-modernization to 14 with the arrival of BigChar. Figure 2. Sales in volume from April to July 2017 According to CRH managers, the green charcoal production has decreased with the USAID grant. Prior the BigChar installation, the company produced about 5 to 6 tons of charcoal per work day; now the charcoal production is 1 to 1.5 tons per work day. As production has decreased, less green charcoal is sold with support received from USAID. To restore the production levels to 6 tons, the machine would need to run 24 hours a day. While it has the capacity, a considerable limitation is the necessary manpower. This approach would require adding several eight-hour shifts and sufficient security and transportation; a considerable challenge in that location. While the company usually sells everything it produces, stakeholders in the field report unreliable production output. Repeated breakdowns cause interruptions in the supply for days and weeks at a time. Once the issues are resolved and the supply is restored, additional marketing and sales resources are needed to publicize that the company is operational again. Production varies with seasons as well since solar energy is required to dry the final products. Unfortunately, when briquettes are needed most during the rainy season, production decreases due to the lack of sunlight needed to dry bagasse and green charcoal. The company tried an alternative means to dry the briquettes without much success. The machine bought for this purpose burnt too much briquettes while charcoals inside the device failed to receive the amount of heat needed. 28 51.5 72 5 156.5 Apr-17 May-17 Jun-17 Jul-17 Total Monthly sale in number of 30 Kg sacks USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 18 The company’s large distribution network was reduced with the decrease in production and sales. The green charcoal produced by the company was sold on site or through the few kiosks located in the city of Cap-Haitian and its suburbs. The “Chabon Boul” is no longer sold by street vendors and motorcycles. More recently, a new distribution network has been established in Port-au-Prince and absorbs more than 50% of the production, according to the sales manager. On the left is the char and the "BigChar" machine and on the right, the Pyrolysis hearth where refined bagasse is transformed into "char" I. 19 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV Machine used to enhance the sugarcane bagasse 4.1.4 RAW MATERIALS AVAILABILITY Originally, during the project design, several raw materials including agricultural waste and sugar cane bagasse were targeted for green charcoal production. In the northern plain, agricultural waste is not as continuously available as cane bagasse. According to several informants, the bagasse available can readily support several green charcoal factories the size of CRH. For example, the head of one of the big distilleries visited in the area claims that it is possible to provide CRH up to 10 carts of bagasse a day when his business is running at full capacity (this quantity of bagasse can meet the need of CRH for the day). With dozens of distilleries in the northern plain, production has suffered just one slowdown due to a scarcity of bagasse since the launch of the activity. That time was during the prolonged drought period of 2015. Carriers had to travel several tens of kilometers to supply the factory with bagasse. Stocks of the various usual suppliers were exhausted because of low sugarcane production. A disadvantage in the use of bagasse lies in the fact that it must be dried for use. In the rainy season, it is difficult to dry bagasse collected because the drying done under the sun. Even in good weather, drying bagasse in open air can take several days, delaying the green charcoal production. The project has been successful in promoting bagasse as a byproduct of distilleries. Distilleries are small companies that make clairin from sugar cane, an edible alcohol with a concentration of about 20 o. Normally, 20% of the bagasse is used as energy source to boil sugarcane juice transforming it into clairin, and the remaining 80% goes to waste. As an effect of CRH activities, bagasse now has a price; the suppliers who usually burned it before to get rid of it, now recognize its value and haggle regarding the price. Distillery owners complained as well that the price paid by CRH for the cart of bagasse is not the same everywhere. A bagasse cart costs on average 500 gourdes (7.91 USD). At the beginning of operations, kits were provided to bagasse suppliers who were engaged in preparing the char themselves. It was soon realized that the char prepared in the 55- gallon metal drums was of poor quality which would have a direct impact on the quality of the finished product. Drums of char Machine to refine the sugarcane bagasse USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 20 were bought according to their weight and some suppliers fraudulently placed stones in the drums to increase the weight of the char to gain more money. The decision was then made to centralize the management of bagasse combustion and char production at the company level which increased the quality of the product and efficacy of the resources. Starch is another essential input in the production of green charcoal. According to information collected, locally produced starch made from cassava, especially in “cassaveries, 7” was used initially. Again, the company would face some suppliers who mixed starch with other products (flours) to increase their earnings. The mixture reduces the quality of the starch and consequently the quality of the resulting green coal. The decision was then made to source corn starch from major US-based industries, a fruitful initiative according to CRH managers. They reported that the cost associated with the acquisition of corn starch is relatively low and can be sustained by the company once the break- even point is reached. 4.1.5 QUALITY AND MARKETING Thereafter the launch of the project, several types of marketing were used to reach potential consumers including radio advertising, banners, posters, and home visits, followed by distribution of small quantities of charcoal to sample. Visits with potential buyers proved to be the most beneficial. Unfortunately, all marketing activities ceased very soon as the demand outpaced the supply. According to stakeholders, consumers are generally satisfied with the price and the quality of the product. The quality of “Chabon Boul” has increased gradually with several key decisions made by the company's management. The decision to centralize the production of the "char" has increased the quality of the resulting coal. The acquisition of new equipment (BigChar and others), allows for better control over the partial combustion of bagasse resulting in a better char dust. The briquettes are no longer produced by hand as they were at the beginning and the decision to change shift from flour- tainted cassava starch to corn starch allows for tighter quality control. According to the consumers interviewed, Chabon Boul is easy to use. There is no need to buy a different, special stove. They estimated that the Chabon Boul intensified the heat produced resulting in a shorter cooking time. Also reported was the Chabon Boul does not produce smoke that darkens cooking utensils. Finally, unlike traditional, bagged wood charcoal, the Chabon Boul briquettes remain intact in the bag. However, consumers visited also claim that Chabon Boul produces a lot of ash. The head of an education center, a very good customer of Chabon Boul, claims to have stopped using it because it produces too much ash. According to the information gathered, if you do not pay attention, the large amount of ash produced is susceptible to blowing around, dirtying the kitchen and beyond. 7 A place where cassava or other cassava products are made. I. 21 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV 4.2 CONCLUSIONS According to all the information shared with the evaluation team and in line with what were planned, the “Chabon Boul” business did not break even with the grant they received from USAID. Already in the fourth modification, it is clearly stipulated that the project did not intend to achieve profitability with this grant and that it would need an additional funding. It said that the funds from this second grant from USAID should be used to install one new pyrolysis retort, which will serve as a demonstration of technology and scale to investors, who would then lend and invest another $1 million to bring the production factory to a sufficient scale to reach financial sustainability. As a result of decreased production and sales, projections have been further modified or adjusted as the company now needs a grant of approximately $ 2.5 million to achieve profitability. According to managers, in its current status, the company will still need external support to survive. Despite the repeated work stoppages, the company has a relatively faithful client base. While there is the chance that CRH may lose some potential customers and market share to propane, the main raw material, sugarcane bagasse, is available in a quantity largely sufficient to operate the current enterprise and after an increase in size. Two major drawbacks with bagasse are 1) in wet weather bagasse cannot be used due to lack of sunshine, and 2) in times of drought, there is shortage of bagasse. The quality of the product, according to the majority of the customers, is good and they are generally happy. The inconvenience of too much ash is offset by a competitive price, relatively shorter cooking time, and no smoke. It is important to note that green coal as produced by the project "chabon boul" has a future in the North region. Customers are satisfied with the quality and even without marketing activities the demand for the product continues to increase. Finally, even with the limited information that was communicated by the project, the evaluation team identified serious inadequacies in the financial projections of the activity. 4.3 RECOMMENDATIONS The evaluation team recommends that the industry: • Has better expense appraisal to better estimate the break- even level and increase the level of transparency in the communication of financial information; • Regularizes the “Chabon Boul” production in order to increase customer loyalty (repeated shutdowns due to energy or other problems affect the market); • Regularizes the price of the bagasse cart to reduce frustrations; • Continues to improve the quality of the product (specifically the complaint of ashes during the burning process); • Find better ways to dry bagasse and charcoals (Drying in open air is unreliable during raining seasons). 4.4 SYNTHESIS OF LEARNIN G FROM THE IMPLEMENTATION OF THE “CHABON BOUL” ACTIVITY The “Lessons Learned” workshop offered an opportunity for reflection after the completed project. It was valuable to reflect on what worked well with the project and what could be improved upon from the perspective of representatives from the project team. The evaluation team worked with a reduced USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 22 number of CRH staff due to the difficult circumstances faced by the enterprise. Project documents were consulted as well in this sense. Leadership. One of the main lessons learned revolves around decisions made by the management team. One such decision was the rotation of the production team for better results and better personnel management. The multiple tasks performed by the production team vary with some being far more demanding than others. There is the handling of the machine that refines dried bagasse, the preparation of the starch, the mixture of char, starch and other materials needed, monitoring the temperature of the BigChar to make sure everything is working fine. Since the company decided to give the same salary to everyone, it initially created frustration. The managers chose to rotate the 14-member production team and better distribute the workload. This improved the dynamics of the team and reduced their frustration. Research. Consistent conversations with consumers lead to ongoing research on how to improve the product to meet their requirements and achieve client satisfaction. Resulting improvements in char production include smokeless green charcoal and improved materials in its composition. Marketing. As mentioned above, several marketing methods have been used by the company to promote the "Chabon Boul." The marketing method of visiting potential consumers of the product and providing a sample amount to test at home yielded the best results; people like to test the product before making the decision to buy. After having tested it, the purchase decision followed almost automatically. Insufficient product supply was a barrier to marketing efforts. Sales strategy. Imitating the traditional charcoal marketing circuit directly in competition with the "Chabon Boul," sales managers sought to understand and then facilitated a rapid integration of the market. In general, traditional charcoal is available in depots in peri-urban areas, in the open markets where it is distributed in detail by retailers, through street vendors who use animal backs or their heads to transport them to consumers, etc. The "Chabon Boul" brought small touches of innovation, such as initially selling it in wheelbarrows through the streets. During the rainy season, the lots cannot be exposed in the open and the traditional charcoal is generally sold in bags during these periods - "Chabon Boul" followed the same method. Quality control. Performing quality control throughout the entire production chain increases the quality of the final product. This quality control cannot be trusted to the many actors in the production chain, but by a company manager with control mechanisms implemented at all stages. CHR had initially trained distillery partners to produce the char the company would buy from them according to weight. 55-gallon metal drums were distributed to the managers of various distilleries; the partners added other heavier products to the char and even stones to increase the weight of the drums. Not only did this reduce the quality of the coal but also damaged the machines on several occasions. Centralized production of char at the company has subsequently increased the quality. Repeated stoppages of production. Repeatedly stopping production is detrimental to the range of customers the industry would like to have. Stoppages were most often due to equipment breakdown or problems related to lack of funds. Before starting operations again, the company must take some time to re-engage customers or even relaunch some advertising incurring additional costs. According to the participants in the Lessons Learned workshop, consumers who simply switched to traditional charcoal I. 23 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV could easily revert back to Chabon Boul unlike those who may have converted their kitchens to propane. Advertisements for "Chabon Boul" tout that the environment is protected by using it. But when motivated consumers can no longer find this "Chabon Boul" and are forced to use traditional charcoal again, they become disenchanted by the company. According to the participants, it would be good, if possible, to maintain a small production core in order to keep the customers abreast of ongoing activities and smooth transitions between stoppages. I. 1 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV ANNEXES A1. SCOPE OF WORK PURPOSE OF THE EVALUATION The purpose of the evaluation is to assess whether the Green Charcoal Activity was able to achieve its intended outcome, which is to produce a viable, affordable and cost-effectiveness product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. This evaluation shall provide pertinent information to USAID and its partner on the overall impact of the Green Charcoal Activity shall serve as a basis for considering whether or not this activity can be replicated on a larger scale. The primary stakeholders for this evaluation include: USAID/Haiti, Carbon Roots International (CRI), Carbon Roots Haiti (CRH). SUMMARY INFORMATION Strategy/Project/Activity Name Green Charcoal for Haiti Implementer Carbon Root Haiti S.A (CRH) Cooperative Agreement/Contract # AID-521-F-15-00005 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) S838,000.00 Life of Activity [June 2015- May 2017] Active Geographic Regions [North] Development Objective(s) (DOs) [Economic Security Increased] USAID Office Economic Growth and Agricultural development USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 2 BACKGROUND A. Description of the Problem, Development Hypothesis (es), and Theory of Change According to the best available data, over 90 percent of Haitian households use charcoal – produced from burning wood - as their primary source of energy. This dependency on charcoal and firewood propels destructive effects such as, negative health outcomes, deforestation, increased greenhouse gas emissions, erosion, flooding, decreased arable land, declining crop yields, and reduced incomes—thereby reinforcing reliance on polluting, tree-consuming charcoal. Viable alternative cooking technologies are not being adopted fast enough to stop or even slow deforestation, and forest decline has led to a rise in charcoal prices, in part due to the rapidly growing black market for illegal charcoal smuggled into Haiti from the Dominican Republic. In this broader context, the dependency on charcoal and firewood becomes a major factor in the environmental degradation of Haiti and poses a serious threat to the lives and livelihoods of Haitian citizens. The Green Charcoal for Haiti activity was designed to address this problem by demonstrating that cooking briquettes made from agricultural waste can be a viable, affordable, cost-effective alternative to the use of traditional charcoal, that will help reducing deforestation in Haiti and increasing rural incomes. Additionally, the use of green charcoal would reduce carbon emissions compared to the traditional charcoal, and it burns somewhat cleaner, thus providing some health benefits for its users. B. Summary Strategy/Project/Activity/Intervention to be evaluated The Green Charcoal for Haiti is a 24-month, $838,000 program implemented by CRH, a Haiti-based for-profit social business affiliated with a U.S.-based non-profit organization, Carbon Roots International (CRI). Since 2013, USAID/Haiti has supported Stage 1 and 2 grants (a total of $938,000) to CRI and CRH through the Development Innovations Ventures (DIV) Annual Program Statement (APS) to support their plan to replace the use of charcoal fuel in Haiti with a green alternative fuel. Graphic 1 below summarizes the result framework of the Green Charcoal for Haiti activity as presented in the project’s Monitoring and Evaluation (M&E) Plan. As illustrated, the overall goal of the activity is to create sustainable jobs and reduce deforestation, with an Intermediate Result of achieving financial sustainability at the briquette production factory. To attend this goal, the project has three intertwined strategic objectives: 1) Increase briquette production, 2) Expand retail distribution network, 3) Increase sales of green charcoal briquettes. Every ton of green charcoal replaces destructive wood charcoal, and creates jobs, income, and value all along the local value chain. In this framework, achieving financial sustainability results in more jobs, higher incomes, and a reduction in deforestation. To achieve its strategic objectives, CRH has designed the project around 3 specific core activities: 1) Activity 1 consists of the procurement of agricultural waste supplied by micro- agricultural entrepreneurs and local rum distilleries. CRH has established sourcing agreements with the local distillers and the local farmers, who are paid for supplying the sugarcane bagasse to CRH. A group of farmers is then trained by CRH to burn and convert the bagasse into char dust, at CRH’s factory site in Quartier Morin. CRH currently has a network of over 100 active char dust suppliers in Northern Haiti. I. 3 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV 2) Activity 2 consists of the production of the green charcoal briquettes. Roughly 90% of the char collected is run through a grinder and crushed to dust. This char dust will then be mixed with starch into a slurry, and run through commercial roller briquette equipment that produces roughly 2,000 kilograms of green charcoal briquettes per day. The resultant green charcoal briquettes are dried for 3-5 days, and then packaged for transport and sale. 3) Activity 3 consists of the sales of the briquettes in the local markets, using three main channels: a) negotiated contracts with existing charcoal wholesalers, who have their own sophisticated distribution networks, b) CRH-branded distribution points and boutiques, established in Cap Haitian and Port-au-Prince, c) a network of individuals who are trained to become independent mobile street retailers. Carbon Roots Haiti also sells to individual consumers of charcoal in and around Cap-Haitien through several distribution points and boutiques. At local markets in Cap Haitien, charcoal is sold by women trained by CRH to become mobile street retailers. Beneficiaries of the project include: - Local rum distillers and micro-agricultural entrepreneurs, who will earn significantly more than the national minimum wage, and several times more than the national average daily income. - Smallholder farmers who are trained as part-time producers of charcoal dust, and will earn additional earnings from producing the char at the CRH facility. - Charcoal retailers, mostly women, who will sell green charcoal briquettes at a market discount and still earn roughly 15% profit on their sales. - End users, who will save between 25-40% of their income by using green charcoal. Graphic 1: Summary of Green Charcoal for Haiti Result Framework USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 4 C. Summary of the Project/Activity Monitoring, Evaluation, and Learning (MEL) Plan It is important to underline that no baseline study had been conducted at the inception of this program. However, as part of the M&E Plan, the activity collected baseline data and key monitoring indicators to track the progress of the program towards meeting the expected results. From the award date to today the program has achieved the following results, as of September 2016: • 3.95 tons of briquettes produced daily on average, from 0.5 tons at the beginning of the project. • 41 direct jobs, including CRH staff, from 45 jobs at the beginning of the project. • 5 active distribution points, from a total of 6 at the beginning of the project. • 41 indirect jobs created, including street retailers, from 72 at the beginning of the project. • $86,157 of sales revenue, compared to $8,815.32 at the beginning of the project. • $3,100.8 of supplier earnings, compared to over $12,000 at the beginning of the project. • 1,936 tons of wood saved, compared to over 104.9 tons at the beginning of the project. Below is a list of documents which will be useful to this evaluation. Soft copies of these documents will be handed to the evaluation team in due time. 1. Activity program description and any modifications made to it. 2. Activity Monitoring and Evaluation Plans 3. Activity work plan 4. Milestone Reports EVALUATION QUESTIONS 1. To what extent the Green Charcoal for Haiti Project was able to achieve a positive net revenue? If yes, what were the most important factors contributing to this achievement? If not, what were the main constraints leading to this non- achievement? 2. What lessons learned can be derived from the Green Charcoal for Haiti business model to reduce the dependency on firewood and traditional charcoal? EVALUATION DESIGN AND METHODOLOGY It is expected that the evaluation team propose a research design to address the evaluation questions and a plan for collecting and analyzing the data. Given the time and the nature of the evaluation questions, the evaluation design should be based solely on a combination of qualitative method techniques. Key informant interviews, focus group interviews, field observation, and in-depth review of project documents are among the techniques that the evaluation team should consider when addressing the evaluation questions. I. 5 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV The proposed evaluation design should: (1) Draw on existing quantitative and qualitative data (e.g., implementing partners' quarterly reports and data) as well as new qualitative data collection and analyses (e.g., analysis of data collected through interviews, focus groups, and field observation); (2) Describe the methodology and/or approach for collecting the data; (3) Use an appropriate combination of mixed methods to analyze both quantitative and qualitative data -- e.g., triangulation of data from a variety of sources -- such that the evaluation produces findings, conclusions, and recommendations grounded in evidence; and (4) Include steps the evaluator will take to assess and describe the quality of the data used for the evaluation. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work plan: Within 1 week of the award of the contract, a draft work plan for the evaluation shall be completed by the lead evaluator and presented to the Contracting Officer’s Representative (AOR/COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities. 2. Evaluation Design: Within 1 week of approval of the work plan, the evaluation team must submit to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR) an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design; and (5) a dissemination plan.USAID/Haiti program and technical offices and relevant stakeholders are asked to take up to 10 business days to review and consolidate comments through the COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 days. 3. Final Presentation: The evaluation team is expected to hold a final presentation in person to discuss the summary of findings and recommendations to USAID. This presentation will be scheduled as agreed upon after the evaluation fieldwork. 4. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, Program and EGAD offices will have 10 business days in which to review and comment on the initial draft, after which point the COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days hence, and again the Program and EGAD offices will review and send comments on this final draft report within 5 business days of its submission. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 6 5. Final Evaluation Report: The evaluation team will be asked to take no more than 10 business days to respond/incorporate the final comments from the aforementioned USAID offices. The evaluation team leader will then submit the final report to the COR. All project data and records (including Focus Group and Key Informant summary reports) will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. EVALUATION TEAM COMPOSITIO N The Evaluation Team shall comprise of an evaluation Team Leader (TL). The Team Leader (TL) is ultimately responsible for the overall management of the evaluation team and the final products, in conformity with this Scope of Work. The TL must be an experienced evaluation expert, with a documented track record of 10 years’ experience in the field of evaluation. The TL shall also have demonstrated experience in evaluating finance-related projects. The TL shall have a Master’s Degree in Economics, environment, or other related field and must possess excellent writing and interpersonal skills. The Team Leader will be responsible for planning the evaluation, coordinating the implementation of the evaluation, assigning evaluation responsibilities and tasks, and authoring the final evaluation report. He/she must be fluent in English and French. The Team Leader may propose additional staff (with CVs) to carry out this evaluation. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. EVALUATION SCHEDULE Timing (Anticipated Months or Duration) Proposed Activities June 1 to June 16, 2017 Preparation of the work plan and evaluation design June 19 June 30 USAID review of the work plan and evaluation design July 3 to July 14 Data Collection July 17 to July 21 Data Analysis I. 7 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. July 24 to August 4 Report writing August 7 to August 18 USAID review of Draft Report August 20 to August 24 Incorporate USAID comments and prepare Final Report ESTIMATED LOE IN DAY S Position Preparation Travel to/from Northern region/PAP In-Country Data Collection Finalization of Report Total LOE in days Team Leader 10 2 15 20 47 Data collector 0 2 15 0 17 Data collector 0 2 15 0 17 Totals 10 8 60 35 113 USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 8 The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information, properly identified and listed; and • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. • Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. • Summary information about evaluation team members, including qualifications, experience, and role on the team. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 8 • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. 8 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. I. 9 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person- level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action- oriented, practical, and specific. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 10 A2. DATA COLLECTION TOOLS Focus group with Green Charcoal retailers Thank you for agreeing to participate. We are very interested in hearing your views on the Green charcoal for Haiti Activity. We hope to learn things that will orient interventions for future programming. All that will be said during the one-hour meeting has great value and we would like the opinion of each of you on each question. If you do not want to answer a question, it is your right not to respond. However, be certain that the information provided is completely confidential and will only be used for one purpose: to help us understand the situation of the “Chabon Boul” Project. With your permission, we will take note of everything that was said during the meeting. We strongly wish that you attend the meeting. You just have to raise your hand. 1. What do you know about the “Chabon Boul” project? 2. How did you find out this information? 3. How were you selected as retailers? Did you sell traditional charcoal before? 4. How do you participate in the project? Which activities were implemented? 5. For the last six months have you noticed an increase or decrease in demands of Green Charcoal? Why? 6. What worked very well and what could be done better? 7. What are the biggest challenges that the project faces? 8. Why would a customer choose to buy green charcoal instead of traditional charcoal? 9. What kind of training did you receive and how useful was that training? 10. Overall, how satisfied are you with the sale of the Green Charcoal and what type of feedback do you have from customers? 11. Considering your available funds, can you buy from the project factory any amount of Green Charcoal that you want to buy and at any time of the year? 12. Did you use a credit program to participate in the project? How easy is it to get credit? 13. What are some suggestions for improving the project in the future? I. 11 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV Key Informant Interview: Local rum distillers and Micro-Entrepreneur agricultural processors Thank you for agreeing to participate. We are very interested in hearing your views on the Green charcoal for Haiti Activity. We hope to learn things that will orient interventions for future programming. All that will be said during the 30 to 45 minutes meeting has great value and we would like the opinion of each of you on each question. If you do not want to answer a question, it is your right not to respond. However, be certain that the information provided is completely confidential and will only be used for one purpose: to help us understand the situation of your institution. With your permission, we will take note of everything that will be shared during the meeting. 1. What do you know about the Green Charcoal for Haiti Activity? 2. How did you find out this information? 3. How were you selected as sugarcane bagasse provider? 4. How do you participate in the project? (Meetings, trainings?) / Is the relationship with the project done individually or through any association? Please explain. 5. To what extent can we say that sugarcane bagasse are available throughout the year? 6. To what extent are you / were you involved in the conversion of sugarcane bagasse into char? Tell us a little. 7. What unit of measure used to sell the sugarcane bagasse/char to the project? How much can a distillers or micro-entrepreneur sell on average per week / per month? 8. What worked very well and what could be done better? 9. What are the biggest challenges that the project faces? 10. What kind of training did you receive and how useful was that training? 11. Overall, how satisfied are you with the sale of the sugarcane bagasse? 12. What are some suggestions for improving the project in the future? USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 12 Key Informant Interview with selected end users (informal roadside cooks, schools, organizations) Thank you for agreeing to participate. We are very interested in hearing your views on the Green charcoal for Haiti Activity. We hope to learn things that will orient interventions for future programming. All that will be said during the 30 to 45 minutes meeting has great value and we would like the opinion of each of you on each question. If you do not want to answer a question, it is your right not to respond. However, be certain that the information provided is completely confidential and will only be used for one purpose: to help us understand the situation of your institution. With your permission, we will take note of everything that will be shared during the meeting. 1. What do you know about the Green Charcoal for Haiti Activity? 2. How did you find out this information? 3. How long has your institution used Green Charcoal? What prompted you to make the decision to use Green Charcoal? 4. In your opinion what worked very well with this initiative and what could be done better? 5. What are the biggest challenges that the project faces? 6. Overall, how satisfied are you with the Green Charcoal? ( Ease of use, duration, cooking results) 7. How do you compare Green Charcoal with traditional charcoal or any other charcoal on the market? 8. Do you think that this activity will continue in the northern department? Why? If not, what should be done to make the production of green charcoal viable? 9. What are some suggestions for improving the project in the future? 10. To what extent has the intervention contributed to positive changes in your life? 11. Have there been any unintended or negative changes that can be attributed to the intervention? I. 13 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV Key Informant Interview with branded brick‐and‐mortar green charcoal boutique vendors Thank you for agreeing to participate. We are very interested in hearing your views on the Green charcoal for Haiti Activity. We hope to learn things that will orient interventions for future programming. All that will be said during the 30 to 45 minutes meeting has great value and we would like the opinion of each of you on each question. If you do not want to answer a question, it is your right not to respond. However, be certain that the information provided is completely confidential and will only be used for one purpose: to help us understand the situation of your institution. With your permission, we will take note of everything that will be shared during the meeting. 1. What do you know about the Green Charcoal for Haiti Activity? 2. How did you find out this information? 3. Do you have a boutique experience before joining the project? If so, what was this activity? Did you sell traditional charcoal before? 4. What do you observe in the profile of users buying Green Charcoal in the Boutique? 5. Do you have an agreement with CRH to sell the Green Charcoal? Please explain. 6. Any specific thoughts/suggestions regarding the market of Charcoal and the project you could share with us. 7. For the last six months have you noticed an increase or decrease in demands of Green Charcoal? Why? 8. Have you been able to meet the demand for Green Charcoal throughout the year? 9. What worked very well and what could be done better? 10. What are the biggest challenges that the project faces? 11. Why would a customer choose to buy green charcoal instead of traditional charcoal? 12. What kind of training did you receive and how useful was that training? 13. Overall, how satisfied are you with the sale of the Green Charcoal and what type of feedback do you have from customers? 14. What are some suggestions for improving the project in the future? USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 14 Key Informant interview with CRH key personnel Thank you for agreeing to participate. We are very interested in hearing your views about the “Chabon Boul” project. We hope to learn things that will orient interventions for future programming. All that will be said during the meeting has great value and we would like your opinion on each question. If you do not want to answer a question, it is your right not to respond. However, be certain that the information provided is completely confidential and will only be used for one purpose: to help us understand the situation of your institution. With your permission, we will take note of everything that will be shared during the meeting. We strongly wish that you accept the interview. 1. What were your roles and responsibilities in the “Chabon Boul” project? 2. What are the main activities implemented by the project? 3. What results did the project achieve? 4. To what extent are the inputs needed for the charcoal production available on an ongoing basis? 5. How is the implementation of the “Chabon Boul” project? Biggest challenges? Opportunities? 6. What has been done in terms of advertising? To what extent has it really helped? 7. What worked very well and what could be done better? 8. What are the biggest challenges that the project faces? (Probe for micro-entrepreneurs, staff, retailers, vendors, etc.) 9. To what extent have the grants helped to increase the production, distribution network and sale of Green charcoal? 10. Overall, how satisfied are you with the production and sale of Green Charcoal? 11. According to you, why a user would choose to use Green Charcoal instead of traditional charcoal. 12. What measures are taken to ensure the sustainability/viability of production? 13. Why the choice of sugarcane bagasse? Do you plan to add agricultural waste? Why? 14. To what extent has the distribution network of the product changed over the past 12 months? 15. What are the key factors driving the Green Charcoal market and how did the project act on these factors? (What are the results on the number of potential buyers / the average quantity purchased / the change in the average purchase price). 16. What are some suggestions for improving the project in the future? 17. Have there been any unintended or negative changes that can be attributed to the intervention? I. 15 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV Key Informant interview with CRH personnel involved in the briquette production process Thank you for agreeing to participate. We are very interested in hearing your views about the “Chabon Boul” project. We hope to learn things that will orient interventions for future programming. All that will be said during the meeting has great value and we would like your opinion on each question. If you do not want to answer a question, it is your right not to respond. However, be certain that the information provided is completely confidential and will only be used for one purpose: to help us understand the situation of your institution. With your permission, we will take note of everything that will be shared during the meeting. We strongly wish that you accept the interview. 1. What are the main activities implemented by the project? 2. What results did the project achieve? 3. To what extent are the inputs needed for the charcoal production available on an ongoing basis? 4. What were your roles and responsibilities in the “Chabon Boul” production team? 5. How is the production team organized? How many people are involved? Daily production capacity? 6. What worked very well? 7. What could be done better? 8. What are the biggest challenges that the project faces? 9. To what extent have the grants helped to increase the production, distribution network and sale of Green charcoal? 10. Overall, how satisfied are you with the production and sale of Green Charcoal? 11. According to you, why a user would choose to use Green Charcoal instead of traditional charcoal. 12. What measures are taken to ensure the sustainability/viability of production? 13. What are some suggestions for improving the project in the future? USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 16 Key Informant interview – USAID key personal Thank you for agreeing to participate. We are very interested in hearing your views on the Green charcoal for Haiti Activity. We hope to learn things that will orient interventions for future programming. All that will be said during the 30 to 45 minutes meeting has great value and we would like the opinion of each of you on each question. If you do not want to answer a question, it is your right not to respond. However, be certain that the information provided is completely confidential and will only be used for one purpose: to help us understand the situation of your institution. With your permission, we will take note of everything that will be shared during the meeting. 1. To what extent was the design of the CRH based on a needs assessment and a context analysis? 2. To what extent was the design of the CRH the most appropriate to meet the needs identified? (Alternative objectives?) 3. What kind of support did the USAID provide to CRH to help with the implementation of the “Chabon Boul” project? 4. What results were achieved? How satisfied are you with performance? 5. What worked very well and what could be done better? 6. What are the biggest challenges that the project faces? 7. Have there been any unintended or negative changes that can be attributed to the intervention? 8. To what extent are the project results (impact if any, and outcomes) likely to continue after the project? 9. What are some suggestions for improving the project in the future? I. 17 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV CHABON BOUL LEARNING EVENT QUESTIONNAIRE General issues and communication 1. How clearly defined were the objectives for your work? ___very ____somewhat ___not very ___not at all 2. How clear were you on your role? ___very ____somewhat ___not very ___not at all 3. How adequately involved did you feel in CRH decisions? ___very ____somewhat ___not very ___not at all 4. If you did not feel involved, what decisions did you feel left out of? 5. How efficient and effective were project team meetings? ___very ____somewhat ___not very ___not at all 6. What changes would assist in speeding up in the future while increasing communication? Project/Activity Implementation 7. What organization, structural problems in general were encountered, and how could we have done better in these areas? 8. What was learned about the activity in general? Did the delivered Green Charcoal meet the specified requirements and goals of the project? 9. What was learned about project management? Did the project management methodology work? If not, why not? 10. What was learned about budgeting? Where costs budgets met? If not, why not? 11. What was learned about procurement? Was the schedule met? If not, why not? 12. What was learned about working with sponsors? 13. What was learned about working with customers? Was the customer satisfied with the green charcoal? If not, why not? 14. What was learned about what went well? 15. What was learned about what did not go well? USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 18 16. What was learned about what needs to change? 17. How will/was this incorporated into project? What could be done to improve the process? Closing 18. What were the main bottlenecks on the process? 19. What were the main sources of frustration in the project? 20. What surprises did the team handle during the project? 21. What project events were not anticipated? 22. What should be done over or differently? 23. As the activity will continue, what is the one thing that you would change (related to process, not to technical solutions)? I. 19 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV A3. INSTITUTIONS/PERSON INTERVIEWED DATA SOURCES INSTITUTIONS/GROUP NAME POSITION Sugarcane Bagasse Suppliers 1. Patrick Sainsurin Distilleries owner 2. Jean-Claude Fontil Distilleries owner 3. Hervé Antoine Distilleries owner 4. Johnson Croissy Distilleries owner 5. Eddy Laguerre Distilleries owner 6. Lesly Laguerre Distilleries owner “Chabon Boul” Vendors 1. Loudridge Léandre 2. Nadège Valerin 3. Guerlande Celestin Religious institution “Stella Maris” 1. Sr. Myrlène Charlemagne Director USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 20 Nutritional center “Second Mile Haiti” / 1. Jennifer Schenk Second Mile Haiti director 2. Hérode Jean-Pierre Secon Mile Haiti Administrator 3. Lujie Louis Second Mile Haiti Cook Informal Roadside Cooks 1. Nadege Celestin 2. Marie Osane Mompremier 3. Anne Edmond CRH Personnel 1. Ryan Delanay Project Director 2. Lesly Croissy Production manager 3. Jacques Benjamin 4. Anderson Pierre 5. Bengonel Ambroise 6. Merlande Ifma I. 21 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV 7. Yvanne Maxi Sales coordinator 8. Rose Daphney Antoine 9. Alexandre Anglade Sales manager USAID 1. Harry François COR Chabon Boul project performance evaluation 2. Christine Musset COR Chabon Boul Project USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 22 A4. SUMMARY OF PROJECT HISTORY PROJECT HISTORY INITIAL AWARD AND MODIFICATIONS DATE OF MODIFICATION PURPOSE OF MODIFICATIONS Initial award June 1 st, 2015 N/A Modification 1 June 23, 2015 The purpose of this modification is to change the Agreement Officer’s Representative and update Milestone 2. Modification 2 September 1 st, 2015 The purpose of this modification is to allow payments in US Dollars instead of payments in Gourdes, the local currency. Modification 3 July 19, 2016 The purpose of this modification is to revise the milestone schedule (make a re- arrangement of the completion dates of milestones but still in the range of 18 months of the initial project) Modification 4 November 30, 2016 The purpose of this modification is to: • Extend the completion date from December 1, 2016 to May 31, 2017 • Increase the total amount by $500000 from $338000 to $838000 • Provide obligation amount of $500000. Therefore, increasing the total obligation from $338000 to $838000; • Revised the milestone schedule; • Revised the program description. Modification 5 March 10, 2017 The purpose of this modification is to revise the milestone Modification 6 August 21, 2017 The purpose of this modification is to revise the milestone and extend the award through December 01, 2017.