Final Performance Evaluation of the “CHABON BOUL” Project

DISCLAIMER THIS PUBLICATION WAS PRODUCED AT THE REQUEST OF THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT. IT WAS PREPARED INDEPENDENTLY BY PHANOL PHILIPPE WITH THE ASSISTANCE OF OTHER “CHASE CONSULTING” PERSONNEL. FINAL PERFORMANCE EVALUATION OF THE “CHABON BOUL” PROJECT PHOTO: A BIGCHAR PYROLYSIS RETORT AT THE “CHABON BOUL” FACTORY Photo: a BigChar pyrolysis retort at the “Chabon Boul” factory USAID/HAITI “CHABON BOUL” PROJECT IMPLEMENTED BY CARBON ROOTS HAITI S.A. Final Performance Evaluation Report Contract Number: AID-521-0-00-17-00039-00 Period of performance: 07/21/2017 to 12/13/2017 CHASE Consulting 10, Rue Acacia, Port-au-Prince, Haïti November 13, 2017 DISCLAIMER THIS PUBLICATION WAS PRODUCED AT THE REQUEST OF THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT. IT WAS PREPARED INDEPENDENTLY BY PHANOL PHILIPPE WITH THE ASSISTANCE OF OTHER “CHASE CONSULTING” PERSONNEL. CONTENTS ACRONYMS 1 SUMMARY 2 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS 7 1.1. EVALUATION PURPOSE 7 1.2. EVALUATION QUESTIONS 7 2 PROJECT BACKGROUND 7 2.1 PROJECT DESCRIPTION 8 2.2 THEORY OF CHANGE 10 3 EVALUATION METHODS AND LIMITATIONS 10 3.1. EVALUATION METHODS 10 3.1.1 OVERVIEW 10 3.1.2 DATA COLLECTION METHODS 11 A. IN-DEPTH PROJECT DOCUMENT REVIEW AND COST-BENEFIT ANALYSIS 11 B. FOCUS GROUP DISCUSSIONS (FGDS) AND WORKSHOP 12 C. SEMI-STRUCTURED INTERVIEWS WITH KEY INFORMANT 13 3.1.3 DATA SYNTHESIS AND ANALYSIS 14 3.2 LIMITATIONS 14 4 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 15 4.1 FINDINGS 15 4.1.1 NET REVENUE FROM THE GREEN CHARCOAL ACTIVITY 15 4.1.2 FUNDING, TECHNOLOGY AND SKILLS 15 4.1.3 PRODUCTION 17 4.1.4 RAW MATERIALS AVAILABILITY 19 4.1.5 QUALITY AND MARKETING 20 4.2 CONCLUSIONS 21 4.3 RECOMMENDATIONS 21 4.4 SYNTHESIS OF LEARNING FROM THE IMPLEMENTATION OF THE “CHABON BOUL” ACTIVITY 21 ANNEXES 1 A1. SCOPE OF WORK 1 A2. DATA COLLECTION TOOLS 10 A3. DATA SOURCES 19 A4. SUMMARY OF PROJECT HISTORY 22 I. 1 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV ACRONYMS APS Annual Program Statement BoP Base of the Pyramid CBA Cost-Benefit Analysis CHASE Collectif Haïtien pour l’Avancement du Suivi et de l’Evaluation CRH Carbon Roots Haiti CRI Carbon Roots International DIV Development Innovations Ventures FGD Focus Group Discussions KII Key Informant Interview NPV Net Present Value ROI Return on Investment USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 2 SUMMARY Background Context and Objectives In July 2017, USAID/Haiti awarded CHASE 1 a contract to conduct the performance evaluation of the “Chabon Boul” project implemented in the Northern Department over the last two years. The purpose of the evaluation is to assess whether the Activity was able to achieve its intended outcome to produce a viable, affordable and cost-effective product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. This evaluation shall provide pertinent information to USAID and its partners on the overall impact of the Activity and serve as a basis for considering whether this activity can be replicated on a larger scale. The “Chabon Boul” Project hopes to achieve three major and intertwined strategic objectives: 1) increase briquette production, 2) expand the retail distribution network, and 3) increase sales of green charcoal briquettes. Green charcoal creates financial sustainability through jobs, income, and value throughout the local value chain while achieving a reduction in deforestation. The following questions were provided by USAID to be addressed during the evaluation process: 1. To what extent the Green Charcoal for Haiti Activity was able to achieve a positive net revenue? If yes, what were the most important factors contributing to this achievement? If not, what were the main constraints leading to this non-achievement? 2. What lessons learned can be derived from the Green Charcoal for Haiti business model to reduce the dependency on firewood and traditional charcoal? Evaluation Methodology Only qualitative methods were used for this evaluation. The evaluation team used the following methods: o Secondary data review, including all available project documentation: periodic reports, project design document, documents on the various modifications (See appendices 1), studies carried out within the framework of this project, annual plans, follow-up plans, M&E plans, etc. o 23 Key Informant Interviews (KII) with key stakeholders, including Mission staff, CRH personnel, beneficiaries, local rum distillers and micro-entrepreneur agricultural processors and other market actors; o Interactive workshop with CRH employees to discuss lessons learned from the project implementation; o Cost Benefit Analysis Findings, Conclusion, Recommendations and Lessons Learned 1 CHASE: “Collectif Haïtien pour l’Avancement du Suivi et de l’Évaluation” I. 3 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV A. Net revenue from the green charcoal activity (Question 1) The production of “Chabon Boul” in its present form is not a viable business. CRH has not been able to generate positive net revenue, despite USAID support. According to the project managers, the income from the sale of green charcoal, which represents the only source of revenue for the company, is always less than the sum of the expenses incurred by the activity. The company is now closed due to lack of financial means. The goals for the USAID-awarded 24-month grant period were to: • Grow revenue to achieve positive cash flow at the north Haiti production facility • Upgrade production equipment to commercialize all processing, from pyrolysis to briquette and drying. • Raise private sector financing for capital expenditures on production and transport equipment Key aspects that are considered in the analysis of the company’s operation are funding, production, key inputs, marketing and product quality. REVENUE. A large part of the amount granted was used, as planned, for the acquisition of modern equipment. In early 2017, Carbon Roots Haiti (CRH) installed and commissioned a commercial pyrolysis unit that convert sugarcane bagasse to charcoal. According to the members of the production team, these orientation sessions gave them the necessary capacity to use the machines. When planning the BigChar purchase, it was hoped that the machine could both convert bagasse into green charcoal and provide clean high heat to dry bagasse and green charcoal briquettes coming off the production line. This assumption was incorrect. The current process relies on solar energy over several days (three to five) to dry both the bagasse and the freshly released charcoal from the commercial briquette pressing machine. According to document review and various meetings in the field, the income from the sale of coal still decreased with the financing. According to the project managers, to reach the break-even point, the company must have at least five equipment units (BigChar) like the one bought with USAID funds. Today the company is looking for a grant of about 2.5 million USD to accomplish this benchmark. According to the project proposal, the project designers acknowledged that additional equipment units were needed for the activity to be profitable. As expected, and according to project personnel, the acquisition of the new equipment (the BigChar unit) was more like a demonstration of what the company could do with a more appropriate technology. The acquisition of the new equipment also puts the company in a mode of operation where it is more easily scalable. CRH reported that increasing the price of the ‘Chabon Boul’ product to increase revenue is not a viable option. Currently, an equal volume of the two brands of charcoals (traditional charcoal and Chabon Boul) are sold at the same average price. Monthly sales prior to the current shutdown are far below the projection of a $60,000 monthly sale limit to achieve profitability. With the acquisition of the new equipment, the company has reduced its staff. Therefore, the costs of the operations were reduced. However, the company still operates with more than twenty employees in addition to other expenses such as carrying bagasse (about 350 USD / month), taxes, fuel, repairs, etc. It is difficult for the evaluation team to provide strong conclusions and recommendations without access to information about the company’s monthly expenditures. There is no way to determine how increasing production will affect the future profitability of CRH. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 4 PRODUCTION . According to CRH managers, the green charcoal production has decreased with the USAID grant. Prior the BigChar installation, the company produced about 5 to 6 tons of charcoal per day; now the charcoal production is 1 to 1.5 tons per working day. This is related to the capacity of the machine. As production has decreased, less green charcoal is sold with support received from USAID. In addition to the reduction in production, the various stakeholders in the field also report lack of regularity in production. In fact, repeated breakdowns prevented the company from producing continuously with detrimental consequences on marketing and sales. Conversely, it is encouraging for the company that it usually sells everything it produces. Production varies with seasons as well since solar energy is required to dry the final products. The company tried an alternative means to dry the briquettes without much success. The company’s large distribution network was reduced with the decrease in production and sales RAW MATERIALS . Sugarcane bagasse is used to produce the “Chabon Boul”. According to several informants, the bagasse available can readily support several green charcoal factories the size of CRH. With dozens of distilleries in the northern plain, production has suffered just one slowdown due to a scarcity of bagasse since the launch of the activity. A disadvantage in the use of bagasse lies in the fact that it must be dried for use. In the rainy season, it is difficult to dry bagasse collected because the drying done under the sun. The project has been successful in promoting bagasse as a byproduct of distilleries. As an effect of CRH activities, bagasse now has a price. Distillery owners complained as well that the price paid by CRH for the cart of bagasse is not the same everywhere. At the beginning of operations, kits were provided to bagasse suppliers who were engaged in preparing the char themselves. It was soon realized that the char prepared in the 55-gallon metal drums was of poor quality which would have a direct impact on the quality of the finished product. Drums of char were bought according to their weight and some suppliers fraudulently placed stones in the drums to increase the weight of the char to gain more money. The decision was then made to centralize the management of bagasse combustion (char production) at the company level which increased the quality of the product and efficacy of the resources. Starch is another essential input in the production of green charcoal. According to information collected, locally produced starch made from cassava, especially in “cassaveries, 2 ” was used initially. Again, the company would face some suppliers who mixed starch with other products (flours) to increase their earnings. The mixture reduces the quality of the starch and consequently the quality of the resulting green charcoal. The decision was then made to source corn starch from major US-based industries, a fruitful initiative according to CRH managers. 2 A place where cassava or other cassava products are made. I. 5 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV MARKETING AND PRODUCT QUALITY . Immediately after the launch of the project, several types of marketing were used to reach potential consumers. Visits with potential buyers proved to be the most beneficial. According to stakeholders, consumers are generally satisfied with the price and the quality of the product. The quality of “Chabon Boul” has increased gradually with several key decisions made by the company's management. According to the consumers consulted, Chabon Boul is easy to use. They estimated that the Chabon Boul intensified the heat produced resulting in a shorter cooking time. Also reported was the Chabon Boul does not produce smoke that darkens cooking utensils. Finally, unlike traditional, bagged wood charcoal, the Chabon Boul briquettes remain intact in the bag. However, consumers visited also claim that Chabon Boul produces a lot of ash. According to the information gathered, if you do not pay attention, the large amount of ash produced is susceptible to blowing around, dirtying the kitchen and beyond. B. Synthesis of learning from the project implementation (Question 2) The “Lessons Learned” workshop offered an opportunity for reflection after the completed project. It was valuable to reflect on what worked well with the project and what could be improved upon from the perspective of representatives from the project team. Project documents were consulted as well in this sense. The main lessons are the following: Leadership. One of the main lessons learned revolves around decisions made by the management team. One such decision was the rotation of the production team for better results and better personnel management. The multiple tasks performed by the production team vary with some being far more demanding than others. There is the handling of the machine that refines dried bagasse, the preparation of the starch, the mixture of char, starch and other materials needed, monitoring the temperature of the BigChar to make sure everything is working fine. Since the company decided to give the same salary to everyone, it initially created frustration. The managers chose to rotate the 14-member production team and better distribute the workload. This improved the dynamics of the team and reduced their frustration. Research. Consistent conversations with consumers lead to ongoing research on how to improve the product to meet their requirements and achieve client satisfaction. Resulting improvements in char production include smokeless green charcoal and improved materials in its composition. Marketing. As mentioned above, several marketing methods have been used by the company to promote the "Chabon Boul." The marketing method of visiting potential consumers of the product and providing a sample amount to test at home yielded the best results; people like to test the product before making the decision to buy. After having tested it, the purchase decision followed almost automatically. Insufficient product supply was a barrier to marketing efforts. Sales strategy. Imitating the traditional charcoal marketing circuit directly in competition with the "Chabon Boul," sales managers sought to understand and then facilitated a rapid integration of the market. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 6 Quality control. Performing quality control throughout the entire production chain increases the quality of the final product. This quality control cannot be trusted to the many actors in the production chain, but by a company manager with control mechanisms implemented at all stages. Repeated stoppages of production . Repeatedly stopping production is detrimental to the range of customers the industry would like to have. Stoppages were most often due to equipment breakdown or problems related to lack of funds. Conclusion and recommendations According to all the information shared with the evaluation team and in line with projections, the “Chabon Boul” business did not break even with the grant they received from USAID. Already in the fourth modification, it is clearly stipulated that the project did not intend to achieve profitability with this grant and that it would need an additional funding. It said that the funds from this second grant from USAID should be used to install one new pyrolysis retort, which will serve as a demonstration of technology and scale to investors, who would then lend and invest another $1 million to bring the production factory to a sufficient scale to reach financial sustainability. As a result of decreased production and sales, projections have been further modified or adjusted as the company now needs a grant of approximately $ 2.5 million to achieve profitability. According to managers, in its current status, the company will still need external support to survive. Despite the repeated work stoppages, the company has a relatively faithful client base. While there is the chance that CRH may lose some potential customers and market share to propane, the main raw material, sugarcane bagasse, is available in a quantity largely sufficient to operate the current enterprise and after an increase in size. Two major drawbacks with bagasse are 1) in wet weather bagasse cannot be used due to lack of sunshine, and 2) in times of drought, there is shortage of bagasse. The quality of the product, according to the majority of the customers, is good and they are generally happy. The inconvenience of too much ash is offset by a competitive price, relatively shorter cooking time, and no smoke. The evaluation team recommends that the industry: • Has better expense appraisal to better estimate the break-even level and increase the level of transparency in the communication of financial information; • Regularizes the “Chabon Boul” production in order to increase customer loyalty (repeated shutdowns due to energy or other problems affect the market); • Regularizes the price of the bagasse cart to reduce frustrations; • Continues to improve the quality of the product (specifically the complaint of ashes during the burning process); Find better ways to dry bagasse and charcoals (Drying in open air is unreliable during raining seasons). I. 7 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1. EVALUATION PURPOSE In July 2017, USAID/Haiti awarded CHASE (Collectif Haïtien pour l’Avancement du Suivi et d’Évaluation) a contract to conduct the performance evaluation of the Green Charcoal for Haiti Activity (Chabon Boul Project). The purpose of the evaluation was to assess whether the Green Charcoal Activity was able to achieve its intended outcome to produce a viable, affordable and cost-effective product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. This evaluation shall provide pertinent information to USAID and its partners on the overall impact of the Green Charcoal Activity and serve as a basis for considering whether this activity can be replicated on a larger scale. The primary stakeholders for this evaluation include: USAID/Haiti, Carbon Roots International (CRI) and Carbon Roots Haiti (CRH). This document presents the final evaluation report including detailed methodology for the evaluation and describing key findings, conclusions and recommendations. 1.2. EVALUATION QUESTIONS 1. To what extent the Green Charcoal for Haiti Activity was able to achieve a positive net revenue? If yes, what were the most important factors contributing to this achievement? If not, what were the main constraints leading to this non-achievement? 2. What lessons learned can be derived from the Green Charcoal for Haiti business model to reduce the dependency on firewood and traditional charcoal? 2 PROJECT BACKGROUND The least well-off strata of the population, also called the communities at the Base of the Pyramid (BoP), are caught in a vicious circle. 3 This vicious circle consists of relying on charcoal as their main source of energy, triggering negative health outcomes, deforestation, increased greenhouse gas emissions, erosion, flooding, decreased arable land, declining crop yields, and depressed incomes forcing reliance on polluting, tree-consuming charcoal. According to the best available data 4 , Charcoal is used by 90% of the households from Port-au-Prince and other major cities, and the charcoal chain from production to final consumption employs more than 150 thousand persons. Charcoal is made from trees, with a low conversion efficiency of about only 20% (5 kg of wood for 1 kg of charcoal) and its production contributes to degrade the local environment by adding pressure to the plant stand throughout the countryside which is already over 98% deforested. 3 Information from the “Chabon Boul” project document. 4 Bureau of Mines and Energy: Haiti Energy Sector Development Plan 2007 - 2017 USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 8 Viable alternative cooking technologies are not being adopted fast enough to stop or even slow deforestation, and forest decline has led to a rise in charcoal prices, in part due to the rapidly growing black market for illegal charcoal smuggled into Haiti from the Dominican Republic. In this broader context, the dependency on charcoal and firewood becomes a major factor in the environmental degradation of Haiti and poses a serious threat to the lives and livelihoods of Haitian citizens. The Green Charcoal for Haiti activity was designed partly to address this problem by demonstrating that cooking briquettes made from agricultural waste can be a viable, affordable, cost-effective alternative to the use of traditional charcoal, that will help reducing deforestation in Haiti and increasing rural incomes. Additionally, the use of green charcoal would reduce carbon emissions compared to the traditional charcoal, and it burns somewhat cleaner, lessening the adverse health effects on its users. 2.1 PROJECT DESCRIPTION The Green Charcoal for Haiti is a 24-month, $838,000 program implemented by CRH, a Haiti-based for-profit social business affiliated with a U.S.-based non-profit organization, Carbon Roots International (CRI). Since 2013, USAID/Haiti has supported Stage 1 and 2 grants (a total of $938,000) to CRI and CRH through the (DIV) Annual Program Statement (APS) to support their plan to replace the use of charcoal fuel in Haiti with a green alternative fuel. The overall goal of the activity is to create sustainable jobs and reduce deforestation, with an Intermediate Result of achieving financial sustainability at the briquette production factory. As presented in the project’s Monitoring and Evaluation (M&E) Plan, the “Chabon Boul” Project hopes to achieve three major and intertwined strategic objectives: 1) increase briquette production, 2) expand the retail distribution network, and 3) increase sales of green charcoal briquettes. Green charcoal creates financial sustainability through jobs, income, and value throughout the local value chain while achieving a reduction in deforestation. CRH’s business consists of four activities that comprise the green charcoal value chain: 1. Raw materials purchasing from local suppliers The value chain begins with a network of smallholder farmers and micro ‐ entrepreneur agricultural processors who earn added income by selling their agricultural waste biomass to CRH. In northern Haiti, the primary biomass feedstock used is sugarcane bagasse. Carbon Roots Haiti pays the suppliers for their waste, then transports it from the farm or processing facility to the production site. 2. Processing At a central production facility, CRH staff sorts and dries the feedstock, then pyrolyzes (i.e. carbonize) it into charcoal dust. Prior to February 2017, pyrolysis was achieved using handmade 55 ‐ gallon batch kilns. Utilizing funding from USAID, CRH began upgrading the pyrolysis process to use continuous-feed retorts that also provide heat to streamline drying. After pyrolysis, CRH staff mixes the char dust with binding agents, resulting in a char slurry. The slurry is then pressed into dense briquettes using commercial roller briquette equipment, consisting of: I. 9 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV • A large batch mixer, • A conveyor belt, and • A commercial briquette pressing machine, which uses two opposing cylindrical molds to compress char slurry into pillow briquettes. After processing by the briquetting equipment, the briquettes are transferred to portable racks and dried in the sun for 3 ‐ 5 days. Upon upgrading to a new pyrolysis process, CRH should be able to dry briquettes in a matter of hours. After drying, the briquettes are packed in re ‐ useable sacks that are ready for transport and sale. 3. Marketing Most products available at local markets in Haiti are sold under brand names. However, due to the decentralized nature of wood charcoal production in Haiti, there are no recognized nationwide brands. This presents a competitive advantage for CRH by being able to produce and distribute green charcoal centrally, allowing the business to sell green charcoal under the “Chabon Boul” brand name. Marketing focuses on cost savings, and highlights that green charcoal is a locally ‐ produced product that benefits the environment. CRH engages in many marketing media, including: • Television commercials and shows, • Radio commercials, • Banners and signs throughout the sales region, • Signage on public transportation, • Promotional events. Due to limited income in the BoP, potential customers may be cautious to purchase new products, the results of which they are unsure. To combat this tendency, CRH encourages consumption through special offers and free samples, which has proven successful at spreading awareness and increasing sales at new retail locations. For example, an individual distribution point selling green charcoal will drop the cost of a marmite (roughly one kilogram of charcoal) 40% for several hours, doubling new customer purchases during the sale. CRH continues to test different marketing strategies to determine the best ROI 5 and scaling up those effective tactics. By following up with prospective buyers, they can emphasize savings, appeal to widespread notions of civic duty, and tie CRH’s marketing to the Haitian Government’s reforestation initiatives. 4. Distribution CRH’s distribution strategy is three ‐ pronged: 5 ROI: Return on Investment USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 10 • Wholesale customers: CRH targets restaurants, informal roadside cooks, schools and organizations, selling in 30kg sacks and providing delivery service to the customer. • Women retailers: In Haiti, women traditionally sell charcoal at local markets. CRH’s second distribution strategy reinforces this inclusion of women in market activities by selling green charcoal briquettes to household users through a network of women retailers. These retailers are “based” out of one of CRH’s distribution points (depots/boutiques), where they purchase green charcoal in 30kg sacks, and then sell to household customers in smaller retail quantities. • Direct Sales: Carbon Roots Haiti has launched branded brick ‐ and ‐ mortar green charcoal boutiques, located in high ‐ traffic areas, and serve the household market directly. Generally, household customers purchase in “marmite” units, often patronizing the boutique daily. CRH currently has boutiques in Port ‐ au ‐ Prince and Cap ‐ Haitian. Beneficiaries of the project include: • Local rum distillers and micro-agricultural entrepreneurs, who will earn significantly more than the national minimum wage, and several times more than the national average daily income. • Charcoal retailers, mostly women, who will sell green charcoal briquettes at a market discount and still earn roughly 15% profit on their sales. • End users, who will save between 25-40% of their income by using green charcoal. 2.2 THEORY OF CHANGE CRH’s Theory of Change postulates that deforestation, rural poverty and unemployment, energy demand, and low agricultural yields can be addressed through the deployment of a profitable social business engaged in the production of green charcoal. This enterprise benefits a wide range of stakeholders: • Farmers monetize their agricultural waste by converting it to char dust and selling it to CRH • Women retailers offer a highly-competitive product, enjoy higher profit margins, develop business skills, and build a broad customer base • Institutions such as schools and restaurants, and households have access to sustainable cooking fuel at a price point that is cheaper than traditional charcoal While the pathway to change is complex in a socioeconomic environment such as Haiti, a market-based enterprise that manufactures products locally, for local consumption, is likely the most impactful and sustainable solution. 3 EVALUATION METHODS AND LIMITATIONS 3.1. EVALUATION METHODS 3.1.1 OVERVIEW The evaluation team had to determine whether the “Chabon Boul” Project was able to achieve its intended outcome, which is to produce a viable, affordable and cost-effectiveness product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. The team through analysis had to also help the mission to appreciate the overall impact of the project. To do this, the evaluation team considered using the following methods: I. 11 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV a) Secondary data review, including all available project documentation: periodic reports, project design document, documents on the various modifications (See appendices 1), studies carried out within the framework of this project, annual plans, follow-up plans, M&E plans, etc. b) Key Informant Interviews (KII) and Focus Group Discussions (FGDs) with key stakeholders, including Mission staff, CRH personnel, beneficiaries, local rum distillers and micro- entrepreneur agricultural processors and other market actors; c) Interactive workshop with CRH employees to discuss lessons learned from the project implementation; d) Cost Benefit Analysis. Above methods were used separately and the findings have not been integrated until after data analysis. Much of the document review took place first and findings from this step were used to inform the design and implementation of key informant interview and focus group methods. The same evaluation team was then involved in implementing key informant interview and focus group methods, and the actual data collection and analysis happened congruently over the same period. CHASE met first with the Carbon Roots Haiti personnel involved in monitoring the “Chabon Boul” Project. It was an opportunity to 1) know how the project was doing from the implementers’ perspective, 2) to record long term issues regarding the project and the evaluation, and 3) to identify other influential key stakeholders. This meeting also helped to determine specific geographical mapping of activities. Data collection was carefully planned in collaboration with Carbon Roots Haiti personnel. 3.1.2 DATA COLLECTION METHODS A. IN-DEPTH PROJECT DOCUMENT REVIEW AND COST-BENEFIT ANALYSIS i. Literature review CHASE personnel conducted an extensive review of the literature. The documents review help in understanding both the theory of change behind the project and key project activities and results. The evaluation team drew on existing quantitative and qualitative data collected during the literature review to (1) refine sampling strategy and/or approach for collecting qualitative data, (2) design relevant qualitative data collection tools, and (3) seek compelling evidence to answer the evaluation questions. Relevant documents include, among others: the USAID/Haiti country strategy; “Chabon Boul” project reports such as work plans; PMP; M&E plans; quarterly progress reports and technical reports; the project proposal; and, documents relating to amendments. Ultimately, the literature review helped to gather relevant information to produce and enhance the comprehensive evaluation design. ii. Cost-Benefit Analysis Cost-benefit analysis (CBA) techniques were to be used to compare the total costs of the activity with its revenue, using only monetary units. This would enable the calculation of the net cost or benefit associated with the production of green charcoal. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 12 As a technique, CBA evaluates the overall impact of the activity in quantifiable and monetized terms by adding up the total costs of an activity and comparing it against its total revenue. For this evaluation, Net Present Value (NPV) of the project cost was to be compared with the net present value of its revenue. Decisions would be based on whether there was a net revenue or cost to the approach, i.e. total revenue less total costs. This comprehensive analysis takes a broad view of costs and benefits, including indirect and longer-term effects. This type of review typically begins with an exhaustive list of all the different costs that could arise, even if some will be later excluded: fixed costs included equipment partially mortgaged, rent, advertising, insurance, office supplies, etc. Variable costs included: personnel, training, vehicle (maintenance and repair), fuel, product inputs, etc. To find the business's profit, the team, with support from accounting and leadership teams, would add up all of the total sales of green charcoal the Activity has made in a set period of time (one month) and other revenues including (but not limited to) gain or loss on disposal. The team would then subtract any amount of cash refunded to customers for returns or disputes to find an accurate figure for the total income. Once the team found accurate values for the Activity’s total income and expenses, calculating the profit would simply be to subtract expenses from the income. The data would be calculated monthly on the last six months and averaged to get the monthly figure. It should be noted that the evaluation team was not provided the information necessary to achieve this Cost Benefit Analysis B. FOCUS GROUP DISCUSSIONS (FGDS) AND WORKSHOP i. Focus Group Discussions Focus Group Discussion was planned to be held during the data collection phase with relevant stakeholders to obtain viewpoints and evidence to answer the evaluation questions. Small groups of eight to 12 were set for a one-hour discussion around main topics identified in the design phase. Candidates for focus groups discussions were the charcoal retailers. Please note, the evaluation team was not able to conduct these focus groups. These discussions would have helped determine the status of the project implementation and assess whether the Green Charcoal Activity was able to achieve its intended outcome, to produce a viable, affordable and cost-effectiveness product (cooking briquettes) that can be used as an alternative for traditional charcoal in Haiti. These discussions would have highlighted the lessons learned. ii. Workshop with CRH staff to determine lessons learned A two-hour workshop was organized with Carbon Roots Haiti personnel to discuss about the implementation and to document lessons learned. The purpose of documenting lessons learned was to share and use knowledge derived from experience to promote the recurrence of desirable outcomes and preclude the recurrence of undesirable outcomes. This “Lessons Learned” session documented the cause of issues and the reasoning behind any corrective action already taken to address those issues. For the identified ““Lessons Learned”” where no action I. 13 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV had yet been taken, the team determined next steps and to identified who will be responsible for implementing those measures. Questions discussed during the session included the following: • What was learned about the project in general? Did the delivered Green Charcoal meet the specified requirements and goals of the project? • What was learned about project management? Did the project management methodology work? If not, why not? • What was learned about communication? What changes would assist in speeding up in the future while increasing communication? • What was learned about budgeting? Were costs budgets met? If not, why not? • What was learned about procurement? Was the schedule met? If not, why not? • What was learned about working with sponsors? • What was learned about working with customers? Was the customer satisfied with the green charcoal? If not, why not? • What was learned about what went well? • What was learned about what did not go well? • How will/was this incorporated into project? What could be done to improve the process? C. SEMI-STRUCTURED INTERVIEWS WITH KEY INFORMANT Interviews were conducted with key stakeholders and resources persons such as key CRH personnel, local distillers of rum, end users such as CRH targets restaurants, informal roadside cooks, schools and organizations, and branded brick ‐ and ‐ mortar green charcoal boutiques vendors. The main objective of these interviews was to assess whether the project was 1) able to achieve its intended outcome, which is to produce a viable, affordable and cost-effectiveness product, and 2) to discuss best practices and lessons learned, keeping focus on seeking evidence to answer the evaluation questions. Key stakeholders involved in the project activities and systems in place to boost green charcoal production were visited during data collection. Key Informant Interviews (KII) were conducted with: - CRH staffs : o To collect qualitative data about performance, get more insight about project implementation, challenges and constraints on the field, and document lessons learned (Question 1 and 2) o To collect information on the functioning and achievements of the “Chabon Boul” project from the implementer’s perspective (Question 1) o To have information on what facilitated achievement of performance or what prevented CRH from achieving the expected results (Question 1) o To collect information on effective measures/decisions taken to reduce production costs and increase the viability of the project (Question 1) - Local rum distillers and Micro-Entrepreneur agricultural processors: o Availability of sugarcane bagasse on a continuous basis (Question 1); o Their transformation of sugarcane bagasse into char dust (Question 1); o Their overall learning from the project (Question 2); o Their level of satisfaction with the production of the Green Charcoal (Questions 1 and 2) o How they compare this activity with the traditional charcoal production (Question 1). USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 14 - Green Charcoal end users (CRH targets restaurants, informal roadside cooks, schools and organizations) o To collect information on the ease of access to the product. (Question 1) o To get more insight about their level of satisfaction with the use of Green Charcoal (Question 1) o To have information on how they compare this product with the traditional charcoal (Question 1) o Collect some suggestions for improving the project in the future (Question 1) - Branded brick ‐ and ‐ mortar green charcoal boutique vendors o To collect information on performance, get more insight about project implementation, challenges and constraints in the field, and lessons learned (Questions 1 and 2); o Difference in product sales prices compared to traditional charcoal and what can motivate people to switch from traditional charcoal to Green Charcoal (Question 1); o Their level of satisfaction with the sale of the Green Charcoal and the type of feedback they have from customers (Questions 1 and 2). o Above described methods and techniques were used to collect data and provide answers for the two (2) evaluation questions. An exhaustive list of planned KII is presented in the summary table of data collection. 3.1.3 DATA SYNTHESIS AND ANALYSIS Parallel data analysis was conducted, weighting answers from various sources. Data from each source were analyzed independently using content analysis methodology, and the findings were integrated and triangulated into coherent statements about the evaluation questions. A synthesis framework was developed to coherently address the wealth of information yielded daily by the key informants’ interview reports. Evidence was clustered into findings that would lead to valid and relevant conclusions and recommendations about the evaluation questions. Triangulation of data from a variety of sources, including both quantitative and qualitative data from literature review, qualitative data collected on the field, was valued such that the evaluation produced evidence-based findings, conclusions, and recommendations. The contents of the discussion were examined to discover the meaning and its particular implications for the evaluation questions. Every effort to interpret focus group/key informant interview data occurred through analysis of content. 3.2 LIMITATIONS The focus groups scheduled with retailers could not take place because this group has been demobilized for several months and according to the managers contacted, it was not possible to have them for several semi-structured meetings as planned. The cost-benefit analysis was not performed as planned in the design document/evaluation plan. According to the director and other managers contacted, the nature of the contract with the donor and the type of business involved meant that the director was not obliged to share cost information incurred during the operations and chose not to. Therefore, the analysis considers only the limited information available in reports of milestones or collected during field interviews. No accounting documents were shared with the evaluation team. I. 15 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV 4 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 4.1 FINDINGS 4.1.1 NET REVENUE FROM THE GREEN CHARCOAL ACTIVITY The production of Chabon Boul in its present form is not a viable business. CRH, a small enterprise, producing green charcoal (the "Chabon Boul") in the northern department of Haiti, has not been generating positive net revenue, despite USAID support. According to the project managers, the income from the sale of green charcoal, which represents the only source of revenue for the company, is always less than the sum of the expenses incurred by the activity. The company is now closed due to lack of financial means. 4.1.2 FUNDING, TECHNOLOGY AND SKILLS The goals for the USAID-awarded 24-month grant period were to: • Grow revenue to achieve positive cash flow at the north Haiti production facility • Upgrade production equipment to commercialize all processing, from pyrolysis to briquette and drying. • Raise private sector financing for capital expenditures on production and transport equipment A large part of the amount granted was used, as planned, for the acquisition of modern equipment. In early 2017, Carbon Roots Haiti (CRH) installed and commissioned a commercial pyrolysis unit that convert sugarcane bagasse to charcoal, replacing the organization’s reliance on hand-made 55-gallon barrel kilns. The pyrolysis unit is a BigChar 1200 model made by Pyrocal Ltd in Australia. Several training sessions were organized on the use of equipment. According to the members of the production team, these orientation sessions gave them the necessary capacity to use the machines. The BigChar equipment was purchased with the expectation that it could convert bagasse into green charcoal, and provide clean, high heat to dry both bagasse and green charcoal briquettes coming off the production line. Unfortunately, it was discovered that the BigChar cannot reliably dry the bagasse or freshly- manufactured green charcoal. The project relies on three to five days of solar power to dry both the bagasse and the freshly released charcoal from the commercial briquette pressing machine. With no assurances on weather, an alternative for drying and storage must be found. According to document review and various meetings in the field, CRH received all the funds allocated to the project but the income from the sale of coal still decreased with the financing . According to the project managers, to reach the break-even point, the company must have at least five equipment units (BigChar) like the one bought with USAID funds. Today the company is looking for a grant of about 2.5 million USD to accomplish this benchmark. In the project design document, it was clearly stated that the activity would not be profitable with this second grant granted by USAID. The project designers acknowledged that additional equipment units were needed for the activity to be profitable. As expected, and according to project managers, the acquisition of the new equipment (the BigChar unit) was more like a demonstration of what the company could do with a more appropriate technology. The acquisition of the new equipment also puts the company in a mode of operation where it can become scalable. USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 16 Figure 1. Monthly sales for 3 months (Source: CRH Milestone reports) The enterprise relies only on green charcoal sales for revenues. In the past, the char dust was given free to farmers but sold to some agricultural sector institutions as organic fertilizer. CRH reported that increasing the price of the ‘Chabon Boul’ product to increase revenue is not a viable option. Currently, an equal volume of the two brands of charcoals (traditional charcoal and Chabon Boul) are sold at the same average price. The 30-kg bag that looks like most of traditional charcoal bags sells for 500 gourdes 6 . According to various stakeholders, any increase in the price of “Chabon Boul” higher than that of traditional charcoal will scare away the green charcoal potential buyers. As can be seen in Figure 1, monthly sales prior to the current shutdown are well below the projection of a $60,000 monthly sale limit to achieve profitability. With the acquisition of the new equipment, the company has reduced its staff. Therefore, the costs of the operations were reduced. However, the company still operates with more than twenty employees in addition to other expenses such as carrying bagasse (about 350 USD / month), taxes, fuel, repairs, etc. Since the company did not aim for profitability with this second grant, the numbers featured in Figure 1 do not come as a surprise. However, it is difficult for the evaluation team to provide strong conclusions 6 500 Gourdes is the equivalent of 7.91 USD (1:63.18) 209 331 508 33 1081 Apr-17 May-17 Jun-17 Jul-17 Total Monthly sale in US Dollar I. 17 | “CHABON BOUL” PROJECT PERFORMANCE EVALUATION USAID.GOV and recommendations without access to information about the company’s monthly expenditures. There is no way to determine how increasing production will affect the future profitability of CRH. 4.1.3 PRODUCTION As mentioned earlier, this second grant enabled CRH to acquire new equipment and at the same time enabled the company to move from a traditional processing to a less labor-intensive industrial processing. Today, the production team is reduced from more than 34 employees’ pre-modernization to 14 with the arrival of BigChar. Figure 2. Sales in volume from April to July 2017 According to CRH managers, the green charcoal production has decreased with the USAID grant. Prior the BigChar installation, the company produced about 5 to 6 tons of charcoal per work day; now the charcoal production is 1 to 1.5 tons per work day. As production has decreased, less green charcoal is sold with support received from USAID. To restore the production levels to 6 tons, the machine would need to run 24 hours a day. While it has the capacity, a considerable limitation is the necessary manpower. This approach would require adding several eight-hour shifts and sufficient security and transportation; a considerable challenge in that location. While the company usually sells everything it produces, stakeholders in the field report unreliable production output. Repeated breakdowns cause interruptions in the supply for days and weeks at a time. Once the issues are resolved and the supply is restored, additional marketing and sales resources are needed to publicize that the company is operational again. Production varies with seasons as well since solar energy is required to dry the final products. Unfortunately, when briquettes are needed most during the rainy season, production decreases due to the lack of sunlight needed to dry bagasse and green charcoal. The company tried an alternative means to dry the briquettes without much success. The machine bought for this purpose burnt too much briquettes while charcoals inside the device failed to receive the amount of heat needed. 28 51.5 72 5 156.5 Apr-17 May-17 Jun-17 Jul-17 Total Monthly sale in number of 30 Kg sacks USAID.GOV “CHABON BOUL” PROJECT PERFORMANCE EVALUATION | 18 The company’s large distribution network was reduced with the decrease in production and sales. The green charcoal produced by the company was sold on site or through the few kiosks located in the city of Cap-Haitian and its suburbs. The “Chab