Planter les graines : L'impact de la formation sur les producteurs de mangues en Haïti

Planting the seeds: The impact of training on mango producers in Haiti Irani Arráiz Carla Calero Songqing Jin Alexandra Peralta IDB WORKING PAPER SERIES Nº IDB-WP-610 August 2015 Multilateral Investment Fund Inter-American Development Bank A ugust 2015 Planting the seeds: The impact of training on mango producers in Haiti Irani Arráiz Carla Calero Songqing Jin Alexandra Peralta Cataloging-in-Publication data provided by the Inter-American Development Bank Felipe Herrera Library Planting the seeds: the impact of training on mango producers in Haiti / Irani Arráiz, Carla Calero, Songqing Jin, Alexandra Peralta. p. cm. — (IDB Working Paper Series ; 610) Includes bibliographic references. 1. Agricultural development projects—Evaluation—Haiti. 2. Occupational training— Evaluation—Haiti. 3. Agricultural productivity—Evaluation—Haiti. I. Arráiz, Irani. II. Calero, Carla. III. Jin, Songqing. IV. Peralta, Alexandra. V. Inter-American Development Bank. Office of the Multilateral Investment Fund. VI. Series. IDB-WP-610 Copyright © Inter-American Development Bank. This work is licensed under a Creative Commons IGO 3.0 Attribution- NonCommercial-NoDerivatives (CC-IGO BY-NC-ND 3.0 IGO) license (http://creativecommons.org/licenses/by-nc-nd/3.0/igo/ legalcode) and may be reproduced with attribution to the IDB and for any non-commercial purpose, as provided below. No derivative work is allowed. Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the UNCITRAL rules. The use of the IDB's name for any purpose other than for attribution, and the use of IDB's logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC-IGO license. Following a peer review process, and with previous written consent by the Inter-American Development Bank (IDB), a revised version of this work may also be reproduced in any academic journal, including those indexed by the American Economic Association's EconLit, provided that the IDB is credited and that the author(s) receive no income from the publication. Therefore, the restriction to receive income from such publication shall only extend to the publication's author(s). With regard to such restriction, in case of any inconsistency between the Creative Commons IGO 3.0 Attribution-NonCommercial-NoDerivatives license and these statements, the latter shall prevail. Note that link provided above includes additional terms and conditions of the license. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the Inter-American Development Bank, its Board of Directors, or the countries they represent. http://www.iadb.org 2015 Planting the seeds: The impact of training on mango producers in Haiti ∗ Irani Arr´ aiz † , Carla Calero ‡ , Songqing Jin § , and Alexandra Peralta ¶ August 2015 Abstract This paper evaluates the short-term impacts of a development project that aims to increase mango yields, sales of mango products, and the income of small mango farmers in rural Haiti. Various matching methods, in combination with difference-in-difference (DID), are used to deal with the potential selection bias associated with nonrandom treat- ment assignment. Robustness checks are conducted to investigate whether and to what extent the results are affected by the coexistence of other similar projects in the same sites. Rosenbaum bounds analysis is carried out to check the sensitivity of the estimated impacts—based on matching methods—to deviations from the conditional independence assumptions; the relative importance of unobserved factors in the decision to participate. Our results show that in a 16-month period, the project increased the number of young Francique trees planted—a type that has greater market and export potential than tra- ditional mango varieties—and likely encouraged the adoption of best practices. But the project has not yet led to a noticeable increase in total sales. The adoption of improved production practices is too recent to translate into significant changes in production and sales. While the robustness check suggests that the results are not caused by the presence of other similar programs on the same sites, the Rosenbaum bounds sensitivity analysis suggests that the matching results are robust against potential “hidden bias” arising from unobserved outcome variables in some but not all cases. JEL Classification: Q13, Q16, O12. Keywords: Agriculture, impact evaluation, producer cooperative, extension services, Haiti, mango. ∗ This study was coordinated and financed by the Multilateral Investment Fund (MIF) of the Inter-American Development Bank (IDB). The authors are grateful to Technoserve for its support in conducting the study. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the Inter-American Development Bank, its Board of Directors, or the countries they represent. † Inter-American Development Bank. E-mail: iarraiz@iadb.org ‡ Inter-American Development Bank. E-mail: ccalero@iadb.org § Department of Agricultural, Food and Resource Economics, Michigan State University. E-mail: jins@anr.msu.edu ¶ Global Food Studies, the University of Adelaide. Email: alexandra.peralta@adelaide.edu.au 1 Introduction The importance of adopting relevant technologies to reduce poverty and spur economic deve- lopment has long been emphasized by development economists and agricultural practitioners (Feder et al., 1985; Feder and Umali, 1993). The adoption of high-yield varieties (HYVs) and complementary modern inputs and crop management practices during the Asia Green Revo- lution is widely acclaimed as one of the most successful stories in the economic development of South and Southeast Asia (Evenson and Gollin, 2003; Otsuka and Larson, 2012; Pingali et al., 2007; World Bank, 2008). Meanwhile, the low adoption rate of HYVs, fertilizers, irriga- tion technologies, and other agricultural and resource management practices is cited as a key reason for the mostly unsatisfactory performance of the agricultural sector in Sub-Saharan Africa (Djurfeldt et al., 2005; Moser and Barrett, 2006; Ndjeunga and Bantilan, 2005; Otsuka and Kijima, 2010; Otsuka and Larson, 2012). In Latin America and the Caribbean (LAC) considerable advances have been made in the development of HYVs, agricultural mechaniza- tion, and agricultural practices; but economic inequality remains high, and many rural areas still lag behind urban areas (World Bank, 2008). Despite the existence of many proven technologies and improved agricultural and resource management practices, the adoption of such technologies and practices by smallholder farmers is generally low in developing countries (Otsuka et al., 2013; Sunding and Zilberman, 2001; World Bank, 2008). Early studies on technology adoption identify several key obstacles faced by small farmers who might otherwise adopt new technologies: lack of credit, limited access to information, aversion to risk, inadequate farm size, insecure land tenure, a low level of human capital, and poor infrastructure (Feder et al., 1985; Feder and Umali, 1993). For the past decade or so, international organizations (both in and outside government) have done much to encourage farmers to adopt new technologies and agricultural best practices through agricultural development projects. A common feature of such development projects is that they typically aim to overcome multiple constraints and to promote the adoption of multi- ple practices and behavior changes. Another important feature is that project locations and beneficiaries are typically not randomly assigned. Though these features pose considerable methodological challenges to any study of project impacts, many researchers have compiled impact evaluations of agricultural development projects in the past ten years or so (Dil- lon, 2011b; Duflo et al., 2011; Duvendack and Palmer-Jones, 2012; Mendola, 2007; Moser and Barrett, 2006; Nkonya et al., 2012; Wanjala and Muradian, 2013). Yet the research— especially that using rigorous evaluation methods—is disproportionately small relative to the large number of projects implemented (IDB, 2010; World, Bank, 2010). In addition, few studies evaluate the possible short-term impactsincluding the adoption of particular agricul- tural technologies or changes in behavior—or intermediate impacts of projects before their closure (Pamuk et al., 2014; Peralta, 2014). Such impacts are important, signaling future project success and indicating necessary corrections in strategy. To add to this emerging literature, we evaluate the short-term impact of a development project that aims to increase the income of small mango producers in Haiti. Our evaluation is based on data from a baseline survey and a follow-up survey after 16 months of project implementation. Because of the short time period involved, we focus on evaluating a set of short-term outcomes, such as the adoption of a preferred mango variety, improved production and harvest practices, and behavioral changes in farmers’ production and commercialization decisions. Longer-term outcomes such as increases in mango production and sales take longer to materialize due to the life cycle of the mango tree. For example, it takes more than a year 1 for mango trees to mature and become productive (University of Hawai Manoa, 2014). As with many other agricultural development projects, the main empirical challenge to an evaluation of the project is the fact that project beneficiaries were not randomly assigned. Direct comparison of outcomes between project participants and nonparticipants would lead to a biased estimation of the projects impacts. To control for the potential selection bias, we use nonexperimental econometric methods: matching, and a combination of matching and difference-in-difference (DID) methods. Due to differences in survey instruments between baseline and follow-up, we are forced to adopt different evaluation methods for different outcome variables. Panel data are available only for some outcome variables. For other outcome variables, we have data from only the follow-up survey. In particular, we use propensity score matching (PSM) and matching in covariates for the outcomes for which only follow-up survey data are available. In order to assess how robust our matching estimates are to possible hidden bias—caused by the effect of unobserved variables that simultaneously affect assignment to treatment and the outcome variable—we use the Rosenbaum bounds approach. For the outcomes for which we have panel data from both the baseline and follow-up surveys, we estimate the impacts using difference-in- difference, propensity score matching (DID-PSM), and DID-matching in covariates to control for time invariant unobserved heterogeneity. There is also a concern that the presence of other projects with similar characteristics in the project intervention sites could potentially bias our evaluation results. To investigate whether and to what extent the presence of other projects bias our results, we conduct a robustness check by excluding all the observations (in both treatment and control groups) of parties that participated in other projects. 1 The results suggest that the project significantly increased the planting of Francique mango trees (the variety preferred for export), but that these trees are still too young to bear fruit (that is, they are not yet productive) and so have not translated into increases in yields or sales. The project also had a positive impact on the adoption of improved agricultural practices (pruning, tidying, grafting, and fencing), and on preferred commercialization behaviors (with a shift away from selling to middlemen and toward selling to producer business groups [PBGs], or cells). But the results on the adoption of improved practices should be interpreted with caution, since they are sensitive to possible deviations from the identifying unconfoundedness assumption. Finally, our robustness checks suggest that the presence of other projects did not influence the estimated impacts of the project. The rest of the paper is organized as follows. Section 2 describes the project, focusing on the theory of change and the selection of project beneficiaries. Section 3 presents the data and the sample design, and Section 4 proposes different evaluation methods. Section 5 discusses the main impact evaluation results, the robustness checks, and the sensitivity analysis, followed by conclusions and recommendations in Section 6. 1 Without more detailed data on the nature and distribution of other programs between the treatment and control groups, it is not possible to know whether they would bias our results. The robustness checks indirectly assess whether and to what degree the presence of other programs would bias the main results. 2 2 The Project 2.1 Background Haiti is the poorest country in Latin America and one of the poorest in the world. Conditions in the country worsened after it was hit by a 7.0 magnitude earthquake in 2010. Its gross national income (GNI) per capita was $760 in 2012. Eighty percent of its population earn less than $2 per day, and 50 percent live below the poverty line ($1 per day) (World Bank, 2014). More than 60 percent of Haiti’s population depends on agriculture for their livelihood. The country also relies heavily on remittances from the Haitian diaspora and on foreign aid. Haiti’s economy has been slowly recovering since 2010, with most of the economic growth coming from agricultural production, construction, and the garment sector (World Bank, 2014). The poverty seen in Haiti is typical of poor rural areas in developing countries, where agriculture plays a major role in the strategy for increasing agricultural incomes and household wealth (World Bank, 2008). Mangoes are among Haitis main agricultural products, with a high potential for exportation. The country is among the 20 largest mango producers in the world (FAO, 2010). But several constraints—such as inadequate production technologies for commercial mangos, institutional barriers, and inadequate (or lacking) infrastructure— impede the increase of mango production and exports (Casta˜ neda et al., 2010). There are about 100 mango varieties planted in Haiti, but only Francique is exported. Of total production, only between 2.5 and 5.0 percent reach the export market (FAO, 2010). Most exported Haitian mangoes go to the United States. Low-quality yields and damage or spoilage during transportation are among the main reasons for the low export rates of Haitian mangoes. Haiti’s mango producers are predominately smallholder farmers who own 10 trees or less and lack the technical knowledge to produce mangoes of export standards (Casta˜ neda et al., 2010). In addition, farmers usually sell mangoes to middlemen, who are responsible for the harvest and transport of mature mangoes. Farmers lack the knowledge, experience, and technical skills required to perform these activities themselves. Mangoes are usually sold per tree and not by the quantity of actual mangoes produced. Farmers’ lack of access to credit to smooth consumption often leads to the premature sale of mangoes to middlemen, who offer cash in advance. 2.2 The Project The project was launched in 2010 to overcome the production and commercialization cons- traints faced by small mango farmers in Haiti. The aim of the project was to increase the income of smallholder farmers and facilitate their access to the value chain for mango exports. The project promotes the formation of producer business groups (PBGs), or cells. Members of PBGs are trained in good practices in mango production—both harvest and postharvest— and commercialization, and basic business literacy (see Table 1 for a detailed list of project interventions). The project promotes the planting of the Francique mango variety, which is in high demand among potential commercial buyers and has the highest export potential (Casta˜ neda et al., 2010). Participants are expected to adopt the practices promoted by the project. The project is a value chain development project implemented in Haiti by TechnoServe, a nonprofit organization with worldwide experience, with the support of project partners. 3 2.2.1 The Project’s Theory of Change The project seeks to boost farmers’ income through higher mango yields, higher-quality man- goes that adhere to international standards, as well as a better linking of producers to inter- national mango value chains. This combination of effects (increased productivity, improved quality, and commercial linkages) is expected to generate an increase in mango sales and the development of stable commercial relationships between farmers and reliable exporters. Ex- porters, meanwhile, would benefit from a higher-quality and more-predictable mango supply. To achieve the project’s objectives, TechnoServe provides training to local mango producers and connects smallholder producers to exporters via PBGs. In addition to establishing PBGs, the project engages existing farmer cooperatives. According to the project design, producers will see increases in mango sales through a combination of three effects: (i) an increase in mango production, either through an increase in the yield of trees already bearing fruit or through new trees; (ii) a reduction of waste due to better harvest and postharvest practices (unnecessary losses are estimated to affect up to 50 percent of mango production); and (iii) an increase in the price producers are able to obtain in the market, as they begin selling higher-quality mangoes through appropriate channels. In the context of the PBGs set up by the project, TechnoServe provides farmers with training in pruning and tree care, nursery care, and orchard-related extension services. 2 , 3 The assumption is that by pruning trees farmers maintain their health, productivity, and size— and facilitate better harvests and improved fruit quality. The program also seeks to increase volume by planting new saplings and, in 2013, TechnoServe decided to undertake grafting to achieve early increases in mango volumes. Thus, increases in volume are to be achieved via (i) increases in the productivity of existing, producing trees (via practices such as pruning and grafting), and (ii) the production of new trees (from saplings). To reduce mango waste, TechnoServe provides farmers with training in harvest and posthar- vest best practices and in the utilization of local transportation. 4 The assumption is that by properly selecting the harvest period that will maximize mango quality, using the right har- vest tools to reduce spoilage, and properly handling the harvested mangoes and sorting them according to the corresponding sales channels to maximize prices per channel, farmers can reduce losses—which before the project were as high as 50 percent of the total potential harvest. 5 2 Asrey et al. [2013] find that pruning results in significantly higher fruit weight, fruit firmness, total carotenoids, antioxidant capacity, and total phenolic content. Early maturity of fruits is observed from un- pruned trees with faster color change, higher total soluble solids, and lower titratable acidity. The fruits harvested from pruned trees show signs of slower ripening, and lower respiration, ethylene evolution rates, and enzyme activity when compared with fruits from unpruned trees. Both anthracnose and stem-end rot disease percentages are reduced in ripe fruits from pruned trees. 3 The pruning and tree care training module teaches farmers to identify and cut appropriate tree branches and maintain mango tree canopy to maximize fruit quality (targeted to all farmers); the nursery training module teaches farmers the process of cultivating a mango from a seedling, including protection against common problems and building seed stock (targeted to a selection of farmers). Farmers also receive on site visits from technicians to help them rehabilitate their mango orchards (extension services). 4 The harvest and postharvest training module teaches farmers to identify appropriate harvest periods and methods, and to properly sort mangoes according to corresponding sales channels. The local transportation training module teaches farmers the appropriate way to transport their mangoes to the appropriate collection center (both modules target all farmers). 5 Casta˜ neda et al. [2010, p. 8] explain that when farmers presell their fruit to middlemen on farms, the middlemen harvest all fruits from a given tree, whether they are adequately ripe or not: After picking all fruit, middlemen select and leave rejected fruits at the farm, paying only for the 4 To increase the price that farmers can get for their mangoes, TechnoServe provides farmers with training in the creation of PBGs and in business planning. 6 The assumption is that by selling their production appropriately (by dozens instead of by plot or tree, for example) and using the PBGs as channels (rather than negotiating by themselves), farmers can get an overall better price for their harvest. Some of these actions may have immediate effects: for example, the adoption of harvest and postharvest best practices could immediately increase mango sales by improving the quality of the fruit and reducing spoilage during transportation. This is the effort with potentially the highest immediate impact. Other activities supported by the program and related to harvesting could also have immediate effects: better access to credit could potentially increase the quality of mangoes sold, by limiting the need of farmers to sell their mangoes before the appropriate time. Another action that could have immediate effects is the adoption of commercialization best practices: markets for agricultural products are becoming more integrated and concentrated in their structure, and smallholder farmers tend to be excluded from the modern value chain, mainly because of the challenges of complying with quality standards and of providing the quantities required to ensure reliability (Barrett, 2008; Farina et al., 2005; IFAD, 2011). Or- ganizing farmers in groups reduces risk and transaction costs, and increases information flows (IFAD, 2011). It also increases the farmers’ bargaining power for better contract conditions, and ensures reliable quality and quantities of product (Markelova et al., 2009). The third action—the adoption of agricultural best practices (pruning and tree care) and the planting of new Francique trees and grafting of productive, non- Francique trees—is unlikely to generate results in the short run. In fact, the pruning promoted by the program, for example, could negatively affect mango yield in the short run. Davenport [2006] states that severe pruning (done to rejuvenate mango trees so large that the canopy migrates far beyond the reach of harvesters) accompanied by tip pruning (to reduce the flush frequency back to normal) would eliminate flowering and reduce production for about a year. Asrey et al. [2013] designed an experiment to quantify the effects of pruning done to manage canopy size. The fruit yield of pruned trees was found to decrease in the first year compared with the fruit yield of unpruned trees; it increased beyond unpruned trees during the second year. In another experiment, Das and Jana [2013] find that the initiation of fruiting begins after the third year of pruning. Meanwhile, it is too early for new Francique trees and grafted, productive, non- Francique trees to bear fruit—it takes 5–6 years for a new tree or 3–4 years for a grafted tree to start bearing fruit. There are other reasons to believe that the project initiatives undertaken thus far will not yield immediate results. It is well known that it takes time to translate adoption into chosen mangoes. Rejected fruits could be immature, over ripe, bruised or fly infested, with a low chance of commercialization. Mango losses may reach up to 50 percent of the total potential har- vest. Mango is sold to exporters in Port-au-Prince (transportation is arranged with the exporter, and prices vary), however, at the export facilities, it is necessary to re-classify mangoes due to the inappropriate postharvest practices of middlemen (rejects account for around 50 percent). Rejected mangoes are sold to madam sarahs. “Madam sarahs” are retailers, usually women, who sell the mangoes in the local markets. 6 The producer business group (PBG) training module supports farmers in creating and organizing PBGs, which function as intermediaries for smallholder farmers to increase their access to markets (and improve their negotiating power in bulk sale, transportation, specialization of tasks, and so on). In the business planning training module, farmers learn about the sales channels available to smallholder farmers, and the components of a business and marketing plan. Both modules target all farmers. 5 changes in production, sales, and income. As farmers decide which practices to adopt, they must consider risk, profitability, and input constraints (Feder et al., 1985; Minten and Barrett, 2008). Technology adoption involves a process of learning over time (Foster and Rosenzweig, 1996). And finally, farmers adopt new technologies in a stepwise fashion, not all at once (Byerlee and Hesse de Polanco, 1986), even when such practices are promoted as a package. 2.2.2 Targeting of Beneficiaries The key challenge we face in our evaluation analysis is the fact that the project areas and project beneficiaries were not randomly assigned. PBGs were established in areas of high po- tential mango yield, as indicated by the quality and volume of current production, agronomic characteristics, local industry players, land availability, and infrastructure. Farmers eligible to participate in the program grow mangoes, have at least five trees, and farm between 0.5 and 5 hectares of land; they are also required to be members of cooperatives and to live in the communes targeted by the project. But these eligibility criteria have not been strictly enforced; they have instead served as a guide for selecting poor, smallholder farmers as project beneficiaries. Within selected communes, community members were hired by TechnoServe and assigned to recruit farmers. Their recruitment areas were determined by their location and their ease of access to surrounding areas where mango producers were likely to be found. The project specifies that recruiters can travel for a maximum of an hour-and-a-half by foot to promote the project among mango producers and to invite them to join a local rally where they may enroll in the project. The decision to join the project or not is voluntary. Given the way beneficiary communes and farmers were selected, two main sources of bias are likely to arise. Farmers participating in the project may differ from nonparticipants in their observable characteristics due to project targeting, but they may also differ in unobservable characteristics due to self-selection. These sources of bias will be addressed to the extent allowed by the data and the sample design using nonexperimental methods. 3 Data and Method for Impact Evaluation 3.1 Data The data used for the impact evaluation are from a baseline survey in 2012 and a follow-up survey in 2013. Information about the sample design and data collection process was obtained from reports provided by the firms hired to conduct data collection, and some gaps remain. A local survey firm under the guidance of TechnoServe implemented the baseline survey during JuneJuly 2012, prior to the implementation of the project. Two hundred and sixty-eight participants (treatment households) and 510 nonparticipants (comparison households) were randomly selected to be part of the study from eligible mango producers in a total of 12 communes. The treatment households were selected from the communes where project activities were implemented. The TechnoServe database was used to obtain the list of mango producers en- rolled in the project. The comparison households were to be selected from the same communes as the treated ones (Ba et al., 2012). Since a list of farmers was not available for the compa- rison group, households for this group were selected based on their geographical proximity to the treatment households. The location of treated households was defined using as refer- 6 ence the communes’ focal point for animators’ project activities; the sites for data collection were chosen randomly from the list of sites available. 7 The treated households interviewed were chosen from within a virtual polygon—a 1km—side equilateral triangle whose centroid coincide with the focal point of the animator’s project activities—while control households were identified in nearby areas with a high presence of mango production but outside the intervention polygon (Schwartz, 2013). 8 According to a project report, “the sample size was determined to detect a 15 percent point difference in terms of change in key indicators for each group between the time of the baseline and the time of the follow-up survey, with a probability of 0.05 that this change is sample error, and a confidence level of 95 percent. As a result, the sample size was calculated to be 220 mango producers and increased to 250 to take into account possible dropouts in the treatment group. For the comparison group, the sample size was doubled to 500” (Ba et al., 2012). 9 But more observations were collected for the treatment (268) and comparison groups (510) (Table 2). 10 The baseline survey collected information on household demographics, mango production (including costs), and mango-harvesting practices, as well as information on project partici- pation activities (that is, participation in various groups and associations) and the types of training received by participants. Table 2 describes the sample distribution by treatment status and by commune. The number of treated and control observations are not evenly distributed across communes. Table 3 compares socioeconomic characteristics between the treatment and the comparison group. Performing the t-test for equal means of the socioeconomic characteristics between the two groups shows there are statistically significant differences for some variables (7 out of 20 variables). In an average household in the treatment group, the household head is older and more likely to be female. A typical treatment household also plants more non- Francique trees and is more likely to rank mangos as their first source of income and first crop. Finally, the wall materials of the residences of the treatment households are significantly better than those of control households (though the homes are small overall). And compared with the control households, treatment households have less access to water from a river or well. This descriptive evidence reinforces the early discussion that the direct comparison of outcome variables between the treatment and control households leads to biased estimates, and thus indicates the need for alternative evaluation methods to account for the existing differences between the two groups. A different firm was hired to conduct the follow-up survey in October 2013. With inputs from IDB and TechnoServe, the firm made a series of changes to the survey instruments. Only some variables from the baseline survey (on production, commercialization, sales, participation in training activities, and gender of household head) were kept. Additional questions on mango production, crop production, and plot information were newly added to the follow-up survey. Neither the baseline survey nor the follow-up survey collected any commune-level data. The sample size was reduced from the original 778 to 474 (Papyrus, 2013). Due to budgetary constraints a limit of 450 observations was set, 200 treated and 250 nontreated (Papyrus, 7 The animator is the person in charge of contacting, recruiting, and training mango producers. 8 These geographic parameters were defined based on TechnoServe’s experience with the project. 9 Unfortunately, this is the only information that is available to us regarding the sample design and power calculation. 10 It is not clear from the project reports why the sample included more observations with respect to the sample design. 7 2013). But the actual number of observations was 47424 more than planned. Of the 474 observations, 211 were of treatment households and 263 of control households. The sample reduction most affected the observations of control households. The households interviewed in the follow-up were selected randomly from the list of households interviewed for the baseline. The number of treatment and control households was established to maximize statistical power, given the restrictions imposed by the budget. We conducted attrition analysis to determine whether the sample reduction affected our conclusions. An additional 13 observations were not included in the baseline survey and were therefore excluded from our study. The total number of observations that include information from both the baseline survey and follow-up survey is 461205 treatment households and 261 control households. 11 Table 4 shows the distribution of the final observations used in our evaluation analysis by commune and by treatment and control groups. As in the baseline survey, the dis- tribution of observation is uneven across communes, with two extreme cases. While Verettes has eight treatment observations but no control observation, the commune labeled as “other” only has 29 control observations but no treatment observation (Table 4). 12 To check whether the attrited households and panel households are systematically different, we compared their pretreatment household and mango production characteristics within both the treated and control group, using data from the baseline survey (Table 5). Out of the 36 variables compared, we found statistically significant differences for five variables in the treatment group: the age of the household head in three categories (household head aged 4554, 3544, and 55 and more), and household access to water from a well and from a public water source. Out of the same 36 variables, the difference is statistically significant for only four variables in the control group: the age of household head in two categories (household head aged 2534, and 55 and more), the number of Francique trees, and the number of rooms. These results suggest no systematic difference exists between the attrited households and the panel households for the treated and control groups. Therefore, attrition bias is unlikely to affect our evaluation. 3.2 Methods To estimate project impacts we use nonexperimental econometric methods to control for selection biases. To estimate the average treatment effect on the treated (ATT) we use propensity score matching (PSM); matching in covariates; difference-in-difference, propensity score matching (DID-PSM); and DID matching in covariates. We use PSM and matching in covariates for the outcomes for which we do not have data from the baseline, namely the adoption of various production, harvest and postharvest, and commercialization best practices, and use DID-PSM and DID matching for the set of outcomes related to production and sales for which we have data from both the baseline and the follow-up surveys. Matching relies on the conditional independence assumption, or unconfoundedness, and on the assumption of overlap (Heckman et al., 1997; Wooldridge, 2010), which states that the researcher should observe all variables simultaneously influencing the participation decision 11 The fact that the number of observations is bigger in the control group than in the treatment group is generally consistent with the sample requirement for matching analysis (Imbens and Wooldridge, 2009; Khandker et al., 2010; Ravallion, 2009). 12 Unfortunately we do not have further information on the sample design to explain why we observe this distribution of treated and control observations within communes. 8 and outcome variables, and that there is overlap between the probability distributions of treat- ment and control samples. With this method we control for bias on observable characteristics, caused by the targeting of project beneficiaries (for example, based on a set of eligibility cri- teria). We conduct matching on the propensity score using kernel matching (Heckman et al., 1997; Hirano et al., 2003; Jalan and Ravallion, 2003). Kernel matching is a nonparametric method that uses a weighted average of all the observations in the control group to cons- truct the counterfactual outcome for each treated observation (Smith and Todd, 2005). The weights depend on the type of kernel function chosen. An advantage of kernel matching is that it reduces the variance of the estimates by using more information. We also conduct matching in covariates (Abadie and Imbens, 2006; Imbens, 2014). This estimator consists of matching all units, treated and control, using the distance between the values of the covariates for each observation (in our case weighted by the sample variance matrix). If the matching is done with replacement, the order of observations does not matter. The matching can be conducted with one or more observations (one and five in our case); increasing the number of observations improves the quality of matching but increases the variances of the estimates (Smith and Todd, 2005). Since matching multiple covariates can lead to substantial bias, it is combined with bias adjustment to remove most of the bias. This approach uses linear regression to remove the bias associated with differences in the matched values of the covariates (Abadie and Imbens, 2011; Imbens, 2014). Bootstrapped standard errors are calculated for matching estimates to account for the two-step PSM pro- cedure (Abadie and Imbens, 2008), 13 and robust standard errors are estimated for matching on covariates (Abadie and Imbens, 2011; Imbens, 2014). As mentioned earlier, PSM relies on the assumption of unconfoundedness. But it is likely that there are systematic differences in outcomes for participants and nonparticipants due to unobservable characteristics, known as bias on unobservables. While it is impossible to directly address this problem using cross-sectional data, we conduct Rosenbaum bounds analysis to check the sensitivity of our estimates to deviations from the conditional independence assump- tion (Rosenbaum, 2002). For the outcome variables for which we have panel data, we use these to estimate the impacts using the DID-PSM (Smith and Todd, 2005) and DID-matching on covariates. Both methods control for time invariant heterogeneity. 14 We follow a few standard procedures to estimate project impacts using the matching methods (Imbens, 2014; Imbens and Wooldridge, 2009; Wooldridge, 2010). The first step is to estimate the propensity scores (PS) using a probit or logit model. The next step is to check the overlap region of the estimated PS between the treatment and control group. Another step is to trim the observations with PS close to zero and one. We estimate PS using a logit model and test whether higher polynomial terms are needed or not, following Dehejia and Wahba [2002]. With the estimated PS we can check for overlap of the probability distribution between the treated and control groups, by plotting the estimated PS for the two groups. A substantial overlap is crucial in order for the PSM method to work. Failing to identify substantial overlap is a major source of bias in PSM estimates of impacts because the counterfactual group is not similar to the treatment group. Following standard 13 When conducting PSM, sensitivity analysis is also usually conducted to determine that the estimates are not sensitive to different matching methods. But when conducting PSM with the nearest neighbor, we are unable to obtain the correct standard errors for inference. For this reason we do not conduct PSM on the nearest neighbor. 14 We lack data from before the baseline survey to test for parallel trends in treatment and control samples, but we make this assumption as done in other studies without long historical data. 9 matching procedures, we prune the observations with an estimated PS above 0.90 and below 0.10 to improve overlap (Imbens and Wooldridge, 2009; Ravallion, 2009; Wooldridge, 2010). With this trimmed sample we reestimate the PS and conduct the matching again. We conduct a balancing test to check for the similarity of the marginal distribution of the covariates used to estimate the PS. The test aims to determine whether the matching procedures have served the purpose of making participants and nonparticipant groups more similar. Covariates are compared via a measure of standardized bias or normalized differences in means (Imbens and Wooldridge, 2009; Wooldridge, 2010). 15 To assess covariate balance we follow Imben’s rule of thumb regarding percentage bias below 25 percent (Imbens and Wooldridge, 2009; Wooldridge, 2010). A potential concern is that the presence of similar projects in the study sites could po- tentially bias the impact estimates. To check whether our estimated results are affected by this, we conduct a robustness check by reestimating the impacts using a reduced sample that includes only those households that have not participated in other projects, and compare the new results with the ones obtained using the whole sample. Because the conditional independence assumption is a strong one, we carry out a sensitivity analysis to determine how strong an unmeasurable variable must be to influence the selection process and the outcome of interest as to undermine the conclusions. If there are unobservable variables that simultaneously influence participation and outcome variables, matching only based on observable characteristics may lead to biased estimates. In order to determine how sensitive the matching results are to deviations from the conditional independence assumption, we also conduct the Rosenbaum bounds sensitivity analysis (Rosenbaum, 2002), a popular exercise after matching methods (Dillon, 2011b; Ogutu et al., 2014; Rusike et al., 2009). 4 Results In this section, we describe farmers’ participation in various project activities. We then briefly present the results of PS regression and check the quality of matching. Most of our discussion is devoted to the evaluation results based on PSM, matching on covariates, DID-PSM, and DID-matching on covariates. 16 We conclude by discussing results from the robustness check and Rosenbaum Bounds analysis. 17 4.1 Participation in Project Activities Farmers’ decision to participate in the program or not was voluntary. After one year of project activities, over 50 percent of participants had received training in improved production and harvest and postharvest practices, and business and commercialization practices (Table 6). Meanwhile, 87 percent of beneficiaries identified themselves as members of a PBG, and 56 percent or more as involved in PBG activities (Table 8). As can be seen in Table 6, Table 15 We use the normalized mean difference instead of the standard t-test for equal means, because the former does not depend on the sample size. For instance, the t-statistic may be large in absolute value simply because the sample is large, and small differences between sample means are statistically significant even if the absolute difference is substantially small. For more details, please refer to Imbens [2014], Imbens and Wooldridge [2009], and Wooldridge [2010]. 16 For the estimation of the PS, PSM, and DID-PSM methods, we use psmatch2 in STATA (Leuven and Sianesi, 2012). For the matching in covariates and DID-matching in covariates we use nnmatch in STATA (Abadie et al., 2004). 17 To estimate Rosenbaum bounds we use mhbounds in STATA (Becker and Caliendo, 2007) 10 7, and Table 8, households in the control group also participate in activities similar to those promoted by the project, which is explained by the presence of similar projects in the study area (AGRITECH, 2014; Casta˜ neda et al., 2010). We address this issue (a possible source of bias in our results) by conducting a robustness check, which will be discussed in section 4.7. 4.2 Propensity Score Estimation The probability of program participation, or PS, is estimated using a logit model. The depen- dent variable is a dichotomous variable for whether or not the household was a beneficiary of the project. The explanatory variables in the logit model are a set of variables from the base- line survey that include information on household head characteristics, housing, children’s schooling, mango production and its perceived importance to household income, access to water, access to markets, and so on. First we estimate the PS with all the observations in the sample. Following standard procedures for PSM, we trim the sample by eliminating the observations with PS > 0.90 and PS < 0.10, and reestimate the PS based on the trimmed sample (Imbens, 2014; Imbens and Wooldridge, 2009; Wooldridge, 2010). A total of eight nontreated observations were trimmed. We check the specification of the PS using the method suggested by Dehejia and Wahba [2002]. We find that we do not need to include higher polynomials of the variables, or additional variables, for the estimation. Based on the estimated results of the logit model, the beneficiary households were more likely to be headed by a female, to cite mangoes as their main source of income, and to have fewer Francique mango trees than households in the control group (Table 9). Beneficiary households were also likely to lack access to water from a river, spring, or pump, but had residences with more adequate wall materials (Table 9). In order to visually examine how the beneficiary and nonbeneficiary households overlap each other in terms of PS, we present in Figure 1 the predicted probability of selection for the project among both treated and nontreated households. As can be seen in Figure 1, there is overlap for a good range of PS except for a few observations in the very right tail (PS > 0.75). There are fewer control observations to match treated ones at values greater than 0.75 of the estimated PS. This is not an issue for estimating the program impacts using PSM, however, because we are able to find nontreated observations similar to the treated ones within a wide range of estimated PS values. A useful criterion for measuring the quality of matching is to check the treated group against the control group in terms of observed characteristics before and after matching. Table 10 presents the difference for the key variables between the treated and control groups before and after matching. Figure 2 presents the estimated PS with the weighted observationsboth treated and controlby the weights generated for the kernel matching algo- rithm. We find that matching improves the balance between the two groups, as supported by the fact that the absolute value of normalized mean differences between the treatment and control groups is much smaller after matching than before matching for the majority of the variables. In fact, the absolute value for the normalized difference in means (percent bias) for all the covariates is below 25 percent, an indicator of a good balance of covariates (Imbens and Wooldridge, 2009). For PSM and PSM-DID we confirm overlap improvement (see Figure 2). 11 4.3 Impacts on Production and Sales We first present the estimated impacts of the project using DID-PSM and DID matching in covariates for the outcomes related to mango production and sales. 18 We then present the results using PSM and matching in covariates for the adoption of best practices promoted by the project. For both DID-PSM and PSM, we present the results based on kernel matching methods. Table 11 reports the estimated impacts of the project on the total number of Francique mango trees, and whether these trees are productive or immature, as well as the total value of sales. We estimate these impacts using DID-PSM and DID matching in covariates. 19 The results suggest that the program has a significant and positive effect on the total number of Francique trees, and that these trees are young (that is, not yet productive). On average, treated farmer households increased their number of Francique trees by 12.3, and number of immature trees by 12.4. These results are significant at the 10 percent level, respectively, and indicate that those new Francique trees are saplings planted by farmers as a result of project participation. But when comparing the number of productive Francique trees we find no statistically significant difference between the treatment and control groups. Consequently, we do not