Risk-Based Scorecards for Microenterprises: Pilot in Haiti Final Report

RISK-BASED SCORECARDS FOR MICROENTERPRISES: PILOT IN HAITI FINAL REPORT MAY 4, 2017 This report was produced for the United States Agency for International Development (USAID) by Social Impact, Inc. It is made possible by the support of the American people through USAID. The contents of this report are the sole responsibility of Social Impact, Inc. and do not necessarily reflect the views of USAID or the United States Government. Grant No. AID-OAA-F-14-00017 June 30, 2017 ii CONTENTS Acknowledgements iii Acronyms iv Executive Summary v Introduction 1 The Pilot 1 The Risk-Based Scorecard 3 Structure of RBS 4 Competitive Landscape for RBS 4 RBS Rollout in Haiti 5 RBS Training in Haiti 5 Tracking RBS Implementation, Data Capture and Storage 6 Assessment of RBS Effectiveness in Haiti 7 Findings from RBS Rollout and Assessment in Haiti 10 RBS Take-up and Completion Rates 10 Risk Factors Affecting SMEs in Haiti 12 Challenges Faced and Managed during the Pilot 14 Lessons Learned 14 Recommendations 17 Scaling Plan and Next Steps 18 Annex I: Award Milestones 20 Annex II: Sample Size Calculations 21 Annex III: Scorecards Completed by Sogesol Branches for Each Round 22 Annex IV: Example of a Completed Scorecard 26 Annex V: RBS Completion, Client Tracker Protocol 27 Annex VI: Guiding Questions for Key Informant Interviews with Sogesol 29 iii ACKNOWLEDGEMENTS Social Impact is highly appreciative of Société Générale Haïtienne de Solidarité (Sogesol) in Haiti for its partnership in the pilot of the Risk-Based Scorecard and for all the support provided to administer the scorecard to small and micro enterprises. We would like to especially thank Sogesol’s Director, Mr. Evans Baptiste, for all the cooperation and guidance he provided us throughout the pilot, as well as the monitoring officers for their contributions to this study through gathering data and sharing their experiences using the tool, which will inform future tool improvements. We also recognize and thank Dr. David Apgar for developing the Risk-Based Scorecard and for identifying and bringing in Sogesol to participate in the pilot study as Social Impact’s partner and subcontractor. Lastly, we extend our gratitude to the Global Development Lab and Development Innovations Ventures at USAID for the grant, and especially to the AOR for the guidance and support throughout this pilot study. iv ACRONYMS AOR Agreement Officer Representative DIV Development Innovation Ventures EFL Entrepreneur Finance Lab MFI Microfinance Institutions MIS MLEs Management Information System Medium and Large Enterprises RBS Risk-Based Scorecard RCT Randomized Control Trial SOGESOL Société Générale Haïtienne de Solidarité SI Social Impact SMEs Small and Micro Enterprises v EXECUTIVE SUMMARY In many developing countries, a large learning gap exists for small and micro enterprises (SMEs) and the microfinance institutions (MFIs) that help address their financial constraints. SMEs face challenges in identifying and assessing the drivers of successful businesses, and in experimenting with such new approaches as financial planning or productivity-enhancing activities. Similarly, given the limited availability of user-friendly tools, MFIs grapple with learning challenges in effectively assessing SME risks in order to provide services to these enterprises. To address these challenges, Social Impact, Inc. (SI) developed a simple-to-use Risk-Based Scorecard (RBS). The scorecard is a continuous learning tool to help both MFIs and SMEs learn collaboratively and adapt their business strategies to improve their performance. With the tool, SMEs can identify crucial risk factors that affect their businesses. They can estimate the effects of such factors on their sales at the beginning of a given period, and then verify, at the end of the period, whether the estimates held true. By examining the gap between the estimates and realized outcomes, SMEs can learn over time to predict their risks more accurately and adopt strategies to improve their performance. MFIs can also learn to better assess risks facing their clients, and feed this knowledge into reducing non-performing loans. With a grant from Development Innovation Ventures (DIV) of USAID, SI conducted a pilot study with Société Générale Haïtienne de Solidarité (Sogesol), a regulated MFI serving SMEs in Haiti, to roll out the RBS and test its utility in improving performance, both for Sogesol and among SMEs. The pilot lasted 18 months, and was completed in June 2017. To roll out the RBS, SI developed training materials and used them to train Sogesol’s monitoring officers. To assess the effectiveness of the RBS to improve performance, SI designed a randomized control trial (RCT). In the RCT, 253 active Sogesol clients were randomly allocated to a treatment group to receive the RBS, while 252 clients were randomly allocated to a control group that would not receive the RBS until the pilot ended. SI trained Sogesol officers to use the RBS to gather data from its treatment clients, share it with SI for analysis, and document both Sogesol and client (SME) experiences using the RBS. After the training, Sogesol rolled out the RBS and conducted three rounds of data collection from the treatment group over a period of seven months. A low uptake and completion of the RBS among SMEs was observed; therefore, the assessment of the effectiveness of the RBS in improving performance was incomplete. Valuable lessons for understanding the potential of the RBS as a learning tool for MFIs and SMEs emerged from the pilot. The MFI was able to learn about the risks facing SMEs, and to a smaller extent, the effects of such risks on SME sales. Some SME clients used the opportunity to express their appreciation to Sogesol or to communicate their requests and issues related to their relationship with Sogesol. SI and Sogesol learned lessons for improving the RBS design and implementation to increase its uptake and utility as a collaborative learning tool. Additionally, Sogesol officials verbally expressed interest in trying an improved version of the RBS in the future, and scaling it up to more branches if proven effective. 1 INTRODUCTION Financial constraints often limit growth of small and micro enterprises (SMEs) in developing countries, and there is great demand from SMEs for financial services to relax these constraints. Many institutions such as microfinance institutions (MFIs) indeed now strive to meet the demand. However, a large gap persists between demand and supply, due in part to learning challenges. There are limited opportunities for SMEs and for MFIs to learn ways to improve and adapt their business practices, and this problem is especially dire in countries with poor enabling environments for SME growth. On the one hand, SMEs face learning challenges in identifying and assessing the drivers of successful businesses. They have less opportunity to experiment with new approaches compared to larger businesses that can afford the costs of such learning processes as financial planning or analysis activities that improve their productivity. On the other hand, MFIs also grapple with learning challenges, since few tools are available to help them adopt strategies for effectively assessing their target SMEs and providing the demanded services. Therefore, Social Impact, Inc. (SI) 1 developed a simple, user-friendly Risk-Based Scorecard (RBS) designed to (1) build the learning capacity of SMEs regarding their business risks, and (2) equip MFIs with a risk assessment tool to help them develop business strategies that serve their clients well. The RBS is essentially a learning tool intended to help both MFIs and SMEs assess risks and take appropriate and timely actions to increase productivity. The tool can be periodically administered to SME clients by loan officers, enabling collaborative and continuous learning and adaptation. To test the value of the RBS as a learning and adaptation tool to improve SME productivity and MFI efficiency, USAID awarded a Development Innovation Ventures (DIV) Stage 1 grant to SI to conduct a pilot study with a MFI serving SMEs. SI carried out the pilot study with Sogesol (Box 1), a regulated MFI in Haiti that has been serving SMEs for two decades. The pilot study with Sogesol lasted 18 months, and was completed in June 2017. 2 This final report discusses the RBS pilot and lessons learned by the SMEs, by the MFI and by SI on the utility of the RBS. Additionally, this report provides recommendations to improve the tool for future use. THE PILOT USAID awarded the grant to SI in May 2014, and SI began implementation in August 2014 with Findev, an MFI in Azerbaijan. However, Findev pulled out of the pilot in early 2015 due to changes in its mission, objectives and target clients. Therefore, SI sought another MFI to pilot the RBS and discussed collaboration with eight MFIs in Africa, Asia and Latin America. Two of those MFIs reported using some type of scorecard –such as a poverty scorecard or credit scoring –to assess their clients. Some of them were reluctant to test new tools for fear of increasing their operating costs and disrupting their regular operations with additional burden on their staff. SI’s offer to share in some of the costs, using SI’s grant from DIV, was not considered adequate by these MFIs to pilot the tool, and most of them therefore declined SI’s offer to particpate in the pilot. In January 2016, SI was able to resume implementation of the pilot after securing another partner, Sogesol, a regulated MFI in Haiti. Sogesol was interested in testing the RBS, as they had previously worked with the Entrepreneurial Finance Lab (EFL) to implement a risk assessment scorecard; this scorecard was intended to help Sogesol understand risks faced by its clients to reduce its risky loan portfolio. The scorecard was based on psychometric models that assessed the integrity, business management and financial skills of only the small 1 Social Impact is a global development management consulting firm based in Virginia, USA with a mission to help global development organization and programs be more effective at improving people’s lives through monitoring, evaluation strategic planning and capacity building services. 2 SI developed the tool with its own resources. USAID/DIV grant covered the data collection and analysis costs. Sogesol also shared in some of the costs by providing training space, access to its clients, use of its equipment and a management staff’s time to select and monitor the data collectors. SI, using the grant, paid for the labor and travel/phone expenses of the data collectors. 2 enterprise borrowers with loans above 50,000 HTG (US$ 720). Within the first few months of implementation, it was evident that the pilot was slow at obtaining results, and the model was not proven to be robust enough for the type of clients with whom Sogesol engages. More specifically, Sogesol did not believe that the statistical model piloted by EFL added value to the risk assessment model Sogesol had tested and applied for almost a decade. Indeed, Sogesol tried to combine both - EFL and Sogesol - models to develop a better tool, but the results were not satisfactory. Therefore, Sogesol terminated the use of the scorecard in 2015 after four years of pursuing its implementation. Nonetheless, Sogesol continued to search for simple tools to conduct risk assessments for both small and micro enterprise clients. As such, when SI approached Sogesol in late 2015 and demonstrated the RBS to its senior management staff, the MFI was interested in collaborating with SI to pilot it. The purpose of SI’s pilot study with Sogesol involved testing the validity of the RBS as a collaborative learning and adaptation tool for SMEs and MFIs. Specifically, the objectives of the pilot study included the following: Objective 1: Train Sogesol on the RBS and roll out the RBS; Objective 2: Through an assessment, test whether clients’ learning through the RBS could help increase their sales by at least two percentage points in a year, and whether through such increased sales, the MFI’s non-performing loans could be reduced by 0.2 percentage points; and Objective 3: Document lessons learned from the pilot to help with future applications of the RBS by MFIs for their SME clients. The pilot with Sogesol began in early 2016. SI, in consultation with its Agreement Officer Representative (AOR) at USAID, established six milestones to track the award progression over a period of 18 months from January 2016 to June 2017. The inception report formed the first milestone. The other five focused on the rollout and assessment of the RBS tool. These deliverables included training, assessment design and sampling and data collection during milestone periods two, three, and four. Assessment data analysis and report writing formed milestone five, and milestone six included a final report on lessons learned. By late 2016, the SI team realized that uptake and completion rates of the RBS were too low to fully evaluate the RBS tool for its impact on SME sales and on the MFI’s portfolio risk. Therefore, Objective 2 could not be fully achieved and milestone five could not be met. 3 However, valuable lessons emerged to help improve RBS utility for SMEs and MFIs in the future, as discussed later in this report. 3 Annex I outlines these milestones in further detail. Box. 1. About Société Générale Haïtienne de Solidarité (SOGESOL) Since November 2000, Sogesol has worked toward its mission to promote Haitian entrepreneurship by adapting traditional bank services to the needs of SMEs. The majority of its clients are small and micro business owners and agricultural producers, of which many are women. As of December 2015, Sogesol was serving nearly 35,017 borrowers, with an average loan size of $502 and total outstanding loans equal to US$35.3 million. Sogesol has 16 branches, of which six are in metropolitan zones and 10 in rural areas. Sogesol is an independent, commercial and a regulated microlending institution that uses a “service company” model. Under this model, Sogesol provides loan origination and credit administration services to SOGEBANK. The loans are booked at SOGEBANK, but Sogesol has primary responsibility for promoting, evaluating, approving, tracking and collecting them. Sogesol’s shareholders include SOGEBANK (50.18%) and ACCION International (9.12%), with the remaining share held by individuals. References: ACCION: Sogesol. https://www.accion.org/our- impact/sogesol Groupe SOGEBANK: Sogesol. https://www.sogebank.com/qui-sommes- nous/sogesol/ 3 THE RISK-BASED SCORECARD Large businesses can often afford to invest in learning and analytical tools to improve their performance. While SMEs also need such learning tools, they cannot afford to develop or buy them, nor are the available tools tailored for SME needs in terms of utility and simplicity. To address this challenge, in 2014 Dr. David Apgar (Senior Director at SI during the time) started developing the RBS to provide SMEs with a simple and easy-to-use tool for risk assessments. The RBS was piloted in 2016 with SMEs through Sogesol, a MFI serving SMEs in Haiti. The RBS is a spin-off of a simple Goals Screen algorithm 4, also developed by Dr. Apgar (See Box 2). Goals Screen was developed to help medium and large entrepreneurs (MLEs) and business executives find the best ways to reach their business goals. Dr. Apgar has been testing and refining the Goals Screen for over a decade in many countries with MLEs and b usiness executives. Based on these experiences, Dr. Apgar claims that, after six sessions, most Goals Screen users significantly change their approach to reaching their goal and predict results more accurately. By the tenth session, most users report significant improvements in sales or cash flow growth. The well-tested and refined Goals Screen provided a strong foundation for RBS development. However, the tool needed to be simplified and made more affordable for use by SMEs and by MFIs that are typically smaller than commercial banks and financial firms. As many MFIs are now familiar with scorecards, SI developed the RBS as a simple scorecard. The simplicity is such that MFIs need not make any major management information system (MIS) modifications to roll out the tool, nor do they require extensive training. Rather, MFIs’ loan officers can easily embed the tool into their regular client visits and can administer it using pen and paper. As RBS requires only a handful of rules and requirements to implement, loan officers need only two days of training before rolling it out. SME owners are expected to learn the RBS thoroughly over the course of 12 monthly sessions. To be effective in improving performance, users must occasionally be creative in identifying new risk factors, and that kind of creativity cannot be taught through 4 See http://www.goalscreen.com for more details on Goals Screen Application. Box. 2. Goal Screen Methodology The Goals Screen application is an Assumption-Based Metrics (ABM) tool that focuses on the most important factors. This tool stands in contrast to commonly-used Balanced Scorecards, which require voluminous data, a challenge in many developing countries for small businesses and small financial firms. The ABM begins with business owners’ logic and intuition about what most drives success, and then tests those intuitions, as described in these three simple steps: 1. Business owner lists all major assumptions the business needs to forecast performance against the business objective. This list includes both areas within business owner’s control and areas that are outside of it. 2. Business owner identifies worst-case scenarios for each of those assumptions, then estimates the worst-case outcome for the factor if that scenario were to occur and the worst-case impact it would have on the ultimate business objective. 3. Business owner ranks assumptions by worst-case impact. The owner can typically notice a large drop-off in impact after four to five factors, and thereby identifies the priority factors and the priority metrics in the scorecard. The resulting metrics from the ABM are expected to help the business owner discern whether the business is progressing towards the objective. These metrics also force the owner to test whether the assumptions are justified. By repeatedly using the tool, the business owner is expected to learn to refine the assumptions regarding which factors are central to achieving business objectives. See http://www.goalscreen.com for more details on Goals Screen Application. 4 the RBS. However, the RBS captures the mechanics of learning from results in a simple, intuitive process using a bare-bones trial-and-error methodology. STRUCTURE OF RBS The RBS contains two parts, to be administered over a defined period such as a month, quarter, or year. At the beginning of a period (month/quarter/year), Part 1 of the tool requires the SME client to identify a maximum of four risk factors it faces (e.g. weather, customer visits, raw-material prices) and provide an estimate of expected outcome (e.g. sales, profits) if the worst-case scenario was realized for each risk factor. For instance, political unrest could be an identified factor/threat, and as such the client could estimate that their sales would reach 35,000 HTG (~$510) under the current political situation. These scenarios help users estimate the outcome they should expect for each factor. At the end of the specified period, Part 2 of the tool requires the client to report actual sales and accuracy for each risk factor against the predicted outcome. Once all the information is entered, the RBS automatically calculates (1) the performance gap between estimated and actual outcome (e.g. sales) for each factor, and (2) the performance gap not explained by the factors identified in Part 1. The difference between an actual and estimated outcome (e.g. sales) represents the unexplained performance gap and demonstrates how realistic the underlying risk factors and business concepts are. By reducing the unexplained performance gap, the client can learn over time to narrow the actual outcome gap. If a large outcome gap remains after the client has minimized the gaps for identified risk factors, then the client is expected to realize that he/she must identify another important risk factor that may be missing. The cycle can be repeated for any number of periods. In each period, the client is given the opportunity to revise the assumptions in Part 1 based on the learning from the previous period. Over time, the unexplained performance gap is expected to decrease, signifying the client is better able to identify risk factors and how the factors constrain sales, and take appropriate actions to manage the risks. See Annex III for an example of a completed scorecard. COMPETITIVE LANDSCAP E FOR RBS Grameen Foundation’s poverty scorecards and the SEEP tool are often mentioned as alternatives to performance scorecards for SMEs. However, neither turns out to be useful for diagnosing business problems based on monthly or quarterly results. Grameen’s scorecards assess welfare rather than business problems, and while the SEEP FRAME tool is optimized for MFIs, it is too technical for microenterprises. There is also little evidence of an impact on productivity growth from the standard scorecards. This is because most scorecards assume one risk factor and compare results of that factor with the target. In contrast, the RBS includes alternative scenarios, as well as targets for each key factor. Thus, these standard scorecards provide little guidance when the result for a risk factor is far from its target; it is difficult to discern how much of the outcome is due to that risk factor, and how much is due to overaggressive or underaggressive targets for the outcome or to missing risk factors. Such occurrences effectively break the learning process required of scorecards to be effective to improve performance. The RBS also provides SMEs with a quick and simple learning opportunity on risks facing them. For example, based on the identified risk factors and scenarios, a large unexplained gap in perfomance implies that the factors driving results need more attention, and that the assumptions behind the alternative scenarios need revisions. Small unexplained gaps imply that the business concept works, but the outcome estimates may need some small adjustments. Thus, when updated with each set of results, RBS has the potential to support sustained improvements in business growth. 5 This is because every set of results leads not only to a 5 Apgar, David. 2008. “Relevance: Hitting Your Goals by Knowing What Matters.” 5 reconsideration of priorities for SMEs, but also to a change in RBS factors, targets or scenarios. Over time, therefore, results increase the predictive power of the simplified models embodied in those factors, targets, and scenarios. In short, this learning process has potential as a substitute for the more formalized learning and analytical processes that large enterprises can afford. Some MFIs have tried to build SMEs’ planning capacity by providing them with simple income statements and balance sheets, often prepared by loan officers during weekly or monthly visits using pen and paper. The problem with financial reports is that they focus on such outcomes as price, volume, direct costs and indirect costs, rather than causal factors that drive those outcomes. In randomized trials in the Dominican Republic, standard fundamentals-based accounting training for microentrepreneurs is shown to have no effect in improving their sales volume and prices received, and in reducing costs. 6 Indeed, periodic business advice to microenterprises would represent a relevant alternative to the RBS. However, the cost-effectiveness of such advice is hard to benchmark because of the wide variety of forms of advice. For example, recent research finds powerful effects of consulting services offered to Mexican businesses – including microenterprises – on the number of employees and total factor productivity. 7 Since the interventions tested in the randomized control trial by Bruhn, Karlan, and Schoar cost about $12,000 per firm over one year, however, they are hard to compare with an intervention that costs $120, or 1% of that amount. The consulting services tested in the trial are estimated to have provided benefits equal to $36,000 – three times the investment. However, such returns are shown to be hard to achieve with smaller firms on significantly smaller outlays. A benefit of RBS over such business advising services is that it systematizes one sort of business advice: namely, the use of results to fuel changes to major components of a business model and to the tacit or explicit business plan underlying it. While the RBS does not cover the range of forms of business advice now available, it does have the potential to contribute to a standard of advice on inferential management logic, because they are to an extent replicable. Indeed, the RBS provides a simple way to interpret outcomes as a product of both effort and planning for the sake of adapting tactics to improve performance. The scorecards can be revised to increase their predictive power without extra data collection or special technology. RBS ROLLOUT IN HAITI Prior to the roll out, during a trip to Haiti in December 2015, SI assessed the interest and capacity of Sogesol to use the RBS. Based on the initial discussions, the rollout of the RBS was achieved through a subcontracting agreement between SI and Sogesol, signed in early February 2016. Under the subcontract, SI trained Sogesol on the RBS, designed an assessment, analyzed the data on RBS effectiveness and prepared a report to submit to USAID and share with Sogesol. For its part, Sogesol administered the RBS, using its monitoring officers as data collectors and monitoring them for quality and compliance. Sogesol shared selected information from its client database to design an assessment, periodically collected and sent the data to SI for analysis, and documented the experiences and challenges experienced by both Sogesol staff and by SMEs in using the RBS. RBS TRAINING IN HAITI Based on the initial assessment of Sogesol’s capacity to roll out the RBS, as well as further discussions with Sogesol, SI developed training materials in the form of PowerPoint presentations and a Microsoft Excel version of the RBS. Sogesol’s director, Mr. Evans Baptiste, provided feedback on all the materials. In March 2016, SI delivered a one-day training in French, followed by one-on-one follow-up sessions, or “office hours,” 6 Drexler, Alejandro, Greg Fischer, and Antoinette Schoar. 2010. “ Keeping it Simple: Financial Literacy and Rules of Thumb.” < http://dev3.cepr.org/meets/wkcn/7/784/papers/FischerFinal.pdf>. 7 Bruhn, Miriam, Dean Karlan, and Antoinette Schoar. 2013. “Impact of Consulting Services on Small and Medium Enterprises: Evidence from a Randomized Trial in Mexico.” World Bank Policy Research Working Paper. 6 for two days in Haiti. Training participants included six Sogesol monitoring officers representing six of Sogesol’s metropolitan branches. The training focused on the structure of the RBS and its administration to clients, as well as data capture and reporting. Sogesol’s Director was present throughout the training to act as a co-facilitator, and was available during the follow-up office hours to provide guidance and any needed clarification to the monitoring officers. SI held the training in one of Sogesol’s training rooms in Petion Ville, Sogesol’s headquarters, and delivered it in a plenary session style using PowerPoint and supporting materials such as handouts and interactive exercises. The first part of the training focused on a thorough introduction on the purpose, use, advantages and rationale of the RBS. The SI trainer facilitated this section using flip charting techniques to explain the possible risk factors affecting SMEs. To fully engage the monitoring officers, SI used interactive training methods such as asking participants to provide examples of real-life risk factors affecting their clients. The second part of the training put the RBS into practice by centering on an illustrative client (in the example, a grocery store owner). Through exercises, participants experimented with adjusting the risk factors affecting the client’s sales to understand the variance in sales between actual sales and sales projected at the beginning of the month. The monitoring officers participated in a role-play exercise in which one participant acted as the monitoring officer calling the client to administer the RBS, and the other as the client receiving the call. Sogesol considered it a helpful exercise for equipping the monitoring officers with a better understanding of how the RBS implementation would work. The SI trainer wrapped up the training by explaining the deliverables that would be due to SI moving forward. After the training, the SI trainer remained in Haiti for two days in order to be readily available for any questions from the monitoring officers. As the monitoring officers were not located at the Petion Ville branch, it was not practical for them to follow-up in person with the SI trainer during those days. (Later, the officers brought questions directly to Sogesol’s Director, who then relayed their queries to SI.) Given that the SI trainer was not engaged with monitoring officers during these two days, she used the time to work directly with Sogesol’s Director to develop tools and templates for use by the monitoring officers in tracking and compiling data. To minimize costs and facilitate use, Sogesol and SI agreed on bi-monthly administration of the RBS over a seven-month data collection period. Ideally, the RBS must be administered in additional rounds over a longer timeline to track whether clients improve their learning in accurately predicting risk factors and increasing their sales. However, given the SME client base and the time limitation of one year for the pilot, Sogesol suggested limiting the administration to approximately three periods in a year. Details of the rollout are discussed below. Sogesol also suggested phone calls from monitoring oficers as a mode for contacting SMEs and administering the RBS to reduce costs. For the EFL scorecard, Sogesol used a web based application with an initial face-to-face meeting for some new clients. TRACKING RBS IMPLEMENTATION , DATA CAPTURE AND STORAGE RBS can be administered easily with pen and paper. However, Sogesol wanted to use electronic tablets for data collection and establish a trackable system such that data can easily be captured, stored, and incorporated into their loan appraisals if needed. Therefore SI, in collaboration with Sogesol, designed a simple and trackable system to implement RBS with the following elements: • Scripts for the phone calls made by monitoring officers to their clients to expain the RBS and help clients complete it. • Unique RBS for each client in Excel format, uploaded on Sogesol’s electronic tablets to be completed and updated during each check-in by the monitoring officers. The Excel spreadsheets allow Sogesol to easily merge the scorecards with the MFI’s management information system. • Simple naming conventions captured in each client’s RBS, saved in an Excel spreadsheet for each client and transferred to a Dropbox account that was shared with SI. The monitoring officers were familiar with Dropbox and did not reqiure additional training on it. Nonetheless, SI provided guidance 7 on how to upload the Excel Spreadsheets to Dropbox including the proper use of naming conventions. • A client tracking protocol for the monitoring officers to record their challenges in administering the RBS at each check-in with their clients. The tracker is also expected to help the monitoring officers understand low or sporadic response rates and take actions to improve response rates (see Annex V for the client tracker protocol). SI also provided the monitoring officers at Sogesol with a user-friendly guide in French with step-by-step instructions to facilitate their self learning. At the end of each period, the SI team retrieved the data in Dropbox for each of the six branches and for each SME client. SI’s review process included cleaning the data using SI’s unique data quality toolkit and conducting follow-up calls with Sogesol’s Director to clarify any ambiguities in the data and to provide guidance on how to increase client response rates and store the RBS. The efforts of Sogesol to increase client responses that were recorded in the client tracker (shown in Annex V) were also studied by SI to draw lessons for improving the RBS in the future. ASSESSMENT OF RBS EFFECTIVENESS IN HAIT I The second objective of the pilot involved an assessment to test whether clients’ learning could help increase their sales by at least two percentage points in a year; and whether through such increased sales, the MFI could reduce its non-performing loans by 0.2 percentage points. To conduct the assessment, SI designed a randomized control trial (RCT) involving a treatment group that received the RBS and a control group that did not receive the RBS during the pilot. The RCT was designed to test the hypothesis that use of the RBS would increase the treatment group’s average sales by at least two percentage points more than the control group, and decrease non-performing loans among the treatment group by 0.2 percentage points relative to the control group, contingent on increase in sales. SI’s sample size calculations, conducted prior to the pilot, required approximately 500 clients, equally divided between treatment and control groups to test the hypotheses (see Annex II for details). SI determined the primary sampling unit to be a Sogesol client who satisfies the following criteria: (i) is active: the client has a current loan outstanding with Sogesol; (ii) is a small business owner and not a salaried employee; and (iii) has not completed paying back their loan. To roll out the RBS quickly and start data collection for the assessment immediately after the training, SI sought to establish a preliminary sample prior to the training. However, the SI team faced a delay in obtaining a simple number count of clients at each bank branch that met the selection criteria. Therefore, SI had to wait until March 2016 during its visit to Haiti for the training to work on the sampling. During the trip, SI coordinated with Sogesol’s Director to start the sampling process. After some additional delays due to challenges in retrieving data from the MIS and preparing a codebook defining each variable in the database, SI finalized a sample in July 2016. Starting with the list of Sogesol clients, SI first dropped all clients who did not fit the sample criteria. Then, SI used a simple random sampling procedure to obtain a list of 505 clients that satisfied the criteria. Table 1 displays the number of clients chosen for the pilot by branch and treatment status. 8 Table 1: SME SAMPLE SIZE SI gathered baseline data on the 505 clients chosen for the assessment from the MIS database. Using the MIS data instead of a client survey helped minimize data collection fatigue and costs. This was possible because the sample was comprised of active clients who had already filled out loan applications and from whom Sogesol regularly gathered financial information. The MIS database provided demographic information on clients, as well as data on length of client relationship, type of business operated, monthly sales data, loans outstanding, loan type and size, number of loans received by a client, current loan type, and repayment record. The baseline characteristics of the treatment and control groups for the key features are shown in Table 2. It is to be noted that, across the seven key variables, only gender of owners differed significantly between the treatment and control groups. SI decided this difference could be managed during the analysis phase through disaggregation of results by gender or through controlling for gender in any regressions measuring treatment effect. Therefore, the balance between the two groups was not an issue affecting the internal validity of the experiment. 8 TABLE 2: SME SAMPLE CHARACTERISTICS AT BASELINE Variable Type Variable Control Treatment p-value Continuous Average Age in years 39.0 (0.56) 39.1 (0.51) 0.863 Loan amount in HTG 61,927 (178,337) 52,550 (148,771) 0.522 Outstanding balance on loan in HTG 40,488 (123,821) 32,086 (63.467) 0.338 Number of previous loans with current loan type 3.2 (3.6) 3.2 (2.9) 0.957 Monthly Sales in HTG 247,885 (682,520) 269,298 (1,117,358) 0.795 Nominal Owner is Female (%) 54.0% 61.5% 0.087* 8 Random assignment in a RCT is expected to balance all baseline characteristics between the treatment and control groups (thus eliminating selection bias). But, it is possible, particularly with small samples, that random assignment can, by chance, yield unbalanced groups. Therefore, balance check between treatment and control groups along key baseline characteristics could help establish validity of counterfactual. Branch name Number of Active Clients Served Monitoring Officer Control Sampled Clients Treatment Total Petion-Ville 1,674 Monitoring Officer I 42 42 84 Carrefour 2,058 Monitoring Officer II 42 42 84 Bois-Verna 931 Monitoring Officer III 42 42 84 Rue du Quai 914 Monitoring Officer IV 42 42 84 Croix-des- Bouquets 712 Monitoring Officer V 42 43 85 Cite Soleil & Delmas 1,432 Monitoring Officer VI 42 42 84 Total 7,721 6 252 253 505 9 Loan was taken in 2016 (% with ‘Yes’) 40.5% 44.8% 0.322 Source: Sogesol MIS data, July 2016. Note: Standard deviations given in parentheses. For continuous variables, a two independent samples t-test (2 tailed) was conducted to test the difference between treatment and control groups. For nominal variables, Chi-square test was conducted. Under p values, ***, **, and * represent statistical significance at 1%, 5% and 10%, respectively. To test the hypotheses under Objective 2, SI planned to use the following data: (i) RBS data from the treatment group on risk factors and sales, gathered over the course of three data collection periods, and (ii) data from Sogesol’s MIS on repayment for both treatment and control groups and sales for the control group at the end of the third period. The results from the assessment, including RBS uptake, are discussed below. In addition to quantitative data, SI used key informant interviews (KIIs) with Sogesol’s Director to gather qualitative data on the strengths and weaknesses of the RBS, challenges in rolling out the RBS and recommendations for improving the tool to increase its utility and potential for scale-up. The KIIs were conducted by phone and by email prior to the last phase of the pilot and immediately after termination of the pilot (see Annex VI for the KII protocols). The above assessment methodology involves some risks that may limit lessons learned and recommendations to be drawn from the pilot. The major risks include the following: (1) The low uptake and compliance by both treatment and control group clients limits the ability to assess the effectiveness of the RBS. (2) The short period of seven months for rolling out the RBS and collecting data may not have been a sufficient time period for client learning. (3) The prototype nature of the tool, which may need adaptation and redesign based on learning at each round, can affect the comparability of client data from one period to another. (4) The qualitative data are limited to information obtained from the monitoring officers through Sogesol’s Director, as well as the Director’s own perspectives. No client-level qualitative data were gathered by SI for the assessment to provide client perspectives. (5) Given the short period of the pilot and funding constraints, no cost-benefit analysis was conducted to assess the value for money of the tool for scale-up. Despite these limitations, the pilot offers valuable lessons on areas for improvement in the design and process of RBS implementation. 10 FINDINGS FROM RBS RO LLOUT AND ASSESSMENT IN HAITI To examine effectiveness of the RBS in improving performance of SMEs and the MFI, data were gathered in three rounds in 2016 from 253 randomly sampled clients in the treatment group, as detailed below: • Round 1 contained scorecards from clients with which Part 1 was initiated prior to October 15, 2016. • Round 2 contained scorecards from clients with which Part 1 was initiated between October 15 and November 30, 2016. • Round 3 contained scorecards from clients with which Part 1 was initiated between December 1 and December 31, 2016. 9 RBS TAKE-UP AND COMPLE TION RATES As shown in Table 3, during Round 1, it was not possible to contact all of the treatment sample, despite the continuous follow-up by phone by Sogesol. The average contact rate in Round 1 across all six branches was 52 percent, and the response rate for clients completing both parts 1 and 2 was 11 percent. Both the contact and response rates decreased further in the subsequent two rounds of data collection. In Round 2, only 24 percent of the treatment sample could be contacted, and only three percent completed both parts 1 and 2. In Round 3, 43 percent of the treatment sample was contacted, and only three percent completed parts 1 and 2 of the RBS. TABLE 3: RBS TAKE UP AND COMPLETION AMONG THE TREATMENT SAMPLE, BY ROUNDS Rounds Assigned to treatment No. Clients Contacted Scorecard Completion Part 1 Only Scorecard Completion Parts 1 & 2 No. % (to assigned) No. % (to assigned) No. % (to assigned) Round 1 253 131 52 38 15 28 11 Round 2 253 60 24 16 6 7 3 Round 3 253 108 43 25 10 7 3 9 Round 1 and round 2 are slightly longer than the intended month for data collection due to slow startup and delays in implementation. 11 In all three rounds, monitoring officers reported clients not picking up the phone as the most difficult challenge in contacting and recruiting clients for the study. During Round 1, incorrect phone numbers, clients not picking up the phone, inability to clearly explain the purpose and use of the scorecard to the clients, and survey fatigue were reported as reasons for low uptake (see Figure 1). In response, SI provided a capacity building training remotely to the monitoring officers to provide a better understanding of the purpose and use of the RBS and equip the officers to better explain the tool to their clients. Additionally, in all three rounds, SI found the monitoring officers did not consistently reach out to clients three times, as prescribed in the SI protocol for follow-up to increase uptake. The reason for the low follow-up may be due in part to the limited experience among Sogesol staff in conducting such follow-up exercises. As shown by this data, monitoring officers faced a major challenge in establishing contact with clients. Furthermore, once contact was established, it was difficult to convince clients to use the RBS, to obtain adequate responses to the RBS, and to motivate the client to continue in the pilot. Indeed, after three rounds, only seven sampled clients completed both parts 1 and 2 of the scorecard for at least two rounds of data collection. Of those seven clients, only one client completed parts 1 and 2 of the scorecard across all three rounds of data collection. To gain a better understanding of the low uptake and completion of the RBS, SI conducted key informant interviews (KIIs) with Sogesol’s Director during the third round of data collection, as well as at the end of the project. It is important to note that the monitoring officers were not accustomed to checking in with clients as regularly as once a month, as requested under SI’s RBS pilot. Additionally, as explained by Sogesol’s Director, Sogesol usually conducts its own evaluation and analysis of its clients’ performance. As such, monitoring officers reported that the RBS was too complex for them to use, and the complexity was an even greater challenge for their clients. They found Part 1 of the RBS to be relatively simple and easy for clients to complete, as it only asked them to list the factors that affect their businesses and estimate the loss in sales considering the worst-case scenario for a given factor. However, both the monitoring officers and the clients found Part 2 of the RBS very difficult to complete. To link the worst-case scenario estimates in Part 1 with Part 2 (as well as automate production of results for easy analysis) SI designed Part 2 of the scorecard such that clients estimate the percent decrease in actual sales for each factor at the end of the period. By design, when both parts are complete, the variance between the estimates of sale value at the beginning of the period from the percentage value at the end of the period will auto- populate and tell the story of how well the client understands the effects of external risks on their sales. As such, Part 2 of the scorecard, which involved estimating the decrease in sales, was very challenging to understand and use for both the monitoring officers and the clients. This underscored the importance to revise the RBS to improve its utilization while automating the process. The low uptake and low completion of all the essential parts of the RBS across the rounds severely limited SI’s ability to analyze the data, as originally planned, to statistically test whether the RBS helped SMEs increase their sales over time and whether the resulting increase in sales led to reduction in non-performing loans. However, several valuable lessons emerged from the pilot, including a better understanding of the risk factors Feedback from Sogesol’s Director: The client did not understand the pilot; the client is no expert; the client will not be able to determine sales from previous month; the section of the scorecard on percentage created a lot of confusion. Did not pick up phone59%Call didn't go through19% Other reason 11% Wrong number7% Client too busy4% FIGURE 1 : REASONS FOR NOT COMPLETING RBS DURING ROUND 1 12 perceived by SMEs to affect their sales, and whether and how any learning was occurring among the MFI and SMEs through use of the RBS. RISK FACTORS AFFECTI NG SMEs IN HAITI As shown in Table 4, the majority of the risk factors reported by the sampled SMEs were external, and thus the SMEs have limited control over these factors. TABLE 4: FACTORS PERCEIVED TO AFFECT TREATMENT CLIENTS Factor Round 1 Round 2 Round 3 Total (Number of RBS selecting the factor) Weather (rain, flood, drought, cyclone) 30 11 7 48 Unrest (demonstration) 3 3 8 14 Unemployment 1 0 0 1 School closing/opening 4 1 0 5 Increase in prices 0 1 2 3 Political crisis 36 5 13 54 Social gatherings (party) 0 0 2 2 Machinery issues (broken, obsolete) 1 0 4 5 Business lacks novelty 0 0 1 1 Insecurity 12 6 9 27 Inflation 1 0 0 1 Illness 21 7 4 32 Holidays 0 1 9 10 Theft 21 5 11 37 Fire 7 6 5 18 Economic crisis 4 0 0 4 Death 2 3 1 6 Customer demand (irregular) 2 0 0 2 Cashflow 0 1 0 1 Burglary 1 2 0 3 Bad reception (phone) 1 0 1 2 Accident 3 0 2 5 Grand Total 150 52 79 281 SI also examined the data to discern whether the factors changed between the rounds – in other words, were these factors temporary phenomena, or recurring events? Results are shown in Table 5. From the list of 22 factors listed in Table 4, only 10 factors were found to be selected in more than one round, and all of these factors appear to be outside SMEs’ control. Nonetheless, there is still valuable learning potential for SMEs through the use of RBS in such situations. While SMEs do not have much control over the incidence of these external factors, through awareness of the most important factors affecting their businesses and through learning to predict how much such factors might affect their businesses, SMEs could develop contingency plans and test them during crisis to avoid drastic reductions in sales. MFIs can also learn about the factors that might affect their clients’ sales and institute strategies to help their clients service their loans better through, for example, loan rescheduling or new loans to build back lost business. These types of actions on the part of MFIs could help reduce non-performing loans. 13 TABLE 5: RECURRING RISK FACTORS AFFECTING TREATMENT CLIENTS, AS REPORTED IN MORE THAN ONE ROUND Factor Round 1 and 2 Round 2 and 3 Rounds 1, 2, and 3 (No. of RBS selecting the factor) Weather (rain, flood, drought, cyclone) 12 3 8 Unrest (demonstration) 5 2 2 Increase in prices 1 2 1 Political crisis 22 1 5 Insecurity 10 3 8 Illness 8 4 2 Holidays 4 2 3 Fire 2 2 1 Death in family 2 1 1 Burglary 2 1 2 Grand Total 68 21 33 Interestingly, while SME clients listed many external factors, such as natural disasters and political unrest, Sogesol was able to continue its operations during Hurricane Matthew. 10 SI maintained constant communication with Sogesol to check on operation status and data collection. Sogesol informed SI that the hurricane did not affect their operating areas or their data collectors. Phone services were functioning normally, except for some temporary disruption. However, Sogesol cautioned that if the Hurricane would have affected their operating areas, the effects could have been severe. SI could have tested learning and adaptation through use of the RBS, as planned, if adequate data were available. However, as only seven panel data points were available, no statistical tests could be conducted to test the effectiveness of RBS. The sales data provided by the seven panel clients (who participated in Part 2 of the RBS for at least two rounds) indicated no clear pattern in terms of the effect of RBS on SMEs sales (Table 6). Therefore SI, in consultation with the AOR at DIV, terminated any follow-up by Sogesol to gather further data for the assessment, and could not meet Milestone 5 of the award. TABLE 6: UNEXPLAINED PERFORMANCE GAPS IN SALES OVER TIME AMONG THE PANEL OF TREATED CLIENTS Client No. Round 1 Round 2 Round 3 Change 1 45.3 90.4 81.5 Not improved 2 95 40 NA Improved 3 20.8 59.8 NA Not Improved 4 8,740 NA 14,750 Not Improved 5 11 NA 23 Not Improved 6 -5 NA 4.5 Improved 7 -1.9 NA -5 Not Improved 10 Hurricane Matthew hit Haiti on October 4, 2016 and killed approximately 900 people and destroyed 90% of the southern region and the town of Jeremie. Hurricane Matthew was the most powerful Caribbean storm to have hit Haiti and the region in a decade. Still affected and not fully recovered from the 2010 earthquake, followed by a cholera epidemic, Hurricane Matthew destroyed Haiti and lives of many Haitians even more. 14 CHALLENGES FACED AND MANAGED DURING THE P ILOT SI anticipated several challenges which could affect the overall implementation and assessment of the RBS. The non-exhaustive list of challenges included language issues, implementation delays, inadequate data from the MIS database to create the sample and to assess changes in SME sales and revenue and default rates, communication breakdowns, low uptake of the RBS, and sample attrition and non-compliance leading to reduced assessment rigor and power to measure effectiveness. There were minimal language barriers during this pilot. SI’s program manager speaks French and could communicate with Sogesol and conduct trainings in French. The Director at Sogesol was also proficient in English and could communicate well with SI and co-facilitate during the training when participants needed explanations in Creole. There were many issues due to delays in rollout. SI planned on eighteen months for implementation and assessment of the RBS, with an assumption that rollout could begin within three months of partnering with Sogesol. While SI was able to train Sogesol officers within three months, the actual rollout could not begin until August 2016, which was four months after the training. To minimize the effect of the delays, SI worked closely with Sogesol during and after the training to ensure that Sogesol staff adequately understood how to use the tool. SI also sought updates through emails and phone calls with Sogesol to troubleshoot any problems as they arose. SI initially proposed weekly calls to ensure Sogesol was invested in the program and to let Sogesol know that SI would support it as needed. However, this was logistically challenging due to the busy schedule of Sogesol’s director. Also, the monitoring officers were not located at Sogesol headquarters, but rather in other branches, which made contact with them difficult. Therefore, SI decided on a monthly update call with Sogesol to discuss non-pressing problems, as well as regular email communication to troubleshoot pressing issues. The biggest threat to the overall pilot was low uptake, incomplete data and attrition in terms of non- continuance. This highly limited the statistical power of the assessment. Therefore, the assessment did not have enough power to identify any statistically significant changes in the sales data, even though it may exist in a larger sample. As a result, SI could not fully assess the RBS effectiveness in this pilot. LESSONS LEARNED Valuable insights emerged from the pilot although SI could not complete the assessment of the RBS as planned at inception. In addition to the data gathered for the assessment, SI conducted KIIs with Sogesol by email and phone using semi-structured questionnaires. These KIIs served to document Sogesol’s and its clients’ experiences with the RBS and their recommendations for improving it. The SI team conducted interviews with the six monitoring officers via Sogesol’s Director, and examined the tracking records maintained by officers based on client RBS feedback. No client-level qualitative data were gathered by SI to provide client perspectives. In this section, we detail key lessons learned to help with recommendations to modify the RBS for use by MFIs and SMEs to improve RBS users’ performance through learning and adaptation. Buy-in from MFIs is essential. While SI asked nine MFIs across the globe to help pilot the RBS, only Sogesol agreed to do so, as discussed earlier in the report. Sogesol bought into the use of the RBS because the MFI is genuinely interested and has indeed dedicated its resources in the past to collaborate with external partners such as Entrepreneurial Finance Lab (EFL) to roll out risk assessment scorecards. Sogesol had found EFL’s tool not to be useful, and therefore discontinued its use after four years. Therefore, when SI approached Sogesol with the RBS, it was open to trying out a simple new tool in its continuous pursuit of risk assessment tools to improve performance. While the buy-in from MFIs is crucial to initiate the use of RBS, operational issues could challenge RBS rollout and effectiveness. As discussed in this report, there were several challenges faced during the 15 pilot due to operational processes used in rolling out the tool. Lessons that emerged from such operational issues are discussed below. Contact frequency and length of the pilot: Sogesol monitoring officers mentioned that the reasons they were required to frequently contact their clients to update the RBS were not made clear to them. As explained earlier in this report, Sogesol is not accustomed to such frequent check-ins with their clients. As explained by Sogesol, the check-ins with their clients and assessments were not conducted frequently, or as explained in their own words “the analyses are not conducted as regularly as required for RBS and certainly not monthly”. Therefore, Sogesol considered that frequent contact with clients would involve considerable resources and may also discourage their clients from using the RBS due to interview fatigue and few advantages to offset the costs. So, SI agreed with Sogesol on three rounds of contact over a period of seven months. Indeed, Sogesol encountered challenges in convincing the SME clients to recurrently use the RBS, even when attempting only three rounds of contact. Nonetheless, the decision to use limited contact with SMEs for RBS proved ineffective. The previous EFL scorecard used by Sogesol involved larger sized SME clients than those targeted by the RBS, and therefore the limited contact with that tool was sufficient. However, most SMEs are unfamiliar with such tools and need more frequent follow-up and a longer period of rollout to learn to use the tool, as well as to appreciate the MFI’s dedication to help them improve their performance through use of the tool. Indeed, at the time of the RBS design, SI predicted that SME owners could be expected to learn the RBS thoroughly only over 12 consecutive monthly sessions. Based on this experience, Sogesol suggests that officers’ first contact with SME clients should be a face-to-face interaction, in which they speak for 10-15 minutes to motivate clients and to avoid the challenge of clients simply not answering the phone. Additionally, Sogesol believes that such personal contact would improve clients’ accountability for completing the RBS. The contact frequency and continous use requirements were discussed at length with Sogesol’s management at the time of pilot initiation. However, it is the monitoring officers who ultimately administer the tool. SI provided a day-long training for monitoring officers in Haiti, during which the requirements were discussed. However, Sogesol officers, at the end of the pilot, mentioned that the training was too short and was inadequate for them to clearly understand the requirements of the RBS. Therefore, while buy-in from Sogesol management existed, the implementing officers needed more training and discussions to appreciate the requirements of the RBS in order for it to be effective with SMEs on the operational level. Although SI provided periodic remote trainings to the monitoring officers, SI did not provide any booster training on the mentioned RBS requirements. Uptake and continued use of RBS by clients. The RBS is designed as a learning tool, and its effectiveness is expected to be realized through repeated use which generates learning, upon which businesses can adapt their strategies. Therefore, it is important to find ways to motivate clients to take up and continue the use of the RBS for at least 12 periods, preferably with a monthly cycle (rather than bi-monthly, as used in the pilot, or quarterly, as used by many scorecards). Also, face-to-face interaction with clients to help them learn the tool, instead of phone calls as used in the pilot, could have helped motivate clients to take up and continually use the RBS. By conducting face-to-face discussions with clients about their business and business practices once every month, the monitoring officers could gain considerable familiarity with each business and client. Clients would also benefit from this increased familiarity, as the officer would be able to give advice customized for their business and business practices. The investment of time that each loan officer makes, therefore, would act as a kind of insurance policy encouraging the clients to remain part of the program. Feedback from Sogesol’s Director: The scorecard design requested that the client be contacted twice a month (beginning and end). It also requested to save the excel file and to then complete the second part of the scorecard at the end of the month. This is what made the work complicated for the monitoring officers. We could have instead uploaded the scorecard to our database or created just one excel file. 16 Gap between training and roll out. During the first few weeks of the pilot, some officers mentioned that the instructions were not clear enough for explaining the purpose and use of the RBS to the clients. Given the three-month delay between the training and rollout of the RBS, officers forgot much of the content from discussions held during the training. There was no booster training to refresh officers’ memories, and the guides provided to the officers by SI were not clear to them. The result of this was that the officers took more than 30 minutes during phone calls to gather data from the clients and fill out the RBS with its many factors and scenarios. The SMEs found it inconvenient to answer such long calls during their busy hours of business, and lost interest, refusing to answer or return calls. SME owners expressed that they were tired of the investigation, especially since they had to respond to questions they didn’t fully understand. Some clients chose to participate, but did so unenthusiastically, expressing that Sogesol called repeatedly to ask the same questions over and over. Through KIIs, Sogesol’s Director explained that over time the scorecard became burdensome to their clients. Equipment used for RBS. As discussed before, the RBS can be administered using paper and pencil. Sogesol wanted to electronically capture the data and easily merge it with their existing information systems, so tablets were used for administering the RBS and entering data. During the training, however, tablets were not used, so officers didn’t have practice using them for the RBS. Additionally, there were issues encountered with the electronic tablets during the rollout, and SI’s remote instructions were inadequate for officers to quickly resolve issues with the malfunctioning tablets during the phone calls with clients. There were lessons on RBS design as well. The responses from Sogesol and its clients indicated the many challenges in using the RBS. The clients had a difficult time answering the questions to complete the scorecard, as well as predicting sales for worst-case scenarios and later reporting on decrease in sales due to the selected factors. The officers also struggled to enter data in percentages in the second part of the scorecard, both in Excel format and on tablets. The task of selecting a percentage on the bottom half of the scorecard created a lot of confusion for both the clients and the monitoring officers. With the EFL scorecard, the monitoring officers or the MLE client would complete several pages of scorecard questions through a mobile application, and their answers were stored automatically to be later downloaded from the EFL website. Clients and the officers could not tamper with the data after they were entered. The files used with RBS, despite extreme simplicity, was considered by Sogesol as a bit heavy and repetitive to handle. Officers had to fill out the file for each client and save the file at the beginning of the period, then return to the same saved file to complete it at the end of the period. Each officer had a list of 45 to 50 clients, each with their own scorecard file. In addition, officers had to complete client trackers for follow-ups. This system was burdensome for the officers. Learning is important to increase utility of risk assessment tools. While the uptake and completion of the RBS was low for the pilot, Sogesol stated that the RBS provided them with an opportunity to learn more from their clients about the risks facing them. Additionally, some clients used the opportunities of contact with Sogesol to express their appreciation or to communicate requests which had not been shared before. Sogesol learned that the majority of their clients encountered difficulties estimating the percentage variance between sales of the previous month compared to actual sales. Sogesol’s Director also explained that the variance estimate would always be subjective if the client is not able to obtain monthly sales estimates. The low level of education among Sogesol’s SME clients was found to have affected their response quality. Despite the numerous explanations by the monitoring officers, the few clients that agreed to uptake had a hard time listing the factors that influence their sales, which prolonged interviews, frustrated clients, and forced officers to work harder to elicit responses. Sogesol’s Director explained that although they do not provide a formal and systematic client education on financial literacy to their clients, they have adapted their credit methodology to their small business clients (lower educated clients). Additionally, Sogesol explained that their credit officers do provide ad hoc advice to their clients as needed on decisions that may affect their business profitability. The limited ability of the clients to predict effects of risks and understanding of the RBS were among the likely reasons for the low uptake and completion of the RBS, as perceived by Sogesol. Sogesol has now recognized the need for educating clients and building awareness of RBS prior to rollout of the tool. 17 Due to the learning potential inherent in the RBS, although the pilot was unable to test RBS effectiveness, Sogesol expressed interest in trying out an improved RBS in the future. RECOMMEND ATIONS Based on the lessons that emerged from the pilot, we offer the following recommendations relevant for scorecard design, and for initiating and implementing the RBS. RBS Design. These recommendations were provided by Sogesol. Since most SMEs are informal businesses run by less educated owners, developers must realize that simplification of the RBS for their initial use is needed. This can be done by reducing the number of factors to only one primary or crucial risk factor for estimating the outcomes under the worst-case scenario. It was difficult for the first-time users to grapple with all the factors that they perceive to affect their businesses. Over time and through experience, clients and MFIs will begin to understand the tool better and will become comfortable in using more factors and scenarios. In short, it is important that clients walk before they are asked to run. As for data capture features, developers must ensure files are user-friendly. It might be better to have a single file with multiple pages - one for each client containing the information at the start and end of a period for each round –which can be used throughout the duration of the use of RBS. This way, once the information is entered, officers must only navigate the pages within one file to complete the information at each round. In addition, if the tracking file can be automated to provide a summary report on the clients contacted/not contacted at the beginning and at the end of the period and completion status for each client, this will reduce the reporting burden for the officers. RBS Initiation Scorecard developers should discuss candidly and in detail the requirements of the RBS with both the management and operations staff at the MFI. In doing so, both the developer and the MFI’s management and operations staff should consider the clientele targeted by the RBS to determine the frequency of contacts and length of intervention most appropriate for ensuring RBS take-up and continuous use. RBS Training Scorecard developers should produce more and better training materials and instructions to administer the RBS, and should build in more than one day of training for monitoring officers, as many of them may not be accustomed to administering such scorecards. Piloting the tool and equipment in the field should be a part of the training, in addition to classroom training that incorporates real-life examples. The officers should receive clear guidance from the scorecard developers on how to motivate their clients to take up RBS and continue to use them to continuously learn to assess risks and take actions in order to improve their performance. MFIs should roll out RBS immediately after the training. This will help avoid losses in officers’ skills due to memory lapses that can occur when there is a long gap between training and rollout. If delays in rollout are unavoidable, developers should provide gap training to recapitulate the earlier discussions. In addition to the initial trainings, booster trainings should also be provided after the roll out as a refresher, 18 so officers have an opportunity to thoughtfully consider their practices based on their real-life learning. These booster trainings could emphasize the importance of contact frequency and continuous use for promoting RBS effectiveness. These trainings are especially needed when uptake is very low and clients are reluctant to use RBS continuously. RBS Implementation Prior to rollout, MFIs must address any issues related to availability and working conditions of equipment used by the monitoring officers to administer the scorecard, such as electronic tablets. Initial contact with clients should involve face-to-face meetings to explain the RBS and train the client on the tool. After securing clients’ buy-in for continuing with the RBS, the officers could contact them by phone to update the RBS. In short, the mode of administering the RBS should involve a combination of phone and face- to-face interactions, rather than only by phone as in the pilot. Once the clients take up RBS, sharing results and discussing lessons drawn from those results in a face-to- face meeting is ideal, at least for the first few periods. In addition, a learning event at Sogesol in which the clients could share their experiences using the RBS with one another would be helpful in providing an opportunity for clients to learn from one another and for keeping their motivation up. Such events can also be useful for obtaining client feedback, which can feed into a redesign of the RBS should the need arise. If MFIs have any ongoing client education programs, scorecard developers may consider the opportunity to integrate the RBS into existing programs. If no such client education programs are available, the MFIs and scorecard developers may explore ways to develop easy-to-use educational materials for the SME clients on the benefits of risk assessments in general and the RBS in particular. SCALING PLAN AND NEX T STEPS Sogesol appreciated the learning that occurred with the RBS pilot. However, based on the challenges in rolling it out, Sogesol staff decided not to scale up the tool in its current form. The MFI expressed it would have considered expanding the tool to rural areas had it been easier to implement and the uptake been higher and proven effective in improving performance of both SMEs and the MFI. The pilot was only implemented in the metropolitan zones of Haiti. Since the data collection ran only for seven months and was also terminated due to lack of data on impacts – both caused by very low RBS uptake and completion - the pilot was unable to provide any clear results on RBS effectiveness. Therefore, Sogesol said that they could not determine whether the tool would be useful if scaled up. Whereas Sogesol learned about the risk factors affecting their clients, they did not draw lessons for what could be done by Sogesol to minimize the risks and related effects identified by their clients. They mentioned that the next iteration of the RBS or any other client risk assessment tool must also help them to identify areas for improvement for their clients, such that Sogesol can facilitate training or client education, provide advice to clients to help them improve their performance, or design products and services to help clients manage their risks better to improve their performance. It is therefore important for any scorecard developer to discuss with MFIs how to use data to effectively address or minimize the identified risks (factors) faced by their clients through their products and services. As discussed above, Sogesol provided many recommendations to improve RBS design to increase uptake and utility of the tool to MFIs and SMEs, and both SI and Sogesol learned valuable lessons on the process of rolling out RBS. Redesign of RBS based on the lessons from this pilot would require considerable financial resources to be deployed by the scorecard developers. Additionally, the scorecard developers would need to expend 19 resources to run small-scale pilots with a diverse set of MFIs prior to rollout of the revised tool to avoid a repeat of the issues encountered in this pilot. SI, without external funding to redesign the tool, is unlikely be engaged in RBS redesign or in carrying out further pilots of the tool. Even if external funding becomes available for SI to redesign the tool, MFIs including Sogesol may likely require co-funding from external sources to effectively pilot or take up the revised RBS tool before it is proven effective. It is clear from this pilot that scaling up of RBS by MFIs must be guided by proven results. The results must be validated through periodic evaluations to assess the effectiveness of the RBS for MFIs to make appropriate investment decisions to scale up. Therefore, external sources that consider supporting RBS redesign and pilots, or any scorecard for use by SMEs, should also insist on assessing the effectiveness of the tool and consider funding such assessments. Since the RBS is a learning tool to help improve SME performance, assessments should incorporate clear learning components and prepare SMEs for adaptation based on the learning. In other words, assessment designs should avoid any rigidity that could restrain the unique learning and adaptation feature of scorecards. 20 ANNEX I: AWARD MILES TONES The Stage 1 grant was executed through a Fixed Obligation Grant to Social Impact, Inc. (AID-OAA-F-14- 00017). Fixed Obligation Grants (since revised to Fixed Amount Awards) include a disbursement structure for project deliverables associated with distinct milestones. According to USAID’s Automated Directives System (ADS), Chapter 303: “Milestones are for a verifiable product, task, deliverable, or goal of the recipient.” A copy of the award milestones associated with the Stage 1 award are provided below for reference. Milestone Num. Approximate Time Deliverable 1 Start date + 2 weeks This milestone requirement is considered fulfilled when the following actions have occurred and the deliverables and/or associated narrative have been submitted to the AOR for review/concurrence: • Updated project implementation plan narrative accompanied by a Gantt chart. 2 Start date + 3 months This milestone requirement is considered fulfilled when the following actions have occurred and the deliverables and/or associated narrative have been submitted to the AOR for review/concurrence: o Risk-based scorecard implementation begun o Submit sample RBS • Implementation report to include: o Status of Sogesol monitoring officers trained in the use of RBS o Status of baseline data on Sogesol microenterprise clients 3 Start date + 6 months This milestone requirement is considered fulfilled when the following actions have occurred and the deliverables and/or associated narrative have been submitted to the AOR for review/concurrence: • Submit updated implementation report to include: o Number and location of Sogesol branches and loan officers offering and completing SI RBS 4 Start date + 10 months This milestone requirement is considered fulfilled when the following actions have occurred and the deliverables and/or associated narrative have been submitted to the AOR for review/concurrence: • Submit updated implementation report • Submit outline for project evaluation report 5 Start date + 14 months This milestone requirement is considered fulfilled when the following actions have occurred and the deliverables and/or associated narrative have been submitted to the AOR for review/concurrence: • Draft project evaluation results which addresses project outcomes with its stated goal to help 250 microenterprises increase their average sales or profitability at least two percentage points more than the growth rate of an experimental control group • Draft scaling strategy and next steps for RBS 6 Start date + 18 months This milestone requirement is considered fulfilled when the following actions have occurred and the deliverables and/or associated narrative have been submitted to the AOR for review/concurrence: • Final report o The report should summarize project results, assess the success to date, lessons learned and any challenges and successes that occurred during implementation and any scaling up strategy. 21 ANNEX II: SAMPLE SIZE CALCULAT IONS For this experimental design, we assume standard values for the parameters to calculate sample size: the confidence level of the test is set at α = 0.05, representing 95% confidence of the estimated impacts; a power of κ = 0.80 represents an 80% likelihood that we will correctly conclude an effect where it indeed exists. Average annual sales revenue data of Sogesol clients reported on their loan applications in 2015 was used to calculate sample size powered to detect 2% change in annual sales through use of the RBS. Our power calculations demonstrate that our test will be sufficiently powered at the standard 80% to detect a change of 0.28 standard deviations given a sample of at least 200 clients in the treatment group receiving monthly scorecards. This means that we would be able to detect an increase in annual sales growth rate of 0.76 percentage points, from 5.0% to 5.76%. Given our expected sample size of 200 and assuming a relatively extreme rate of 20% attrition that decreases the sample size to 160, we would be able to detect a minimum effect of 0.31 standard deviations, which is roughly equivalent to an increase in growth of 0.84%. The expected change for non-performing loan rate is not calculated due to a lack of data. Table 7 below shows the sample sizes required for various minimum effect sizes. TABLE 7 : SAMPLE SIZE REQUIRED FOR VARIOUS MINIMUM DETECTABLE EFFECT SIZES OF ANNUAL SALES GROWTH Minimum Detectable Effect (Standard Deviations) Minimum Detectable Effect (Sales Growth) Total Sample 0.50 1.35 percentage points 64 0.40 1.08 percentage points 100 0.31 0.84 percentage points 160 0.28 0.76 percentage points 200 By comparing the change in outcomes for the randomly assigned treatment and the control group, the effect of the RBS on Sogesol’s SME clients can be assessed. 22 ANNEX III: SCORECARDS COMPLETED BY SOGESOL BRANCHES FOR EACH ROUND ROUND 1 Of the 243 clients assigned to receive the scorecard at round 1, 38 clients (15%) completed part 1 of a scorecard, of which 28 completed both parts 1 & 2 (11%). Among the six branches, Carrefour had the lowest completion rate for clients completed parts 1 & 2 (1 client, 2%) and Bois Verna had the highest (10 clients, 24%). TABLE 8: IMPLEMENTATION OVERVIEW AT ROUND 1 (PRIOR TO OCTOBER 15) Branch Treatments Assigned Clients Contact Scorecard Part 1 Only Scorecards Part 1 & 2 No. No. % No. % No. % Bois Verna 42 22 52% 10 24% 10 24% Carrefour 42 5 12% 4 10% 1 2% Croix des Bouquets 43 23 53% 9 21% 5 12% Delmas, Cite Soleil 42 20 48% 4 10% 4 10% Petion Ville 42 33 79% 7 17% 4 10% Rue du Quai 42 28 67% 4 10% 4 10% Total 253 131 52% 38 15% 28 11% FIGURE 2: COMPARISON OF PERCENT AGE OF CLIENTS WHO C OMPLETED SCORECARDS PART 1 VERSUS PARTS 1 & 2 BY EACH BRANCH AT ROUND I 23 ROUND 2 Of the 243 clients assigned to receive the scorecard at round 2, 16 clients (6%) completed part 1 of a scorecard, of which 7 completed both parts 1 & 2 (3%). Among the six branches, three branches (Croix des Bouquet, Delmas, Cite Soleil, Rue du Quai) did not have any clients completing part 1 & 2 of the scorecards. Bois Verna had the highest completion rate with 5 clients completing scorecards (12%). TABLE 9: IMPLEMENTATION OVERVIEW AT ROUND 2 (OCTOBER 15TH - NOVEMBER 30) Branch Treatments Assigned Clients Contact Scorecard Part 1 Only Scorecards Part 1 & 2 No. No. % No. % No. % Bois Verna 42 15 36% 5 12% 5 12% Carrefour 42 13 31% 3 7% 1 2% Croix des Bouquets 43 7 16% 0 0% 0 0% Delmas, Cite Soleil 42 21 50% 6 14% 0 0% Petion Ville 42 1 2% 1 2% 1 2% Rue du Quai 42 3 7% 1 2% 0 0% Total 253 60 24% 16 6% 7 3% FIGURE 3 COMPARISON OF PERCENT AGE OF CLIENTS WHO C OMPLETED SCORECARDS PART 1 VERSUS PARTS 1 & 2 BY EACH BRANCH AT ROUND 2 24 ROUND 3 Of the 243 clients assigned to receive the scorecard at round 3, 25 clients (43%) completed part 1 of a scorecard, of which 7 completed both parts 1 & 2 (3%). Among the six branches, three branches (Carrefour, Croix des Bouquet, and Delmas, Cite Soleil) did not have any clients completing part 1 & 2 of the scorecards. Bois Verna had the highest completion rate with 3 clients completing scorecards (7%). TABLE 10: IMPLEMENTATION OVERVI EW AT ROUND 3 (DECEMBER 1- DECEMBER 30) Branch Treatments Assigned Clients Contact Scorecard Part 1 Only Scorecards Part 1 & 2 No. No. % No. % No. % Bois Verna 42 20 48% 12 29% 3 7% Carrefour 42 7 17% 2 5% 0 0% Croix des Bouquets 43 21 49% 0 0% 0 0% Delmas, Cite Soleil 42 1 2% 1 2% 0 0% Petion Ville 42 28 67% 5 12% 1 2% Rue du Quai 42 31 74% 5 12% 3 7% Total 253 108 43% 25 10% 7 3% FIGURE 4: COMPARISON OF PERCENT AGE OF CLIENTS WHO C OMPLETED SCORECARDS PART 1 VERSUS PARTS 1 & 2 BY EACH BRANCH AT ROUND 3 25 TABLE 11: PROPORTION OF PANEL CLIENTS IN THE DATASE T Branch Treatments Assigned No. of Scorecards Round 1,2 (Parts 1 & 2) No. of Scorecards Round 1,3 (P1 Only) No. of Scorecards Round 2,3 (P1 Only) No. of Scorecards Round 1,2,3 (P1 Only) Bois Verna 42 2 2 0 1 Carrefour 42 0 0 0 0 Croix des Bouquets 43 0 0 0 0 Delmas, Cite Soleil 42 0 0 0 0 Petion Ville 42 1 0 0 0 Rue du Quai 42 0 2 0 0 Total 253 3 4 0 1 26 ANNEX IV: EXAMPLE OF A COMPLET ED SCORECARD 27 ANNEX V: RBS COMPLETION, CLIENT T RACKER PROTOCOL CLIENT TRACKING FORM Instructions: Complete this tracking form for each client on a monthly basis. Branch Name Administrator Name Client ID Client Name ____________________ ____________________ ____________________ ____________________ Interview attempt (allow up to 2 attempts each month) 1 2 3 (optional) Date (dd/mm/yyyy) ____/____/_______ ____/____/_______ ____/____/_______ Time started Time finished Interview result code |___|___| |___|___| |___|___| Interview result codes 01 … Scorecard completed 02 … Client refused to participate 03 … Scorecard not started because client was busy - schedule call back 04 … Scorecard not started because client was not available - schedule call back 05 … Scorecard partially completed - client refused call back 06 … Scorecard partially completed - schedule call back 07 … No response - did not pick up phone 08 … Not reachable - call did not go through 09 … Wrong number 96 … Other (specify) ________________________________________________ CALLING SCRIPT Hello. My name is [Administrator Name] and I am calling you on behalf of Sogesol. We are administering a brief survey that will take about 10 minutes of your time. We appreciate your participation in this interview. Please be assured that the information you are providing us for will remain confidential and only be available to Sogesol and its partner organization in Washington DC. No personal identifying information will be released publicly. The purpose of this survey is to help you better understand the risk factors that are affecting your sales every month. For example, we will look at how the climate or political situation in Haiti can be affecting the performance of your sales. As such the survey will help you better understand your business to help you increase your sales at the end of the month. The information you provide us will NOT affect your loan with Sogesol. We will be calling you every month for about 8 months to gather this information on your sales. 28 Before we begin, please let me know if you have any questions I can help you answer now? Mr. Evans Baptiste can also answer your questions at any time. Please reach him at: (phone number). Do you agree to participate? [ ] Yes [ ] No If yes, administer the scorecard If no, thank the participant for their time. But before ending the call ask the participant if you could kindly ask their reason for refusing to participate in the scorecard. Fill in Interview result codes. Schedule call back if applicable. 29 ANNEX VI: GUIDING QUESTIONS FOR KEY INFORMANT INTERVIEWS WITH SOGE SOL SI's first round of KII questions for Sogesol (February 2017) 1. What are the major challenges faced by the monitoring officers during implementation of the scorecard? 2. What worked well during implementation of the scorecard? 3. What are the lessons learned during implementation of the scorecard? 4. What are the changes the loan officers would recommend if they were to use the scorecard again for a similar purpose? SI’s second round of KIIs with Sogesol (April 2017) 1. What did you learn from this collaboration with SI? Select all that apply. i. Setting up risk based scorecards ii. Administering risk based scorecards iii. Uses of risk based scorecard for making business decision iv. Evaluating for client outcomes v. None of the above 2. If none of the above, did you learn anything from this project? Please describe your learning. 3. In this project, you collaborated with SI. Would you enter such a collaboration in future with SI or anyone else like SI? Yes/No i. If yes, what changes would you suggest to make it work better? a. More and better trainings b. More and better training materials and instructions to reference c. Easier to manage scorecards (i.e. less factors) d. Other_________________ ii. If No, why and what changes would you suggest to make it work? iii. Would you collaborate with a non-SI type institution (give some examples)? 4. Are you planning on scaling up the implementation of this scorecard (i.e. expanding it more broadly)? i. If yes, what is your scaling strategy? Who are the new stakeholders? How did you bring in the new stakeholders to support the scale up? What is the timeline? ii. If you are not going to scale up, why? 5. Do you believe that this scorecard, if expanded to all of Sogesol (scaled up), would help: i. Increase your client base 30 ii. Improve your competitiveness in the market iii. Improve your ability to screen applicants better iv. Reduce late payments and defaults v. Increase sales of your clients vi. None of the above vii. Other ______________ 6. If none of the above: i. Why was the tool not useful to scale up? ii. What are the relevant lessons you can give for other institutions like you wanting to implement risk based scorecards? 7. Were external factors (i.e. hurricane Matthew, political situation in Haiti etc.) a limitation to carry out the project? If yes, explain how. 8. In the absence of this project, would you have tried using a risk based scorecard like this at all? Yes/No. Please state reasons for the response. SI’s follow up Questions based on Sogesol’s responses for the KIIs above (May 2017): 5. What are the reasons why the implementation of the risk scoring project with “Entrepreneur Finance Lab” could not be completed? 6. What was the greatest motivation for your loan officers (SUPADs) to complete the calls with customers to have the RBS filled out? Why were the RBS completion rates low? 7. What kind of follow-up (e.g. second training) would have been effective after the first (March 2016) training by SI?