Tableaux de bord axés sur les risques pour les microentreprises : Rapport final du projet pilote en Haïti

RISK - BASED SCORECARD S FOR MICROENTERPRISES: P ILOT IN HAITI FINAL REPORT MAY 4, 2017 This report was produced for the United States Agency for International Development (USAID) by Social Impact, Inc. It is made possible by the support of the American people through USAID. The contents of this report are the sole responsibility of Social Im pact, Inc. and do not necessarily reflect the views of USAID or the United States Government. Grant No. AID - OAA - F - 14 - 00017 June 30, 2017 ii CONTENTS Acknowledgements iii Acronyms iv Executive Summary v Introduction 1 The Pilot 1 The Risk - Based Scorecard 3 Structure of RBS 4 Competitive Landscape for RBS 4 RBS Rollout in Haiti 5 RBS Training in Haiti 5 Tracking RBS Implementation, Data Capture and Storage 6 Assessment of RBS Effectiveness in Haiti 7 Findings from RBS Rollout and Assessment in Haiti 10 RBS Take - up and Completion Rates 10 Risk Factors Affecting SMEs in Haiti 12 Challenges Faced and Managed during the Pilot 14 Lessons Learned 14 Recommendations 17 Scaling Plan and Next Steps 18 Annex I: Award Milestones 20 Annex II: Sample Size Calculations 21 Annex III: Scorecards Completed by Sogesol Branches for Each Round 22 Annex IV: Example of a Completed Scorecard 26 Annex V: RBS Completion, Client Tracker Protocol 27 Annex VI: Guiding Questions for Key Informant Interviews with Sogesol 29 iii A CKNOWLEDGEMENTS Social Impact is highly appreciative of Société Générale Haïtienne de Solidarité ( Sogesol ) in Haiti for its partnership in the pilot of the R isk - B ased S corecard and for all the support provided to administer the scorecard to small and micro enterprises. W e would like to e specially thank Sogesol’s Director, Mr. Evans Baptist e, for all the cooperation and guidance he provided us throughout t he pilot, as well as the monitoring officers for their contributions to this study through gathering data and sharing their experiences using the tool , which will inform future tool improvements. We also recognize and thank D r. David Apgar for developing t he Risk - Based Scorecard and for identifying and brin g ing in Sogesol to participate in the pilot study as S ocial Impact’s partner and subcontractor . Lastly, w e extend our gratitude to the Global Development Lab and Development Innovations Ventures at USAID for the grant, and especially to the A O R for the guidance and support throughout this pilot study. iv A CRONYMS AOR Agreement Officer Representative DIV Development Innovation Ventures EFL Entrepreneur Finance Lab MFI Microfinance Institutions MIS MLE s Management Information System Medium and Large Enterprises RBS Risk - Based Scorecard RCT Randomized Control Trial SOGESOL Société Générale Haïtienne de Solidarité SI Social Impact SMEs Small and M i c r o Enterprises v EXECUTIVE SUMMARY In many developing countries, a large learning gap exist s for small and m i c r o enterprises (SME s ) and the microfinance institutions (MFI s ) that help address their financial constraints. SMEs face challenges in identifying and assessing the drivers of successful businesses, and in experimenting with such new approaches as financial planning or productivity - enhancing activities. Similarly, given the limited availability of user - friendly tools, MFIs grapple with le arning challenges in effectively assessing SME risks in order to provide services to these enterprises. T o address these challenges , Social Impact , Inc . (SI) developed a simple - to - use Risk - Based Scorec ard (RBS) . The scorecard is a continuous learning tool to help both MFIs and SMEs learn collaboratively and adapt their business strategies to improve their performance. With the tool, SMEs c an identify crucial risk factors that affect their businesses . They can estimate the effects of such factors on their sales at the beg i nning of a given period, and then verify , at the end of the period, whether the estimates held true . By examining the gap between the estimates and realized outcomes, SMEs c an learn over time to predict their risks more accurately and ad o pt strat egies to improve their performance. MFIs can also learn to better assess risks facing their clients, and feed this knowledge into reducing non - performing loans. With a grant from Development Innovatio n Ventures (DIV) of USAID, SI conducted a pilot study with Société Générale Haïtienne de Solidarité ( Sogesol ) , a regulated MFI serving SMEs in Haiti, to roll out the RBS and test its utility in improving performance, both for Sogesol and among SMEs. The pi lot lasted 18 months , and was completed in June 2017. To roll out the RBS, SI developed training materials and used them to train Sogesol’s monitoring officers . To assess the effectiveness o f the RBS to improve performance, SI designe d a randomized control trial (RCT) . In the RCT, 25 3 active Sogesol clients were randomly allocated to a treatmen t group to receive the RBS , while 25 2 clients were randomly allocated to a control group that would not receive the RBS until the pilot en ded. S I trained Sogesol officers to use the RBS to gather data from its treatment clients, share it with SI for analys is, and document both Sogesol and client (SME) experiences using the RBS . After the training, Sogesol rolled out the RBS and conducted three rou nds of data collection from the treatment group over a period of seven months. A low uptake and completion of the RBS among SMEs was observed; therefore, the assessment of the effectiveness of the RBS in improving performance was incomplete. Valuable l ess ons for understand ing the potential of the RBS as a learning tool for MFIs and SMEs emerged from the pilot. The MFI was able to learn about the risks facing SMEs , and to a smaller extent , the effects of such risks on SME sales. S ome SME clients used the opportunity to express their appreciation to Sogesol or to communicate their requests and issues related to their relationship with Sogesol . SI and Sogesol learned lessons for improving the RBS design and implementation to increase its upt ake and utility as a collaborative learning tool. Additionally, Sogesol officials verbally expressed interest in trying an improved version of the RBS in the future, and scaling it u p to more branches if proven ef f e ctive. 1 INTRODUCTI ON Financial constraints often limit growth of small and m i c r o enterprises (SME s ) in developing countries, and there is great demand from SMEs for financial services to relax the se constraints. Many institutions such as microfinance institutions (MFIs) indeed now strive to meet the demand. However , a large gap persists between demand and supply , due in part to learning challenges . There are limited opportunities for S MEs and for MFI s to learn ways to improve and adapt their business practices, and this problem is especially dire in countries with poor enabling environment s for SME growth. On the one hand, SMEs face learning challenges in identifying and assessing the drivers of succe ssful business es. They have less opportunity to experiment with new approache s compared to larger businesses that can afford the costs of such learning processes as financial planning or analysis activities that improve their productivity. On the other han d, MFIs also grapple with learning challenges , since few tools are available to help them adopt strategies for effectively assess ing their target SMEs and providing the demanded services. Therefore, Social Impact, Inc . (SI) 1 developed a simple, user - frie ndly Risk - Based Scorecard (RBS ) designed to (1) build the learning capacity of SMEs regarding their business risks , and (2) equip MFIs with a risk assessment tool to help them develop business strategies that serve their clients well. The RBS is essentiall y a learning tool intended to help both MFIs and SMEs assess risks and take appropriate and timely actions to increase productivity. The tool can be periodically administered to SME clients by loan officers, enabling collaborative and continuous learning a nd adaptation . To test the value of the RBS as a learning and ad ap tation tool to imp r ove SME productivity and MFI efficiency, USAID awarded a Development Innovation Ventures (DIV) Stage 1 grant to SI to conduct a pilot study with a MFI serving SMEs. SI carried out the pi lot study with Sogesol ( Box 1 ), a regulated MFI in Haiti that has been serving SMEs for two decades . The pilot study with Sogesol lasted 18 months , and was completed in June 2017. 2 This final report discusses the RBS pilot and l essons learned by the SMEs, by the MF I and by SI on the utility of the RBS . Additionally, this report provides recommendations to improve the tool for future use. THE PILOT USAID awarde d the grant to SI in May 2014, and SI began implementation in August 2014 with Findev, a n MFI in Aze rbaijan. However , Findev pull ed out of the p ilot in early 2015 due to change s in its mis s i on , objectives and target clients. Therefore, SI sought another MFI to pil ot the RBS and discussed collaborat ion with eight MFIs in Africa, Asia and Latin America. Two of those MFIs reported using some type of score card – such as a poverty scorecard or credit scoring – to assess their client s . S ome of them were reluctant to test new tools for fear of increasing their operating costs and disrupting their regular operations with additional burden o n their staff . SI’s offer to share in some of the costs, using SI’s grant from DIV, was not considered adequate by these MFIs to pilot the tool , and most of th em therefore declined SI’s offer to particpate in the pilot. In January 2016, SI was able to resume implementation of the pilot after securing another partner, Sogesol, a regulated MFI in Haiti. Sogesol was int e rested in testing the RBS , as they had previo usly worked with the Entrepreneur ial Finance Lab (EFL) to implement a risk assessment score card ; this scorecard was intended to help Sogesol understand risks faced by its clients to reduce its risky loan portfolio. The scorecard was based on psychometric models that assessed the integrity, business management and financial skills of only the small 1 Social Impact is a global development management consulting firm based in Virginia, USA with a mission to help global development organization and programs be more effective at improving people’s lives through monitoring, evaluation strategic planning and capacity building services. 2 SI developed the to ol with its own resources. USAID /DIV grant covered the data collection and analysis costs. Sogesol also shared in some of the costs by providing training space, access to its clients, use of its equipment and a management staff’s time t o select and monitor the data collectors. SI, using the grant, paid for the labor and travel/phone expenses of the data collectors. 2 enterprise borrowers with loans above 50,000 HTG (US$ 720) . Within the first few months of impleme ntation, it was evident that the pilot was slow at obtaining results , and the model was not proven to be robust enough for the type of clients with whom Sogesol engages . More specifically, Sogesol did not believe that the statistical model piloted by EFL a dded value to the risk assessment model Sogesol had tested and applied for almost a decade . Indeed, Sogesol tried to combine both - EFL and Sogesol - models to develop a better tool, but the results were not satisfactory. Therefore, Sogesol terminated the use of the score card in 2015 after four years of pursuing its implementation. Nonetheless , Sogesol continued to search for simple tools to conduct risk assessments for both small and micro enterprise clients. As such , when SI approached Sogesol in late 2015 and demonstrated the RBS to its senior management staff, the MFI was interested in collaborating with SI to pilot it. The purpose of SI’s pilot study with Sogesol involved testing the val idity of the RBS as a co llaborative learning and adaptation tool for SMEs and MFIs. Specifically, the objectives of the pilot study included the following: Objective 1: Train Sogesol on the RBS and roll out the RBS; Objective 2: Through an assessment, test whether clients’ learning through the RBS could help increase their sales by at least two percentage points in a year, and whether through such increased sales, the MFI’s non - performing loans could be reduced by 0.2 percentage points; and Objective 3 : Document lessons learned from the pilot to help with future applications of the RBS by MF Is for their SME clients. The pilot with Sogesol began in early 2016 . SI, in consultation with its Agreement Officer Representative ( A O R ) at USAID, established six milestones to track the award progres s ion over a period of 18 months from January 2016 to June 2017. The inception report formed the first milestone. The other five focused on the roll out and assessment of the RBS tool . These deliverables included training, assessment design and sampling and d ata collection during milest one periods two, three, and four . A ssessment data analysis and report writing formed milestone five, and milestone six included a final report on lessons learn ed . By late 2016, the SI team realized that uptake and completion rates of the RBS were too low to fully evaluate the RBS tool for its impact on SME sales and on the MF I’s portfolio risk. Therefore, O bjective 2 could not be fully achieved and milestone five could not be met. 3 However, valuable lessons eme rged to help improve RBS utility for SME s and MFIs in the future , as discussed later in this report. 3 Annex I outlines these milestones in further detail. Box. 1. About Société Générale Haïtienne de Solidarité (SOGESOL) Since November 2000, Sogesol has worked toward its miss ion to promote Haitian entrepreneurship by adapting traditional bank services to the needs of SMEs. The majority of its clients are small and micro business owners and agricultural producers, of which many are women. As of December 2015, Sogesol was servi ng nearly 35,017 borrowers, with an average loan size of $502 and total outstanding loans equal to US$35.3 million. Sogesol has 16 branches, of which six are in metropolitan zones and 10 in rural areas. Sogesol is an independent, commercial and a regulate d microlending institution that uses a “service company” model. Under this model, Sogesol provides loan origination and credit administration services to SOGEBANK. The loans are booked at SOGEBANK, but Sogesol has primary responsibility for promoting, eval uating, approving, tracking and collecting them. Sogesol’s shareholders include SOGEBANK (50.18%) and ACCION International (9.12%), with the remaining share held by individuals. References: ACCION: Sogesol. https://www.accion.org/our - impact/sogesol Groupe SOGEBANK: Sogesol. https://www.sogebank.com/qui - sommes - nous/sogesol/ 3 THE RISK - BASED SCORE CARD Large businesses can often afford to invest in le a r ning and analytical tools to improve their performance. While SMEs also need such learning tools, they cannot afford to develop or buy them , nor are the available tools tailored for SME needs in terms of u tility and simplicity. To address this challenge, in 2014 Dr. David Apgar ( Senior D irector at SI during the time) started developing the RBS to provide SMEs with a simple and easy - to - use tool for risk assessments. The RBS was piloted in 2016 with SMEs thro ugh Sogesol, a MFI serving SMEs in Haiti . The RBS is a spin - off of a simple Goals Screen algorithm 4 , also developed by Dr. Apgar (See Box 2). Goals Screen was developed to help medium and large entrepreneurs (MLEs) and business executives find the best ways to reach their business goals. Dr. Apgar has been testing and refining the Goals Screen for over a decade in many countries with MLEs and b usiness executives. Based on the se experiences, Dr. Apgar claims that , after six sessions, most Goal s Screen users significantly change their approach to reaching their goal and predict results more accurately . By the tenth session, most users report signi ficant improvements in sales or cash flow growth. The well - tested and refined Goals Screen provided a strong foundation for RBS development. However, the tool needed to be simplified and made more affordable for use by SMEs and by MFIs that are typically smaller than commercial banks and financial firms. As m any MFIs are now familiar with scorecards , SI developed the RBS as a simple scorecard. The simplicity is such that MFIs need not make any major management information system (MIS) modifications to ro ll out the tool, nor do they require extensive training. Rather, MFIs’ loan officers can easily embed the tool into their regular client visits and can adm in i ster it using pen and paper. As RBS requires only a handful of rules and requirements to implement , loan officers need only two days of training before rolling it out. SME owners are expected to learn the RBS thoroughly over the course of 12 monthly sessions. To be effective in improving performance, users must occasionally be creative in identifying n ew risk factors, and that kind of creativity cannot be taught through 4 See http://www.goalscreen.com for more deta ils on Goals Screen Application. Box. 2. Goal Screen Methodology The Goals Screen application is an Assumption - Based Metrics (ABM) tool that focuses on the mo st important factors. This tool stands in contrast to commonly - used Balanced Scorecards, which require voluminous data, a challenge in many developing countries for small businesses and small financial firms. The ABM begins with business owners’ logic and intuition about what most drives success, and then tests those intuitions, as described in these three simple steps: 1. Business owner lists all major assumptions the business needs to forecast performance against the business objective. This list include s both areas within business owner’s control and areas that are outside of it. 2. Business owner identifies worst - case scenarios for each of those assumptions, then estimates the worst - case outcome for the factor if that scenario were to occur and the wors t - case impact it would have on the ultimate business objective. 3. Business owner ranks assumptions by worst - case impact. The owner can typically notice a large drop - off in impact after four to five factors, and thereby identifies the priority factors and the priority metrics in the scorecard. The resulting metrics from the ABM are expected to help the business owner discern whether the business is progressing towards the objective. These metrics also force the owner to test whether the assumptions are jus tified. By repeatedly using the tool, the business owner is expected to learn to refine the assumptions regarding which factors are central to achieving business objectives. See http://www.goalscreen.com for more details on Goals Screen Application. 4 the RBS . However, the RBS capture s the mechanics of learning from results in a simple, intuitive process using a bare - bones trial - and - error methodology . STRUCTURE OF RBS The RBS contains two parts , to be administered over a defined period such as a month, quarter, or year. At the beginning of a period (month/quarter/year), P art 1 of the tool requires the SME client to identify a maximum of four risk factors it faces ( e.g. weather , customer visits, raw - material prices) and provide an estimate of expected outcome (e.g. sales, profits) if the worst - case scenario was realized for each risk factor. For instance, political unrest could be an identified factor/threat, and as such the cli ent could estimate t hat their sales would reach 35, 000 HTG (~$510) under the current political situation. These scenarios help users estimate the outcome they should expect for each factor . At the end of the specified period, P art 2 of the tool requires th e client to report actual sales and accuracy for each risk factor against the predicted outcome. Once all the information is entered , the RBS automatically calculates (1) the performance gap between estimated and actual outcome (e . g. sales) for each facto r, and (2 ) the performance gap not explaine d by the factors identified in P art 1. The difference between an actual and estimated outcome (e . g. sales) represents the unexplained performance gap and demonstrates how realistic the underlying risk factors and business concepts are. By reducing the unexplained performance gap, the client can learn over time to narrow the actual outcome gap. If a large outcome gap remains after the client has minimized the gaps for identified risk factors, then the client is ex pected to realize that he/she must identify another important risk factor that may be missing. The cycle can be repeated for any number of periods . In each period, t he client is given the opportunit y to revise the assumptions in P art 1 based on the learning from the previous period. Over time, the unexplained performance gap is expected to decrease, signifying the client is better able to identify risk factors and how the factors constrain sales , and take appropriate actions to manage the risks. See Annex III for an example of a completed scorecard. COMPETITIVE LANDSCAP E FOR RBS Grameen Foundation’s poverty scorecards and the SEEP tool are often mentioned as alternatives to performance scorecards for SMEs. However, neither turns out to be useful for diagnosing business problems based on monthly or quarterly results. Grameen’s scorecards assess welfare rather than business problems , and while the SEEP FRAME tool is optimized for MFIs, it is too technical for microenterprises. There is also little evid ence of an impact on productivity growth from the standard scorecards. This is because most scorecards assume one risk factor and compare results of that factor with the target. In contrast, the RBS includes alternative scenarios , as well as targets for e ach key factor. Th us, these standard scorecards p rovide little guidance when the result for a risk factor is far from its target ; it is difficult to discern how much of the outcome is due to that risk factor , and how much is due to overaggressive or undera ggressive targets for the outcome or to missing risk factors. Such occurrences effectively break the learning process required of scorecards to be effective to improve performance . The RBS also provides SMEs with a quick and simple learning opportunity on risks facing them. For example, based on the ident i fied risk factors and scenarios, a large une xplained gap in perfomance implies that the factors driving results need more attention , and t hat t he assumptions behind the alternative scenarios need revisions. Small unexplained gaps imply that the business concept works , but the outcome estimates ma y need some small adjustments. Thus, when updated with each set of results, RBS has the potential to support sustained improvements in busines s growth . 5 Th is is because every set of results leads not only to a 5 Apgar, David. 2008. “Relevance: Hitting Your Goals by Knowing What Matters.” 5 reconsideration of priorities for SMEs, but also to a change in RBS factors, targets or scenarios. Over time, therefore, results increase the predictive power of the simplified models embodied in those factors, targets, and scenarios. In short, this learning process has potential as a substitute fo r the more formalized learning and analytical processes that large enterprises can afford. Some MFIs have tried to build SMEs’ planning capacity by providing them with simple income statements and balance sheets, often prepared by loan officers during weekly or monthly visits using pen and paper. The problem with financial reports is that they focus on such outcomes as price, volume, direct costs and indirect c osts , rather than causal factors that drive those outcomes. In randomized trials in the Dominican Republic, standard fundamentals - based accounting training for microentrepreneurs is shown to have no effect in improving their sales volume and prices receive d, and in reducing costs. 6 Indeed, periodic business advice to microenterprises would represent a relevant alternative to the RBS. However, the cost - effectiveness of such advice is hard to benchmark because of the wide variety of forms of advice. For example, recent research finds powerful effects of consulting services offered to Mexican businesses – including microenterprises – on the number of employees and total factor productivity. 7 Since the interventions tested in th e randomized control trial by Bruhn, Karlan, and Schoar cost about $12,000 per firm over one year, however, they are hard to compare with an intervention that costs $120, or 1% of that amount. The consulting services tested in the trial are estimated to have provided benefits equal to $36,000 – three times the investment. However, such return s are shown to be hard to achieve with smaller firms on significantly smaller outlays. A benefit of RBS over such business advising services is that it systemati zes one sort of business advice: nam ely , the use of results to fuel changes to major components of a business model and to the tacit or explicit business plan underlying it. While the RBS does not cover the range of forms of business advice now available, it does have the potential to contri bute to a standard of advice on inferential management logic , because they are to an extent replicable. Indeed, the RBS provide s a simple way to interpret outcomes as a product of both effort and planning for the sake of adapting tactics to improve perform ance . The scorecards can be revised to increase their predictive power without extra data collection or special technology. RBS ROLLOUT IN HAITI Prior to the roll out, during a trip to Haiti in December 201 5 , SI assessed the inter est a nd capacity of Sogesol to use the RBS. Based on the initial discus s ions, the roll out of the RBS was achieved through a subcontracting agreement bet ween SI and Sogesol, signed in early February 2016. Under the subcontract, SI trained Sogesol on the RBS, designed an assessment, analyzed the data on RBS effectiven e ss and prepared a report to submit t o USAID and share with Sogesol. For its part, Sogesol administered the RBS , using its monitoring officers as data collectors and monitoring them for quality and compliance . Sogesol shared selected information from its client database to design an assessment, periodically collected and sent the data to SI for analysis, and documented the experiences and challe nges experienced by both Sogesol staff and by SMEs in using the RBS. RBS TRAINING IN HAITI Based on the initial assessment of Sogesol ’s capacity to roll out the RBS , a s well as further discussions with Sogesol, SI developed training materials in the form o f PowerPoint presentations and a Microsoft Excel version of the RBS . Sogesol’s director , Mr. Evans Baptiste, provided fe e dback on all the materials. In March 2016, SI delivered a one - day training in French , followed by one - on - one follow - up sessions, or “office hours,” 6 Drexler, Alejandro, Greg Fischer, and Antoinette Schoar. 2010. “ Keeping it Simple: Financial Literacy and Rules of Thumb.” < http://dev3.cepr.org/meets/wkcn/7/784/papers/FischerFinal.pdf >. 7 Bruhn, Miriam, Dean Karlan, and Antoinette Schoar. 2013. “Impact of Consulting Services on Small and Medium Enterp rises: Evidence from a Randomized Trial in Mexico.” World Bank Policy Research Working Paper. 6 for two days in Haiti. Training participants included six Sogesol monitoring officers representing six of Sogesol’s metropolitan branches . The t raining focused on the structure of the RBS and its administration to client s, as well as data c apture and reporting. Sogesol’s Director was present throughout the training to act as a co - facilitator , and was available during the follow - up office hours to provide guidance and any needed clarification to the monitoring officers. SI held the training in one of Sogesol’s training rooms in Petion Ville , Sogesol’s headquarter s, and delivered it in a plenary session style using PowerPoint and supporting materials such as handouts and interactive exercises. The first part of the training focused on a thorou gh introduction on the purpose, use, advantages and rationale of the RBS. The SI trainer facilitated this section using flip charting techniques to explain the possible risk factors affecting SMEs. To fully engage the monitoring officers , SI used interacti ve training methods such as asking participants to provide examples of rea l - life risk factors affecting their clients. The second part of the training put the RBS into practice by centering on an illustrative client (in the example, a grocery store owner ). Through exercises, participants experimented with adjusting the risk factors affecting the client’s sales to understand the variance in sales between actual sales and sales project ed at the beginning of the month. The monitoring officers participated in a role - play exercise in which one participant acted as the monitoring officer calling the client to admi ni ster the RBS , and the other as the client receiving the call. Sogesol considered it a helpful exercise for equipping the monitoring officers with a b etter understanding of how the RBS implementation would work . The SI trainer wrapped up the training by explaining the deliverables that would be due to SI moving forward. After the training, the SI trainer remained in Haiti for two days in order to be rea dily available for any questions from the monitoring officers. As t he monitoring officers were not located at the Petion Ville branch , it was not practical for them to follow - up in person with the SI trainer during those days. (Later, the officers brought questions directly to Sogesol’s Director , who then relayed their querie s to SI. ) Given that the SI trainer was not engaged with monitoring officers during these two days , s he used the time to work directly with Sogesol’s Director to develop tools and temp lates for use by the monitoring officers in tracking and compiling data. To minimize costs and facilitate use , Sogesol and SI agreed on bi - monthly administration of the RBS over a seven - month data collection period . I deally, the RBS must be administered in additional rounds over a longer timeline to track whether clients improve their learning in accurately predict ing risk factors and increasing their sales. However , given the SME client base and the time limitation of one year for the pilot, Sogesol sug gested limiting the administration to approximately three periods in a year. D etails of the rollout are discussed below. Sogesol also suggested phone calls from monitoring oficers as a mode for contacting SMEs and administering the RBS to reduce costs . For the EF L scorecard, Sogesol used a web based a pplication with an initial face - to - face meeting for some new clients. T RACKING RBS I MPLEMENTATION , DATA CAPTURE AND STORAGE RBS can be adminis t ered easily with pen and paper. However , Sogesol wanted to use electronic tablets for data collection and establish a trackable system such t hat data can easily be captured, stored, and incorpor ated into their loan appraisals if needed. Therefor e SI , in collaboration with Sogesol , designed a simple and trackable syste m to implement RBS with the following elements: • Scripts for the phone calls made by monitoring officers to their clients to expain the RBS and help clients complete it. • Unique RBS for each client in Excel format , uploaded on Sogesol’s electronic tablets t o be completed and updated during each check - in by the monitoring officers. T he Excel spreadsheets allow Sogesol to easily merge the scorecards with the MFI ’ s management information system. • Simple naming conventions c aptured in each client’s RBS , save d i n an Excel spreadsheet for each client and transfer red to a Dropbox account that was shared with SI. The monitoring officers were familiar with Dropbox and did not reqiure additional training on it. Nonetheless , SI provided guidance 7 on how to upload the Exce l Spreadsheets to Dropbox including the proper use of naming conventions. • A client tracking protocol for the monitoring officers to record their challenges in administering the RBS at each check - in with their clients. The tracker is also expected to help the monitoring officers understand low or sporadic response rates and take actions to improve resp onse rates (s ee Annex V for the client tracker protocol). SI also provided the monitoring officers at Sogesol with a user - friendly guide in French with step - by - step instructions to facilitate their self learning. At the end of each period, the SI team retrieved the data in Dropbox for each of the six branches and for each SME client. SI’s review process included cleaning the data us ing SI’s unique data quality toolkit and conducting follow - up calls with Sogesol’s Director to clarify any ambiguities in the data and to provide guidance on how to increase client response rates and store the RBS. The efforts of Sogesol to increase client responses that were recorded in the client tracker ( shown in Annex V) were also studied by SI to draw lessons for improving the RBS in the future. A SSESSMENT OF RBS E FFECTIVENESS IN HAIT I T he second objective of the pilot involved an assessment to test whether clients’ learning could help increase their sales by at least two percentage points in a year; and whether through such increased sales, the MFI could reduce its non - performing loans by 0.2 percentage points. To conduct the assessment, SI design ed a randomized control trial (RCT) involving a treatment group that received the RBS and a control group that did not receive the RBS during the pilot. The RCT was designed to test the hypothesis that use of the RBS would increase the treatment group’s average sales by at least two percentage points more than the control group, and decrease non - performing loans among the treatment group by 0.2 percentage points relative to the control group, contingent on increase in sales . SI’s sample size calculations, conducted prior to the pilot, required approximately 500 clients, equally divided between treatment and control groups to test the hypotheses ( see Annex II for details) . SI determined the primary sampling unit to be a Sogesol client who satisfies the following criteria: (i) is active : the client has a current loan o utstanding with Sogesol; (ii) is a small business ow ner and not a salaried employee; and (iii) has not completed paying back their loan. To roll out the RBS quickly and start data collection for the assessment immediately after the training, SI sought to establish a preliminary sample prior to the training. However, the SI team faced a delay in obtaining a simple number count of clients at each bank branch that me t the sel ecti on criteria . Therefore, SI had to wait until March 2016 during its visit to Haiti for the training to work on the sampling . During the trip, SI coordinated with Soge s ol’s Director to start the sampling process. After some additional delays due to challenges in retr ie ving data from the M IS and preparing a codebook defining each variable in the database, SI finalize d a sample in July 2016. Starting with the list of Sogesol clients, SI first dropped all clients who did not fit the sample criteria. The n, SI used a simple random sampling procedure to obtain a list of 505 clients that satisfied the criteria. Table 1 displays the number of clients chos en for the pilot by branch and treatment status. 8 Table 1 : SME SAMPLE SIZE SI gathered baseline data on the 505 clients chosen for the assessment from the MIS database . Using the MIS data instead of a client survey helped minimize data collection fatigue and costs . This was possible because the sample was comprised of active clients who had already filled out loan applications and from whom Sogesol regularly gather ed financial information . The MIS database provided demographic information on clients , as well as data on length of client relations hip, type of business operated, monthly sales data, loans outstanding, loan type and size, number of loans received by a client, current loan type , and repayment record. The baseline characteristics of the treatment and control groups for the key features are shown in Table 2 . It is to be noted that, across the seven key variables, only gender of owners differed si gnificantly between the treatment and control groups . SI decided this difference could be managed during the analysis phase through disaggregation of results by gender or through controlling for gender in any regressions measuring treatment effect. Therefore, the balance between the two groups was not an issu e affecting the internal validity of the experiment . 8 T ABLE 2 : SME S AMPLE C HARACTERISTICS AT B ASELINE Variable Type Variable Control Treatment p - value Continuous Average Age in years 39.0 (0.56) 39.1 (0.51) 0.863 Loan amount in HTG 61,927 (178,337) 52,550 (148,771) 0.522 Outstanding balance on loan in HTG 40,488 (123,821) 32,086 (63.467) 0.338 Number of previous loans with current loan type 3.2 (3.6) 3.2 (2.9) 0.957 Monthly Sales in HTG 247,885 (682,520) 269,298 (1,117,358) 0.795 Nominal Owner is Female (%) 54.0% 61.5% 0.087* 8 Random assignment in a RCT is expected to balance all baseline characteristics between the treatment and control groups (thus eliminating selection bias). But, it is possible, particularly with small samples, that random assignment can, by chance, yield unbalanced groups. Therefore, balance check between treatment and control groups along key baseline characteristics could help establish validity of counterfactua l. Branch name Number of Active Clients Served Monitoring Officer Control Sampled Clients Treatment Total Petion - Ville 1,674 Monitoring Officer I 42 42 84 Carrefour 2,058 Monitoring Officer II 42 42 84 Bois - Verna 931 Monitoring Officer III 42 42 84 Rue du Quai 914 Monitoring Officer IV 42 42 84 Croix - des - Bouquets 712 Monitoring Officer V 42 43 85 Cite Soleil & Delmas 1,432 Monitoring Officer VI 42 42 84 Total 7,721 6 252 253 505 9 Loan was taken in 2016 (% with ‘Yes’) 40.5% 44.8% 0.322 Source: Sogesol MIS data, July 2016. Note: Standard deviations given in parentheses. For continuous variables, a two independent samples t - test (2 tailed) was conducted to test the difference between treatment and control groups. For nominal variables, Chi - square test was conducted. Under p values, ***, **, and * represent statistical significance at 1%, 5% and 10%, respectively. To test the hypotheses under O bjective 2, SI planned to use the following data: (i) RBS data from the treatment group on risk factors and sales , gathered over the course of three data collection periods, and (ii) data from Sogesol’s MIS on repayment for both treatment and control groups and sales f or the control group at the end of the third period. The results from the assessment, including RBS uptake , are discuss ed below. In add ition to quantitative data, SI used key informant interviews (KIIs) with Sogesol ’s Director to gather qualitative data on the strengths and weaknesses of the RBS, challenges in rolling out the RBS and recommendations for improving the tool to increase its utility and potential for scale - up. The KIIs were conducted by phone and by email prior to the last phase of the pilot and immediately after termination of the pilot ( see Annex VI for the KII protocols). The above assessment meth odology involves some risks that may limit lessons learned and recommendations to be drawn from the pilot . The major risks include the following: (1) The low uptake and compliance by both treatment and control group clients limits the ability to assess the effectiveness of the RBS . (2) The short period of seven months for rolling out the RBS and collecting data may not have been a sufficient time period for client learning. (3) The prototype nature of the tool, which may need adaptation and redesign based on learnin g at each round , can affect the comparability of client data from one period to another. (4) The qualitative data are limited to information obtained from the monitoring officers through Sogesol’s D irector , as well as the Director’s own perspectives . No client - level qualitative data were gathered by SI for the assessment to provide client perspectives. (5) Given the short period of the pilot a nd funding constraints, no cost - benefit analysis was conducted to assess the value for money of the tool for scale - u p. Despite the se limitations, the pilot offer s valuable lessons on areas for improvement in the design and process of RBS implementation. 10 FINDINGS FROM RBS RO LLOUT AND ASSESSMENT IN HAITI To examine effectiveness of the RBS in improving performance of SMEs and the MFI , data w ere gathered in three rounds in 2016 from 253 randomly sampled clients in the treatment group, as detailed below: • Round 1 contained scorecards from clients with which Part 1 was initiated prior to October 15 , 2016. • Round 2 contained scorecards from clients with which Part 1 was initiated between October 1 5 and November 30, 2016. • Round 3 contained scorecards from clients with which Part 1 was initiated between December 1 and December 31, 2016. 9 RBS T AKE - UP AND COMPLE TION RATES As shown in Table 3 , during Round 1, it was not possible to contact all of the treatment sample , despite the continuous follow - up by phone by Sogesol . The average contact rate in Round 1 across all six branches was 52 percent, and the response rate for clients completing both parts 1 and 2 was 11 percent . Both the contact and response rates decreased further in the subsequent two rounds of data collection. In Round 2, only 24 percent of the treatment sample could be contacted , and only three percent completed both parts 1 and 2. In Round 3, 4 3 percent of the treatment sample was contacted , and only three percent completed parts 1 and 2 of the RBS. T ABLE 3 : RBS T AKE U P AND C OMPLETION AMONG THE T REATMENT S AMPLE , BY R OUNDS Rounds Assigned to treatment No. Clients Contacted Scorecard Completion Part 1 Only Scorecard Completion Parts 1 & 2 No. % (to assigned) No. % (to assigned) No. % (to assigned) Round 1 253 131 52 38 15 28 11 Round 2 253 60 24 16 6 7 3 Round 3 253 108 43 25 10 7 3 9 Round 1 and round 2 are slightly longer than the intended month for data collection due to slow startup and delays in implementation. 11 In all three rounds, monitoring officers reported clients not picking up the phone as the most difficult challenge in contact ing and recruiting clients for the study. During Round 1, incorrect phone numbers, client s not picking up the phone, inability to clearly explai n the purpose and use of the scorecard to the clients, and survey fatigue were reported as reasons for low uptake (see Figure 1 ). In response , SI provided a capacity b uilding training remotely to the monitoring officers to provide a better understanding of the purpose and use of the RBS and equip the officers to better explain the tool to their clients. Additionally, in all three rounds, SI found the monitoring officers did not consistently reach out to clients three times , as prescribed in the SI protocol for follow - up to increase uptake . Th e reason for the low follow - up may be due in part to the limited experience among Sogesol staff in conducting such follow - up exercises. As shown by this data, monitoring officers faced a major challenge in establishing contact with clients. Furthermore, o nce contact was established , it was difficult to convince clients to use the RBS, to obtain adequate responses to the RBS , and to motivat e the client to continue in the pilot. Indeed, after three rounds, only seven sampled clients completed both parts 1 and 2 of the scorecard for at least two rounds of data collection. Of those seven clients, only one client completed parts 1 and 2 of the scorecard across all three rounds of data collection. To gain a better understanding of the low up take and completion of the RBS, SI conducted key informant interviews (KIIs) with Sogesol ’s Director during the third round of data collection , as well as at the end of the project. It is important to note that the monitoring officers were not accustomed to check ing in with clients as regularly as once a month , as requested under SI’s RBS pilot. Additionally, as explained by Sogesol’s Director, Sogesol usua lly conducts its own evaluation and analysis of its clients’ performance . As such, monitoring officers reported that the RBS was too complex for them to use, and the complexity was an even greater challenge for their clients . They found Part 1 of the RBS t o be relatively simple and easy for clients to complete , as it only asked them to list the factors that affect their business es and estimate the loss in sales considering the worst - case scenario for a given factor. However , both the monitoring officers and the clients found Part 2 of the RBS very difficult to complete. To link the worst - case scenario estimates in P art 1 with Part 2 (as well as automate productio n of results for easy analysis) SI designed Part 2 of the scorecard such that clients estimate the percent decrease in actual sales for each factor at the end of the period. By design, when both parts are complete , the variance between the estimates of sale value at the beginning of the period from the percentage value at the end of the period w ill auto - populate and tell the story of how well the client understands the effects of external risks on their sales. As such, Part 2 of the scorecard , which involved estimating the decrease in sales , was very challenging to understand and use for both the m onitoring officers and the clients. This underscored the importance to revise the RBS to improve its utilization while automating the process. The low uptake and low completion of all the essential parts of the RBS across the rounds severely limited SI’s ability to analyze the data, as originally planned, to statistically test whether the RBS helped SMEs increase their sales over time and whether the resulting increase in sales led to reduction in non - performing loans. However, several valuable lessons eme rged from the pilot , including a better understand ing of the risk factors F eedback from Sogesol’s Director : The client did not understand the pilot; t he client is no expert; t he client will not be able to determine sales from previous month; t he sect i on of the scorecard on percentage created a lot of confusio n. Did not pick up phone 59% Call didn't go through 19% Other reason 11% Wrong number 7% Client too busy 4% F IGURE 1 : R EASONS FOR NOT C OMPLETING RBS DURING R OUND 1 12 perceived by SMEs to affect their sales, and whether and how any learning was occurring among the MFI and SMEs through use of the RBS . RISK FACTORS AFFECTI NG SME s IN HAITI As shown in Table 4 , the majority of the risk factors reported by the sampled SMEs were external, and thus the SMEs have limited control over these factors. T ABLE 4 : F ACTORS P ERCEIVED TO A FFECT T REATMENT C LIENTS Factor Round 1 Round 2 Round 3 Total (Number of RBS selecting the factor) Weather (rain, flood, drought, cyclone) 30 11 7 48 Unrest (demonstration) 3 3 8 14 Unemployment 1 0 0 1 School closing/opening 4 1 0 5 Increase in prices 0 1 2 3 Political crisis 36 5 13 54 Social gatherings (party) 0 0 2 2 Machinery issues (broken, obsolete) 1 0 4 5 Business l ack s novelty 0 0 1 1 Insecurity 12 6 9 27 Inflation 1 0 0 1 Illness 21 7 4 32 Holidays 0 1 9 10 Theft 21 5 11 37 Fire 7 6 5 18 Economic crisis 4 0 0 4 Death 2 3 1 6 Customer demand (irregular) 2 0 0 2 Cashflow 0 1 0 1 Burglary 1 2 0 3 Bad reception (phone) 1 0 1 2 Accident 3 0 2 5 Grand Total 150 52 79 281 SI also examined the data to discern whether the factors changed between the rounds – in other words, were these factors temporary phenomena, or recurring events? Results are shown in Table 5 . From the list of 22 factors listed in Table 4, only 10 factors were found to be selected in more than one round , and all of the se factors appear to be outside SMEs’ control . Nonetheless , there is still valuable learning potential for SMEs through the use of RBS in such situations. While SMEs do not have much control over the incidence of the se external factors , through awareness of the most important factors affecting their business es an d through learning to predict how much such factors might affect their business es , SMEs could develop contingency plans and test them during crisis to avoid drastic reductions in sales. MFIs can also learn about the factors that might affect their clients’ sales and institute strategies to help their clients service the