Utilisation de la gestion axée sur les données pour améliorer la performance des fournisseurs de services d'eau

USING DATA-DRIVEN MANAGEMENT TO IMPROVE WATER SERVICE PROVIDER PERFORMANCE A USAID Water and Sanitation Project Learning Note Project Title: USAID Water and Sanitation Project Sponsoring USAID Office: USAID/Haiti’s Office of Infrastructure, Engineering and Energy (OIEE) Contract Number: AID-OAA-I-14-00049/720521-18F00001 Contractor: DAI Global, LLC Date of Publication: 15 March 2022 Revision: Author: mWater and DAI This publication was produced by the USAID Water and Sanitation project under Contract No. AID-OAA-1-14-00049/720521-18F00001 at the request of the United States Agency for International Development. This document is made possible by the support of the American people through the United States Agency for International Development. Its contents are the sole responsibility of the author or authors and do not necessarily reflect the views of USAID or the U.S. Government. Table of Contents EXECUTIVE SUMMARY ..................................................................................................................................................... 1 INTRODUCTION AND BACKGROUND................................................................................................................................ 2 APPLYING A DATA-DRIVEN APPROACH TO UTILITY MANAGEMENT ................................................................................. 3 REVERSING THE SPIRAL OF DECLINE............................................................................................................................................... 3 WHAT IS DATA-DRIVEN MANAGEMENT?........................................................................................................................................ 3 THE UNIQUE DATA NEEDS OF WATER UTILITIES ................................................................................................................................ 5 DEMOCRATIZING ACCESS TO DATA ANALYTICS................................................................................................................................. 5 MOBILE-FIRST DATA MANAGEMENT ............................................................................................................................................. 5 DESIGN AND IMPLEMENTATION OF THE UTILITY DATA SYSTEM ...................................................................................... 7 DEFINING THE PROBLEM ............................................................................................................................................................ 7 STRENGTHENING PEOPLE AND PROCESSES...................................................................................................................................... 8 ALIGNING INCENTIVES.............................................................................................................................................................. 10 KEY PERFORMANCE INDICATORS ................................................................................................................................................ 10 LEARNING AND ITERATION........................................................................................................................................................ 11 USING DATA TO IMPROVE SERVICES ...............................................................................................................................13 BOTTOM UP RATHER THAN TOP DOWN ....................................................................................................................................... 13 BUILDING A DATA-DRIVEN CULTURE............................................................................................................................................ 14 CONCLUSION...................................................................................................................................................................15 REFERENCES CITED..........................................................................................................................................................16 Executive Summary The USAID Water and Sanitation Project in Haiti is a five-year (2017-2022), project to improve urban water and sanitation in Haiti. It works in partnership with Haiti’s National Water and Sanitation Authority (DINEPA), its regional offices (OREPAs), and local water utilities (CTEs) to improve the performance of urban and rural water service providers by using a data-driven management approach. This project initially targeted 5 utilities for support, but by the end of the project DINEPA decided to expand the approach to all 27 urban water utilities in Haiti. In this learning note, we define the key elements of the data-driven model, discuss how they were applied in Haiti, and share some insights from the experience to help others apply a similar approach. The percentage of households in Haiti with piped water into their dwelling declined between 1990 and 20151. With flat tariffs and declining customer base, the water utilities were caught in a ‘spiral of decline,’ common in many rapidly urbanizing countries that lack the infrastructure and resources to provide reliable service to all customers. Recent case studies by the World Bank2 have shown that it is possible to reverse this decline by setting initially small goals or public commitments that can be accomplished within existing resources and delivering on them by rigorously monitoring performance against targets. This process also forms the basis of data-driven management. The core activity of data-driven management is the process of setting goals, measuring key outcomes, and using this information to update assumptions and plan future actions. Data-driven organizations must address all the steps in the data value chain: generation, collection, processing, analysis, sharing, and use of information. Recent advances in mobile enabled software have democratized the tools of data-driven management, reducing the cost and expertise needed to collect and analyze data. But the most important resources needed to adopt a data driven approach are people and processes, which were strengthened in Haiti using the strategies outlined in Box 1. Box 1. Strategies for building institutional capability to support data-driven management • Clearly define the problems leading to poor service delivery with stakeholders at all levels • Strengthen people and processes to ensure that data is used by frontline workers • Define a manageable set of no more than 10 key performance indicators • Align incentives for staff around data collection • Plan for multiple cycles of learning and iteration • Take a bottom-up approach focused on the routine day-to-day data needs of workers, which can be aggregated to inform national indicators • Build local data expertise in the private sector and within the service provider • Create a data-driven culture through monthly indicator reviews, awards, and recognition An analysis of the support for improving data systems provided by the USAID Water and Sanitation Project shows that at the beginning of the five-year effort, numerous workshops and trainings were conducted to build a shared understanding of problems and potential solutions. In the final two years, this support shifted to technical outputs, such as improved software functionality or integrations between data systems, and one-on-one or small group technical support provided by the local private sector data consultants trained by the project. We identified several factors that contributed to the successful expansion of the data-driven management approach from a few utilities to a national performance monitoring system. These include developing a shared understanding of problems, the process changes required, and human resources needed to address them, an iterative approach that builds on quick wins and incorporates learning over time, and the creation of a data-driven institutional culture. Page 1 Introduction and background In establishing the Sustainable Development Goals, the United Nations High-Level Panel called for a ‘data revolution’ that leverages mobile and other emerging technologies to enable real time monitoring of development results3. Effective monitoring is considered one of the key building blocks of a strong water, sanitation, and hygiene (WASH) sector and a recent review of external support programs for WASH found that at least 42% included activities to strengthen monitoring and regulation4. Despite the emphasis placed on improved monitoring by governments and development partners, the data landscape in most low-resource countries is characterized by failed or obsolete national management information systems, national or regional mapping campaigns that have not been updated in years, and upwardly focused reporting systems that do not provide useful data or insights back to local decision-makers and water managers. The need for better informed decision-making in the water sector is even more clear at the local level, where water service providers struggle with complex challenges. Haiti, like many low-income countries, is not on track to reach the Sustainable Development Goal (SDG) 6 target for safely managed drinking water services by 20301. In addition to common sector challenges such as inadequate financing and a lack of sufficient revenue to support routine operations and maintenance, Haiti has experienced major natural and human-caused disasters and shocks, including the earthquake near Port-au-Prince in 2010, Hurricanes Matthew and Irma in 2016 and 2017, a serious cholera outbreak, and the earthquake that struck the western peninsula of Haiti in 2021. In its efforts to reach SDG 6, Haiti is grappling with a problem common in low-resource regions: the failure of designated service providers to provide reliable piped water to residents, leading to a decline in the number of people who use piped water (Figure 1). As a result, many Haitians have turned to the private sector to purchase expensive bottled or trucked water, or to less expensive water sources of questionable quality5. This trend has eroded public confidence in the ability of the government to provide basic services and threatens the progressive realization of safely managed water services for all. Non-Piped Piped Figure 1. Access to piped and non-piped water supplies in Haiti (WHO/UNICEF). The USAID Water and Sanitation Project is reversing this decline in public services in Haiti by improving the capacity and performance of public utilities and private sector partners, who are essential actors in the delivery of services in rapidly growing urban and peri-urban areas of the country. Working in partnership with DINEPA (the Haitian national water and sanitation agency), regional water authorities, and utility managers and staff, the program works to strengthen sector performance through improvements to operations, maintenance, and financial management. In this learning note, we present findings from one key aspect of the USAID Water and Sanitation Project: the implementation of a data-driven model to drive a turn-around in public utility performance. We detail the iterative, problem-driven approach taken to develop this system and identify factors that lead to successful use of data for decision-making in public utilities. Page 2 Applying a data-driven approach to utility management To most private sector businesses, the term “data driven management” might be synonymous with “management.” The shift toward digital and cloud based services to manage and derive value from data has defined business and management trends over the past two decades. Digital water utility solutions is already a $5B per year market and is expected to double by 20306. Yet many of these tools have remained out of reach for low-income countries due to a lack of financial resources to invest in data systems and a gap in technical expertise and awareness among staff. Fortunately, this divide is being bridged by new cloud and mobile technologies coupled with innovative business models that can improve the performance of public service providers. In this section, we discuss the challenges faced by Haiti’s utilities, which are common to many utilities in low-resource regions, and how a data-driven approach can help address them, as demonstrated through the USAID Water and Sanitation Project in Haiti. Reversing the spiral of decline When service quality is poor due to a lack of adequate water supply or delivery capacity, citizens become reluctant to pay user fees, choosing instead to invest in private supplies or to tap illegal connections. This further reduces revenue that the service provider needs to operate, maintain, and improve or expand water services. This negative feedback cycle has been referred to as a “spiral of decline.” 7 When service providers are stuck in this downward spiral, it is difficult to derive value from their existing assets and nearly impossible to finance improvements because of their poor creditworthiness. While there are examples of utilities in low resource countries that have successfully turned around their performance, as documented by the World Bank 8 and others 9,10, there are many more which have not yet managed to reverse the downward spiral of service. The challenge before the sector is to identify mechanisms that can reliably and demonstrably initiate virtuous cycles of increasing performance in public utilities under differing political and institutional contexts. Low revenue due to poor services Service quality decreases Users invest in private infrastructure Revenue further decreases Figure 2. The "spiral of decline" leading to poor performance (after Galaitsi 2016). The World Bank conducted in-depth case studies of successful utility turnarounds in low-resource settings and used their findings to identify a common pathway (the Utility Turnaround Framework) that emphasizes using currently available resources to set initially small commitments and then deliver on them, leading to increased autonomy and implementation capacity for utility managers and staff. The tangible improvements, in turn, result in improved confidence and willingness to pay on the part of customers, greater autonomy from oversight authorities, and enhanced creditworthiness. One of the first actions that successful utilities take when implementing a turnaround is to upgrade the management information system (MIS), since having a functioning data system is critical to understanding how to target actions and then measure the changes that result from these initial commitments. The foundational role played by data in successful utility turnarounds inspired the USAID Water and Sanitation Project team to implement a data-driven approach in Haiti. What is data-driven management? At its most basic level, data-driven management is the process by which an organization takes an action, Page 3 measures the results, and updates its assumptions and future actions in response to what happened. Data driven management is a concept frequently used in the business sector to describe both a process and a corporate culture of being “data-driven.” It has parallels in the international development sector in the emerging practices of adaptive management for development programming and problem-driven iterative adaptation for building state capability11. Becoming data-driven requires that an organization collect, analyze, and act on data to make decisions. Data-driven approaches rest on the foundation of a robust system to collect and interpret data about important outcomes. The data value chain is a useful framework to help ensure that data is successfully captured and turned into actionable information. Much like a commercial product, data must be produced, distributed, and used to create value. There have been various conceptualizations of the steps in the data value chain and how they relate to deriving value from data 12–14. We defined a version of the data value chain for the water utility context (Figure 3) that reflects the circular feedback cycle inherent in data-driven management models, as well as the asset management process that utilities follow15. Generate Collect Use must generate data. For example, when customers make a payment, this must be recorded in a ledger or database that will allow the commercial office to update account balances and record the transaction as revenue. The organization also needs to collect this data in some way that it can be used for analysis. This step may seem easy to perform with the wide array of mobile data collection apps now available, but effective collection requires standard definitions that are applied consistently across the organization or sector. Otherwise, useful data will become stuck at the collection step. Data that is collected through different systems, including operations and maintenance logs, commercial accounting systems, and geographic information systems (GIS), often requires processing before it can be analyzed. This can include review and validation by technical staff, aggregating by site or location, and formatting for analysis. The more systems that are used by the utility, the more complex this step becomes. Large utilities often require an entire office to keep the various systems of record for each data type in sync. For this reason, there is a broad industry trend toward the consolidation of these functions under a single platform or data center. Analysis is the process by which data about individual events are transformed into information, often in the form of key performance indicators (KPIs). Digital technologies can help compensate for capacity gaps by automating the analysis process, avoiding bottlenecks that arise when others are waiting on an Share Process Analyze analyst to update results. It is also increasingly common to couple the analysis process with mechanisms to share data in real time, through online dashboards, reports, and alerts. A utility often must differentiate their reporting for different Figure 3. The data value chain model for turning data into useful information. The key concept in putting the data value chain into practice is to understand that the human and technological systems used to generate information must address every link in the chain. To begin with, key events that occur in the daily operations of the utility audiences, including internal staff, regulators and water authorities, financial institutions and donors, and the public. Although managers may like to see visually appealing dashboards showing KPIs across the organization, often the most useful type of sharing is to provide specific data to frontline workers as they go about their daily tasks. This can take the form of alerts, filtered reports, and digital work orders. Page 4 Finally, information only has value to an organization if it can use it. While there has been much attention devoted to the value of data for planning and policy at the district or national level, much of the progress made toward better services comes in the form of “small wins,” or minor changes that agents make as the result of learning16. Therefore, a key task in data use is to connect the primary agents to information that they need, when and where they need it. The unique data needs of water utilities At the beginning of the USAID Water and Sanitation Project, research carried out using the organizational capacity assessment tool (OCAT) methodology found that managers and staff were fully aware of their own challenges but lacked the specific information that would enable them to make changes. For example, a national cadastral exercise had been conducted to map every water connection, but that data was not available to the utilities and was not integrated with the commercial databases on billing and accounts payable. Many of the survey-based data collection apps used in the WASH sector are not well-suited to the task of actively managing and updating utility data, which has limited the ability of utilities to use the data collected. Based on our user research in Haiti and with other mWater partners worldwide, we concluded that a utility information system should have the following capabilities: - Ability to map and update physical infrastructure, including vertical assets (water supply, storage, and treatment works) and horizontal assets (pipes). - Flexible form design and editing tools. - Ability of staff to create new analyses and visualizations without writing code. - Import and export of data in useful formats, including XLX, CSV, and shape file. - Capacity to automatically transfer data to and from other data systems, including commercial and accounting software. - Localizable to different languages. - Unlimited data transfer capacity to ensure sustainability under local revenues. - Industry standard security protocols, such as encrypted data transfer and role-based access controls for different types of data. These core capabilities do not necessarily need to be provided through a single digital platform, and often they are not. However, if separate solutions for different tasks are used then the organization needs to provide for the human resource capacity to keep the data synchronized. The organization should also designate a single “system of record” for each type of data to minimize conflicts. Democratizing access to data analytics In response to the high cost of commercial information technology and data analytics software, the information and communication technology for development (ICT4D) sector has developed open source software and low-cost or free software-as-a service (SaaS) delivered via the internet, often taking advantage of the decreasing cost of accessing the internet via mobile devices in developing countries. However, much of the ICT4D sector remains focused on donor-driven projects and once-off interventions, rather than on supporting governments with generalized tools for data management. Early software tools developed for international development applications were focused on data collection, rather than complete solutions for analysis and reporting. Recently, this has changed with the introduction of integrated data management platforms such as DHIS2, mWater, and others. With these options available at little or no cost to users, technical assistance efforts of development partners can shift away from developing new software and toward the challenge of how to effectively introduce data-driven approaches. Mobile-first data management The majority of internet users in low- and middle income countries access the internet through a mobile connection and 94% of the global population Page 5 now lives in an area covered by a mobile broadband connection17. The rapid adoption of smartphones in these contexts offers an opportunity to develop mobile solutions that leapfrog traditional back-office data management software models. A mobile-first approach, in which most or all data are available on the mobile device, is also more resilient to common issues such as unreliable electrical power and poor internet connectivity during periods of peak demand. To help address the unique needs and challenges of data management in Haiti, the project selected mWater because it was already a trusted platform in use by government and donor organizations in Haiti and was cost effective, with no per-user or other recurring software fees. mWater operates a free and centrally hosted data management platform that provides a mobile app for field agents to collect and view data, a web-based data management portal, and the ability to deploy a custom management information system (MIS) for use by all the staff of an organization. The mWater platform already met most of the requirements identified by DINEPA during the problem definition phase. Where gaps in functionality existed, the USAID Water and Sanitation Project was able to work with mWater to develop new features that then became available to all 120,000+ users of the free mWater platform. This collaboration model helps to ensure the sustainability of the data systems because new programs can build on past contributions and users can influence the future evolution of the platform. That said, as we describe in the next section, the sustainability of a data-driven approach relies much more on the work done to align human capacity and organizational processes than the technology platforms selected. Utilities should strive to document the data model, operating procedures, and interfaces involved in the management information system so that it will be easier to move to a new technology platform in the future, if necessary. As a mobile-first technology, most of the core features in mWater, such as creating or editing an asset or record, can be performed using the mobile app, even when working offline or in areas with poor data coverage. This enables front line workers to make changes directly to mapped infrastructure and customer data, rather than simply recording data that must be manually transferred to the back office. This ensures that field staff always have the most up to date information on their mobile device, allowing them to bring the appropriate spare parts and tools to the job site. The mobile-first approach also helps to increase buy-in because the frontline workers can correct problems themselves without needing support from the IT department. Figure 4. The mWater mobile app allows utility workers to view piped network infrastructure and add updates or new connections. Page 6 Design and implementation of the utility data system In this section we document the process used to implement the data-driven utility management approach in Haiti. The implementation approach was informed by the literature on adaptive management and Problem-Driven Iterative Adaptation (PDIA)18, as well as lessons derived from lean and agile management strategies that are more commonly found in the technology sector. The implementation process is summarized below in Figure 5. Each phase took roughly one year to complete and involved frequent workshops, field support visits, and specialized support from the project team to help utilities with specific needs. By project end, the utility management system was fully under local control and rolled out to all 26 urban water utilities in Haiti. Figure 5. Implementation process for the WATSAN utility management system. Defining the problem Problem definition is too often neglected or rushed in the design of water sector monitoring systems. A common mistake is to define the problem as the lack of a preferred solution or “best practice.” Yet, by taking the time to define the context-specific problems related to poor service delivery in Haiti through dedicated workshops and site visits, our partners discovered a set of underlying causes that were related to data but also rooted in organizational processes and staff performance. Identifying these root causes allowed the team to relate aspects of poor performance that they have observed for themselves to an objective or desired outcome of the data system, rather than simply imposing a set of standard sector indicators. Well-defined problems are helpful when dealing with complex challenges and uncertainty around the most effective policies. This is clearly the context in Haiti, where public service delivery has failed in most areas of the country and civil strife and natural disasters frequently disrupt projects. This concept of using a problem or crisis to build a shared vision for the path forward is frequently taught and studied in the fields of political science and public management. In Haiti, problem definition was conducted through meetings with representatives from all levels of the five initial participating utilities, the four regional water authorities that oversee them, and DINEPA. In each meeting, participants were encouraged to identify management problems that intersect with data and to prioritize them based on the severity of the problem and the potential to make progress on a solution. The technical team consolidated this focus group feedback, categorized the needs, and calculated the average priority ranking for each one. The prioritized needs (Figure 6) were then presented to stakeholders for review and validation. Page 7 Priority ranking (average) Needs assessment rankings 1 Piped network and client management Commercial management system 2 Water quality monitoring and reporting 3 Customer service system Meter management 4 Leak management 5 $14,000 $12,000 5-Year Total Cost for Software $10,000 $8,000 $6,000 $4,000 $2,000 $0 Free and open source software Engineering design tools Hydraulic analysis tools GAP Figure 6. Priority ranking of needs identified by utility and water authority stakeholders in Haiti (lower score means higher priority). The highest-ranking data management needs were for better piped network and client management to address the lack of effective revenue collection; improving the poor functionality rate of public kiosks; and improved commercial management to track financial performance. Lower priority needs were more focused on quality-of-service issues, including water quality monitoring, customer service, meter reading, and leak management. In response to the identified needs, the team carried out a market landscape assessment to determine if existing software solutions could meet some of these needs, since the mWater platform lacked some of those functions at the time. The landscape assessment found that there were indeed high-quality, but expensive, software solutions for piped network mapping. However, there was a clear market gap for software that could provide the more basic functionality required at an affordable cost (Figure 7). Although open source software is often described as “free,” the use these tools still result in significant costs for the organization. These costs include hosting services for server-based applications, IT support for frequent updates and security patches; and training and technical support for staff. For example, the open source QGIS application provides very advanced GIS capabilities, but it requires a high level of expertise to operate and is not integrated with mobile apps. 0 2 4 6 8 10 12 14 16 Feature Score Figure 7. Landscape assessment for piped water network mapping and analysis software. Feature scores were based on whether the solution offered various features for mapping piped networks, optimization. hydraulic modeling, and advanced analysis and Since mWater already provided most of the basic functionality, including mobile mapping and updating of monitoring sites, the project team decided to invest in expanding the capabilities of mWater instead of funding bespoke software development. The most notable upgrades were the inclusion of a pipe drawing feature, improved asset management functionality, and the development of customer management and meter reading features. Although the customer management system developed in mWater could have met the needs of the utilities, DINEPA was already using a locally developed platform. Rather than create a parallel system, the project team decided to create a software integration so that data from the other platform was automatically reflected in mWater, allowing it to power dashboards and key performance indicator calculations. Strengthening people and processes The rapid pace of innovation in digital data collection and visualization technology makes it easy to lose sight of the fact that data are collected and used by Page 8 people as they go about their daily tasks within an organization. This issue is illustrated visually by the iceberg analogy of Figure 8. The most visible part of the utility data system is the technology used: the dashboards, apps, and detailed GIS maps. The people and processes needed to keep data up to date remain under the surface, but they are the key resource and the limiting factor in most organizations19. Figure 8. The iceberg analogy for utility data management (modified from Vitasovic 2020). Data-driven management relies on institutional capacity to collect, analyze, and make use of data. The USAID Water and Sanitation Project followed an adaptive management approach, allowing technical assistance to evolve with the increasing implementation capacity of partners. There were four main components to the capacity building assistance provided: • Workshops: Formal events with diverse groups of participants drawn from different levels of the organization, aimed at developing a shared understanding of problems, needs, and plans. • Trainings: In-person (or remote, during the COVID-19 pandemic) sessions to help staff become proficient in specific software or data analysis skills, usually following a training-of-trainers approach to diffuse knowledge across larger organizations. • Technical outputs: New software features, integrations to other systems, data collection and visualization tools, and data migration assistance provided by mWater to address specific gaps or needs. • Support: On-the-job training, technical assistance, and mentoring to develop local data management expertise, within both the utilities and the local private sector. To better quantify the capacity-building support provided over the course of the project, we analyzed each major support activity completed over the five year project (Figure 9). In the first year, support was characterized by national and regional workshops organized to define needs, priorities, and key performance indicators. Formal trainings took place throughout the first 3 years, decreasing in 2020 due to COVID-19 travel restrictions. Fortunately, mWater recruited and trained two local data consultants prior to the pandemic and these local experts, complemented by remote support from the international team, were able to provide additional on-the-job training, attend monthly data review meetings, and respond to routine technical support requests. Figure 9. Capacity-building activities on data management carried out each year of the project. Page 9 As the system was scaled up to all CTEs in Haiti during 2020, DINEPA took the lead role in training. This allowed the external support effort to focus on data collection tools, web-based data visualization sites for internal use and the public, integrations with existing commercial and water quality databases, and new software features to support meter reading, geospatial analysis, and asset management. These technical outputs were developed in close partnership with the intended users and often tested in small groups before rolling out to the entire organization. Aligning incentives Many water sector monitoring initiatives have failed due to assumptions that operators or district-level officials will contribute data out of a sense of duty or shared commitment (often expressed during a kick off workshop), or simply because monitoring was mandated by national policies20–22. In contrast, most successful examples of sustained, data-driven improvements in WASH services involve setting clear commitments that are supported by monitoring systems that can measure progress and hold staff accountable for performance8. Therefore, it is critical that staff incentives be identified and formalized to ensure the sustainability of any new data system. One key strategy for formalizing incentives introduced by DINEPA was to update the terms of reference (ToRs) for contracted staff, such as local water technicians, to include monitoring tasks in their responsibilities. In addition, the timeliness of submitting monthly reports was included as one of the high-level metrics that are reviewed by regional authorities each month. These changes made clear to staff that monitoring was not just one of their many responsibilities but was also a priority to management. Informal incentives also play a role in developing a data-driven culture. For example, DINEPA began giving awards at their annual performance review to the utilities who had submitted the most on-time or accurate data. This sense of friendly competition was also fostered by the design of the dashboards in the management information system, which allows managers to see how their performance compares to other similar utilities in their region. Key performance indicators Most utilities can effectively manage up to 10 high level key performance indicators. WASH indicators used in international development tend to be numerous and complex, reflecting the many different priorities of donors, finance institutions, and civil society organizations. We can point to numerous examples of complex WASH indicator systems with 20 or more indicators that are calculated from even more lower level indicators and factors23,24. In contrast, Sustainable Development Goal Target 6.1 for safely managed water services requires just 6 relatively simple indicators25. A data-driven organization needs to focus on the most critical data to learn and iterate on actions taken, and key performance indicators must reflect that focus. In Haiti, key performance indicators were identified from a much larger set of indicators previously collected by DINEPA or used on various donor funded projects. The process required several workshops with representatives from all levels of the organization, resulting in a set of 11 key performance indicators that will be collected and reviewed by all CTEs and OREPAs monthly. The CTE report that informs these indicators contains much more detailed data on all aspects of financial and technical operations, but DINEPA determined that these key performance indicators were most critical to their data-driven management efforts. The prioritized approach allowed the smaller CTEs with less technical capacity to focus on the most important data first and build up their business processes over time. An iterative design-test-revise strategy carried out by the five early adopter utilities allowed DINEPA to test and validate the methods for collecting and aggregating these indicators using real data. Once the data collection systems, data analytics, and visualizations were deemed ready, DINEPA rolled out the system nationally using a training of trainers approach that built on the expertise of the early adopters. Page 10 Table 1. Key performance indicators for urban water utilities in Haiti. Key Performance Indicators Commercial Active subscribers Collection efficiency - current accounts (%) Collection efficiency - accounts in arrears (%) Collection efficiency - overall (%) Financial Revenue Expenses Operating ratio (%) Technical Total production (m3/month) Service continuity (hours/week) Residual chlorine tests conforming (%) Functioning kiosks (%) The 11 key performance indicators agreed to by the national observatory and regional water authorities are presented in Table 1 These indicators are evenly divided between commercial operations, finance, and technical performance. For the most part, these indicators follow the standard definitions set forth by the International Benchmarking Network (IBNET). In some cases, a standard indicator needed to be localized due to differences in definitions. For example, to address the high number of accounts in arrears, DINEPA defined a separate collection efficiency just for those delinquent accounts. Also, although the usual definition of operating ratio is to divide the income by the expenses, in Haiti, it was historically taken as the inverse: expenses divided by income. The process of defining indicators took longer than expected, largely due to the need to better define the data fields and calculations, and then to map them to the various business processes in place, which varied greatly across large and small utilities. One of the most important insights from this process was the need to precisely define the data that goes into calculating an indicator. Some participants who had worked in their jobs for years discovered that they and their coworkers had different understandings of commonly used terms. We refer to this process of working through detailed definitions of terms as “the good struggle,” because in our experience it can involve hours or days of tedious discussions and sometimes heated debate, but the result is greater consensus among team members and buy-in from leaders. By localizing international standards to reflect the values and traditions of the utility sector in Haiti, participants were able to develop a monitoring system that generates actionable data that is well understood by the managers who need to use it for decision making. It is important to note that localization of the indicators did not mean that each utility created its own monitoring system, which would have made it very difficult to scale to all 26 utilities from the initial group of five early adopters. The goal was to identify a set of standards that represented the priorities of the Government of Haiti for water service provision. Some of the more advanced utilities could go beyond the minimum and collect more data, while other less capable ones were allowed to omit certain indicators until they could build up their capacity and resources. Learning and iteration With a well-defined set of indicators, data collection sources, and methods identified, the next step was to train the group of five utilities to use the system developed with mWater. We created an initial draft of the monthly reporting form along with detailed documentation on data sources and definitions, translated into both French and Haitian Creole. In addition, we developed a set of dashboards that allow the same aggregated data to be viewed at the level of the utility, the regional water authority, and the entire country. The initial field testing of the system resulted in dozens of actions to modify the forms and visualizations. These were completed in some cases in real time by technology staff embedded in the field offices, while others required minor software updates. The key strategy used in this phase was rapid Page 11 iteration. This is critical because when field agents have a problem that prevents them from using a new system, they often stop using it until it is fixed. If the fix does not come quickly, staff will revert to the old ways of doing business and can lose faith in the likelihood of the new approach succeeding. The process of making changes to the design of forms, indicator calculations, and dashboards was enabled by the “no code” mWater software environment. This means that anything the user sees can be changed by administrators via a simple visual interface, rather than requiring a software engineer to modify computer code, which is costly and time-consuming. For example, the mWater platform allows indicator calculations to be viewed and updated using a visual interface. These calculations are automatically performed whenever new data comes in, offering a real time view to everyone in the organization. This feature allowed for rapid iteration and improvement during the validation phase, which was much more effective than a more traditional design-build-deploy software development approach. The rapid iteration strategy combined with the visual “no code” editing environment reduced the timeframe for releasing updates from months or weeks down to days or even hours. Page 12 Using data to improve services “Proximity focusses attention on small wins. After all, large wins are really an accumulation of those small wins; of minor changes that result from ongoing learning.” – S. Abimbola16 National WASH monitoring efforts have typically focused on generating data for regional or national planning, which is an important long-term goal. But most of these systems in low-resource countries fall into disuse, largely as the result of a lack of human capacity, financial resources, and incentives. A recent review of the evidence regarding WASH systems approaches found that few studies even report on the service delivery outcomes27. Data-driven management offers an alternative approach that can link outcomes to specific actions taken, while also supporting the routine data needs of the proximate actors involved in service delivery. Bottom up rather than top down In contrast to the district, regional, or national sector monitoring approaches often promoted in the WASH sector, the focus of the Haiti system started at the utility level and scaled upward to provide data to regional managers, ministry officials, donors, and the public. This approach has several benefits. First, there is a reduced risk of “premature load bearing,18” where stakeholders expect too much of the system too soon, resulting in highly visible failure. By starting with a smaller group of utilities who were committed to change and had additional technical support from the USAID Water and Sanitation Project, it was possible to identify and fix many issues early on that would have been more difficult to deal with at a larger scale. A second benefit of starting at the utility scale is that the system addresses everyday problems that the frontline workers experience as they go about their work, such as how to keep track of customer balances or remembering which type of pipe is installed at a leaking water main. This practical focus provides natural incentives for using the data system that go beyond job descriptions. The system helps the staff to do their job, while also informing higher level sector performance indicators, as shown in Figure 10. Figure 10. The “bottom up” approach: using service provider data to inform SDG6. Page 13 Building a data-driven culture Data-driven management is as much a culture change as it is a system for managing business activities. The final implementation phase, Expansion and Institutionalization (Figure 5), was largely driven by pressure from DINEPA itself. The authority recognized that the most efficient approach for adopting the new utility management system was to roll it out to all urban utilities in the country at once. DINEPA asked if USAID would provide support to their internal training and technical staff for a nationwide implementation. This request was quickly granted and additional on-call technical assistance was provided by the project team. This effort to scale up data-driven management nationwide was characterized by the following strategies: Building local data capacity. After seeing that use of data systems peaked after each visit from one of our international staff, we realized that a better approach is to engage local data and technical expertise so that these visits become routine. Whereas some technical assistance organizations set up country offices that are dependent on external funding, mWater prefers a distributed management approach. This is done by recruiting local entrepreneurs and tech startups, training them to use mWater technology, and providing a direct line for on-call support by the international technical team. After the end of project support, these local experts are available to engage directly with governments and service providers, which helps to develop the private sector data ecosystem. In Haiti, we engaged one independent consultant and one small firm as data consultants to engage on a weekly or monthly basis with the supported utilities. This is much less costly than an international visit or setting up an office, and it directs more of the project resources into the local economy. The local data experts also allowed the project to continue technical assistance activities during the COVID-19 pandemic with minimal disruption. Training of super users. This strategy involved an intensive training, conducted over several weeks by the local data consultants, to train certain individuals to an expert level in skills such as data management, visualization, and reporting. These “super users,” who are staff members working in the utilities, regional water authorities, and DINEPA, can resolve more complex issues encountered by field staff, develop new forms and visualizations, and assist in training and field support. We found that the more typical classroom-style large group trainings were insufficient to develop the necessary expertise. The super user trainings were conducted with small groups of 2-5 participants at their workplaces over 6 days or more. More advanced administrator-level topics were emphasized, including survey creation and deployment, data visualization, and organizational data management processes. This more in-depth training allows utility staff to turn to their own colleagues when they need help with technical issues rather than requiring external support. Monthly performance reviews. Each month, the local data consultants meet with senior management at each utility for a key performance