(2018) Les conditions préalables au financement axé sur les résultats peuvent-elles être établies dans les États fragiles?

Results-based financing (RBF) mechanisms such as conditional cash transfer (CCT) programs, school grants, and teacher and school incentives are increasingly being used around the world to improve education results and outcomes and move away from the procurement of “more of the same” inputs. 1 However, for RBF to work, certain preconditions must be met, the most important of which is the need to build the technical capacity of countries to define and implement RBF mechanisms and to develop monitoring systems that measure the results. One of the most important preconditions of RBF is a functioning education management information system (EMIS) so that results can be accurately monitored and reported. 2 Without an EMIS to precisely measure both learning outcomes and intermediate indicators of educational quality, it is impossible for education ministries to effectively incentivize improvements. A crucial question is whether these types of preconditions can be established in countries affected by Can Preconditions for Results-Based Financing Be Established in Fragile States? Functional information systems can be difficult to establish in countries affected by fragility and conflict. Yet, information systems are important for measuring learning outcomes and intermediate indicators of educational quality. REACH funded the development of a quality assurance system (QAS) to help the Government of Haiti establish the necessary preconditions for the adoption of RBF. Based on a note provided by Juan Baron and Melissa Adelman The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. HAITI SEPTEMBER 2018 RBF EDUCATION RESULTS-BASED FINANCING EVIDENCE Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized fragility, conflict, and violence (FCV) as a first step towards implementing RBF programs in these countries. Using RBF may be an attractive option to consider in some FCV countries given that donor and local exhaustion may have set in after decades of investments in the education sector that have not produced the expected results. The flexibility of RBF approaches also means that they might be the financing modality that is best suited to the challenging and ever-changing environment in FCV countries, allowing all stakeholders to keep an eye on desired results while adjusting the actions being taken to achieve those results when necessary. This question is very relevant given the fragility of institutions and regulatory frameworks in FCV countries as well as the weaknesses in their monitoring and governance systems. The Results in Education for All Children (REACH) Trust Fund at the World Bank provided funding to the Government of Haiti with the goal of establishing the preconditions for the adoption of RBF in the Haitian National Ministry of Education and Professional Training (Ministère de l’Education National et de la Formation Professionnelle, MENFP). To this end, the grant funded the development of a quality assurance system (QAS) based on specific standards for the most important dimensions of educational quality in the country. The idea was to include clear indicators for each quality dimension that would make it possible to measure education results on the ground. The grant also funded a series of complementary activities aimed at strengthening the technical capacity of MENFP staff to define and measure quality. By developing a QAS for all primary schools in the country, the grant aimed to improve governance, enhance the data systems needed to measure results, and establish the preconditions necessary to introduce an RBF mechanism in the education sector in Haiti. CONTEXT Although there have been considerable increases in enrollment in recent years, the education sector in Haiti still faces many challenges. 3 The net primary enrollment rate rose from about 60 percent in the early 2000s to between 70 and 80 percent in 2012. 4 Because there are not enough public schools to meet this increased demand, private providers have stepped in to fill the gap and now operate over 80 percent of primary schools. 5 However, there is little oversight of these providers nor are they held accountable for the quality of their teaching or learning. The public sector plays a similarly limited role in both the provision and regulation of all other levels of education as well. The Government of Haiti has been able to raise enrollment rates by making increasing access to primary education one of its top funding priorities. The government has been financing tuition waivers for students attending private schools through the donor-funded Education for All (Éducation Pour Tous, EPT) Project and its Tuition Waiver Program since Haiti With over 80 percent of primary schools operated privately, there is little oversight to maintain quality. 2 RFB EDUCATION | EVIDENCE 2007 and through the government- funded Universal, Free, and Obligatory Education Program (Programme de Scolarisation Universelle, Gratuite, et Obligatoire, PSUGO) since 2011. At their peak between 2011 and 2014, these two programs financed the school fees of over 60 percent of all primary students in Haiti, with PSUGO accounting for about 20 percent of domestic public spending on education. 6 However, these gains in access are at risk as these programs have not yet been established as a permanent part of Haiti’s education system. Despite these gains in enrollment, student achievement remains very weak. The average Haitian child enters first grade nearly two years late due to a combination of household financial constraints and restrictive school policies that require children to receive at least one year of pre-primary education in order to prepare for first grade. Once in school, 13 percent of first graders repeat the year, and only about half will reach the sixth grade. Student learning is also very weak. For example, a 2016 nationally representative early grade reading assessment found that 42 percent of students near the end of their second-grade year could not read a single word of Haitian Creole. 7 Furthermore, a 2015 fourth grade math assessment by the MENFP in public and private schools found that student scores were only slightly better than random guessing. This shows that the learning crisis has also affected the Haitian education system. 8 The idea behind the QAS was to help the MENFP to tackle some of the toughest challenges in the sector and to develop a system that would make it possible to link financing to results in the future. The QAS would be designed to cover both public and private primary schools. The approach also included complementary activities to strengthen the technical capacity of MENFP staff to develop the QAS—in particular, to establish standards for learning conditions and for student learning, as well as diagnostic assessments linked to those standards. Although the MENFP was already putting substantial emphasis on improving the quality of education, different actors had different definitions of quality. It was critical to establish a consensus on a definition of educational quality before specific learning standards could be developed. Since multiple previous attempts to develop accreditation systems for non-public schools had failed, the work under the REACH grant focused on developing a definition of quality that would apply to both public and non-public schools. The final QAS design addressed several issues, starting with the WHY WAS THE INTERVENTION CHOSEN? HAITI 3 definition of learning standards. Standards of learning were defined to prioritize the most important elements of the curriculum and ensure that any learning assessment that is developed is aligned with the curriculum. The QAS also established the necessary tools to measure the school learning conditions—outputs and intermediate outcomes—that are needed to improve learning. By establishing systems to collect data on learning inputs and outcomes, the QAS would be setting the stage for strengthening the monitoring and evaluation of interventions such as the tuition waiver programs and to increase the accountability of both private and public schools. The establishment of the QAS would also provide donors and policymakers with the data that they needed to measure progress and results and to target interventions. To increase the capacity of the MENFP, the World Bank team in Haiti worked with the ministry to identify and carry out specific tasks that would lead to the development of the QAS. The process of building this capacity was intended to emphasize the MENFP’s ownership of each activity and to increase the likelihood that it would sustain the QAS over time. It also brought together diverse stakeholders and incentivized them to coordinate their work. The QAS was also intended to empower local actors to get more involved in the decisions that impact learning in schools, to prioritize resource use, to increase government accountability, and to meet the local population’s most urgent needs. By developing a QAS for all primary schools in the country, the REACH Trust Fund hoped to improve governance, enhance data systems to measure results, and establish the preconditions necessary to put in place an RBF mechanism to improve the quality of education in Haiti. HOW DID THE INTERVENTION WORK? The QAS was developed by the MENFP through a series of workshops, meetings, and continuous technical assistance from the World Bank. The developers took into account relevant international experiences that could be appropriately adapted to the Haitian context and built on technical work done by the MENFP and other partners. So far, what has been accomplished includes: (a) the development of an officially endorsed framework for educational quality with a holistic perspective that includes conditions for learning in schools and student learning outcomes; (b) the creation of validated measurement instruments for assessing the quality of learning; and (c) a nationally representative pilot test of these instruments that will yield baseline measurements for the system. The QAS is now the guiding framework for providing support to both public and private primary schools under the IDA-financed Providing an Education of Quality in Haiti (PEQH) Project and is likely to be used by other donors to channel financing to schools as well. An important aspect of the REACH grant was to strengthen the capacity of the MENFP to collect and analyze education data. The REACH funds have been used to reinforce the statistical and information functions executed by the ministry’s Directorate of Planning and External Coordination (Direction de la Planification et de la Coordination Externe, DPCE). In particular, REACH is supporting a coordinated effort among donors and MENFP directorates to build and sustain a functional EMIS. Using frameworks developed by the World Bank’s Systems Approach for Better Education Results (SABER) initiative, REACH funds were used to carry out a diagnostic to identify strengths and weaknesses in data collection and information management in the Haitian education sector and to support the MENFP’s design for an EMIS. A grant from the Trust Fund for Statistical Capacity Building, as well as IDA funds under the PEQH Project, will be used to implement this design. REACH funds have also been used to strengthen the capacity of The QAS aimed to help the government improve governance and enhance data systems by including both public and private schools. 4 RFB EDUCATION | EVIDENCE MENFP staff to define standards for learning conditions and to develop assessments to measure conditions in schools against these standards. These standards define quality education in five dimensions: (a) the school principal’s leadership; (b) the school environment; (c) pedagogical management and support; (d) the management of human and physical resources; and (e) community participation. The standards for each of these dimensions are described in Table 1. An important goal of the QAS was to build Haiti’s capacity to measure student learning in primary education. To achieve this, the REACH grant also supported workshops attended by international and national experts, teachers, and MENFD staff to develop learning standards aligned with the curriculum and funded an accompanying assessment of third grade reading in Haitian Creole. These learning standards were added to the QAS to provide clear milestones for the development of reading skills in the third grade. These standards can also be used to develop assessments to strengthen the capacity of teachers to assess their own students, to measure learning in a transparent and technically sound way, and to adjust their pedagogical improvement plans. These standards were defined across five levels of learning as seen in Table 2. Table 1. Standard for Each of the Dimensions of School Quality in Haiti Quality dimension Dimension description D1 School principal leadership School principals focus their practices towards student learning, thus requiring that they monitor results and carry out a strategic and planned vision. D2 School environment The school, the administration, and teachers follow procedures and practices that are clear in terms of the organization, preparation, implementation and evaluation of teaching, considering the needs of all students by taking concrete actions so that students achieve their learning objectives and reach their potential. D3 Pedagogical Management and Support The school has policies, procedures, and strategies in place that favor an environment that is respectful, organized, safe, and appropriate for student learning and the school community. Additionally, the school supports and promotes the spiritual, ethical, moral, emotional, and physical development of its students. D4 Management of Human and physical resources The school has policies, procedures, strategies, and practices in place for the management of its staff and educational resources. D5 Community Participation The school has presents policies, procedures, strategies, and practices in place for creating and developing the participation of stakeholders within the educational community. Table 2: Five Levels of Learning Levels of learning Description Advanced Pupils can make simple inferences: they are able to draw conclusions that are obvious, to find the meaning or subject of a theme or main idea of a text, to make inferences that an event is the cause of another one, to make simple inferences regarding the main characters, their attributes, feelings, or motivations, and are able to relate two pieces of information contained in two adjacent sentences. High Pupils can collect explicit information in a text: they are able to identify a word or idea already mentioned in a text and are able to retrieve information in a word or in a sentence. Average Pupils can read words and short sentences: they can recognize the image corresponding to the sentence or word that they read by themselves, identify the words read aloud by the teacher, and identify words written in uppercase or lowercase. Low Pupils can recognize letters and syllables: they are able to recognize images corresponding to short oral messages (a sentence or a word), to recognize syllables, to identify letters written in upper and lower case, to identify the first letter of a word and the initial sound of a word, as well as written characters among other symbols (such as numbers or signs). Below Average Pupils at this level do not demonstrate the skills measured by this test in Creole. These pupils have difficulties with the knowledge and skills at the low level. HAITI 5 School performance in Haiti varies significantly across the five dimensions of educational quality. To assess progress towards the achievement of educational quality standards for each school, a set of questionnaires were developed, pre-piloted, and piloted in 2016 and 2017, one for school principals, one for teachers, and one for school characteristics. The data collected by these questionnaires was aggregated into a school’s profile consisting of the school’s scores on indicators that provide an overview of its level of development in each dimension. The average school performance ranged from 28 percent in the “Management of Human and Physical Resources” dimension to 62 percent in the “School Environment” dimension (Figure 1). These school profiles, in combination with measurements of learning, will be valuable inputs into the school improvement plans (SIPs) that each school is required to develop with the participation of their local community. There is potential for policymakers to use RBF to encourage schools to achieve the goals set out in their plans as measured by the indicators in the QAS. Some of these improvements might actually be inputs, such as constructing proper facilities for learning (as in Dimension 2. School Environment). While school facilities are often strictly a final outcome, the lack of proper classrooms and latrines can prevent students from being able to effectively attend school and learn. The assessment of third grade student learning in Haitian Creole was piloted in 20 schools in Port-au- Prince in 2017. Almost 70 percent of students in the pilot had an “Average” level of reading skill or lower (Figure 2). Only 2 percent of students in the pilot were at the “Advanced” level, which is defined as having the skills expected by the official third grade curriculum, including being able to recognize letters and syllables, to read words and short sentences, to collect explicit information in a text, and to make simple inferences. In other words, if third grade students were reading at the expected third grade level, most students would have exhibited the “Advanced” level of reading in the assessment. WHAT WERE THE RESULTS? Figure 1: Average School Scores in the Pre-Pilot 01020304050607080 D5 D4 D3 D2 D1 Quality dimension Achievement of standard (%) Figure 2: Students’ Competencies in Reading (Creole) (Pre-Pilot Data) Percentage of students (%) 0 10 20 30 40 50 HighAdvanced AverageLowBelow low Competency level 6 RFB EDUCATION | EVIDENCE WHAT WERE THE LESSONS LEARNED? The QAS has established standards for what a high-quality school should look like in multiple dimensions. This has helped to move the dialogue away from focusing on enrollment numbers, inputs, and vague pronouncements on the state of education towards a constructive discussion about what is working and what is not and how improvements can be made. It has also shifted the focus of the policy and technical dialogues in Haiti away from inputs and towards results and helped the MENFP to envision how to improve the quality of education to the benefit of Haiti’s children. While development organizations tend to focus on student learning as the ultimate and most important outcome, it is very difficult to “move the needle” and substantially boost learning in an enduring way without understanding and addressing the complex and interrelated aspects of what makes a good school. The QAS assesses not only student learning outcomes but also the intermediate indicators that make up the learning conditions within each school, including the quality of the infrastructure, teaching, school leadership, and the overall interpersonal climate. Given the prevalence of private schools in Haiti and the long history of weak public sector management, the QAS is also acting as a guiding framework through which the government can both improve the quality of public service delivery (by prioritizing its spending according to clearly identified needs) and regulate the private sector (by making public financing conditional on schools meeting high quality standards). The continued interest and engagement that the World Bank team has experienced from the MENFP and other sector stakeholders demonstrates that there was clearly a substantial latent demand for a dialogue on results. The development of the QAS is helping to legitimize the MENFP’s efforts to regulate the education sector by creating clear, specific, and realistic standards that will apply to both public and private schools. At the same time, moving towards RBF is a process that will take time and multiple iterations. Creating a sustained consensus about quality standards despite shifting priorities and political pressures will require ongoing technical assistance, dialogue, and knowledge sharing. Moreover, the QAS is a work in progress, as the realities of implementation mean that the tools and approaches must be regularly adjusted and refined to make them work on the ground. Based on the experience of developing the QAS in Haiti, it seems that successfully implementing RBF in a low-income, fragile country requires at least three elements: (a) laying the groundwork for defining and measuring educational quality, as the World Bank team in Haiti has done over the past two years, and remaining persistent despite the inevitable setbacks and slowdowns that come from instability, shifting priorities, and emergencies on the ground; (b) recognizing that “results” can often include what we often dismiss as “inputs,” which can actually matter a great deal for a school’s learning outcomes (for example, having decent, functional latrines); and (c) recognizing that many low-income countries may already have policies that define fairly lofty ideals for standards but that having a dialogue on RBF can actually be more useful than introducing aspirational outcomes for bringing policy closer to reality. The QAS has been able to shift the focus of policy and technical dialogues in Haiti away from inputs and towards results and helped the Ministry understand how to improve the quality of education. HAITI 7 CONCLUSION The set of activities funded by REACH successfully created a quality assurance system (QAS) in Haiti that collects information on five dimensions of learning conditions in schools, provides schools with clear standards to meet, and gives them information about their progress towards reaching these standards. Now that this information is being measured by the QAS, policymakers can decide to offer monetary incentives to schools, teachers, or other relevant stakeholders to make the necessary efforts to improve these learning conditions and outcomes. REACH also supported the development of learning standards for reading in Haitian Creole. The QAS data collection instruments have been piloted and revised and have become important tools for measuring educational quality in Haiti. These activities have had several benefits: (a) they have increased the Haitian government’s capacity to measure educational quality; (b) they have advanced the technical design and piloting of accreditation processes and improved the governance and regulation of the school system; (c) they have made it easier to target programs to address schools’ needs based on the best available data; (d) they have empowered local actors (such as school leaders, principals, teachers, and parent associations) to become more involved in school decisions; and (e) they have established the foundation for introducing an RBF mechanism to incentivize school improvement. By establishing the QAS in Haiti as a reliable mechanism for measuring learning and intermediate indicators of educational quality, REACH has shown that the preconditions for RBF can be established in FCV countries. 1 World Bank. 2015. The Rise of Results-Based Financing in Education. Washington, D.C.: World Bank Group. http://www.worldbank.org/content/dam/Worldbank/Brief/Education/RBF_ResultsBasedFinancing_v9_web.pdf 2 Holland, Peter A. and Jessica D. Lee. 2017. Results-Based Financing in Education: Financing Results to Strengthen Systems. Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/en/715791489054110215/pdf/113265-REVISED-RBF-Approach-Final-Digital-Apr21.pdf 3 Adelman, Melissa A., Juan Baron, and David Evans, 2016. Why school enrollment is not enough: A look inside Haiti’s classrooms. Washington, D.C.: World Bank Group. https://medium.com/world-of-opportunity/why-school-enrollment-is-not-enough-a-look-inside-haitis-classrooms-34780e9614c7 4 World Bank. 2015. Better spending, better services—a review of public finances in Haiti (English). Public expenditure review (PER). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/246141467030212959/Better-spending-better-services-a-review-of-public-finances-in-Haiti 5 Observatoire National de la Pauvreté et de l’Exclusion Sociale. 2014. Investing in People to Fight Poverty in Haiti: Reflections for Evidence-based Policy Making. World Bank, Washington, .DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/21519 6 Adelman, Melissa A. and Peter A. Holland, 2015. Increasing access by waiving tuition: evidence from Haiti (English). Policy Research Working Paper No. WPS 7175. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/823841468035508642/Increasing-access-by-waiving-tuition-evidence-from-Haiti 7 Ministère de l’Éducation Nationale et de la Formation Professionnelle. 2016. Évaluation des apprentissages de base en lecture des élèves de la deuxième année fondamentale. 8 World Bank. 2018. World Development Report 2018: Learning to Realize Education’s Promise. Washington, D.C: World Bank. doi:10.1596/978-1-4648-1096-1. PHOTO CREDITS: Cover: “Education in Haiti” by GPE/Chantal Rigaud, license: CC BY-NC-ND 2.0 Page 3: “Source de la Grace Jumecourt Children’s Village, SDLG” by The Global Orphan Project, license: CC BY 2.0 Page 6: “Pre-primary section of the Ecole mixte de la Convention baptiste de Hinche” by GPE/Chantal Rigaud, license: CC BY-NC-ND 2.0 Page 7: “Education in Hati: Collège Mixte Joinvil in Port au Prince” by GPE/Chantal Rigaud, license: CC BY-NC-ND 2.0 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany through the Federal Ministry for Economic Cooperation and Development. worldbank.org/reach reach@worldbank.org