AYITI: Èske telefòn entelijan ka amelyore lekòl yo?

Learning what works for better programs and policies from EVIDENCE to POLICY Haiti has made substantial improvements in educational access over the last decade and 90 percent of children of primary school age now are enrolled in school. Most of the schools are privately run, often through a foundation or religious institution. Although the government doesn’t administer these private schools, the Ministry of Educa- tion assigns inspectors to visit and monitor both private and public schools. Nonetheless, inspections don’t occur regularly. The country’s poor road infrastructure makes it hard for inspectors to get to the schools, and they often don’t have a car or enough gasoline to go to the schools. They also each get about 35 schools to monitor, making it hard to visit all on a regular basis. The Ministry of Education was interested in improv- ing its approach for monitoring schools. A World Bank-led team worked with the government to develop a pilot to test whether technology could improve monitoring by allow- ing for “virtual” inspections. Haiti seemed like a particularly good candidate for this technology-based solution since it has high overall usage of mobile phone technology, with nearly Context Over the last decade, millions of children have been able to go to school because countries made free primary school education a priority. What happens now that they are in school? Are they learn- ing? A lot needs to happen to make sure that children can get a quality education. To start, education officials must be able to manage schools effectively and monitor what’s going on, especially whether teachers are showing up for work and what they’re doing once they are in school. The availability of new technolo- gies, such as smartphones, offers new routes for monitoring teacher absenteeism and for improving communication be- tween schools and central authorities. But can information and communications technologies, known as ICTs, really change how schools—and school officials—act? The World Bank is committed to making sure all chil- dren have the chance to get a primary school education. Suc- ceeding requires helping governments discover innovative solutions to problems that might keep kids from learning or keep them out of school. In Haiti, World Bank researchers worked with the Ministry of Education to evaluate a pilot program to use digital technology to help keep track of teachers in school and enable more effective monitoring of schools. The plan was to use smartphones to take pho- tos in school of teachers and send the photos to a central server that could be accessed by school inspectors and ministry officials. The problem was that implementing the program was difficult and compliance was low. In the end, there was no effect on teacher absenteeism or stu- dent learning, in part because of the problems getting the program up and running. The evaluation highlights the challenge of using technology in countries with limited infrastructure and serves as a cautionary tale that tech- nology alone cannot solve the complex challenges many school systems face. HAITI: Can Smartphones Make Schools Better? EDUCATION December 2015 102710 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized This policy note summarizes the World Bank paper, forthcoming, “Can Information Technology Improve School Effectiveness in Haiti? Evidence from a Field Experiment,” Melissa Adelman, Moussa P. Blimpo, David Evans, Atabanam Simbou, Noah Yarrow. Support for the evaluation was provided by the Strategic Impact Evaluation Fund and the Bank Netherlands Partnership Program. Results Evaluation EDUCATION The program faced challenges from the start, including delays and technical problems that made it hard to implement it as planned. There were so many problems getting schools ready for the pilot that the program ended up starting months late. Instead of running for the whole of the 2013–2014 school year, the pilot was only in effect for four months. This short implementation period reduced the chance of seeing any change in teacher behavior or student learning. The problems included administrative and procurement ones, such as difficulties identifying and hiring survey firms and unrelated teacher strikes. Once the program was launched, school officials reported difficulties using the new technology, despite having received training beforehand. School directors were given cell phones with digital cam- eras, a data connection subscription and a software pro- gram for uploading the photos and sending information about teacher absences to the central server, which could be accessed by administrators who had the password. The cell phone company provided training to school directors on how to use the phones to take picture and how to up- load the photos to the central server. School inspectors, who were in charge of checking that the uploaded photos Researchers randomly assigned 200 public and private primary schools either to a control group or to partici- pate in the program. In the 100 schools assigned to re- ceive the program, directors received a smartphone with a built-in system that allowed them to send information about the school to a centralized server, including dai- ly photographs of teachers to verify they were at work. School inspectors could access the server in real time for efficient supervision. Both the inspectors and the school directors received training from the research team and the cell phone company on how to use the phones and software. Baseline data collection started in November 2012. During this initial round, the research team collected detailed data on the school and the school director, and ad- ministered an Early Grade Reading Assessment (EGRA) to a sample of 15 students at each school. After the pro- gram was launched in January 2014, the research team collected data on absenteeism and other basic informa- tion from all the schools. In May and June of 2014, as the program ended, the research team collected endline data from all the schools and administered the EGRA to another random sample of students in all 200 schools. Researchers also conducted an in-depth qualitative study in a small sample of schools after the program ended, interviewing 37 teachers, 14 directors, and four inspec- tors about the operational challenges of the program and their perceptions of it. 70 cell phones per 100 people. The plan was to use this technology to complement in-person visits by inspectors and make it simpler for the education ministry to moni- tor schools, while giving school directors clear information about which of their teachers were and were not turning up. An impact evaluation, funded in part by the World Bank’s Strategic Impact Evaluation Fund, was built into the program to measure the effect. were of the teachers who were supposed to be teaching (the uploaded photos were checked against a pre-existing database of teacher photos), were trained in how to retrieve the information and match the photos. But school directors and inspectors still weren’t always able to use the new technology and directors reported that they had difficulty keeping the phones charged, uploading the data, and using the software. In interviews, all but one school director reported technical difficulties with the pro- cess of taking and uploading the photos. Inspectors also reported significant technical difficulties on the receiving end that led to their failure to use the database of pho- tos. And when school directors asked for a technician to come out and help—which was supposed to be available as part of the program—they usually didn’t get a response. It turned out the technical team, which was supposed to be provided by the cell phone service provider, wasn’t ready in time to handle the questions. In the end, very few schools actually used the program with any regularity. Schools in the pilot program only used the technology 30 percent of the time during the four-month implementa- tion period. Part of the reason was clearly due to the tech- nical difficulties with the actual technology, but it’s also possible that teachers viewed it as unfair. In Haiti, teachers’ salaries are often delayed or not been paid altogether, so there’s little incentive for teachers to allow themselves to be held accountable when the system itself doesn’t hold itself accountable. Teachers also didn’t get any extra com- pensation for taking part. The promised phone credits never materialized because of logistical and administrative problems. The program also took time—whether it was checking that the smartphone worked or taking and uploading photos —and school directors who managed more than one school were less likely to participate, as compared with directors who managed just one school. Similarly, school directors who also had other jobs were less likely to use the program. The technology seemed to have introduced additional problems, including a drop in inspections among schools that received the mobile technology. Thirty-five percent of the schools in the treatment group had not received any inspector visit six months into the school year, while only 29 percent of schools in the control group had not received any inspector visit. This may sug- gest that inspectors use the program as a substitute for ac- tual school visits, rather than as a complement. However, it’s also possible that the inspectors used the technology to identify the schools in greatest need of support visits and then concentrated their efforts on those specific schools, but researchers were not able to verify if that was the case. More research will be done in this area. The program had no impact on student learning— nor on teacher absenteeism. Students’ test scores did not improve as a result of the pro- gram, which was unsurprising given the program was im- plemented for just four months and use of the technology was limited during that period. The program didn’t improve management practices such as record keeping, likely also be- cause so few schools used the technology, nor did it have an impact on teacher absenteeism. The real lesson learned from the evaluation was that things are never as simple as they seem. The new technology was more complex than it seemed for school directors and inspectors. Teacher absenteeism, also, turned out to be not as much of a problem as initially assumed. The surveys done for the evaluation found that the rate of absence was usually between five percent and 10 percent. That’s actually quite low for a low-income country, where rates can range from around 11 percent in Peru to 25 percent in India. Across the developing world, technological solutions are often viewed as a panacea, capable of solving some of the most critical problems developing countries are fac- ing. In the education sector, many school systems are looking to technology to increase accountability and improve student learning. Although there have certainly been successes with technological innovations in the ed- ucation sector, governments should be aware that tech- nology-based programs that work in one context may not work in another, and innovations with the poten- tial for success must be rigorously tested on the ground to ensure that the local infrastructure can absorb and sustain them. In addition, as this evaluation and others show, infrastructure is only part of the equation: Mak- ing sure that teachers, school managers and inspectors are on board with the program—and willing to carry it out—is equally important. Without care, technologies like smartphones, swipe cards, and time clocks could use up precious government resources, without any ben- efit to schools. Conclusion EDUCATION The Strategic Impact Evaluation Fund, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help alleviate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health and job opportunities for people in developing countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief. The Evidence to Policy note series is produced by SIEF with generous support from the British government’s Department for International Development. THE WORLD BANK, STRATEGIC IMPACT EVALUATION FUND 1818 H STREET, NW WASHINGTON, DC 20433 Produced by the Strategic Impact Evaluation Fund Series Editor: Aliza Marcus Writer: Laura Burke