Audit of USAID/Haiti's Education Activities (Report No. 1-521-11-005-P)

OFFICE OF INSPECTOR GENERAL AUDIT OF USAID/HAITI’S EDUCATION ACTIVITIES AUDIT REPORT NO. 1-521-11-005-P JUNE 6, 2011 SAN SALVADOR, EL SALVADOR Office of Inspector General June 6, 2011 MEMORANDUM TO: USAID/Haiti Mission Director, Carleene Dei FROM: Regional Inspector General/San Salvador, Catherine Trujillo /s/ SUBJECT: Audit of USAID/Haiti’s Education Activities (Audit Report No. 1-521-11-005-P) This memorandum transmits our final report on the subject audit. We have carefully considered your comments on the draft report in finalizing the audit report and have included your response in Appendix II. The report contains two recommendations intended to improve the effectiveness and implementation of USAID/Haiti’s education activities. Management decisions have been reached on both recommendations. Please provide the Audit Performance and Compliance Division (M/CFO/APC) with evidence of final action to close the recommendation. I want to express my appreciation for the cooperation and courtesy extended to my staff during the audit. U.S. Agency for International Development Embajada Americana Urb. y Blvd. Santa Elena Antiguo Cuscatlan, Depto. La Libertad San Salvador, El Salvador Tel. (503) 2501-2999—Fax (503) 2228-5459 www.usaid.gov/oig CONTENTS Summary of Results ................................................................................................... 1 Audit Findings............................................................................................................. 3 USAID/Haiti Did Not Have a Plan for End-Use Checks of Classrooms .................. 3 Implementer Lacked a Process for Evaluating Training Effectiveness ................... 4 Evaluation of Management Comments..................................................................... 5 Appendix I – Scope and Methodology ...................................................................... 6 Appendix II – Management Comments ..................................................................... 8 Abbreviations The following abbreviations appear in this report: MOE Ministry of National Education and Vocational Training AIR American Institutes for Research PHARE Haitian Program Supporting Education Reform EDC Education Development Center, Inc. IDEJEN Haitian Out-of-School Youth Livelihood Initiative NGO nongovernmental organization ADS Automated Directives System SUMMARY OF RESULTS In Haiti, supply and demand for education are out of synch. Although a high social demand for schooling is present, the capacity of the state to provide education services and govern the education sector is weak. More than 85 percent of primary schools are managed by churches, nongovernmental organizations, and for-profit operators with minimal government oversight. In addition, primary school enrollment rates are low (close to 50 percent) compared with the average in Latin America and the Caribbean (94 percent). The January 12, 2010, earthquake that hit Haiti brought further difficulties to the education sector. A postearthquake assessment by the Ministry of National Education and Vocational Training (MOE) revealed that 80 percent of the schools in the west of Haiti were destroyed or damaged. The Ouest (West) Department includes the capital Port-au-Prince and is the department with 60 percent of all education activity. Given the immediate need for school structures before any improvement in education could be undertaken, USAID/Haiti moved to provide funding for the construction of provisional school structures. After the earthquake, USAID redirected the efforts of its two main implementers of education activities. USAID directed the American Institutes for Research (AIR), which was implementing the Haitian Program Supporting Education Reform (PHARE), to erect provisional school structures and provide training to educators until the end of AIR’s task order in July 2011. USAID/Haiti directed its second main implementer, Education Development Center, Inc. (EDC), to develop the capacity of the Haitian Out-of-School Youth Livelihood Initiative (IDEJEN) so that it could become an independent, stand-alone, nongovernmental organization (NGO) by the end of EDC’s agreement in March 2011. In January 2010, USAID/Haiti sent authorization letters to AIR and EDC to expend already obligated funds for postearthquake efforts. At the time, AIR had approximately $5 million and EDC $1.3 million in unexpended obligations. Later, USAID/Haiti obligated close to another $10.8 million for AIR to expend on the refocused education activities through July 2011, and another $1.2 million for EDC’s postearthquake activities through March 2011. As of December 31, 2010, total education obligations and expenditures for both programs combined (pre- and postearthquake) came to $44.2 million and $33.9 million. The objective of the audit was to determine whether USAID/Haiti’s education activities have strengthened the capacity of the Ministry of National Education and Vocational Training to increase access to quality basic education for Haitian children and out-of-school youth. The Office of Inspector General could not determine whether the USAID/Haiti-funded activities have strengthened the capacity of the Ministry of National Education and Vocational Training due to scope limitations imposed on the audit. Political unrest following contested election results caused the scope limitations. Despite the scope limitations, the auditors obtained reasonable assurance that AIR completed 322 classrooms. The audit also determined that EDC advanced the goal of IDEJEN’s becoming an independent, stand-alone, local NGO. Further, the audit disclosed two weaknesses in the implementation of the postearthquake programs:  USAID/Haiti did not have a plan in place to perform end-use checks of classrooms (page 3).  AIR lacked a process for evaluating the effectiveness of its training (page 4). 1 The audit team recommends that USAID/Haiti: 1. Develop and implement a procedure to conduct intermittent end-use checks for classrooms procured under the program (page 4). 2. Require its implementing partners to establish and implement procedures for systematically following up with training participants to assess the impact and effectiveness of training (page 4). Detailed findings appear in the following section. USAID/Haiti agreed to implement the recommendations and has developed specific plans to address them. Management decisions have been reached on both recommendations. Our evaluation of management comments is on page 5. Appendix I contains information on the audit scope and methodology; USAID/Haiti’s comments are included in Appendix II. 2 AUDIT FINDINGS USAID/Haiti Did Not Have a Plan for End-Use Checks of Classrooms Automated Directives System (ADS) 324.5.6, “End-Use Checks,” requires missions to confirm that USAID-procured commodities are being used as specified in agreements. ADS states that the mission shall carry out, or arrange to have carried out, end-use checks on commodities to confirm their use in accordance with the requirements of the underlying agreement. It is not necessary that every commodity be checked. The mission may determine a representative sampling or percentage of commodities to be checked. Contrary to ADS 324.5.6, the mission did not carry out end-use checks to verify that the completed classrooms were being used for their intended purpose. USAID/Haiti reported that under PHARE, AIR had constructed 322 classrooms (like the one shown below). PHARE officials had hired a local construction subcontractor and were keeping records on each temporary school site built. The designs were prefabricated to provide for a cement foundation, metal roofs, and partitioned classrooms with desk space for up to 50 children. The audit team visited one of the school sites with 18 new classrooms for primary and secondary school students. At left stands a school structure built on the site of classrooms that were destroyed in the earthquake. At right secondary school students attend classes in a USAID-constructed classroom. (Photos by Office of Inspector General, December 2010) USAID/Haiti provided evidence demonstrating that it performed monitoring during the construction of the schools and in some cases hosted public ceremonies to hand over the schools. However, after the schools were handed over, the mission did not carry out end-use checks to verify that the classrooms continued to be used as originally intended. This important step was overlooked because USAID/Haiti did not put a plan in place to perform these types of checks. It should be noted that USAID/Haiti’s Financial Management Office mentioned this as a finding in its recent Federal Managers’ Financial Integrity Act of 1982 annual certification. Moreover, although no plan is in place for end-use checks, there is evidence that the USAID/Haiti Education staff is conducting site visits at certain school sites. 3 AIR planned to construct an additional 300 classrooms by July 2011, bringing the total to 622 classrooms. USAID/Haiti funded the construction of these 622 classrooms to allow children to attend school, providing basic educational services to help them cope with the postdisaster situation and return to normalcy as quickly as possible. AIR’s technical proposal states that the transitional structures are expected to meet standards leading to the provision of a safe, secure, and healthy environment for learning. Given that many Haitians are still living in tents, the risk of misuse of the school structures is high. To mitigate the risk that schools built with USAID funding might be used as shelter and not as schools, we make the following recommendation. Recommendation 1. We recommend that USAID/Haiti develop and implement a procedure to conduct intermittent end-use checks for classrooms procured under the program. Implementer Lacked a Process for Evaluating Training Effectiveness ADS 253.3, “Policy Directives and Required Procedures,” require missions that expend USAID funds for participant training to design and implement the training for results and impact. Specifically, missions must report on their participant training activities as part of their broader performance measurement, evaluation, and reporting requirements. According to ADS 203.3.2, “Performance Management,” USAID missions and offices and their assistance objective teams are responsible for measuring progress toward the results identified in the planning stage to achieve foreign assistance objectives. Under PHARE, AIR was providing training to educators but had not developed a formal system for evaluating the effectiveness of the training. The training programs being conducted were both pedagogical and psychosocial in nature. As of November 30, 2010, AIR reported that it had delivered training to 56 individuals after the January 2010 earthquake. However, because AIR did not develop a systematic process to obtain feedback from those attending training, AIR cannot determine the effectiveness of the training. USAID and PHARE officials agree that feedback is needed to evaluate the effectiveness of the training. As a result, USAID/Haiti and its partner do not have reliable information on the degree to which the training programs were having the desired impact or could be tailored to the needs of the participants. Assessing the impact of training may yield greater returns on training investments and provide management better information to determine future training needs. Recommendation 2. We recommend that USAID/Haiti require its implementing partners to establish and implement procedures for systematically following up with training participants to assess the impact and effectiveness of training. 4 EVALUATION OF MANAGEMENT COMMENTS In response to the draft report, USAID/Haiti agreed to implement and has developed specific plans to address both recommendations. Regarding Recommendation 1, the draft audit report recommended that USAID/Haiti develop and implement a procedure to conduct intermittent end-use checks for classrooms procured under the program. The mission agreed with the recommendation and has already developed and implemented a plan of action to address the issue. As of April 18, 2011, USAID/Haiti reported that it had performed end-use checks on 19 of the 56 schools built after the January 12, 2010, earthquake (or 143 out of 322 classrooms). According to the mission, its education team selected these schools randomly and visited without notifying the implementer. The mission stated that it would make available to the Regional Inspector General’s Office a final report on the end-use checks on or about April 29, 2011. USAID/Haiti also indicated it would perform end-use checks on the new schools that will be inaugurated at the end of June 2011. On the basis of the mission’s described actions, a management decision has been reached on Recommendation 1. Regarding Recommendation 2, the draft audit report recommended that USAID/Haiti require its implementing partners to establish and implement procedures for systematically following up with training participants to assess the impact and effectiveness of training. USAID/Haiti agreed with the recommendation. The mission stated that it is working with the implementer to ensure that appropriate tools are used to measure the results and impact of training. USAID/Haiti indicated it would also carry out random spot checks of various trainings for the remainder of the project and report to the Inspector General’s Office by the end of May 2011. On the basis of the mission’s described actions, a management decision has been reached on Recommendation 2. 5  Appendix I SCOPE AND METHODOLOGY Scope The purpose of the audit was to determine whether USAID/Haiti’s education activities have strengthened the capacity of the Ministry of National Education and Vocational Training to increase access to quality basic education for Haitian children and out-of-school youth. The Regional Inspector General/San Salvador conducted this performance audit in accordance with generally accepted government auditing standards, 1 except for the following impairments to the scope of the audit: Because of the political unrest caused by the election results, from December 7, 2010, through December 14, 2010 (the date the audit team left Haiti), audit team members were not permitted to leave the premises of their hotel or the U.S. Embassy. As a result, the audit team was not able to conduct planned interviews with Ministry of Education officials and implementing officials, nor could the audit team conduct planned field site visits to observe the work taking place. Because of these scope impairments, we based our conclusions on the items we were able to test prior to the political unrest and on documentation that the implementers provided during and after the fieldwork. In assessing the evidentiary value of these sources and materials, we considered the scope limitations above and obtained corroborating information whenever possible. As a result, the Office of Inspector General could not determine whether USAID/Haiti- funded activities have strengthened the capacity of the Ministry of National Education and Vocational Training. Audit fieldwork was conducted at USAID/Haiti and implementing partner offices in Port-au-Prince on December 6 and 7. From December 8 to December 14, audit fieldwork at the sites of the implementers was suspended, and communication was conducted through teleconferences and e-mails. The audit covered the period January 12, 2010, through December 14, 2010, and focused on the implementation of PHARE by AIR and of the IDEJEN project by EDC. The focus was on the activities undertaken after the earthquake because previous work was not likely to be verifiable. In planning and performing this audit, we assessed the mission’s controls related to its education activities. The management controls identified included the mission’s performance plan and report, program progress reports, a strategy document, and the fiscal year 2010 self- assessment of management controls as required by the Federal Managers’ Financial Integrity Act of 1982. 2 We began reviewing the program results reported by AIR for the number of classrooms built and the training provided after the earthquake. We tested supporting documentation for 195 of the 322 classrooms and visited 18 of the classrooms. We were not able to perform a review of the training because of the political unrest described above, caused by the election results. We also began assessing whether EDC was on track in preparing IDEJEN to be an independent, stand- alone, local NGO. 1 Government Auditing Standards, July 2007 Revision (GAO-07-731G). 2 Public Law 97-255, as codified in 31 U.S.C. 3512. 6  Appendix I Methodology To determine whether AIR and EDC were achieving the goals stated above, the audit team interviewed USAID/Haiti staff to gain an understanding of the program’s history and status. The audit team reviewed relevant agreements, modifications, program descriptions, progress reports, and performance plan reports. The audit began reviewing the work accomplished as reported in the implementing partners’ reports and began comparing actual accomplishments with the specific targets as defined in the task order, agreement, technical proposals, and progress reports. This comparison entailed examining supporting records, including documentation of classrooms built, for evidence that the project had, in fact, achieved its intended results. Before the political unrest, we also interviewed staff of both implementing partners and conducted one site visit to verify that 18 classrooms had been built. To determine whether the mission reported accurate and complete information, we interviewed mission and implementing partner personnel. We judgmentally selected 66 of the 322 classrooms said to have been built under PHARE (AIR), but were able to visit only 18 classrooms. We also made a larger judgmental selection of 195 classrooms for documentation review. We also requested documents from IDEJEN that would indicate its progress in becoming an independent NGO—for example, documentation of alternative funding sources, a strategy, etc. The audit team was not able to complete this work because of the political unrest. In addition, we reviewed applicable policies, procedures, and management controls related to management for results, including ADS Chapters 203, 253, and 324. We evaluated the mission’s compliance with relevant program management controls and policies. 7  Appendix II MANAGEMENT COMMENTS To: Catherine Trujillo, RIG/ San Salvador From: Carleene Dei, Mission Director Date: April 18, 2011 Subject: Mission Response to the Draft RIG’s Audit of USAID/Haiti’s Education Activities (Draft Report No. 1-521-11-00X-P) This memorandum represents USAID/Haiti’s formal response to the draft audit report of RIG’s Audit of USAID/Haiti’s Education Activities (Draft Report No. 1-521-11-00X-P). USAID/Haiti appreciates the time and effort of the RIG staff in carrying-out this audit and for the professional and cooperative manner in which the audit was conducted. General Comment: We agree with all two (2) recommendations provided in the audit report and believe they will enable us to improve the Education activities in strengthening the capacity of the Ministry of National Education and Vocational Training to increase access to quality basic education for Haitian children and out-of-school youth. The audit report outlines two specific recommendations for USAID/Haiti’s education activities. The first recommendation is regarding end-use checks required by Automated Directives System (ADS) 32.5.6, which requires missions to confirm that USAID-procured commodities are being used as specified in agreements. The second is regarding USAID policy ADS 253.3, which requires missions that expend USAID funds for participant training measure trainings for results and impact. As requested, action plans and timeframes are developed below in order to implement all the recommendations cited in the report in a timely manner. Recommendation No. 1: We recommend that USAID/Haiti develop and implement a procedure to conduct intermittent end-use checks for classrooms procured under the program. The timing of the RIG education audit in December 2010 was at the end of a tumultuous year. Although a rigorous plan had been put in place for the construction of semi-permanent schools, USAID/Haiti had not put forward a plan to conduct end-use checks. After the RIG visit, the team developed a plan of action to address the issue swiftly. To date, USAID/Haiti has performed 8  Appendix II end-use checks on 19 out of the 56 schools built after the January 12, 2010 earthquake (or 143 out of 322 classrooms). The education team visited randomly selected schools in Port-au- Prince, Léogâne, and Petit-Goâve, interviewing school directors, teachers, students, and carefully checking to make sure USAID procured investments are used for the intended purpose as outlined in the agreement. The team selected these schools randomly and visited without notifying the implementer to make sure to adhere to the strictest statistical integrity. The team intends to finalize this report after a final trip to schools in Jacmel. After the visit to Jacmel, the education team will have visited every geographic area that USAID/Haiti has made investments under the current contract. A finalized report of the end-use checks will be made available to the Regional Inspector General’s Office by April 29, 2011. USAID/Haiti will also perform user- end checks on the new schools that will be inaugurated at the end of June 2011. Recommendation No. 2: We recommend that USAID/Haiti require its implementing partners to establish and implement procedures for systematically following up with training participants to assess the impact and effectiveness of training. USAID/Haiti agrees with the Audit recommendation to require the implementing partner to establish and put in place procedures to measure effective of trainings. The USAID/Haiti education team is working with the implementer to ensure appropriate tools are used measure the results and impact of the educator trainings sessions. USAID/Haiti will carry out random spot checks of various trainings for the remainder of the project and will report to the Inspector General’s Office by the end of May 2011. Beyond the important need to adhere to legal directives, the recommendations outlined by Regional Inspector General’s report are in line with USAID/Haiti’s general effort to measure impact, carefully track investments, and chronicle lessons learned. USAID/Haiti thanks the Regional Inspector General’s Office for the thoughtful report prepared and the cooperation extended to USAID/Haiti staff through this necessary process. 9 U.S. Agency for International Development Office of Inspector General 1300 Pennsylvania Ave., NW Washington, DC 20523 Tel: (202) 712-1150 Fax: (202) 216-3047 www.usaid.gov/oig