Rechèch kalitatif pou amelyore evalyasyon pwogram Finansman Baze sou Rezilta yo: Pwomès la ak Reyalite a

QUALITATIVE RESEARCH TO EN HANCE THE EVALUATION OF RESULTS-BASED FINANCING PROGRAMMES: THE PROMISE AND THE REALITY D I S C U S S I O N P A P E R F E B R U A R Y 2 0 1 6 Fabian Cataldo Karina Kielmann 103670 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized QUALITATIVE RESEARCH TO ENHANCE THE EVALUATION OF RESULTS-BASED FINANCING PROGRAMMES: The Promise and the Reality Fabian Cataldo & Karina Kielmann February 2016 Health, Nutrition and Population (HNP) Discussion Paper This series is produced by the Health, Nutrition, and Population Global Practice. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. For information regarding the HNP Discussion Paper Series, please contact the Editor, Martin Lutalo at mlutalo@worldbank.org or Erika Yanick at eyanick@worldbank.org . RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org . © 2016 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. ii Health, Nutrition and Population (HNP) Discussion Paper Qualitative Research to Enhance the Evaluation of Results-Based Financing Programmes: The Promise and the Reality Fabian Cataldo a & Karina Kielmann a,b a London School of Hygiene and Tropical Medicine, London, UK b Institute for Global Health and Development, Queen Margaret University, Edinburgh, UK Paper prepared for the Health Results Innovation Trust Fund World Bank, Washington, DC, February, 2016 Abstract : This Discussion Paper presents the approach, findings, and recommendations from a desk review of the qualitative research conducted within Results-Based Financing programmes (RBF) under the Health Results Innovations Trust Fund (HRITF). The review included 17 studies conducted in Benin, Burundi, Cameroon, DRC, Ethiopia, Haiti, Kenya, Kyrgyzstan, Nigeria, Rwanda, Tajikistan, Tanzania, Zambia, and Zimbabwe. The studies reveal a body of high quality work that is consistent with the conceptual framework of RBF schemes, supported by political will, resources, and research capacity. Strengthening the added value of qualitative inquiry in on-going and future qualitative studies may be enabled by small shifts in thinking and practice, in line with a qualitative research paradigm. First, in order to better ground research in an existing country and system specific context, some interrogation of constructs and posited relationships in the existing conceptual framework for intervention/evaluation may be required. Second, to enable more in-depth and richer data that documents working practices and relations under RBF schemes, training of local researchers should place stronger emphasis on entry to the field, gaining trust, building rapport, and sustaining a dialogue with key informants. Third, smaller, more intensive and focused studies targeting fewer sites and smaller samples - but addressing a wider range of methods and informants within the health system - are likely to yield richer data that can support the understanding of how health workers and managers are responding to schemes, and what impact schemes have on service volumes and outputs. Keywords : Results-based financing, Qualitative, Methods. Disclaimer : The findings, interpretations and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Executive Directors, or the countries they represent. Correspondence Details : Dr. Fabian Cataldo - email: fabiancataldo@gmail.com Dr. Michael Kent Ranson - email: mranson@worldbank.org iii Table of Contents RIGHTS AND PERMISSIONS ........................................................................................ II FOREWORD .................................................................................................................. V ACKNOWLEDGMENTS ............................................................................................... VI PART I – APPROACH AND METHODOLOGY: ............................................................. 7 I NTRODUCTION .............................................................................................................. 7 METHODOLOGY ............................................................................................................. 8 Purpose .................................................................................................................... 8 Selection Criteria ...................................................................................................... 8 Methods ................................................................................................................... 9 Desk-based review: .............................................................................................. 9 Semi-structured telephone interviews: .................................................................. 9 Ethical Considerations ............................................................................................ 10 PART II– RESULTS:..................................................................................................... 11 Conceptualisation ................................................................................................... 11 Logistics and Planning ........................................................................................... 14 Training .............................................................................................................. 14 Sampling ............................................................................................................ 15 Recruitment of Study Participants ....................................................................... 15 Data Quality Control ........................................................................................... 16 Ethical Considerations ........................................................................................ 16 Data Collection ....................................................................................................... 16 Study Sites ......................................................................................................... 16 Study Participants ............................................................................................... 17 Methods .............................................................................................................. 18 Data Collection Instruments ................................................................................ 19 Structured questioning ........................................................................................ 20 Open-ended questioning .................................................................................... 21 Analysis and Presentation of Results ..................................................................... 21 Knowledge Translation ........................................................................................... 23 Limitations .............................................................................................................. 24 PART III - CONCLUSIONS ........................................................................................... 25 PART IV – OPPORTUNITIES AND RECOMMENDATIONS TO STRENGTHEN THE QUALITY OF QUALITATIVE RESEARCH IN RBF STUDIES ...................................... 26 Conceptualisation of Studies .................................................................................. 26 Logistics and Planning ........................................................................................... 27 Data Collection ....................................................................................................... 30 REFERENCES.............................................................................................................. 32 APPENDIX: DOCUMENTS INCLUDED IN THE REVIEW ............................................ 34 iv FOREWORD The Health Results Innovations Trust Fund (HRITF), established in 2007 with funding from Norway and the UK, supports the design, implementation, monitoring and evaluation of Results-Based Financing (RBF) programmes with a particular focus on improving maternal and child health outcomes for accelerating progress towards reaching MDGs 1c, 4 and 5. In addition, the HRITF supports activities that build country institutional capacity for RBF and broaden the evidence base for implementing successful RBF mechanisms. A portfolio of rigorous impact evaluations (IEs) have been designed to demonstrate whether these programmes can improve the quantity and quality of health services delivered as well as health outcomes at the population level. As a complement to these IEs, a number of qualitative studies have been carried out to learn about processes of implementation and intermediate components in the causal pathway. Process evaluations and studies of political economy have been supported under the Learning from Implementation programme of work. Additionally, many smaller qualitative studies have been conducted in association with HRITF implementation design or impact evaluation. This Discussion Paper focuses on a synthesis review of the qualitative components of RBF-related studies and IEs commissioned by the World Bank under the Learning from Implementation programme and through HRITF. v ACKNOWLEDGMENTS The authors would like to thank the following individuals and institutions for their timely and pertinent contributions to this review: Dr. Clare Chandler, Prof. Janet Seeley, Dr. Dinesh Nair, Dr. Michael Kent Ranson, the Health Results Innovations Trust Fund, the World Bank, the London School of Hygiene and Tropical Medicine, and the study staff and investigators interviewed during this review. The authors are grateful to the World Bank for publishing this report as an HNP Discussion Paper. vi PART I – APPROACH AND METHODOLOGY: I NTRODUCTION Results Based Financing (RBF) is defined as a cash payment or non-monetary transfer made to a national or sub-national government, manager, provider, payer or consumer of health services after predefined results have been attained and verified (World Bank 2013). RBF is an umbrella term that encompasses various types of interventions that target beneficiaries (for example, conditional cash transfers), providers (for example, performance-based financing), and country governments (for example, cash on delivery) (Musgrove 2010). Project experiences to date show that multiple systems components and their interactions affect health workers' motivation and capacity to implement activities for improved service delivery (World Bank 2014). Likewise, the health-care seeking behaviours of those who are meant to benefit from the health system interventions are determined by many factors, and may be influenced by ‘demand-side’ incentives. Programmes utilizing RBF involve complex health systems interventions (Oxman & Fretheim 2008). Bringing in an RBF scheme is inevitably going to impact on issues around financing, governance, and management, including of the health workforce more directly. Dealing with the human impact of these interventions also means dealing with unpredictability. This is where the question of qualitative research comes in: looking at the subjective experiences of health care workers (HCWs) and patients accessing the facilities, the agency of the actors involved, and the way multiple systems components impact on human experience in the context of improved service delivery. There is consensus that qualitative research methodologies can enhance the understanding of how interventions are implemented within the context of local health systems ( Green & Thorogood 2013) , and how they do or do not work towards desired outcomes . As a result, the impact evaluations have increasingly used qualitative research methodologies to understand how RBF mechanisms work, and what intermediate components are relevant in the causal pathways between intervention and outcomes. However, to maximize the potential for qualitative research to generate relevant and meaningful data about processes and mechanisms of effect, research has to be fit-for- purpose, adapted to local context and capacity, asking the right questions, and using appropriate, rigorous, and ethical methods of data collection and analysis. Questions arise in the context of the evaluation framework of RBF projects: What constitutes ‘good qualitative research’? Where does it add value to the evaluation of RBF schemes? How is qualitative research best evaluated? A recent review by Reynolds et al. (2011) identified two dominant narratives in the current literature looking at the quality of qualitative research: one focusing on the outputs and the other on the processes of qualitative research. They recommend that the strengths of both the output-oriented and process-oriented approaches be brought together to create evaluation guidance that reflects core principles of qualitative research, but also responds to expectations of the global health field for explicitly assured quality in research (Reynolds et al. 2011). 7 In this synthesis review of a few RBF studies supported by HRITF, we do not attempt to assess the validity 1 of results achieved as part of RBF evaluations, as we did not access ‘raw’ data collected. Instead, we focus our approach on the processes involved in obtaining, managing, and analysing the data. METHODOLOGY A desk-based review was conducted focusing on the qualitative components of 17 RBF studies linked to HRITF (see Appendix 1). This review was complemented by a more in- depth investigation of six studies through a case study approach (Stake 1995, Crowe et al. 2011, Yin 2014). Purpose The objectives of the synthesis review were to assess the research approach and design, methodological rigor, reporting, and conceptual depth of the qualitative component of studies focusing on RBF; explore opportunities to improve the quality of work conducted as part of the qualitative elements of RBF projects; and offer recommendations to the HRITF team to improve the quality of qualitative projects focusing on RBF initiatives. Moreover, the additional case study approach aimed to gather further information on the experiences, challenges, and perceived benefits and outputs of conducting qualitative research within six RBF-projects conducted between 2011 and 2015. Selection Criteria Projects were selected because they included a qualitative approach or methods focusing on RBF, and were funded directly or indirectly by HRITF. There were 17 studies included in the synthesis review. These studies were identified in collaboration with the HRITF team. Our review included studies conducted in the following countries: Benin, Burundi, Cameroon, DRC, Ethiopia, Haiti, Kenya, Kyrgyzstan, Nigeria (n=4), Rwanda, Tajikistan, Tanzania, Zambia, and Zimbabwe. Six of these studies (Cameroon, Ethiopia, DRC, Nigeria, Zambia, Zimbabwe) were selected for a more in-depth assessment. These studies were selected either because they were included in the ‘Learning from Implementation’ programme (World Bank 2014) or in relation to pragmatic criteria, which included the stage of the research at the time of the review (that is, having reached at least the data analysis stage) and the type of documentation available including, for example, study protocols, Institutional Review Board (IRB) submissions, data collection tools, reports and publications. Note however, that the studies were not directly comparable with respect to the application of qualitative techniques for data collection, management, and analysis. They were developed at different times by different teams, hence do not constitute a coherent portfolio. 1 Validity here refers to whether the qualitative data adequately and accurately reflect the reality that they were intended to describe. 8 Methods In the review of the qualitative components of these projects, a flexible and pragmatic approach was adopted, taking into account the processes involved in carrying out qualitative research as embedded in the broader aims of the projects. The methods employed were a desk-based review of available documents for each of the 17 studies followed by a semi-structured telephone interview with at least one investigator of six research projects selected for a more in-depth assessment (that is, principal investigator, or co-investigators if the principal investigator was not available). Review and data collection tools were developed in order to review each study and conduct the interview. The first section of this tool focuses on a descriptive profiling to situate the qualitative elements in relation to the larger research or evaluation. The second section was designed to examine the quality of qualitative research, mainly focusing on study processes prior to, during and after data collection. The third section focused on identifying opportunities to strengthen the place and quality of qualitative research as well as any ‘missed opportunities’ to strengthen the qualitative component, and identifying where and how the qualitative endeavour may be limited by the overall research approach. The first and third sections of the tools were developed by the authors of this review. The second section (focusing largely on study processes) was adapted from a tool initially developed for the ACT consortium (Reynolds et al. 2013). Each sub-section was adapted for this review and follows a similar pattern of investigation (i.e. following the study process from conception to dissemination). As part of the tool development, a more in-depth case study approach and interview guide were developed by Karina Kielmann and Fabian Cataldo, and reviewed by all investigators. Desk-based review: The desk-based review of documents related to the 17 studies was conducted by one of the investigators and six of these projects (that is, Learning from Implementation studies selected for a more in-depth case study approach) were cross-reviewed by a second investigator, discussing any divergent findings. Semi-structured telephone interviews: The target group for the telephone interviews included project leads, investigators and/or researchers of six selected projects for a more in-depth case study approach. Following on from the desk-based review, one of the review investigators scheduled a telephone interview with the Principal Investigator, Co-Investigator or project leader for each of the six projects. Five telephone interviews were conducted with project leads and/or investigators, and one investigator was not available for an interview but provided responses in writing. Each interview lasted between 40 and 60 minutes, and focused mainly on information that was not readily available at the time of the desk-review. Data collected during the semi- structured interviews helped to situate and contextualize specific aspects of the research project, elicit more information in relation to the context of the research, the strengths and weaknesses identified in planning, logistics and implementation of the research, and how the research contributed to understanding processes and outputs for each project. 9 Ethical Considerations This review was approved by the Ethics Review Committee of the London School of Hygiene and Tropical Medicine (Ref. 8803). Rules for informed consent were respected throughout data collection, and data gained through telephone interviews were anonymized to protect confidentiality of the informants. 10 PART II– RESULTS: The results section is organized around the processes that characterize the development and conduct of qualitative research (Figure 1) - namely conceptualisation, planning, data collection, analysis, and knowledge translation (Kielmann et al. 2011). This review focused mainly on issues around conceptualisation, planning and data collection, as limited access was available to the ‘raw’ data and documentation of the dissemination processes for each study. Figure 1: Cycle of research enquiry Source: Authors The majority of the RBF studies reviewed were conducted in African countries (n=14); others were located in Tajikistan, Kyrgyzstan, and Haiti. Most were small- to medium-sized studies, generally conducted in the context of an impact evaluation. Overall, studies aimed to explore and document the experiences of health care workers (HCWs), patients, and decision makers in relation to RBF implementation. In the following sections we cite documents listed under Appendix 1 when referring to a specific example from one of the studies – for example, when referring to the research proposal from the Benin study, we use the citation ‘1.1: Benin’. Conceptualisation All studies were cross-sectional – they tended to use mixed methods, and some were explicitly described as ‘case studies’ (for example, 9.1: Nigeria, 16.1: Zambia). In several studies (for example, 3.1: Cameroon, 9.1: Nigeria, 17.1: Zimbabwe) the RBF conceptual framework (World Bank 2013; Figure 2) constitutes an explicit starting point for the evaluation and/or qualitative component of the study, although, as stated above, projects did not necessarily adopt a coordinated approach to applying the framework to the research. Conceptualisation Logistics and Planning Data Collection Analysis and presentation of results Knowledge Translation 11 Figure 2: RBF Conceptual Framework World Bank (2013) Using Results-Based Financing to Achieve Maternal & Child Health: Progress Report. The RBF conceptual framework (World Bank 2013; Figure 2) was recently developed by the HRITF team at the World Bank. It was primarily set up in an effort to rationalise and improve RBF interventions. This model identifies several contextual levels linked to RBF interventions: individual/behavioural, health facility (HF), health system (HS), community and political economy. We used these categories to determine the contextual level at which the 17 studies included in the review were situated. Most of the studies were located at the level of Health Facilities (n=9). Others were focusing on individual behaviours (n=2), community (n=4) and political economy (n=2). 2 Several studies included in the review used the RBF conceptual framework to explore the impact of RBF schemes on the health system, and more specifically on HCWs, and in turn how changes in organizational and HCWs’ behaviours influence service outputs. The Zimbabwe study (17.1), for example, uses the RBF framework in order to articulate a key study design assumption, which links RBF interventions to health outputs, quality of health services provided, HCWs motivation, and improved access to services in the community. 2 Studies focusing on community and political economy were differentiated as follows: by community, we refer to studies that specifically looked at barriers or enablers to uptake of services by beneficiaries. By political economy, we refer to studies that examine the responsiveness and political will amongst key policy donor and high-ranking government officials towards RBF schemes. 12 In one of the studies conducted in Nigeria, an investigator described how the RBF framework was used and ‘customized’ during the conceptualisation of the study: “There are some things I wanted to improve in that conceptual framework: the conceptual framework’s focus on how performance changes with PBF [Performance Based Financing] . For example they talk about behavior change at the health centre level but as we are interested in why that behavior change happens, we needed to dig further into details of management practices at the health centre. So instead of focusing on the effects or changes that we can see in the health centre, we wanted to see why those changes actually can happen. So I customized this framework a bit.” (9.7: Nigeria) When adopted in mixed method studies 3 , qualitative research components can be placed at different stages of a project. Timing is often linked to the purpose of the research. In some of the studies we reviewed, qualitative research was conducted early on as a way of exploring context and understanding on-going processes, for example, in the case of the social assessment conducted in Tanzania prior to RBF interventions (15.1: Tanzania). In other studies, the research was used primarily to explain quantitative trends observed, for example, variation in the performance of health facilities under RBF schemes, as determined through a set of service output indicators. An example of this is the Nigeria study providing a case study analysis on the best vs. poor performers based on results from an on-going evaluation into HF-related performance linked to RBF pilot schemes (9.1, 9.6: Nigeria): “We wanted to investigate ‘what is going on?’ or ‘why this is going well, why this is not going well, why there is a difference in performance?’. So qualitative analysis can really unpack these things.” (9.7: Nigeria) Another example from Cameroon illustrates how the qualitative element was intended to generate contextual data in order to complement quantitative data from a larger impact evaluation: “We felt that doing this qualitative study and making sure that we do cover these different contexts would contribute to the overall impact evaluation and help us once we do the survey to interpret these results.” (3.3: Cameroon) We found no instances in which the qualitative research served to help identify or operationalise the constructs relevant to understanding the impact of RBF schemes on health systems components, and specifically the health workforce. Such research would have helped to understand locally relevant definitions and sources of ‘motivation’, functional as well as more context and culture-specific dimensions of ‘quality’ in performance, but also specific constructs to characterise organisational culture, including, for example, dimensions of management and leadership style – hierarchical, vertical, 3 By ‘mixed methods’ studies, we refer to studies which employ a mix of quantitative and qualitative methods to triangulate data collection on a particular topic. As using methods that assess or measure variables of interest through different assumptions, questions, and ways of eliciting data generates different types of data, triangulation allows for a variety of perspectives as well as multiple dimensions of the phenomenon at hand to be explored. 13 horizontal – that reflect broader societal norms based on gender, occupation, status and so on. The design of the six Learning from Implementation studies appeared to be better supported in terms of resources and more emphasis on qualitative components as part of the overall research approach. This was expressed by several of the investigators interviewed: “I’m not sure how far we would have actually gone in doing this qualitative study if funding was not made available by HRITF.” (3.3: Cameroon) In the 17 studies we looked at, only a few were assessing context before implementation of the scheme; the majority were designed to supplement the process or impact evaluation mid-way through implementation, and a few used performance indicators for HFs to examine retrospectively what could explain variations. Like any other models, the RBF conceptual framework contains assumptions and hypotheses, which frame the methodology. It is instrumental in supporting several of the assumptions made in the qualitative work, for instance the causal relation between the provision of monetary incentives and behaviour change, HCWs motivation, and/or HF overall performance. These assumptions are also reflected in the data collection instruments that were used in the studies reviewed - as discussed in Section 4.3: Data Collection . Logistics and Planning A critical component of research is logistics and planning leading to data collection itself. Even when the conceptual approach and methodology is sound, it may be impossible to conduct the research as planned because of the specific constraints relating to resources, time, local research capacity, and gaining access and trust in specific sites. We present some considerations in relation to training, sampling, recruitment, data quality and ethics processes. Training In terms of capacity to undertake the research, some of the project documents we reviewed included details of training organized as part of the preparation for data collection. Research teams were often structured around task team leaders and senior consultants with technical expertise, but not necessarily in qualitative research. These were supported by one or more local investigators and several field researchers hired on a short-term basis (for example, 4.1: DRC, 3.1: Cameroon, 9.1: Nigeria, 17.1: Zimbabwe). The responsibility for data quality, analysis and reporting often lay with external consultants. Those collecting qualitative data, that is, research assistants or fieldworkers, were generally trained through short intensive workshops led by one of the international investigators. From the limited documentation available, we noted that the training focused on correct conduct in the field, however this did not necessarily include an emphasis on creating and sustaining relations in the field. 14 We were pleased to see good practice examples of pilot and pre-testing of instruments, for instance in the Zambia study focusing on HCWs motivation, where detailed training plans and a report were available to review (16.8, 16.9: Zambia). Sampling In terms of sampling of research participants, and bearing in mind that the logic of qualitative sampling is different from quantitative sampling, we noted that some of the strategies were not clearly described. While large probability samples attained through random sampling techniques are appropriate in quantitative studies that aim to provide some conclusions regarding how representative a trend is of the wider population, or what differences observed are statistically relevant, qualitative sampling is generally purposive; sample sizes are, on the whole, much smaller, because the aim is to select information- rich cases that can explain variation across a set of informant groups/sets that the researcher deems relevant to the outcome of interest. Other than stratifying the sample of informants according to facility performance (for example 9.3: Nigeria), other features differentiating the sampling of informant groups were rarely mentioned. An ‘ideal’ focus group discussion (FGD) has a clear strategy for recruitment of participants based on the research questions, including details of the socio-demographic or professional profile of research participants. Inclusion criteria for FGDs were not always clearly stated in the studies included in the review, and the sampling strategies for community members and gatekeepers were often based on numeric rather than substantive considerations. An important point is the lack of justification for a relatively large number of respondents interviewed in some of the studies reviewed. A baseline qualitative study in Kyrgyzstan, for example, conducted interviews with 106 individuals (8.1: Kyrgyzstan), while the Cameroon qualitative study had planned to include 168 individual interviews in addition to 67 FGDs (3.1: Cameroon). In some instances, fewer interviews would have been likely to have generated similar findings, and would have been adequate for reporting on the processes. In addition, generating a more manageable volume of data may allow for a more in-depth reflection based on the information gathered. Recruitment of Study Participants Protocols for recruitment of participants were in place. A protocol from one of the Nigerian studies, for instance, adequately describes how to recruit research participants (12.3: Nigeria). Generally, research participants were not directly approached, but fieldworkers went to gatekeepers first or used snowballing approaches to recruit study participants. In several instances, it is not clear how gatekeepers such as ‘community leaders’ actually represent the communities around them, and how they are defined by the community themselves (for example, ‘elected representatives’, volunteers, political or religious leaders, etc.). The inclusion of relatively large samples of study participants has implications for study participant recruitment. Having to recruit large numbers of informants may compromise procedures that are critical for enabling good quality, in-depth data including, for example, adequate processes of rapport building, as well as finding the time and space conducive for the conduct of an in-depth interview and adequate time to do justice to the data. 15 Data Quality Control Few studies included detailed information on how to ensure data quality, which involves considerations ranging from creating conditions that are conducive for a good interview, to adequate procedures for recording and reporting of data, as well as reviewing the quality of the data. One study for which we had ample documentation, namely the Zambia project, provided concise, clear protocols on communication (16:5), data collection (16:4), data storage (16:6) and selection of informants (16:7). Having good protocols in place is not a guarantee, however, for obtaining ‘thick’ data – in reference to Geertz’s “thick description” (1973), that is data that provide rich, in-depth contextual information. This often relies on more intensive interviewing and rapport- building techniques that may be hard to acquire within a relatively short period of time. Ensuring consistently good quality of data may also be contingent on being able to perform ‘reality checks’ on the data periodically to assess whether the data collection guides and instruments as well as the interviewing techniques are eliciting information that is in-depth, coherent, and ‘makes sense’ in the light of what is known about the context. Hence, another question relevant to this review was when should one assess the quality of the data – often, transcripts are only available for review at the end of data collection rather than mid-way through data collection, allowing for the team to critically review procedures as well as quality of data obtained. Ethical Considerations Most studies included in this review have undergone ethical review by the relevant national or international ethics review committees. However, what is mainly being reviewed by institutional review boards are standardized consent forms, data storage protocols, and issues of confidentiality that are concerned with mitigating institutional risk rather than ethical issues faced in the field during data collection. 4 These ethical issues involve rules of conduct in the field to be adhered to by researchers, but they also relate to the broader framing of projects. Given that researchers were often linked with the evaluation of a specific RBF scheme, it remains unclear from the review of documents and interviews with team members how data collectors and fieldworkers were perceived by informants. An ethical issue which would need to be addressed in the context of these studies is the possible association between RBF schemes and researchers (who may have been perceived as auditors, monitors, or sponsors), which may lead to potential conflict of interest and bias. Data Collection Study Sites The selection of study sites for the qualitative components of the studies was mostly associated with existing RBF schemes and/or impact evaluations. Sites were often selected along the same logic as that used for the impact evaluation, for example, to enable comparisons or uncover reasons for differences in outcomes in relation to location 4 Guidelines for considerations in building relations with and responsibilities towards research participants can be found in the “Ethical Guidelines for Good Research Practice” published by the Association of Social Anthropologists of the UK and Commonwealth (see http://www.theasa.org/ethics/guidelines.shtml). 16 (urban or rural), facility type (dispensary, health centre, or district hospital; for example, 6.1: Haiti) or in relation to performance criteria (high or low performance; for example, 9.1: Nigeria). Study Participants Study participants included in the studies reviewed were predominantly frontline health care workers. Figure 3: Study participants’ profile Source: Authors In most studies (14 out of 17), HCWs were the main unit of data collection and analysis. In five of these studies, HCWs were the exclusive focus of the qualitative data collection. This is justified through reference to the RBF conceptual framework (Figure 2), which pinpoints HCWs as the locus of behavioural change, mainly in relation to their motivation and performance. Nine of the 14 studies that focused on HCWs as study participants also included patients and community members. These studies included patients to explore, for instance, what changes could be made to the RBF intervention (for example, 11.1: Nigeria), or factors underlying the success or failure of RBF models on the demand-side (e.g. 13.2: Rwanda). Several studies that included community members did so to elucidate local perspectives and experiences in relation to RBF schemes (for example, 7.3: Kenya), or to evaluate more broadly the potential for RBF in the context of a specific location (for example, 2.1: Burundi). Relatively few studies included a broader spectrum of health worker cadres that represent or could speak to the dynamics of organizational change. For example, the protocol for one of the Nigerian projects (9.1: Nigeria) included mid-level managers and senior nursing staff as study participants. Another category of study participant included specific target groups that often served as a pool for key informant interviews (KII). For instance ‘community leaders’ were study participants in a number of projects. In the Cameroon study (3.2, 3.3: Cameroon), they included the president or leader of the community’s women’s group and community members who served as the community representative on health centre committees. In HCWs+Patients+Communit y (n=9) HCWs only (n=5) Decision/Policy Makers (n=2) Households (n=1) 17 the Rwanda study, they included presidents of community health care workers cooperatives, heads or deputy-heads of HF, and district health officers (13.2: Rwanda). One study (5.2, 5.3: Ethiopia) focused exclusively on high-level stakeholders, who are donor representatives in and outside the country, as well as government officials to assess the pre-implementation context for the RBF scheme. Methods A relatively limited range of qualitative methods was used, mainly interview methods relying on reported experience of what happened as opposed to more participatory and embedded methods that might document what is going on . The choice of methods seemed to be often made on pragmatic rather than methodological grounds. In most protocols, there was no explicit justification, for example, for the inclusion of FGDs in addition to - or instead of - individual interviews. Figure 4: Choice of methods Source: Authors As illustrated in Figure 4, most studies (16 out of 17) used individual interviews, and these were mainly used in combination with FGDs (n=9). A few studies (n=3) used interviews as a stand-alone method. This was the case for a study relying on stakeholder interviews to examine the pre-implementation context of RBF (5.2: Ethiopia), or KIIs to explore appropriate policy and institutional options for mitigating risks and improving the chances for successful implementation of RBF schemes (10.1: Nigeria). Individual interviews were also used exclusively in one study examining motivation amongst HCWs (16.11: Zambia). FGDs were often combined with individual interviews, for instance to explore the experience of RBF implementation amongst HCWs and local population groups (3.1: Cameroon). In Haiti, FGDs were used to triangulate data around revenue and human resources from a questionnaire (6.1: Haiti). In Kyrgyzstan, FGDs were used to gather information with women attending primary care facilities to elicit local perception of RBF schemes (8.3: Kyrgyzstan). In Benin, the research team conducted direct observations (in addition to questionnaires, FGDS, and interviews) to explore the attitude of health providers towards patients during consultations, in addition to data on time and movement mapping within the HFs (1.1, 1.2: Semi-struct. interviews and FGDs (n=9) Semi-struct. interviews and Questionnaires (n=2) Semi-struct. interviews and Direct Obs. (n=1) Household Survey (n=1) Structured or Semi-struct. interviews only (n=3) 18 Benin). These observations seemed constrained, however, by the attempt to quantify observed practices, instead of describing patient-providers interactions during patients’ visits to the HFs. Data Collection Instruments For some, but not all studies, data collection instruments were available for review (see Appendix 1). The guides and instruments reviewed are listed in Table 1. Table 1: Data collection instruments reviewed Source: Authors Most instruments for data collection are semi-structured; the categories of interest are set by researchers in advance, and the instrument contains a mix of closed- and open-ended questions (e.g. 9.2: Nigeria, 13.2: Rwanda, 16.2: Zambia). This is generally also the case for tools intended for in-depth interviews, which ideally should use open-ended questions, or simply a topic guide (list of areas to discuss and probe) to elicit more in-depth information. The instruments are organised according to a comprehensive set of themes that implicitly reflects the RBF conceptual framework. Hence, for example, questions directed at health workers on motivation relate to the assumed relationship between receiving monetary incentives linked to performance and improved motivation for HCWs (for example,16.2: Zambia). Some of the studies reviewed used concepts that derived from the RBF framework, but did not operationalise these further. In principle, it is good practice to use a conceptual framework to guide methodology and research questions, however, it is important to ensure that concepts, especially more abstract ones such as ‘autonomy’ (for example, 4.2: DRC), ‘changes in attrition of HCWs’ (for example, 7.2: Kenya), ‘performance’ (e.g. Data collection tool Number Studies Structured questionnaires: 3 1.2: Benin 1 2.3: Burundi 1 12.2: Nigeria 1 17.2: Zimbabwe Semi-structured interview guides: 1 7.2: Kenya 2 4.1: DRC 4 9.2: Nigeria 2 12.2: Nigeria 1 13.2: Rwanda 1 16.2: Zambia FGD guides: 2 13.2: Rwanda 1 17.2: Zimbabwe Observation guide: 1 1.2: Benin KII guides: 1 10.1: Nigeria 1 7:2: Kenya 7 17.2: Zimbabwe 19 3.1: Cameroon), are appropriately operationalized for use in data collection instruments. This means ensuring that the variables chosen to assess the concept are robust and qualitatively or quantitatively measurable, as well as locally meaningful. In one of the studies, there was thoughtful discussion of the literature that could be drawn on to develop indicators for assessing managers’ competencies – as good practice, a further question would be how to ensure that the dimensions adapted from available literature/tools would be locally appropriate and applicable (9.1: Nigeria). Structured questioning There are substantial issues with survey-based studies that try to understand how people think and behave. For some of the concepts used in the studies, such as ‘quality of care’, ‘job satisfaction’, and ‘motivation’, researchers used Likert scales (Likert 1932) to quasi- quantify informants preferences or ranking of value/importance attached to a particular concept (for example,12.2 Nigeria, 16.2: Zambia, 17.2: Zambia). While such tools may be powerful when carefully developed and tested for analytical validity in particular settings, under psychometric methodologies, there are also limitations to applying such methods across different settings and without the necessary lead time in developing meaningful measurements. A key limitation is that such scales assume that the concept is understood by the informant in the same ways as the researcher and that a numeric ranking is meaningful. Scales such as these can, in some instances, provide a relatively good sense of how informants evaluate the impact of a particular intervention on their subjective experience – yet results are difficult to compare as concepts such as ‘job satisfaction’ are not standardised across informants (what makes one person more or less ‘satisfied’ in relation to his/her job is not the same for the next person). Further, we noted that some instruments used the same scale for a number of questions – which may lead to ‘fatigue’ and a tendency to gravitate towards the mean on the part of the informant. Additionally, such scales for measuring satisfaction are known to tend towards a positive bias, which limits their utility. In addition to basing questions around concepts that may or may not be fully operationalised, many of the data collection instruments assumed, rather than probed, relationships between and among concepts, for example, in relation to the effects of specific incentives on motivation and job satisfaction. This is evident in questions such as ‘Do you think adding an extra financial incentive will improve your performance?’ (16.2: Zambia, 17.2: Zimbabwe); or ‘Have you mobilized the community to assist you in increasing the delivery of MCH services’ (16.2: Zambia); or ‘ Avez-vous pu constater des changements dans la structure depuis le début du versement des primes?’ 5 (4.2: DRC) . In these instances, the informants were asked to comment on a presumed situation rather than being given the chance to iterate how they saw the situation and its impact on their working lives. In addition, there were a number of questions that were abstract in nature and did not focus on the concrete experience of the informants, and his or her working practices. Some examples of this were: ‘Do you think standards of care at this HF can be improved?’ (17.2: Zimbabwe);‘Would you think that the PBF has influenced the experience of patients?’ (7.2: Kenya); ‘How do you see the motivation of health workers in general?’ (12.2: Nigeria). 5 ‘Have you noticed changes in the facility since incentives started to be paid out?’ 20 Open-ended questioning The interview guides were structured following a set of topics that covered a number of the areas laid out in the framework. In principle, this is good practice because it ensures consistency between the framework and the data collected. However, in practice, this meant that some of the instruments did not lend themselves to an interview that would encourage informants to speak openly and reflect on what was being said: these instruments lacked a natural or organic flow and tended to move from one topic to the next without adequate links or integration of themes. For instance, the semi-structured interview guide for Zambian HCWs (16.2: Zambia) is divided in clear sections, however these jump from descriptive questions (that is, ‘work profile and motivation factor’) to perception of challenges (that is, ‘understanding challenges and coping up’), and to broad context (i.e. socio-economic and cultural context for MCH care), jumping back to questions about personal satisfaction, motivation and future prospects. Another point noted was the interspersing of hypothetical questions that asked informants to comment on what they would have liked to see or experience, or how they felt about something that was yet to happen in the future