Impact evaluations are increasingly used as a tool to establish what works, why and under what circumstances in a variety of development sectors. However, doubts have been voiced as to the feasibility and desirability of carrying out impact evaluation in contexts of conflict and fragility. Some evaluators and practitioners in this field raise four main concerns: (i) it is unethical to identify a comparison group in situations of conflict and fragility; (ii) it is too operationally difficult to do so; (iii) impact evaluations do not address the most important evaluation questions; and (iv) they are too costly.
This paper argues that it is both possible and important to carry out impact evaluations even in settings of violent conflict, and it presents some examples from a collection of impact evaluations of conflict prevention and peacebuilding interventions. The paper examines the practices of impact evaluators in the peacebuilding sector to see how they address evaluation design, data collection, and conflict analysis. Finally, it argues that such evaluations are crucial for testing assumptions about how development interventions affect change which is important for understanding the results on the ground.
Key findings:
- In designing an impact evaluation it is important to carefully consider how to establish a counterfactual, analysing what is ethical and feasible in the particular context. Different researchers have established a counterfactual using experimental and quasi-experimental designs. Examples of these designs are: individual randomization; group-based randomization; and quasi-experimental designs. The unpredictability of the situation in which many peace building and conflict prevention impact evaluations take place sometimes calls for flexibility in the design and implementation of the evaluation.
- Carrying out impact evaluations in conflict-affected settings can be risky and methodologically challenging. All evaluations in which primary data are being collected through human interaction could in themselves be seen or perceived as a type of intervention. This fact has potential implications both for the reliability of the evaluation results and for the safety of the evaluation personnel and those being evaluated. Consequently, evaluations must address the operational and methodological consequences of the risk of violence. More specifically, in order to deal with this challenge, it is advisable that the evaluation itself include a conflict analysis in order to assess the intervention and to ensure that the evaluation process and product is conflict sensitive. Broadly speaking, there are three reasons for evaluation teams to conduct conflict analyses: 1) to assess the relevance and impact of the program; 2) to assess the risks of negative effects of conflict on the evaluation design and process; and 3) to assess the risks of the evaluation exacerbating conflict (conflict sensitivity).
- Part of what underlies the ethical concern about impact evaluations is the premise that assignment to a comparison or control group implies ‘not receiving a benefit’. This is not necessarily the case for two reasons. First, the comparison group can be receiving a treatment with which another competing intervention is being compared. Second, it is important to examine the assumption that receiving a development intervention, or more of one, is always a benefit. The reality is that the effectiveness and impact of a large number of development interventions have yet to be proven. Another concern raised is about randomizing a program’s activities across possible beneficiaries instead of selecting according to other criteria. In a conflict-affected setting, prioritizing certain beneficiaries could be important for defusing volatile situations or prioritizing quick wins. On the other hand, in cases where a program cannot be implemented across all individuals immediately, randomization of eligible individuals can in fact be more ethical and politically feasible than determining who benefits first and who later, especially in a sensitive situation where particular choices can be construed as being politically motivated.
