This paper highlights the importance of measuring the impact of peacebuilding interventions on rule of law and security institutions. It shows that there are a range of feasible approaches and methodologies, and that the approach taken will need to vary from case to case. Several approaches and methodologies can be combined in order to build on their individual strengths and mitigate their weaknesses. It is important to promote a culture of learning in evaluations.
Approaches to measuring the impact of support to rule of law and security institutions in peacebuilding contexts range from those that can demonstrate attribution to those that can evaluate contribution. The impact assessment methodologies identified include impact evaluation, theory-based impact evaluation, contribution analysis, outcome mapping, the Most Significant Change technique, and RAPID outcome assessment.
While much remains to be learnt in this area, the study finds that the best way to measure impact is to combine several approaches and methodologies in order to build on their individual strengths and mitigate their weaknesses according to the context. The selection of a combination of approaches and methodologies to be applied should be based on four criteria: 1) the purpose of the evaluation (accountability or learning); 2) the questions the evaluation seeks to answer (What impact? Why was the impact such?); 3) cost-effectiveness; and 4) the specific constraints of the peacebuilding context. In addition, the study finds that:
- There are a range of relevant approaches available. Beyond the scientific-experimental approach to evaluations, theory-based approaches (i.e. testing the theory of change) and participatory approaches can provide the basis for establishing a suitable counterfactual. (A counterfactual – an understanding of the situation of the beneficiaries had the intervention not taken place – is necessary for measuring impact in the sense of attribution.) Actors seeking to measure impact can therefore adopt alternative approaches that may be more amenable to the complexities of peacebuilding environments and might have lesser costs or skill requirements than those traditionally promoted for impact measurement.
- Small steps can be taken to strengthen traditional evaluation approaches to focus more on impact. Impact evaluation can be made less expensive by using alternative techniques such as post-intervention project and comparison groups with no baseline data. Participatory methodologies, which are often dismissed as being too time-consuming, can also be made more user?friendly: for instance, in the case of outcome mapping, progress markers can be integrated into surveys.
- Measuring impact can be a significantly political undertaking. There is a risk that an evaluation may shed light on the failings of an intervention to achieve its desired impact, and in that case there needs to be clarity on whether all actors are willing to confront this reality and what they can do with this information. This requires realistic expectations of both the intervention and of the evaluation.
- Attempting to measure impact is – or ought to be – more expensive and time-consuming that an evaluation at a lower level of the results chain. There is no need to measure impact on a yearly basis. In fact, it has been suggested that impact should be measured when the intervention has been in place for long enough to show observable effects, when the scale of the intervention (numbers and cost) justifies measuring impact, and/or the evaluation can contribute to new understandings on what works and what does not.
Building capacity for evaluations is fundamental for the professionalism and effectiveness of security and justice institutions, and yet is often neglected. Using elements of participatory approaches can help to build this national capacity.
In moving forward, there is also a need to promote a ‘culture of learning’ as opposed to a ‘culture of blame’ in evaluations. Establishing a culture of learning involves communicating the importance of evaluations as a learning mechanism, and requires that evaluation results are actually used to support adjustments in policy and practice. Procedures for ensuring this is the case should be included in any approach to measuring impact – including timelines and responsibilities. Moreover, mechanisms for sharing findings with national (and other international) partners should be developed.
Piloting various approaches will lead to a better understanding of what works for different actors operating in distinct contexts. In addition, applied research is needed on the implementation of theoretical and methodological approaches to measuring impact. This includes the compilation of detailed knowledge on evaluations that have used different techniques in the area of support for rule of law and security institutions.
