At present there is little guidance on monitoring and evaluating (M&E) security sector reform (SSR). This report, published by Saferworld, considers what the main challenges are in this area. It also looks at what should be measured, what resources can be drawn on when developing guidance, and who would use this guidance. It focuses particularly on donor-supported SSR programmes, but with reference throughout to local ownership of, and capacity for, M&E activities.
Monitoring is defined as a function that uses collection of data on indicators to provide management and stakeholders with indications of the achievement of objectives. Evaluation is defined as the systematic and objective assessment of an on-going or completed project.
The research identified a large number of challenges and specific issues relating to the M&E of SSR:
- Within general, ‘familiar’ challenges, the most significant is the frequent resistance to M&E processes, with project staff and even management often perceiving it to be burdensome, unnecessary or threatening.
- SSR-specific challenges include the fact that security and justice systems are often complex, there is no one ideal model for a security system, and SSR touches on matters that are often entrenched in cultures of secrecy.
- With respect to donor policy, evaluations should consider coherence across agencies, sectors and policies, but should recognise the many practical obstacles to integrated approaches to SSR.
- Challenges related to donor bureaucracy include poor local ownership of M&E, intentionally ambiguous programme goals, lack of coordination between donor departments, and high staff turnover and workload.
- Many of the problems that arise in the M&E of SSR programmes can be traced back to poor programme design.
- M&E should always be sensitive to the context, and this is particularly important in fragile and conflict-prone environments.
Beyond these challenges, the fundamental questions for the M&E of SSR are what exactly should be measured, and how?
- M&E can be carried out at several distinct levels of analysis. These are programme/institutional, sector/system, country/context level, and policy.
- Often, the underlying theory of change informing the design of an SSR programme is opaque. Evaluations should explore theories of change in more detail.
- This report proposes eight evaluation criteria to inform the design of evaluations. They are relevance/appropriateness, effectiveness, efficiency, impact, sustainability and ownership, coherence, coordination/linkages, consistency with values.
- How M&E is carried out is just as important as what is done. Donors need to raise the issue of M&E at the design stage and explore the attitudes of local actors towards what kind of M&E should be undertaken and why.
The report groups relevant publications on M&E into four categories. These are SSR-specific material, performance measurement in developed countries, M&E in fragile and conflict-affected environments, and general M&E publications.
Essential points to include in any guidance aimed at improving M&E practice for SSR work include:
- Definitions and purposes of M&E
- The importance of M&E for effective SSR
- What is specific about the M&E of SSR
- Levels and focuses for M&E
- Theories and dimensions of change
- Evaluation criteria and prompt questions
- Forming legitimate and skilled evaluation teams
- Participation in and ownership of M&E processes
- Building the capacity of partner country M&E systems
