This evidence mapping was conducted to identify the empirical evidence on the outcomes of security and justice (S&J) interventions ranging from security reform to training border guards and to update an original evidence mapping published in September 2015. The findings are presented in the form of an evidence gap map, which provides an accessible and visual representation of where the evidence for S&J programming is more abundant or limited.
The evidence mapping included a rigorous review of the existing S&J evidence base and identification of new evidence and material in an expanded search since 2015. Selected sources, including journal indices, online research and evaluation repositories, resource centres and experts were interrogated. Studies that explored interventions and results were selected for inclusion in a database (available to download) and coded according to the publication type, thematic focus, intervention, output and outcome categories.
Features of the evidence base
Research design: The majority of studies continue to be mostly observational in nature, with only a 1% differential compared to 2015 (94%). Most of these used the same techniques (interviews, focus groups, ethnography, historical analysis and political economy analysis). Fifteen experimental or quasi-experimental studies were identified, an increase on the eight from 2015, and twenty-eight secondary studies (up from fifteen) covering a mixture of literature reviews, and annotated bibliographies. Having mainly observational case studies means a lack of cross-country comparisons, but a depth of knowledge about specific countries. Additionally, the very small number of experimental studies means there is limited information on the effectiveness of S&J interventions.
Publication form: The largest group (46%) of documents remain peer-reviewed journal articles. Thirty-five per cent of studies are classified as ‘other’ reports; these include analytical reports and case studies and were mostly published by academic organisations, think tanks and non-governmental organisations. Whilst few evaluations (12%) continued to meet the inclusion criteria, this is double (6%) the previous 2015 map. Those that did meet the criteria include thematic evaluations of donor security and justice programming, synthesis programme evaluations and other individual programme or country evaluations. This suggests that evaluations may be improving in quality, but there is scope to improve this further.
Geographic scope: This has not materially changed since the 2015 map. Western Africa is by far the most studied region. South Asia and Eastern Africa are the next most studied regions. There are very few studies exploring Central Asia, East Asia, Eastern Europe and Northern Africa.
Thematic focus: Policing remains by far the most studied theme, followed by access to justice/legal empowerment, justice sector reform, legal reform and non-state actors. Individual studies often explore interventions that addressed more than one theme/sector. The evidence base on interventions aimed at, or including, non-state actors showed a relatively large increase. This seems to reflect an increased role for non-state actors in approaches like community policing and local justice. There is still a lack of evidence in some important areas like intelligence and border security. Within the ‘access to justice/legal empowerment’ theme, most of the existing evidence is for the former and there is very little evidence on actual legal empowerment.
Gap map analysis
Evidence gaps and areas of strength appear where one would expect them, and more or less in line with the 2015 analysis. The gaps appear where the outputs and outcomes are less tangible and more difficult to operationalise and measure (section 6). The evidence is scarce for a number of output and June 2019 5 outcome categories that are justice-specific, whilst stronger evidence can be found on outcomes and outputs that are more general and applicable across a wider range of intervention types. Detailed definitions for each of the intervention, output and outcome categories are tabulated on pages 45- 51. In terms of the evidence, ‘abundant’ is coded in green on the tables on pages 21-23, whereas ‘limited’ is coded as red. Short- and medium-term results, which are tangible, operational and directly related to intervention categories, are labelled ‘outputs’, while longer-term and more indirect results are labelled ‘outcomes’. The distinctions were based on DFID’s Security Sector Reform (SSR) Theory of Change (ToC).
Interventions
- The evidence is abundant for: capacity-building of organisations; strategic/statutory frameworks and legislation; community-based approaches; and restructuring of the security and justice (S&J) sector. The evidence base on non-state actors has also improved since 2015. Many studies provide examples in which a combination of these interventions is used.
- The evidence is limited for: preventative interventions (integrated efforts to prevent violence and crime); Disarmament Demobilisation and Reintegration (DDR); and integrated political engagement/activities that promote political will. Evidence is also scarce for displaced/refugee interventions. There were zero impact evaluations of mutual legal assistance. This was discussed during the research period because it subsequently does not show up in the database but is clearly a gap.
Outputs
There have been no significant changes in the balance of where evidence is focused since 2015:
- The evidence is abundant for: the capacity of state and non-state organisations to deliver; strategic frameworks; confidence, trust or satisfaction on the part of citizens in providers; and roles, coordination and dialogue amongst organisations and agencies. These are considered to be key outputs for S&J programming, they are relatively tangible, and are directly related to intervention categories.
- The evidence is limited for: application, compliance with and interpretation of laws; and state/non-state linkages. These are both justice-specific outputs. Outcomes • The evidence is abundant for ownership of reforms by national and local stakeholders; resource allocation/funding stability and sustainability; access to provision; stability and outbreaks of conflict or violence; and human rights measures implemented to improve compliance. These outcomes are considered core tenets of S&J and/or are relatively tangible. • The evidence is limited for: incentives for improved service delivery amongst security and justice actors; actual crime rates; legal awareness and confidence; gender-based violence rates; and judicial redress to protect rights. The evidence is particularly limited for longer-term development outcomes such as: economic development (local or national); poverty reduction; access to land, inheritance and property rights; access to public services and economic resources.
