What lessons does Uganda’s long-running Justice Law and Order Sector (JLOS) development programme offer in creating workable monitoring and evaluation (M&E) frameworks for justice/law and order programming? A key lesson identified in this report is that indicators should be: high-level and cross-sectoral to encourage holistic thinking; and combined so as to minimise the risk of perverse incentives. National-level M&E systems should be supplemented with local ones, and the time, money and staff allocated to M&E needs to be commensurate with the programme size, duration and ambition.
JLOS involves a sector-wide approach (from which the armed forces and Ministry of Defence are absent), and donor assistance is primarily delivered via sector budget support. JLOS has, over a number of years, developed sophisticated M&E arrangements that include routine monitoring, periodic reviews and indicators which measure the impact of both the sector’s member institutions and the sector as a whole.
Moves to develop a reform programme for the entire justice, law and order sector began in Uganda in November 1999, and attempts to establish suitable indicators began in 2000.
- In its first stage, JLOS had two sub-sectors on Commercial Justice and Criminal Justice, and provided no obvious means through which to track the performance of the sector as a whole, or of individual institutions in terms of purpose / impact.
- A mid-term evaluation (2004) highlighted M&E weaknesses, including: lack of capacity; inadequate public feedback; overlapping reporting systems; insufficient user surveys; institution-based reporting; unharmonised information flow between the sector and its institutions; ineffective and underused complaints mechanisms; Management Information System and case management systems not integrated; and inappropriate indicators.
- The JLOS’ next investment plan (2006/7-2010/11) uses sector-wide objectives and indicators. The intention is to combine a Policy Matrix and an Institutional Policy and Indicators Matrix to form a sector monitoring framework.
Lessons from the JLOS experience include the need to link indicators with poverty reduction and national development frameworks, and to incentivise M&E (for example, through output-focused budgeting). It is also important to:
- Address political as well as technical issues as part of M&E: Progress requires high-level evaluation, and programming must be adapted where necessary.
- Develop data collection, analysis and handling capacities: JLOS’s sector-wide approach to M&E, which places GoU mechanisms and institutions at its centre, relies heavily on self-reporting by member institutions, with limited verification through other means and sources. Increased peer review and capacity building in data collection and management are needed.
- Carry out baseline and tracker surveys: While data needs will change over time, some consistency should be maintained so that time-sequenced comparisons are still possible.
- Promote public and civil society engagement: Empowered users and CSOs are more likely to provide a check on the sector and to agitate for and sustain changes. Their views also provide a useful counter to what may otherwise be an inward-looking M&E system.
- Define M&E terminology: Clarity of meaning and broad-based agreement on key terms at the national level is important.
- Evaluate donor as well as sector performance: The use of direct budget support by donors in the case of JLOS means that evaluation of donor performance and sector performance needs to be clearly distinguished.
- Balance the requirement for M&E with other priorities: M&E must support and enhance the work of the sector and strengthen its efficiency without burdening its officials and institutions.
- Allow M&E systems and culture time to develop and mature: The JLOS experience suggests that M&E needs to develop incrementally in both its technical and cultural aspects.
