Page contents
- Evaluating governance programmes
- Governance indicators
- M&E in fragile and conflict-affected states
- M&E of empowerment and accountability
- M&E of policy influence
- M&E in governance and social development sub-sectors
Monitoring and evaluation tools and approaches are applied in a range of programmes and contexts. This section provides a guide to key resources on using monitoring and evaluation tools in relation to specific governance and social development programmes.
Evaluating governance programmes
To date, most impact evaluations have focused on economic and social development programmes in the fields of health, nutrition, water, agriculture and education. Relatively few impact evaluations have been conducted on governance programmes. This is due to several factors. Governance programmes are complex and combine a number of objectives, many of which may be difficult to measure. There is also a shortage of reliable baseline data for many governance interventions. Despite these problems, experimental methods such as Randomised Control Trials have been used to evaluate corruption, community development and election programmes. Where it is not possible to use experimental methods, evaluators should consider using quasi-experimental or mixed methods approaches.
Garcia, M., 2011, ‘Micro-Methods in Evaluating Governance Interventions’, Evaluation Working Papers, Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung, Bonn
Although billions of dollars have been invested on improving governance in developing countries in the past decade, few of the programmes that have received funding have been subjected to rigorous impact evaluation. The aims of the paper are to: (i) discuss the challenges in evaluating governance programmes, (ii) identify some aspects of governance that have been rigorously evaluated, and finally (iii) provide practical recommendations based from previous evaluation experience.
Access full text: available online
In recent years, a growing number of studies have used experimental methods to assess the impact of a range of governance programmes and processes. The most common sub-sectors for RCTs are elections, community-driven development and service delivery. Although there are a number of challenges associated with using RCTs to measure the impact of governance programmes, these experiments also present ‘immense opportunities’.
Moehler, D., 2010, ‘Democracy, Governance and Randomised Development Assistance’, Annals of the American Academy of Political and Social Science, vol. 628, no. 1, pp.30-46
Are field experiments useful in studying the impact of development assistance on democracy and governance (DG)? This article reviews recent and ongoing DG field experiments. It considers the prospects for and obstacles to the development of a useful body of experimental evidence on the political economy of development. There are significant challenges related to the difficulty of generalising from small samples and micro-level projects. However, although the field experiments have tended towards village-level interventions, they suggest that impact evaluations can potentially address higher-level interventions and theories.
Access full text: available online
E-Gap is a research network for scholars and practitioners engaged in field experiments on topics of governance, politics and institutions. Its website provides news and information on current research in this area.
Governance indicators
One of the most contentious debates surrounding the M&E of governance programmes is the use of governance indicators. Increasing emphasis on the need to measure ‘good governance’ and how it relates to poverty reduction has led to a proliferation of worldwide data sets, guidelines and frameworks for assessment. There is considerable debate about the validity of different methodological approaches to measurement, and increasing recognition that measuring governance is itself a political process.
UNDP, 2007, ‘Governance Indicators: A User’s Guide’, Second Edition, United Nations Development Programme, Oslo
Demand for ways to measure different aspects of democracy, human rights and governance is increasing. This has led to a rapid growth in the number of indicators to assess government performance, the quality of institutions and people’s perceptions. This guide offers advice on where to find and how to use governance indicators.
Access full text: available online
When used carelessly, governance indicators can ‘misinform and mislead’. It is important to recognise that governance indicators are used for a range of purposes and can be distinguished according to their source and objectivity, level of impact, comparability, degree of aggregation and ‘actionability’. The following paper presents several pitfalls associated with the use of governance indicators and proposes guidelines to ensure more careful use. It highlights the need for providers of indicators to be subject to greater transparency, scrutiny and peer review.
Williams, G., 2011, ‘What Makes a Good Governance Indicator’, Policy Practice Brief 6, The Policy Practice
What are the most common pitfalls associated with the use of governance indicators? How can these be avoided? This brief examines the different types of governance indicator, their use and their misuse. Users need to ask critical questions about what exactly indicators are measuring, and to remember the limited ability of broad governance indicators to demonstrate causality. Process indicators are most relevant to understanding causal mechanisms. Providers of indicators should ensure full transparency in the publication of methodology and source data, and should be subject to greater scrutiny, evaluation and peer review.
Access full text: available online
Much of the debate around governance indicators has focused on country-level governance monitoring and assessments. Relatively few studies have addressed the question of which kinds of governance indicators are most useful for governance programming. Meso-level indicators that link project/programme results to higher-level impacts have been largely absent, and macro-level indicators have sometimes been used by evaluators to judge projects.
In response to these issues, a number of agencies have sought to develop ‘improved’ indicators, by ensuring that these indicators are either more linked to concrete problems or based on a more participatory approach. DFID has recently commissioned an assessment of existing programme-level governance and conflict indicators, (Barnett et al., 2011), which includes a revised list of suggested indicators.
GSDRC, 2010, ‘Evaluation of Governance Programme Indicators’, Helpdesk Research Report, Governance and Social Development Resource Centre, Birmingham
While there is a large body of literature providing advice in selecting and designing indicators, there is little evaluating governance programmes indicators per se. In general, the literature criticises the application of quantitative indicators to governance programmes, or criticises the confusion on which aspects to monitor. Recently, there has been a growth in ‘improved’ indicators. There is a debate surrounding their validity, however. Participatory and mixed method approaches seem to respond best to criticism of governance assessments.
Access full text: available online
Barnett, C., et al., 2011, ‘Governance and Conflict Indicators Report’, ITAD
How can a robust and clear set of indicators be established to monitor effectively the progress of a programme? Which indicators show best that a programme has achieved what it set out to achieve? This study tests the relevance and robustness of a list of indicators for the Department for International Development’s (DFID’s) conflict and governance programmes. The list consists of fifteen separate sets, or suites, of outcome and output indicators and covers a spectrum of related programme areas, including security and justice, elections, civil service reform and corruption. It suggests that a good results framework that enables programme progress to be effectively monitored and explained is critical, particularly in times of financial austerity.
Access full text: available online
In another important initiative, the World Bank has recently sought to catalogue those governance indicators that can be used to measure concrete governance issues that can be addressed directly in governance projects and programmes. The AGI Data Portal consolidates information on actionable governance indicators, provides a one-stop-shop platform to navigate these indicators and their documents, and offers customised tools for data management, analysis and display. Another useful resource is the UNDP’s Governance Assessment Portal, which provides a database on country-level governance assessments, a library of publications on governance assessment and a toolbox of frameworks and indicators.
Further discussion of governance assessments and governance indicators can be found in the Political Economy Analysis Topic Guide.
M&E in fragile and conflict-affected states
M&E faces unique challenges in fragile and conflict-affected contexts. These include:
- Lack of security (both for researchers and the people who talk to them)
- Distrust of outsiders and reluctance to talk, or fear of reprisals for talking
- Shame in acknowledging sexual violence
- Rapidly fluctuating populations (due to migration) and a lack of adequate documentation on the population
- Lack of adequate statistical data to generate baselines
- Government unwillingness to recognise the level of violence or the failure of programmes which might in turn affect aid flows.
Elkins, C., 2006, ‘Monitoring and Evaluation (M&E) for Development in Peace-Precarious Situations’, RTI International, London
How can M&E information systems improve programme impact and assist peaceful development in situations prone to violent conflict? This paper outlines M&E’s status as a unique discipline and describes M&E strategies and tactics implemented in real-world, peace-precarious situations. Even under the stresses of violence and conflict, M&E approaches can help build knowledge of how to advance peace and development.
Access full text: available online
GSDRC, 2007, ‘M&E in Fragile States’, GSDRC Helpdesk Research Report, Governance and Social Development Resource Centre, University of Birmingham
There is a general consensus on the need to move M&E beyond the project level to the sector and strategic level in fragile states. It is important to assess the overall impact of interventions on stabilising fragile states and promoting sustainable peace. Joint evaluations and agreed upon objectives, among government departments (in the case of ‘whole-of-government’ approaches) and with other governments and relevant actors, are useful in providing this comprehensive picture.
Access full text: available online
For more detailed guidance on M&E in conflict-affected areas, see the GSDRC’s Conflict Topic Guide.
For more detailed guidance on measuring and assessing fragility, see the GSDRC’s Fragile States Topic Guide.
M&E of empowerment and accountability
Donors have been increasingly concerned with empowering poor people in their economic, social and political relationships, and ensuring that their programmes (and the governments that they support) are accountable to beneficiaries. These processes are complex and can be difficult to measure. There is a tension for donors between the need to pursue a flexible approach to supporting empowerment processes and the need to demonstrate results and accountability to their constituents. Combining quantitative and qualitative methods and using locally-generated indicators can help to overcome some of these issues.
A growing number of studies have sought to use Randomised Control Trials to measure empowerment and accountability processes and programmes. The following report outlines some of the key challenges associated with using RCTs for these programmes and notes that scholars are divided about the ability of these methods to generate reliable results.
GSDRC, 2011, ‘RCTs for Empowerment and Accountability Programmes’, Helpdesk Research Report, Governance and Social Development Resource Centre
This report examines the extent to which RCTs have been used successfully to measure empowerment and accountability processes and programmes. Field experiments present immense opportunities, but they are more suited to measuring short-term results with short causal chains and less suitable for complex interventions. The studies have also demonstrated divergent results, possibly due to different programme designs. The literature highlights that issues of scale, context, complexity, timeframe, coordination and bias in the selection of programmes also determine the degree of success reported. It argues that researchers using RCTs should make more effort to understand contextual issues, consider how experiments can be scaled up to measure higher-order processes, and focus more on learning. The report suggests strategies such as using qualitative methods, replicating studies in different contexts and using randomised methods with field activities to overcome the limitations in the literature.
Access full text: available online
A more detailed discussion of measuring impact and change in empowerment and accountability programmes is provided in the ‘Empowerment and Accountability’ Topic Guide.
M&E of policy influence
Influencing policy has become increasingly central in the development strategies of aid agencies and NGOs. Monitoring advocacy is challenging, however, because these interventions are highly complex and involve a range of interacting forces and actors. Key responses to these challenges include developing a theory of change, user surveys and collecting significant change stories.
Jones, H., 2011, ‘A Guide to Monitoring and Evaluating Policy Influence’, Background Note, Overseas Development Institute, London
Is it possible to monitor and evaluate effectively efforts to influence policy? This paper provides an overview of approaches to the monitoring and evaluation (M&E) of policy influencing activities. It suggests that while M&E in this field is challenging, information can be generated that can be used to improve programmes and provide accountability for funds. The key is for policy influencing teams to recognise the value of M&E to their work and to incorporate it into their practices from the beginning of a project or programme.
Access full text: available online
M&E in governance and social development sub-sectors
For brief overviews of some of the specific issues associated with M&E in key governance and social development sub-sectors, see the following sections in other GSDRC Topic Guides.