Page contents
- Steps in planning and design
- Ensuring evaluation quality
- Identifying threats to the validity of evaluation findings
- Evaluating multi-donor programmes
- The politics of evaluation
- Promoting the use of evaluation findings
- Strengthening national M&E capacity
- Strengthening national statistical capacity
Badly designed and managed evaluations can do more harm than good: misleading results can undermine the effective channelling of resources for poverty reduction. Designing effective M&E systems is not enough: donors need to commit to supporting M&E at all stages of implementation, which include selecting appropriate indicators, establishing baselines, collecting quality data, and reporting and using findings effectively. Sustaining an M&E system within an organisation also presents considerable ongoing challenges.
Establishing international standards for methodological rigour, ethical practice and efficient management processes in M&E is another critical challenge. Key issues include how aid agencies should oversee evaluations outsourced to consultants, how to build country ownership of M&E processes where there are significant capacity constraints or limited buy-in, and how to co-ordinate evaluations of joint donor programmes effectively.
Steps in planning and design
Monitoring and evaluation activities are usually broken down into the stages of planning, implementation, analysis, dissemination and use.
Kusek, J., and Rist, R., 2004, ‘Ten Steps to a Results-based Monitoring and Evaluation System’, World Bank, Washington DC
Governments and organisations face increasing internal and external pressures to demonstrate accountability, transparency and results. Results-based M&E systems are a powerful public management tool to achieve these objectives. This handbook presents a ten-step model that provides extensive detail on building, maintaining and sustaining a results-based M&E system.
Access full text: available online
Aid agencies largely work to their own internal requirements for reviewing, reporting on and evaluating the inputs, process and results of their activities, producing internal guidance notes that describe the practical steps involved.
United Nations Development Programme Evaluation Office, 2009, ‘Handbook on Planning, Monitoring and Evaluating for Development Results’, UNDP, New York
Access full text: available online
Ensuring evaluation quality
Ensuring the quality and integrity of evaluation design is vital for reaching accurate and reliable conclusions about what works and what does not. A key mechanism for ensuring evaluation quality is to establish internal quality control panels. DFID has had such a panel since 2009, providing expert advice, guidance and quality assurance.
International standards emphasise the need for impartiality, accountability, appropriately skilled experts conducting the evaluation, participation, country ownership and timeliness (evaluations should be appropriately timed to influence policymaking). The OECD DAC Network on Development Evaluation’s principles for evaluation of development assistance are widely cited.
OECD-DAC, 1991, ‘Principles for Evaluation of Development Assistance’, OECD, Paris
This paper presents a set of principles on the most important requirements of the evaluation process. Development assistance is a cooperative partnership between donors and recipients. Both must take an interest in evaluation to improve the use of resources through learning and to ensure accountability to political authorities and the public.
Access full text: available online
Most donors now also make substantial use of the harmonised DAC Quality Standards for Development Evaluation. While the OECD Principles focus on the management and institutional set up of evaluation systems, the Standards inform evaluation processes and products:
OECD-DAC, 2010, ‘Quality Standards for Development Evaluation’, DAC Guidelines and Reference Series, OECD, Paris
The OECD-DAC Quality Standards for Development Evaluation, built through international consensus, provide a guide to good practice. They are not intended to be used as a development evaluation manual, but they outline the key quality dimensions for each phase of a typical evaluation process: defining purpose, planning, designing, implementing, reporting, and learning from and using evaluation results. Principles informing the whole of the evaluation process are transparency and independence; integrity and respect for diversity; partnership, coordination and alignment; capacity development; and quality control.
Access full text: available online
There is also a need to ensure that evaluations are conducted ethically, in a culturally sensitive manner that protects the anonymity and confidentiality of individual informants.
United Nations Evaluation Group, 2005, ‘Standards for Evaluation in the UN System’, UNEG, New York
This document offers solid guidelines for evaluation planning, design, implementation and reporting. Fundamental requirements include: institution-wide support, clearly-defined and transparent responsibilities, appropriately qualified staff, and a constant commitment to the harmonisation and updating of methods used.
Access full text: available online
Identifying threats to the validity of evaluation findings
Significant criticism has been levelled against the development community for failing to adopt methodologically sound approaches to evaluating their activities. These include weak analysis of qualitative data and not paying enough attention to mapping the causal chain from inputs to impacts.
White, H., 2005, ‘Challenges in Evaluating Development Effectiveness’, IDS Working Paper 242, Institute of Development Studies, Brighton
Evaluation has a crucial role to play in today’s results-based culture and in the context of the Millennium Development Goals. How, then, can the quality of evaluation be improved? This paper argues that there has been inadequate investment in methodology, often resulting in low quality evaluation outputs. It discusses techniques in three areas: measuring agency performance; evaluation methods at the project level; and sustainability analysis.
Access full text: available online
The validity and usefulness of an evaluation are determined, among other things, by its statistical validity, use/action orientation, transferability and fittingness.
Bamberger, M., Rugh, J., and Mabry, L., 2006, ‘Strengthening the Evaluation Design and the Validity of the Conclusions’ Chapter 7 in Realworld Evaluation: Working Under Budget, Time, Data and Political Constraints, Sage Publications, California
How can threats to the validity of evaluations be identified and addressed? This chapter outlines some of the most common threats to the validity of both quantitative and qualitative evaluation designs. It offers recommendations on how and when corrective measures can be taken to ensure validity.
Evaluating multi-donor programmes
The Paris Declaration commits donors to co-operation and harmonisation in all stages of the development cycle. Joint evaluations are necessary where multiple agencies are involved in a chain of interventions to pursue similar outcomes, or to understand the combined effects of all interventions across a particular sector.
Joint evaluations present opportunities for donors to pool their technical and financial resources for more rigorous, in-depth and longer-term evaluations, and to reduce the multiple information demands on governments and stakeholders. However, they require agencies to reconcile their often divergent mandates and preferred evaluation approaches.
OECD-DAC, 2006, ‘Guidance for Managing Joint Evaluations’, DAC Evaluation Series, OECD, Paris
Joint evaluations have become central to development practice in recent years. Collective assessment of agencies’ combined work minimises transaction costs for developing country partners and addresses the large aid-giving role of joined-up modalities such as basket funds and joint assistance strategies. This booklet provides practical guidance for making joint evaluations efficient, educational and collaborative.
Access full text: available online
The politics of evaluation
Evaluations are more than a technical process. They have the capacity to determine access to resources and the funding fate of programmes. It is inevitable therefore that they will be subject to pressures from different stakeholders to produce favourable assessments or to avoid addressing sensitive issues.
Bamberger, M., Rugh, J., and Mabry, L., 2006, ‘Reconciling Different Priorities and Perspectives: Addressing Political Influences’, Chapter 6 in Realworld Evaluation: Working Under Budget, Time, Data and Political Constraints, Sage Publications, California
No evaluation can ever be value free and completely objective. Decisions such as what to study, which methods to use and whose criteria define programme success all involve human judgement. This chapter discusses how political factors affect evaluation. It provides a detailed analysis of possible pressures and constraints in evaluation design, implementation, dissemination and use.
Why do the evaluation, monitoring and accountability gaps persist decades after aid agencies started emphasise impact assessment? The following paper argues that the incentives in favour of the status quo are far stronger than the incentives to institutionalise stronger accountability and more rigorous evaluation.
Morton, J., 2009, ‘Why We Will Never Learn: A Political Economy of Aid Effectiveness’, James Morton & Co
Why is M&E still not being carried out effectively? This paper is a practitioner’s response to current debates on M&E and aid effectiveness. It examines the technical side of M&E and the latest thinking on complexity, arguing that current methodological debates are red herrings. It highlights the underlying incentive structures that create the need for M&E, but do not create a sincere demand for the impact assessments that M&E is designed to produce. The aid industry’s default setting is lesson suppression, not lesson learning.
Access full text: available online
Promoting the use of evaluation findings
M&E should ultimately result in improved policy and practice. Yet the findings and recommendations of evaluations are frequently under-used. In order for evaluations to be influential, it is important to consider: how to integrate them into the policymaking cycle; the political incentives to take up findings; and how the report is presented and understood by different stakeholders.
World Bank Operations Evaluation Department, 2004, ‘Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs’, World Bank, Washington DC
Evaluations can be a cost-effective way of improving the performance and impact of development activities. However, they must be conducted at the right time, focus on key issues and present results in an accessible format. This report presents eight examples of evaluations that have had an important impact, and summarises lessons learned.
Access full text: available online
The extent to which impact evaluations are used can be enhanced by ensuring greater stakeholder involvement, ensuring that impact evaluations contain clear policy lessons and that these are disseminated to a wide audience.
Jones, N., et al., 2009, ‘Improving Impact Evaluation Production and Use’, Working Paper 300, Overseas Development Institute, London
What can be learned from the many impact evaluations (IEs) carried out by development agencies in recent years? This review highlights the need and growing demand for greater and more strategic coordination of IE efforts, and notes insufficient attention to diverse methodological approaches to evaluation. It is important in all policy sectors to reflect on the suitability of methods to development questions and to invest in the development of impact evaluations informed by methodological pluralism. Developing country evaluation capacity, early stakeholder involvement in IEs, and the dissemination of clear policy implications should also be supported.
Access full text: available online
Strengthening national M&E capacity
Evaluations of development programmes have historically been driven and designed by donors – primarily to satisfy their own accountability needs. However, it is increasingly recognised that both monitoring and evaluation should be a country-led process, not least because country ownership is a major factor in determining whether evaluation findings are then used in a national context.
Technical barriers to country-led evaluations centre on lack of human and financial resources, but M&E is also a highly political issue. The incentives, or lack of incentives, for evaluations to be conducted (e.g. fear of aid being withdrawn as a result of negative evaluation results) also need to be considered.
There is evidence that capacity development programmes work best when adapted to local governance structures, professional capacity and evaluation culture.
Schiavo-Campo, S., 2005, ‘Building Country Capacity for Monitoring and Evaluation in the Public Sector: Selected Lessons of International Experience’, World Bank Evaluation Capacity Development Working Paper no. 13, World Bank, Washington DC
The Evaluation Capacity Development (ECD) unit of World Bank’s Operations Evaluation Department is designed to help countries strengthen their M&E capacity. The unit targets ‘high-intensity’ support to Uganda and Egypt and various other types of support to an additional 32 countries. This paper from the World Bank collates some of the main lessons learned from ECD activities and outlines major issues to be addressed.
Access full text: available online
Strengthening national statistical capacity
Good statistics are vital for the effective monitoring of development programmes and strategies, and ultimately to support evidence-based policymaking. However, many developing countries lack the institutional capacity or effective systems for gathering data. Many existing statistical systems were predominately designed to meet immediate rather than long-term data needs, and therefore lack coordination.
There is increasing recognition of the need for a strategic approach to statistical capacity development, particularly following the Second International Roundtable on Managing for Development Results in February 2004 and the resulting Marrakech Action Plan for Statistics (MAPS). This emphasised the need to develop National Strategies for the Development of Statistics (NSDSs).
Paris21 Secretariat, 2004, ‘A Guide to Designing a National Strategy for the Development of Statistics (NSDS)’, Prepared by the Partnership in Statistics for Development in the Twenty-first Century (PARIS 21), Paris
There is increasing awareness of the need to strengthen statistical capacity to support the design, monitoring and evaluation of national development plans. National Strategies for the Development of Statistics (NSDS) are designed to achieve this goal. This guide aims primarily to assist developing countries to design their NSDSs, but will also be useful to development partners.
Access full text: available online