Evaluation has a crucial role to play in today’s results-based culture and in the context of the Millennium Development Goals (MDGs). How then, can the quality of evaluation be improved? This working paper from the Institute of Development Studies (IDS) argues that there has been inadequate investment in methodology, often resulting in low quality evaluation outputs. It discusses techniques in three areas of contemporary relevance: measuring agency performance; evaluation methods at the project level; and sustainability analysis.
Evaluation studies must be able to credibly establish the beneficial impact on the poor of official interventions. They must be able to draw out relevant and applicable lessons for policymakers. Evaluation in the three key areas can be improved through monitoring systems that satisfy the Triple-A criteria of aggregation, attribution and alignment, through the use of randomisation or quasi-experimental methods and by embracing the tools already available for sustainability analysis.
Many evaluation studies are data-rich but technique-poor. Problems in evaluation production result from the misuse and under-use of both data and theory:
- There is a lack of explicit attention to the techniques of meta-analysis in studies that involve the aggregation of agency ratings.
- Mean-based quantitative statistics can give a misleading summary. The best practice approach sometimes leads to the misrepresentation of data. Studies may focus on desirable processes and impacts but ignore differences in the quality of best practices or in the conditions that explain disparities in performance.
- There is often weak analysis of qualitative data, including the prevalence of data mining, sometimes formalised in the best-practice approach.
- Common approaches to the problem of attribution involve before versus after comparisons, comparisons with a control group or a combination of the two. However, such approaches have only limited applicability.
- Established methods of tackling sustainability in project appraisal suffer from being too technically sophisticated, distracting from key assumptions and from being most suitable only where variables are clearly defined.
There is thus a need to pay more attention to theory and technique, focussing on the following areas:
- Formal application of meta-analysis in studies that aggregate performance (agency-wide performance, and country and sector studies). This should be measured against the triple-A requirements of attribution, aggregation and alignment.
- The use of techniques to ensure that qualitative data are summarised in a way that reveals, rather than distorts, the patterns in the data.
- Paying greater attention to establishing the control in evaluation design, either through randomisation or through propensity score matching. Both techniques imply taking a prospective approach.
- Analysis of impact which is firmly embedded in a theory-based approach and which maps the causal chain from inputs to impacts.
- Seeking ways to establish impact that ‘open the black box’ and provide lessons about what works and what doesn’t.
- Application of risk analysis to discussions of sustainability using theory-based evaluation (TBE), which seeks to uncover the key assumptions that underlie project design.
