How can impact evaluation identify not just what does – or does not – work, but why? A theory-based approach to impact evaluation maps out the causal chain from inputs to outcomes and impact, and tests the underlying assumptions. Despite wide agreement that this approach will address the why question, it has not often been effectively used. This paper outlines six principles for successful theory-based impact evaluation: (1) map out the causal chain (programme theory); (2) understand context; (3) anticipate heterogeneity; (4) rigorously evaluate impact using a credible counterfactual; (5) use rigorous factual analysis; and (6) use mixed methods.
Most studies make speculations as to the reasons for impact, or differences in impact, rather than using solid empirical analysis. In order to apply the theory-based approach successfully, the following six principles should be followed:
Map out the causal chain or programme theory
- The causal chain links inputs to outcomes and impacts. It embodies the programme theory (or theory of change) as to how the intervention is expected to have its intended impact.
- Such a theory can be found in the traditional log frame. However, a log frame does not make explicit the underlying assumptions, whereas testing assumptions is central to a theory-based approach.
- The programme theory needs to be able to adapt to changing circumstances in the field and to take on board competing theories and unintended consequences.
Understand the context
- The social, political and economic setting in which the programme takes place can influence how the causal chain plays out. It is crucial to understanding programme impact and thus to designing the evaluation.
- It is important to read project documents thoroughly and to be familiar with the broader literature (anthropology and political economy) before beginning evaluation design.
- Contextual understanding also helps in understanding to what extent evaluation findings can be generalised.
Anticipate heterogeneity
- Impact can vary according to intervention design, beneficiary characteristic or the socio-economic setting. Understanding context helps to anticipate possible impact heterogeneity.
- Examining the underlying theory can help to expose possible heterogeneity and allow the evaluation design to anticipate it.
Evaluate impact rigorously using a credible counterfactual
- The appropriate counterfactual is usually a control group, which must be identified in a way that avoids selection bias.
Undertake rigorous factual analysis
- Many of the links in the causal chain are based on factual analysis. Targeting analysis (who benefits from the programme?) is the most common form of factual analysis.
Combine qualitative and quantitative approaches in a single evaluation
- Use of qualitative data involves using focus groups and reading anthropological and political literature to inform evaluation design.
- Spending time in the field is essential for analysing data effectively.
- There should be some action research activities, following up puzzles in the data with additional fieldwork.
NB: This paper has also been published in the Journal of Development Effectiveness. See the article’s abstract.
