There is a debate in the field of impact evaluation (IE) between those promoting quantitative approaches and those calling for a larger range of approaches to be used. This paper highlights four misunderstandings that have arisen in this debate. They involve: 1) crucially, different definitions of ‘impact’ – one based on outcomes and long term effects, and one referring to attribution; 2) confusion between counterfactuals and control groups; 3) confusion of ‘attribution’ with sole attribution; and 4) unfounded criticism of quantitative methods as ‘positivist’ and ‘linear’. There is no hierarchy of methods, but quantitative approaches are often the best available.
Amongst evaluators, ‘impact’ commonly refers to a long-term outcome or long-term effect that took place as an intended or unintended result of a development intervention. However, this is different from an attribution study which seeks to identify the counterfactual value Y (Y0) where the impact is defined as the difference in the indicator of interest (Y) with the intervention (Y1)and without the intervention (Y0). The former definition allows a large number of studies to be termed ‘impact evaluation’. These, however, do not necessarily generate solid evidence of what works. Many such studies explicitly state that they cannot clearly identify impact.
While both definitions are the basis for useful studies, there is a current focus on funding attribution analysis. This type of analysis provides more concrete data about what works and what does not, at what cost: it gives a greater degree of numerical precision and wider consideration to context. However, insufficient attribution studies have been conducted.
A second misunderstanding relates to comparison groups and counterfactuals. Establishing attribution involves dealing with counterfactuals. In attributing changes in outcomes to a development intervention, an explicit counterfactual is likely to be useful. An explicit counterfactual does not necessarily require a comparison group, although often it will. Further confusion has arisen in the IE debate regarding:
- Contribution versus attribution: Many people suggest that it is difficult, if not impossible, to attribute a change in outcomes to a specific intervention since there are so many different factors involved, and that it is therefore better to look for contribution. This argument confuses attribution with sole attribution: analysis of attribution includes contribution in that it allows identification of how much an intervention contributed to an overall change.
- Criticism of quantitative IE for being positivist and linear: Such criticism fails to consider that all policy advice is arguably ‘positivist’. Critics also seem to use ‘linear’ to mean different things – such as a linear relationship (meaning that a unit change in X causes a fixed increment in Y, regardless of the value of X); one-way causation (the assumption that X affects Y without allowing for the fact that Y may also affect X); or that models of the causal chain imply an automatic (or deterministic) relationship. In fact, methodologically sound quantitative analyses will avoid the very faults attributed to them. For example, one-way causation can be avoided through random allocation of the treatment.
Must experimental and quasi-experimental methods be used for attributing observed changes to a specific intervention? Where these methods can be used, they should be. Where they cannot, other approaches can be used, but will not usually give the numerical precision that enhances policy relevance. It is also important to note that:
- Any assessment of attribution needs to deal with the problem of selection bias, and randomised control trials (RCTs) often provide the most effective way of doing this. If not RCTs, then some other quantitative method should be used for attribution analysis.
- A key reason for using quantitative methods is to analyse cost effectiveness or, better, to conduct a cost-benefit analysis.
- Methods can be combined: a theory-based approach will usually combine quantitative and qualitative methods.
NB: This paper has also been published in Evaluation. See the article’s abstract.
