There is a wealth of literature available on theory-based evaluation and impact evaluation (TBE), but experts disagree on whether TBE is a common and clearly defined approach. Some think a common conceptual and operational understanding has been elusive, while others point to a completely consistent basic concept regardless of slight differences in the use of terminology.
Some core features appear consistent across the main accounts of the TBE approach:
- Opening up the black box to answer not simply the question of what works, but also why and how it worked. This is key to producing policy relevant evaluation.
- Understanding the transformational relations between treatment and outcomes, as well as contextual factors and aiming to identify the ‘mechanisms’ that make things happen. This goes from asking whether a programme works to understanding what it is about the programme that makes it work.
- Having two key parts: conceptual (developing the causal model or theory of change that underlies a programme, and using this model to guide the evaluation); and empirical (testing this theory of change to investigate how a programme causes intended or observed outcomes).
- Being issues led, and therefore, methods neutral.
Some of the variations in TBE strategies are:
- Approach to types of theory: whether the black box is empty, full of theories or inhabited by people, and the implications for how to accumulate knowledge and establish the theory of change.
- Approach to causal inference: the realist evaluation approach adopts a generative approach to attribution seen by some as distinct from other (i.e. experimental) designs; other approaches promote the use of a range of techniques and tools to make counterfactual comparisons under the TBE approach.
This review highlights the following key points from the literature.
- Some promote the benefits of applying a TBE approach to experimental designs.
- Much of the guidance proposes the use of mixed methods (quantitative and qualitative) while leaving open exactly how to go about choosing the appropriate design of mixed methods.
- Few studies apply the approach in practice.
- Guidance has been developed on the TBE approach and tools for evaluating complex and complicated programmes.
