Despite decades of investment in social development programmes, we still know relatively little about their net impact. So why are rigorous social development impact evaluations relatively rare? This paper from the Center for Global Development (CGD) aims to address this question and provide recommendations for more and better evidence for policymaking and programme planning. A new, collective approach is needed, in which developing country governments, bilateral and multilateral development agencies, foundations and NGOs work together to define an agenda of enduring questions and fund the design and implementation of rigorous impact evaluations in key sectors.
Although significant resources are devoted to monitoring of programme outputs or process evaluations, there is insufficient investment in impact evaluations. Reviews of evidence that are conducted in the course of policy or programme design routinely reveal that little is known about “what works” and decisions are made on the basis of anecdotal or other weak evidence. Of those studies that have been completed, many are methodologically flawed so that net impact cannot be estimated in a valid way.
Three basic incentive problems need to be overcome to generate more and better impact evaluations:
- A portion of the knowledge generated through impact evaluation is a public good. Such broad benefits are amplified greatly when the same type of programme is evaluated in multiple contexts, and addresses enduring questions. However, the cost-benefit calculation for such an evaluation might not include these benefits, making it look like the impact evaluation is not worthwhile.
- The rewards for institutions, and for individuals within them, come from “doing” not from ‘building evidence’ or ‘learning’. Thus, it is extremely difficult to protect the funding for good evaluation, or to delay the initiation of a project to design the evaluation and conduct a baseline study. The opportunity costs are seen as too great. Consequently, resources that might be devoted to rigorous evaluation are used instead for project implementation.
- There are disincentives to finding out the truth. If policymakers and programme managers believe that future funding depends directly on achieving a high level of success, the temptation will be to concentrate on producing and disseminating ‘success stories’.
The solution to overcoming these incentive problems lies both in strengthening capacity within organisations and in creating a collective approach. Specifically:
- Organisations should commit individually to strengthening monitoring and evaluation systems, dedicating resources to impact evaluations and building capacity in developing countries.
- These efforts by individual organisations should be coupled with collaborative – collective action among actors to coordinate and fund rigorous impact evaluation.
- A committee, standards-based network, secretariat, or other organisation is needed as a focal point to lead a collective action initiative. This ‘council’ should include any developing country governments, development agencies, NGOs, foundations, and other public and private entities willing to engage actively in improving the quality and increasing the quantity of impact evaluations.
- The core functions of the council would be: to establish quality standards; administer a review process for evaluation designs; identify priority topics; and provide grants. Other functions would include: organising and disseminating information; building capacity to produce, interpret, and use knowledge; creating a directory of researchers; and undertaking communication activities.
