This guide presents an evaluation framework that assesses the contribution to recovery associated with post-disaster interventions. It is a method focused on assessing positive and negative changes to the lives of affected people and other local stakeholders, in the medium term following a disaster event. The output of the approach is a report that presents detailed findings, an in-depth analysis drawing on the findings, and a concluding section that discusses the contribution to change generated by post-disaster interventions – including a series of summary statements.
The approach assumes that changes in people’s well-being and livelihoods can be most clearly identified at a household level. Assessing change necessarily involves identifying what the situation was like for households before and after the disaster occurred, as well as the situation following a period of post-disaster recovery and intervention. The guide presents a technique for collecting retrospective data to cover these changes, combining quantitative and qualitative data collection and analysis.
Resources
Impact evaluations need to be based on a realistic availability of resources, both financial and human. There have been some large-scale household surveys looking at pre-disaster data and tracking changes over time; examples of these include longitudinal surveys applied after the Indian Ocean tsunami of 2004 and the Pakistan earthquake of 2005. However, the rigour and expense of these longitudinal studies exceed what most agencies can afford in the way of evaluation.
Control groups
The use of control or comparison groups raises costs and poses additional issues. The logical control groups would be disaster-affected communities that do not receive assistance, but working with such groups in post-disaster situations without providing assistance raises ethical concerns. It may also be difficult to identify genuinely comparable communities.
Baseline/endline
Although the more accepted method for measuring the impact of interventions is to have a baseline/endline, in humanitarian programmes it is often difficult for agencies to carry out a baseline early enough to be able to do a comparison. Life-saving and emergency measures take precedence over data collection in the early stages after the onset of a disaster. Traumatised communities may not wish to participate in surveys and discussion groups that do not appear to bring any immediate relief to their situation. Pre-emergency data may be available from other sources, such as government national household surveys or other agency data. However, it may be difficult for agency staff to extrapolate the information needed for their particular target group or the quality may be such that it is not considered useful.
