Rigorously measuring the impact of VEA is challenging: while inputs, outputs and results may be monitorable, longer-term outcomes that involve complex causal chains often go uncaptured by conventional M&E frameworks (DFID, 2011). Recent secondary reviews have shown that VEA programmes often have poorly articulated or unrealistic theories of change, which can hamper the quality of evaluation (Tembo, 2012; McGee & Gaventa, 2011).
Measuring outcomes against preconceived indicators may miss harder to measure changes in power and relationships (Brook & Holland, 2009). Some argue predefined targets may in some circumstances create pressure to report positive outcomes that ultimately bias the evidence base (Cavill & Sohail, 2007). A recent systematic review called for a holistic approach to capturing the outcomes of empowerment, focusing on multi-dimensional indicators of power in different dimensions of people’s lives (Taylor & Pereznieto, 2014). Some success has been achieved when beneficiaries themselves decide how their empowerment should be measured (Jupp et al, 2010).
Tembo, F. (2012). Citizen Voice and State Accountability: Towards Theories of Change that Embrace Contextual Dynamics (Project Briefing No. 73). London: Overseas Development Institute
This briefing argues that current approaches to Theories of Change (ToCs) are inadequate for citizen voice and accountability interventions: linear ToCs do not capture the complex and dynamic realities of state-citizen relations and of the influences of the wider context on these interactions. It suggests a model for developing ToCs that are better grounded in dynamic socioeconomic and political contexts. The model, which blends outcome mapping and political economy analysis, can facilitate an ongoing process of analysis, intervention and learning. ToCs need to be subjected to a continuous process of construction and deconstruction to improve knowledge of what works and what does not, and in what circumstances.
See full text
McGee, R. & Gaventa, J. (2011). Shifting Power? Assessing the Impact of Transparency and Accountability (Working Paper No. 383). Brighton: Institute of Development Studies.
What does impact mean in relation to accountability programmes and projects? This meta-review argues that current approaches to impact assessment in this field are inadequate: methodological wars are overshadowing key issues of power relations and politics. A learning approach to impact assessment is needed that gives power and politics a central place in monitoring and evaluation systems. Instead of looking at the extent to which the desired impact was achieved, it is important to look at what happened as a result of the initiative, how it happened and why. It is also important to test and revise assumptions about theories of change continually and to ensure the engagement of marginalised people in assessment processes.
See full text
Brook, S. & Holland, G. (2009). A Mixed-method Approach to Measuring Empowerment in the Context of Social Policy Monitoring in Jamaica. Washington, DC: World Bank.
This mixed-method research was designed to understand young people’s sense of empowerment in their relations with the police. It used a Community Score Card (CSC) completed by focus groups, which produced numeric ratings as well as a narrative explanation of the ratings, and rapid assessment peer interviews of individual young people in the communities (ethnographic research). It found that while indicators were useful for measuring the accountability gap, they can be misleading – and even dangerous – if they reduce power relations to a depoliticised relationship between service provider and user. The difficult relations between youth and police in the three study communities were symptomatic of much broader societal problems. This research was timely and cost-effective; not only did it inform policy, but it had an empowering effect on the young people involved in the pilot.
See full text
Cavill, S. & Sohail, M. (2007). Increasing Strategic Accountability: A Framework for International NGOs. Development in Practice, 17(2), 231-248.
This review argues that international NGOs have focused their accountability mechanisms on inputs, activities and outputs, through formal reporting such as evaluations, stakeholder surveys and complaints mechanisms. M&E typically looks at predetermined quantitative indicators, which risks pressuring INGOs to highlight positives and downplay problems. There has also been a lack of effective feedback from the field to headquarters and to local communities and partners. This situation leaves strategic gaps in M&E. For example, the larger purpose of development may be seen as aspirational, and advocacy may be poorly evaluated. As a result of all this, mistakes are often repeated, and learning is not prioritised.
See full text
Taylor, G. & Pereznieto, P. (2014). Review of evaluation approaches and methods used by interventions on women and girls’ economic empowerment. London: Overseas Development Institute.
This systematic review assesses the quality and effectiveness of 254 evaluations of women’s and girls’ economic empowerment. It argues that because these programmes can bring about transformational change in women’s and girls’ lives, evaluations need to apply a holistic approach. This involves not just looking at whether women and girls have increased their access to income and assets, but also their power, agency and control over other areas of their lives. Effective evaluations used multidimensional indicators. The most innovative studies used variables to capture aspects of economic empowerment that one might not normally consider, such as whether a young girl was less likely to have unwanted sex. Mixed method evaluations were effective in various ways: they captured change in more diverse ways, and explored not only what changes occurred, but why and how.
See full text
Jupp, D., with Ibn Ali, S., & Barahona, C. (2010). Measuring Empowerment? Ask Them – Quantifying Qualitative Outcomes from People’s Own Analysis (Sida Evaluation Series). Stockholm: Sida.
This paper presents the experience of a social movement in Bangladesh, which found a way to measure empowerment by letting the members themselves explain what benefits they acquired from involvement and by developing a means to measure change over time. These measures have also been subjected to numerical analysis to provide convincing quantitative data which satisfies the demands of results-based management. The study shows how participatory assessments can empower and transform relationships, while at the same time generating reliable and valid statistics for what were thought to be only qualitative dimensions.
See full text
For further resources, see the supplement on measuring empowerment and accountability.
- DFID (2011). A Preliminary Mapping of the Evidence Base for Empowerment and Accountability. London: DFID See full text