This paper reviews experience in the use of randomised control trials (RCTs) in evaluating transparency and accountability (T&A) initiatives, and where evidence exists, in evaluating Technology-for-Transparency-and-Accountability (T4T&A) initiatives.
The review explores: (i) where and under what conditions RCTs might be the most appropriate approach; and (ii) where other approaches would be more effective and more robust, given the particular characteristics of T&A programmes.
The literature search identified 15 evaluations of T&A initiatives in service delivery and in social accountability in the context of political participation and representation, based on an experimental/RCT design. Three of these were for technology-based (T4T&A) interventions. The 15 evaluations were analysed against criteria encompassing evaluation design, the way the evaluation deals with contribution of the intervention to impact, how well the evaluation is able to explain how the intervention caused an effect, and the extent to which the evaluation explains how the intervention worked.
Key findings:
- The RCTs considered here tend to test alternative mechanisms for T&A, in the context of pilot programmes, to evaluate which of these mechanisms are likely to achieve the desired impact. In some cases there is more than one ‘treatment’ arm where modifications of the same basic design are evaluated against a control group. Otherwise the studies are basic ‘with–without’ evaluations with one treatment arm and one control.
- Overall, when a programme strategy is to be politically transformative, RCTs do not work in evaluating impact in terms of transformation. Neither do they work as a sole method of evaluation where a programme is taking an adaptive and iterative approach. However, as an evaluative tool, RCTs can be an effective and useful means of deciding between different variations in an intervention design in the context of piloting a programme, especially in evaluating a subset of an intervention: for example, where a pilot within an adaptive programme is tested on a small scale before rolling out. In programme design, implementation (treatment) has to be randomisable and there need to be exclusionary factors. This can be difficult in T&A especially if policies apply nationally, for example, or regionally, or at other administrative levels for implementation.
- A recurring theme is the importance of a clearly defined and well-articulated theory of change – both for effective programme implementation and evaluation of impact. While all the reviewed papers had implicit theories of change underlying the T&A initiatives, in many cases articulated via a series of hypotheses that the evaluation aimed to test, the assumptions that lead from cause (the intervention) to effect (the intended impact) are not made explicit, with no alternative, plausible, causal links identified. This makes it difficult to assess how and where initiatives work in the intermediate stages.
- Consideration of context is also limited. For example, the evaluations here do not take account of important factors such as social effects and social networks on people’s behaviour and are therefore limited in drawing learning more broadly as well as missing critical components and potentially key
- contributory factors in the impact pathway. If context is not taken adequately into account then evaluations are of limited external validity as we cannot be sure if they would work if scaled up.
- There is in particular a strong case here for not only paying more attention to the theory of change, encompassing impact pathways, context and underlying assumptions, but also re-examining and evolving these approaches, both in the context of T&A/T4T&A and more broadly. Rather than seeing the development of a theory of change as being something that is ‘done for the donors’, instead it could be instituted as a crucial and evolving tool in design and planning, as a framework for learning and for assessing and managing risk – considering not only impact pathways to positive change but also theories of ‘negative’ change.
- Overall, an RCT approach does lend itself well to technology-based initiatives, even given the challenges in the T&A context described in detail in this and other reviews. It is an ideal method for testing alternative mechanisms and/or technologies – although the three T4T&A RCTs analysed here do not do this. However, the value of this exercise needs to be balanced against cost. For small, one-off, low-cost interventions the generally high cost of doing a high-quality RCT could well preclude its use in evaluation. But if the programme is intended to be scaled up – say to national or even regional level – it could well be worth the investment.
- Longer time frames between implementation and endline evaluation to allow sufficient time for impacts to be manifest, especially where technology is relatively new to users with little known about their propensity to take it up;
- Evaluations at intermediate stages of the implementation process – midline – in order to gauge intermediate impacts;
- Overall design based on a range of methods – qualitative and quantitative, experimental and non-experimental – to complement the RCT component;
- Clearly articulated theories of change, in order to ensure underlying models are correctly specified and to help identify the most appropriate ‘package’ of methods.
This review also suggests that an RCT design for T&A evaluation needs to be improved by: