What are the most common pitfalls associated with the use of governance indicators? How can these be avoided? This brief examines the different types of governance indicator, their use and their misuse. Users need to ask critical questions about what exactly indicators are measuring, and to remember the limited ability of broad governance indicators to demonstrate causality. Process indicators are most relevant to understanding causal mechanisms. Providers of indicators should ensure full transparency in the publication of methodology and source data, and should be subject to greater scrutiny, evaluation and peer review.
The term governance indicator refers to an eclectic set of measures covering the wide range of governance topics. In 2008, 400 separate governance indicators were available. Indicators can be distinguished according to their source and objectivity, level of impact, comparability, degree of aggregation and the extent to which they are actionable. They may also be categorised according to their different users and uses. Process-oriented indicators are most suited to political economy analysis.
When used inappropriately governance indicators can obscure more than they reveal, promote false assumptions about what drives progress in governance and development, and at worst lead to the wrong policy or investment choices. Users need to avoid seven common pitfalls:
- Placing too much faith in numbers and treating governance indicators as fully reliable measures.
- Pushing the indicators beyond their original purpose. Governance indicators are often used in ways that depart from their original purpose or ignore statistical limitations to the data.
- Forgetting the meaning behind the measurement. Governance indicators are commonly cited without adequate explanation of the governance problem being measured and the method of assessment.
- Falling into normative traps. Some governance indicators, when used unquestioningly, can act to reinforce normative assumptions about how countries should be governed, based on models from OECD countries.
- Failing to recognise that ‘Not everything that can be counted counts, and not everything that counts can be counted’. It is vital not to place greater value on what is measurable. Broader and more fundamental problems should not be neglected.
- Tracking symptoms and neglecting causes. An excessive focus on indicators can lead to a superficial governance assessment that merely tracks symptoms and does not address root causes.
- Assuming causation where there is only correlation. It is important to guard against dubious claims of causality based on the simplistic use of indicators.
A key requirement for improved quality will be to subject governance indicators to stronger external scrutiny and challenge. Users need to:
- Take full account of the statistical uncertainty inherent in governance indicators and potential sources of bias. Assertions based on small differences in indicators should be viewed sceptically unless the differences exceed stated margins of error.
- Take careful account of the purpose for which individual indicators were designed and the statistical limitations of the data.
- Be alert to the normative assumptions behind governance indicators, and actively question whether they are appropriate to the context of the country in question.
- Be prepared to include more subjective indicators based on perceptions surveys and expert judgements to capture less tangible aspects of governance.
- Prioritise the development of indicators that are suited to political economy analysis and relevant to explaining the causes of observed patterns of governance.
