The surveys conducted in Afghanistan and other challenging environments tend to polarise opinions, with either too much or too little confidence often being placed in the findings. This paper examines three concerns regarding perception surveys in Afghanistan:
- that the responses are unreliable
- that the samples are unrepresentative of the populace
- that the data is commonly not genuine.
It concludes that surveys’ comparative findings are more reliable than absolute results, and that surveys remain a blunt instrument through which to develop an understanding of opinions. However, the concerns frequently associated with survey research can often be overcome during the analysis phase.
The reliability of the reponses
The paper argues that social desirability biases and attempts to manipulate the research create unquantifiable distortions, so it is not possible to have absolute faith in the results. However, greater degrees of confidence can be retained with regard to the comparative conclusions based upon the (admittedly also problematic) assumptions that (a) these biases are sufficiently constant between locations and over time, and (b) that the research methods are applied consistently.
The representativeness of the sample
A sample may be unrepresentative either because certain locations are outside of the reach of the pollsters, or due to the demographic characteristics of the respondents. The findings are undermined as the limitations of the sample are effectively reduced from ‘the residents of Region A’ to ‘the residents of accessible parts of Region A’. However, the comparative results over time may again retain value provided that: (a) the research methods have been consistent, and (b) the data from locations that have on occasions been beyond the reach of the pollsters are excluded during the analysis phase.
Further, it is possible to correct sample biases through ‘weighting’, a technique that enlarges the unrepresented populations at the expense of those that are overrepresented. Whilst adding considerable complexity, the weighting technique can be used simultaneously to remove multiple biases covering gender, age, ethnicity, income, occupation, and so on. However, this is no panacea given the lack of reliable demographic data upon which to base these calculations, and as the technique is inapplicable where the sample size of specific subsets of the populace is excessively small.
The reliability of the pollsters
With respect to data genuineness, sanity checks can be undertaken during the analysis phase. These involve determining whether the expected associations occur, for instance, between (a) local perceptions of security and actual levels of violence, (b) urban residences and the use of specific media forms (specifically television), and (c) the ethnicity or tribe of the respondents and their political loyalties. It might not be possible to detect low-level fraud through this process, but wide-scale fabrications are revealed by a consistent failure to identify the expected correlations. It is also possible to identify duplicated data during the analysis phase by isolating uncommon responses – e.g. claims to be from a rare tribe, or to have an infrequently large number of offspring.
Conclusions
Comparative findings such as “the respondents from District X are more supportive of the Karzai regime than those from District Y” are more reliable than absolute results like “34 percent of those from District X support Karzai.”
The concerns frequently associated with survey research can often be overcome during the analysis phase through sanity checks and techniques such as weighting. Funding bodies can assist in this process by becoming suitably informed about the practicalities of surveys, and by ensuring that research organisations provide sufficient information about their methods.
Surveys remain a blunt instrument through which to develop an understanding of opinions. It remains necessary for those operating in this environment to place greater value upon the more nuanced insights delivered by qualitative research.
