This paper from MandE News, an online monitoring and evaluation news service, outlines an innovative qualitative monitoring technique known as the ‘most significant change’ (MSC) approach. The MSC technique is a participatory method of collecting and analysing stories from the field which focuses on monitoring intermediate outcomes and impact. It provides a simple means of making sense of a large amount of complex information and is best suited to large-scale, open-ended projects which would otherwise be difficult to monitor easily using traditional methods.
The MSC process involves the following steps: raising interest; defining the domains of change; defining the reporting period; collecting significant change (SC) stories; selecting the most significant of the stories; feeding back the results of the selection process; verifying the stories; quantification; secondary analysis and meta-monitoring; and revising the system.
MSC monitoring is useful for identifying unexpected changes. It requires no special professional skills, encourages analysis as well as data collection and can build staff capacity. It can deliver a rich picture of what is happening and can be used to monitor and evaluate bottom-up initiatives that do not have predefined outcomes.
MSC is better suited to some programme contexts than others and has a number of advantages and drawbacks compared to other forms of M&E:
- MSC is suited to monitoring that focuses on learning rather than just accountability. The types of programmes can gain considerable value from MSC include those that are complex, large, focused on social change, participatory and highly customised.
- MSC may be less appropriate for: capturing expected change; developing good news stories; conducting retrospective evaluation; understanding the average experience of participants; producing an evaluation report for accountability purposes; or for completing a quick and cheap evaluation.
- MSC helps draw valid conclusions through thick description, systematic selection, transparency, verification, participation, and member checking.
- Some of the key enablers for MSC are: an organisational culture where it is acceptable to discuss failures; champions with good facilitation skills; a willingness to try something different; infrastructure to enable regular feedback; and commitment by senior managers.
- Problems with MSC relate to the meaning, significance and relevance of the question, the selection of SC stories, time contraints, and complaints that certain choices are ignored and feedback forgotten. Furthermore, MSC contains a number of biases as well as subjectivity in the selection process.
MSC should be considered a complementary form of monitoring which fills a number of gaps. It tells us about unexpected outcomes, encourages a diversity of views, enables broad participation, puts events in context and enables a changing focus on what is important. Nevertheless there is scope for improvement:
- MSC can be fine-tuned by developing methods for incorporating insights into programme planning, eliciting the views of programme critics, participatory analysis of stories en masse and improving the feedback process.
- Evaluation approaches that would complement MSC include those that provide quantitative evidence of the achievement of predetermined outcomes, evidence of ‘average’ experiences and views of non-participants.
- Further research should focus on: the extent of unexpected changes and negative stories that are reported, and ways of strengthening both the feedback loop and the link between dialogue and programme planning. It might also serve to investigate how to strengthen MSC for use in summative evaluation and combine MSC with deductive approaches.
