The days of assuming that ‘good’ development work always results in reductions in poverty are over. Thankfully. Rigorous measurement is now an integral part of most development efforts – not only to evaluate and understand impact, but increasingly to also inform implementation practices in real time. ‘Learning’, when combined with a willingness to adapt, is a key tenant of the MSD approach. In simple terms this is an inherent recognition that we cannot simply rely on our assumptions of what works and what has value.

There is, however, one notable area of our work which has not faced the same degree of measurement rigor – and strangely this is the question of whether (and how) we are achieving sustainability, the primary rationale for MSD. As readers of these blogs are fully aware, the ambition to achieve sustainable outcomes, at scale, is what drove the development of MSD. This is what underpins the methodological frameworks in MSD and is the key justification for all the additional effort, resources and time typically needed to implement a systemic approach. However, to date, very little attention has been paid to assessing sustainability[1]. There has been relatively minimal emphasis on measuring what, if anything is sustained beyond the lifespan of development programmes and few funders or implementers set aside or make use of resources for an independent assessment of sustainability after a programme has closed.[2] Given MSD’s focus and reliance on measurement and learning, this is a fundamental and somewhat embarrassing gap: using our own measurement mantra, we cannot just assume that MSD and other systemic approaches necessarily result in sustainable outcomes[3]

The good news is that change is afoot. Development funders and practitioners are increasingly recognising this gap – some might say gaping hole – in the learning agenda and a small number of sustainability assessments have taken place in the last few years. This ‘next phase of learning’ includes a sustainability assessment of the Enter-Growth programme in Sri Lanka in 2019, ten years after it finished; an ongoing (first half 2022) self-sustainability assessment of selected MDF interventions in Pakistan, two years after they finished; and a planned sustainability review of MarketMakers in BiH, which will focus on developing insights for achieving sustainable outcomes. This is, we hope, a growing field and lessons from these assessments – both in terms of the factors that contribute to or hinder sustainability, and also how best to evaluate sustainable outcomes, which are markedly different to conventional programme assessments – are emerging. This blog focuses on the latter – in particular our understanding of the key steps to follow when undertaking a sustainability assessment. It is primarily intended for programme managers and funders who might consider commissioning, as well as for the practitioners who are planning to undertake, a sustainability assessment.

Before we focus on the key steps, it is worth noting that not all programmes will lend themselves to a sustainability assessment. Experience to date suggests two key factors to consider:

  • Timing of the assessment: there are no hard and fast rules about how long a period should have passed before a sustainability assessment makes sense. Less than two years means there will not be much meaningful learning, and more than five years for the first assessment will likely undermine the feasibility of the process as people move on and stakeholder memories are fallible. Although this will vary by programme and context, we expect that in most circumstances an assessment after two to three years would be useful, ideally followed up again after a further three to five years. A long-range assessment after ten years or longer can provide unique learning, but the results would likely be richer and more credible if there had also been one or two interim assessments.
  • MRM systems: combined with programme reporting, the existence of an effective, ideally independently audited, MRM system is a crucial foundation for an ex-post sustainability assessment. This is important in building understanding of what was intended, what was done, what was achieved in terms of changes, and what kind of impact these resulted in (if any). The regular use of frameworks which show a programme’s theories of change, such as results chains, make ex-post sustainability assessments substantially easier.

These are the main broad steps we have found useful for ex-poste sustainability assessments of system change programmes:

  1. Be clear about the definition of sustainability that the assessment is going to apply and measure progress against. At face value, defining sustainability may seem obvious, but it is a much-used term often meaning different things to different people, so well worth seeking agreement and clarity. For most MSD programmes the admittedly ambitious understanding of sustainability is:
    • lasting benefits for an identified target group, which are…
    • driven by lasting improvements to system performance, which are in turn…
    • driven by behaviour changes that last for as long as they continue to yield benefits, but then adapt and evolve so that they continue to yield the similar benefits over time.
  1. It is not necessary or particularly useful to assess everything a programme has done, so select the focus and boundaries of the assessment. A transparent set of criteria is helpful to identify the most appropriate focus – these criteria will be influenced by the nature of the programme and the particular interests of the funders, but it is important to include feasibility aspects such as the relative strength of the evidence base, the potential for learning, the degree of distortion that has taken place after the close of the programme (and that will impact any findings) and crucially the practicality of being able to re-trace the programme’s steps and interview key informants.
  2. Map the behaviour changes and performance changes for each system function in each of the selected regions. Using results chains to show the connections between the behaviour and performance changes is helpful.
  3. Clarify the priority questions that the sustainability assessment needs to answer. These will typically fit within the following broad categories:
    • Questions about whether behaviour changes that were facilitated by the programme have been sustained
    • Questions about whether performance changes have been sustained
    • Questions about other behaviour changes in the key system functions, that were not facilitated by the programme; and
    • Questions about attribution and contribution
  1. Develop a research plan including data collection instruments best suited to address the identified questions. If time and budget allow, it is helpful to make use of a combination of methods to explore different issues with different informant groups and triangulate findings. We found that the most interesting learning comes from qualitative research, exploring why changes were or were not sustained and digging into the various ways and extents to which they were sustained.
  1. Undertake the data collection. Key lessons to date are that:
    • Sustainability assessments take time and require flexibility – more so than a conventional programme review process. Conducting a sustainability assessment is analogous to investigative journalism – it requires pulling together fragments of scattered information, triangulating findings (which is particularly important when relying on the long-term memory of research participants), and following up on new pieces of information, which may necessitate changing the fieldwork plan to respond to new leads.
    • Ex-post assessments risk raising expectations of follow-up programmes which may bias informants’ responses and also raise an ethical quandary. Being totally clear about the interview’s purpose is important.
    • Local knowledge and connections are enormously valuable, especially given the passage of time since the intervention took place. When possible, it is helpful to involve members of the original programme team as key informants and to help implement the research plan (contact key stakeholders and so forth). However, independence and objectivity are also crucial, so when feasible we found it was valuable to have interviews conducted by independent researchers.
    • Remaining focused is important – we found it’s easy to get distracted by interesting but generic evaluation issues that are not necessarily relevant to the purpose of assessing sustainability.
  1. Analyse and interpret the results to try to answer the research questions identified at the start of the assessment. It boils down to what has been learnt about the sustainability of the programme’s interventions.

Perhaps the most important conclusion from the work on sustainability assessments to date, is that they are long overdue. Whether you’re a funder or a practitioner, if an ambition to achieve sustainable change and impact drives your programme, how do you know if you are achieving it?

[1] Bailey H and Sutherland Z, ITAD Lessons from an ex-post evaluation – and why we should do more of them. 2019. Available at: https://www.itad.com/lessons-from-an-ex-post-evaluation-and-why-we-should-do-more-of-them/

[2] For the Donor Committee for Enterprise Development (DCED) recommendations on this, see The DCED Standard for Measuring Results in Private Sector Development, Version III (2017) Available at: https://cdn.enterprisedevelopment.org/wpcontent/uploads/DCED_Standard_VersionVIII_Apr17.pdf

[3] Bailey H and Sutherland Z, ITAD Lessons from an ex-post evaluation – and why we should do more of them. 2019. Available at: https://www.itad.com/lessons-from-an-ex-post-evaluation-and-why-we-should-do-more-of-them/

Leave a Reply