For evaluators’ eyes only

In December 2015 I posted slides about eight questions evaluators could ask to determine if a program was ‘complexity-sensitive’. I had been spending a lot of time thinking about whether complexity was an obstacle to creating the kind of social change the Aboriginal controlled organisations and international NGOs I worked were trying to create. You can read the blog and look at the slides here. They will make the most sense to those who, like me, spend a lot of time evaluating programs to figure out what they achieved and how they made a positive difference.

My slides from 2015 have grown into a paper published in July in the British journal Evaluation. If you have access to SAGE journals through a university library, you can access it here. With permission from SAGE publications, I have uploaded the version accepted by the journal editor here.

When I did the original slides, I wasn’t convinced that complexity was an obstacle to program success. Two and a half years later, I am certain it isn’t. Every day what seems reasonable, predictable and normal is vanishing, being replaced with new policies, technologies and moral imperatives. But amidst the turmoil, programs are still making a positive difference.

Evaluators can help programs navigate external and internal changes as well as intransigence by opening up their eyes, and their lines of questioning, using knowledge about the properties of complex adaptive systems. To emphasise that being complexity sensitive isn’t radically different from what we know as good evaluation processes, I have aligned the eight questions with the DAC evaluation criteria of efficiency, effectiveness, relevance and sustainability. Those DAC criteria, incidentally, are being reviewed right now.

The eight questions in the paper, along with the corresponding DAC criteria and CAS properties, are:

 

Was the program… DAC criteria CAS property
grounded in history and current priorities? Relevant Path dependency
informed by the dynamic relationships between implementers and beneficiaries and between within and between units who have different functions? Effective Interdependence and feedback loops
effective in influencing those dynamics to enable the intended change to occur, such as by introducing extrinsic or intrinsic incentives? Effective Interdependence and feedback loops
responsive to external shocks, such as new policies, program funding or process changes, new stakeholders, etc.? Effective External shocks and attractor states
accommodating of the diversity of intended beneficiaries and other agents and units in its design and implementation by employing different approaches depending on capacity and circumstance? Efficient Diversity
monitoring, reviewing and taking action based on regular information to ensure that over time the program was having the intended effect? Efficient Nonlinear outcomes
aware of and supportive of self organising and emergent behaviours relevant to the intervention? Sustainable Emergent behaviour
engaged in what would happen when the program ended? Sustainable Interdependence, feedback loops and path dependency

 

Categories: Blog Evaluation methods Tags: , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.

  • Email

    ann.larson [at] socialdimensions [dot] com.au peter.howard [at] socialdimensions [dot] com.au

    Contact us

  • Phone

    04 2707 0683
    04 3419 5184
    08 9965 3015

    ABN

    38849688220

  • Post

    PO Box 2429 Geraldton Western Australia, 6530

    Subscribe

    Don't miss a post. Subscribe to have new blog posts delivered to your mailbox.


     

  • Site by Us&Them Studios | Log in