- For evaluators’ eyes only (21/07/2018)
- Reconciliation Week and findings from an Aboriginal health evaluation (04/06/2016)
- Evaluation amidst complexity: 8 questions evaluators should ask (04/12/2015)
- To count or not to count: Australian population data (20/02/2015)
- My pick of readings on scaling up health interventions amid complexity (12/12/2014)
For evaluators’ eyes only
Saturday, 21st July
In December 2015 I posted slides about eight questions evaluators could ask to determine if a program was ‘complexity-sensitive’. I had been spending a lot of time thinking about whether complexity was an obstacle to creating the kind of social change the Aboriginal controlled organisations and international NGOs I worked were trying to create. You can read the blog and look at the slides here. They will make the most sense to those who, like me, spend a lot of time evaluating programs to figure out what they achieved and how they made a positive difference.
My slides from 2015 have grown into a paper published in July in the British journal Evaluation. If you have access to SAGE journals through a university library, you can access it here. With permission from SAGE publications, I have uploaded the version accepted by the journal editor here.
When I did the original slides, I wasn’t convinced that complexity was an obstacle to program success. Two and a half years later, I am certain it isn’t. Every day what seems reasonable, predictable and normal is vanishing, being replaced with new policies, technologies and moral imperatives. But amidst the turmoil, programs are still making a positive difference.
Evaluators can help programs navigate external and internal changes as well as intransigence by opening up their eyes, and their lines of questioning, using knowledge about the properties of complex adaptive systems. To emphasise that being complexity sensitive isn’t radically different from what we know as good evaluation processes, I have aligned the eight questions with the DAC evaluation criteria of efficiency, effectiveness, relevance and sustainability. Those DAC criteria, incidentally, are being reviewed right now.
The eight questions in the paper, along with the corresponding DAC criteria and CAS properties, are:
|Was the program…||DAC criteria||CAS property|
|grounded in history and current priorities?||Relevant||Path dependency|
|informed by the dynamic relationships between implementers and beneficiaries and between within and between units who have different functions?||Effective||Interdependence and feedback loops|
|effective in influencing those dynamics to enable the intended change to occur, such as by introducing extrinsic or intrinsic incentives?||Effective||Interdependence and feedback loops|
|responsive to external shocks, such as new policies, program funding or process changes, new stakeholders, etc.?||Effective||External shocks and attractor states|
|accommodating of the diversity of intended beneficiaries and other agents and units in its design and implementation by employing different approaches depending on capacity and circumstance?||Efficient||Diversity|
|monitoring, reviewing and taking action based on regular information to ensure that over time the program was having the intended effect?||Efficient||Nonlinear outcomes|
|aware of and supportive of self organising and emergent behaviours relevant to the intervention?||Sustainable||Emergent behaviour|
|engaged in what would happen when the program ended?||Sustainable||Interdependence, feedback loops and path dependency|