A case study – determining how far local food systems are transitioning positively in Spain
The Challenge
Stone Soup was commissioned by the Fundación Daniel and Nina Carasso (FDNC) to help them evaluate the systems change that they had achieved in Spain through their Sustainable Food programme. One of the flagship projects in this Sustainable Food programme is the ‘Programa SAT’ (Sistemas Alimentarios Territorializados) which aims to create systems change at a local, place-based level through the creation of public-private-NGO partnerships to define and implement public policies that promote local sustainable food systems. The FDNC has funded SAT projects in a significant number of municipalities in Spain. The foundation believed that there had been significant impact achieved through funding of these programmes, but this had not been systematised or evaluated in a rigorous way. FDNC felt that the more ‘traditional’ theory of change and programme/project impact evaluation approaches did not sufficiently encapsulate the systems change dimension. Indeed, there is a broader issue in the impact field of how far the most common frameworks/standards/indicators used to measure and evaluate change do not sufficiently encapsulate or account for systems change generated by different actors. This project is part of a new wave of more innovative evaluation approaches interrogating how to adequately evaluate systems change initiatives which are becoming increasingly important.
The Practice
In a co-participative process, Stone Soup and FDNC undertook a systems mapping process (using the Omidyar methodology) identifying the short-term and long-term objectives for enabling the transition towards sustainable food systems in Spain, the key barriers and enablers and the cause/effect relationship between them. We broke down this cause/effect relationship into three levels: mental, relational and structural based on the important work of Peter Senge, Mark Kramer and John Kania. Through this mapping which we tested with external stakeholders, we were able to identify the foundation’s key levers for catalysing systems change. We developed a list of systems indicators to help FDNC better measure how far local, sustainable food systems were changing/evolving in Spain over the last five years based on their intervention and conducted an initial impact evaluation through extensive qualitative interviews and a literature review. Given the challenges of collecting data from a fragmented ecosystem, we also reviewed what data can/should be collected going forward to be able to better assess systems change using the proposed systemic indicators framework.
Results
The project generated a greater internal strategic clarity within the foundation about how systems change can be achieved in the SAT programme (in particular what the key levers for systems change are) and what the foundation’s contribution has been in the past and can be in the future within the system. An impact monitoring framework was developed to measure systems change which is a resource for FDND as well as the wider community engaged in local sustainable food systems not only in Spain but also internationally. The impact evaluation helped FDNC understand what had been achieved (where sufficient information existed) so far and importantly what information needed to be collected in the future to be in a stronger position to understand how far the system was transitioning.
Insights / Advice to Peers
- Managing complexity: breaking the system down into meaningful parts or themes was key for us to manage inevitable system complexity rather than trying to map the whole local sustainable food system in Spain at once. The systems mapping process itself requires flexibility as the time taken to map is quite difficult to estimate until you start trying.
- Ensure a diversity of voices is included: having a systems mapping team which includes people with diverse experiences, skills and perspectives is important and testing the systems map with those who have not participated in its development is also critical.
- Creating a meaningful indicator framework involves prioritising: not all components/features of systems change can be monitored and it is important to prioritise a few, key indicators which measure systems change. Using both qualitative and quantitative data is important.
- Investing in data collection is critical: in a systems change environment, often individual actors are not incentivised to collect impact data, or do so in a local, fragmented way. A funder such as FDNC is a hugely important actor which can invest in generating the overarching systems-level data that is needed assess more broadly which interventions are working, how and why and where tipping points are being reached.
Further Resources
- Stone Soup Consulting CoLab Award Submission
- Fundación Daniel y Nina Carasso, ‘Los cuaderno SAT, Sistemas Alimentarios Territorializados’, January 2019 (in Spanish)
- Summary of systems mapping process
- Julia Coffman, ‘A Framework for Evaluating Systems Initiatives’ (2007); Marc Cabaj, Evaluating Systems Change Results. An Inquiry Framework (2019); Water of Systems Change, May 2018
Contacts
Leonora Buckland, Principal Consultant, Stone Soup,