Measuring what matters: rethinking impact and evaluation

Jenny: I would describe it as more ad-hoc and project specific. At that time we weren’t carrying out evaluation or story gathering from a whole organisation perspective. We were keen to work with a partner to help us better understand what impact our work was having, and how these impacts were taking place.

We also wanted to develop tools and approaches for gathering stories that were trauma informed and aligned with our values. We didn’t want to use approaches that mined the private lives of our care experienced beneficiaries.

We also need to develop our impact systems – how we capture, how we track and monitor and how we analyse and reflect on what this information is telling us.

Dougie: Blue Cabin’s situation was one that we see often – funders are asking for impact data, but it becomes an activity that is attached to each project separately, often creating quite a large administrative burden (and sometimes being ‘tagged on’ to practice at the last minute).

We wanted Blue Cabin to feel that they had developed an organisational approach to learning and evaluation, that better tells the story of how all the different aspects of the organisation’s work create impact.

Crucially, we also wanted to create something that can be live and develop over time, rather than a one-off commission every few years. I think that’s when our role started to emerge more as a ‘learning partner’, rather than an external evaluator.

Jenny: A huge impact in many ways. Working collaboratively with the tialt team has been a real joy and we have learnt so much through this process. There is a lot of trust between organisations and people – this has underpinned the rigour and critical thinking that has taken place and been at the centre of our collaboration.

Some of the key developments have included:

• We have worked slowly, over time and taken on board the array of suggestions and guidance from tialt about how we can strengthen our evaluation processes and systems. We have met monthly throughout this time to ensure there is a focus on this process and that momentum isn’t lost.
• Outside of these meetings we have collaborated on specific pieces of work including developing participant trackers and evidence trackers.
• We have introduced mid year evaluation check-ins with the core Blue Cabin team so that they can learn about what is emerging from the projects they are delivering. We also meet annually as a wider team to dive into the detail of what the impact of our work has been and reflect on what this means about our work and practice.
• In 2024 we also set about commissioning three Blue Cabin Associate Artists to develop brand new evaluation tools that we can use to gather feedback from participants. This has been an incredibly valuable and insightful piece of work, guided gently and carefully by tialt throughout.

Jenny: The importance of taking things slowly and working together over the long term. This has meant that, as small organisations, there isn’t pressure to work in a contained period of time with a fast deadline. This has also allowed things to percolate and develop over time.

Blue Cabin has learnt a huge amount about the impact of our work, we know now what outcomes emerge and why and we can see patterns and themes emerging. We more clearly understand what our work means to our participants and we can make the case for our work in a more authentic and clear way.

That we can get to the nub of some tricky stuff too. This work involves Blue Cabin and tialt exploring our values, ways of working, ethical frameworks and it matters deeply to both organisations. Due to the trust and strong relationship at the centre of this collaboration we can go to those places and contain them safely in a professional space.

Dougie: Absolutely. Whether we are defining the role as ‘learning partner’, ‘evaluator, or ‘impact measurement’, we’re inevitably going to find things that are not working as well as they could, or grey areas that can feel uncomfortable. But having deep trust, mutual respect and genuinely values-led approaches to our work means that we can navigate this with care.

As tialt we’ve learned a huge amount from the partnership.The collaboration with the artists to create trauma-informed creative methods is a great example of where we’ve considered a range of needs and knowledge and worked to our mutual strengths. I think this came from us (tialt) designing a set of visual cards as a method and sharing with Blue Cabin, who, rightfully, fed back that they didn’t feel they were quite right for the people they work with. From there we thought ‘the artists are the ones doing this work, what can they tell us?’, ‘how can we gently test emerging approaches before finalising things?’, ‘how do these feel in relation to Blue Cabin’s values and practice more broadly’. It took us down a different, but much more appropriate road, and to a place that we all feel we’re genuinely drawing on mutual and complementary expertise. This is at the heart of strong collaboration.