Measuring What Matters: From Conversations to Collective Impact

How can systems change truly be measured? What if the process of measuring impact could actively fuel learning, build trust, and drive collective action?

These were just a couple of the questions we explored in our latest fireside chat featuring Erin Simmons (Chief Operating Officer at Imaginable Futures), Enyi Okebugwu (Senior Program Manager at Imaginable Futures), Jewlya Lynn (Founder of Policy Solve and Senior Advisor on Learning & Impact at IF), and Nathalie Zogbi (Lead Program Manager at Imaginable Futures). Together, they discussed moving beyond traditional KPIs, gathering and integrating conversation-based insights, leveraging AI in qualitative analysis, and reimagining measurement as a tool for learning and connection.

Unlocking Systems Change Insights Through Conversation

Jewlya jumped into the discussion by challenging traditional approaches to measuring impact: “It’s easy when we’re starting to think about impact, to go straight to a theory of change…The problem with that is…you’re centering your own understanding and your own work, but the system is a whole lot more than that.” Instead, she advocated starting with the fundamental questions: “How are we going to use this information, with whom, and for what purpose? What’s going on in the system that we can’t see? What is the information we need to know?”

This learning-first, values-driven mindset sits at the core of Imaginable Futures’ Systems & Impact Conversation Guide, a core data collection tool in our measurement framework that centers dialogue as a primary method for understanding change in complex systems.

Developed with input from partners and inspired by peers, the guide begins with an open prompt. “The first question is incredibly wide open,” said Jewlya. “It just asks: What are the changes you’re observing in the system? Some participants sometimes take 20, 30, even 40 minutes to answer that one question. The rest of the protocol just prompts and builds on things they talked about.” 

Erin highlighted how this approach creates space for deeper insights:

She also added: “The beauty is that we didn’t have to lose the rigor in this choice [to use the Conversation Guide], but we redefined rigor against the actions we knew [based on the results of gathering the data] we could take…and that our grantees and the broader ecosystem might take.”

Nathalie shared how the Conversation Guide has been unexpectedly meaningful for strengthening partner relationships: “We were able to build more intimate rapport with our partners…through the protocol and the framework. Many of our partners said that the questions we asked them were helpful in spurring more strategic thinking and invited them to step away from the day-to-day busyness to think systemically about the systems which we are a part of.”

Enyi offered another powerful example from his own work: “One of our partners reached out in response to the story survey and essentially said: ‘We do have a really interesting story to share, but we wanted to give more nuance and context and also chat live through how our work lines up with what we’re seeing across different efforts.’ This led to greater transparency, trust building, and open dialogue around an emerging concern in the system.”

Leveraging AI While Centering Human Stories

The depth of these conversations created both an opportunity and a challenge. With over 60 conversations generating approximately 1,500 pages of rich qualitative data, the team needed to find a way to quickly translate these rich conversations into actionable insights without losing their nuance and to do so with limited capacity.

While some might view AI as incompatible with a human-centered approach, Jewlya explained how it became a practical tool for moving forward: “It’s always a little controversial to have a conversation about AI, particularly when you’re doing work that is very driven by and connected to the voices of those most affected by the system…We knew if we took a traditional approach to coding and analyzing this data, it would take months…and we would lose momentum. Instead of spending 60 to 80 hours per analysis, the whole process took about 20 hours, meaning we could get it all done in one month.”

To maintain rigor and reduce bias, the team built in several safeguards: staff conducted structured debriefs immediately after each conversation to capture key takeaways; the analyst then reviewed transcripts and notes to identify relevant jargon, quotes, policies, and context-specific language before prompting the AI; and final outputs were assessed against human insights to ensure key patterns weren’t missed or misrepresented.

Embracing the Learning Journey

Throughout the conversation, speakers emphasized the ongoing, iterative nature of the approach and the patience, adaptation, and collaboration critical to its success:

• Nathalie was candid about the challenges: “A lot of our partners have been asked very different kinds of questions from funders for many years…so being asked to take a more systems-oriented view…is a new kind of muscle [for them].” 
• Jewlya’s echoed this, emphasizing process over replication: “Don’t replicate this framework—not because it hasn’t been effective for Imaginable Futures, but because the thing that was most powerful here was the process this team took to figure out what they needed, and the real commitment to looking at the system first rather than centering everything on their own work.”
• Enyi added: “Learning is a nonlinear, constant journey. We’re still adjusting by the week how we undertake our learning work.”
• The fireside chat concluded with Erin saying:

A full transcript of the event is available to read here. The event is part of our The Future We Imagine virtual fireside chat series, where we explore different approaches to advancing equity and systemic change in education and beyond.