Measuring for Impact Beyond The Surface

Having data to understand and increase progress is, no doubt, important across all sectors. Whether your organization's mission is to expand access to learning opportunities, increase workforce opportunities for youth, or to reduce carbon emissions in our ozone layer, we all know data helps us make sense of the world and its complex problems.

But consider how one might measure growing confidence among youth? Or the strength of emerging relationships between racial equity leaders and education policymakers? These questions require us to lean into different kinds of measurement tools, and to listen deeply to the communities we aim to serve. Many of us have seen this photo of an iceberg above the water as being only a tiny part of the larger picture. The metaphor, in many ways, could be applied here: So much change takes place beyond the surface.

Someone wise named Einstein once said: “Not everything that matters can be measured, and not everything that can be measured matters.” In philanthropy, there’s a tendency to overemphasize the importance of easily measurable data, often clashing with the ever-evolving nature of impact itself.

The reality is: Systems change work is complex. The traditional “gold standard” for measuring impact can take many years or even decades; it can require large sample sizes and controlling for different variable factors. Realistically, many of our partners who are working on urgent issues that impact families and communities don't have the luxury of time on their side. Ultimately, embracing all types of data, staying flexible and creating a trusted space for learning can serve as the basis for a more effective understanding of impact.

1. Data is there. We just need to listen to it.

Undoing systemic inequities requires to develop an understanding of the forces at play in a system, including but not limited to the mindsets, power structures and flows of information that underlie relationships and institutions. Further, when working with communities that are historically and presently excluded, quantitative data is often not available. Thus, other forms of qualitative evidence gathered through deep listening are essential. What does it look like to expand the type of information that philanthropies collect, value and use to include qualitative data?

Lean into stories of lived experience

Stories of lived experience are useful data points that can uncover new insights, inform how to interpret existing data, and identify effective solutions. For instance, we know that student parents represent more than one in five college students across the US—at nearly 4 million, they are a fairly sizable student population—and a population whose educational success has massive ripple effects on the next generation. Yet, very few postsecondary institutions track the parenting status of their students, leaving many college leaders in the dark about the reasons why less than half of student parents graduate. Without necessary quantitative data on student parents, we have leaned on stories of their lived experience through Ascend at the Aspen Institute’s “1 in 5” podcast and Nicole Lynn Lewis’s Pregnant Girl book. Not only has their lived expertise helped inform our strategies, investments and impact, but their stories have formed the basis for progress, including the kind that can be easily measured.

2. Listening and flexibility enable the emergence of promising new opportunities.

To identify opportunities that can lead to progress and impact, we need to be flexible and refine our approach based on what we are hearing and learning from our partners. There is no perfect way to assess impact, but we can be better when we form deep relationships with those closest to the issues and work with them to understand what impact and success really mean to them. , and what is most wanted and needed.

In Brazil, we work at the intersection of education and racial equity primarily within the K-12 system. Yet, wWhile listening to our Black and Indigenous partners, it became clear that the potential repeal of Brazil’s Quota Law—an affirmative action law that ensures that Black, Indigenous and low-income Brazilians have access to quality higher education—would have wide-ranging implications across the education system and society. We simply could not risk the unwinding of this anti-racist law that has helped to develop a generation of Black and Indigenous educators, community leaders and so many more.

Understanding the urgency, we responded by moving beyond our K-12 focus to support multiple advocacy partners working to maintain the law. And while we’re still watching to see what our impact will be and if the Quota Law will stand, we’ve learned that to be better partners in advancing systems change, we need to regularly and actively listen and step beyond our “written” strategy when called for.

3. Vulnerability catalyzes learning. Create space for it.

A strong internal culture of learning can position an organization to effectively pre-empt, respond to, or act on ongoing shifts in the system. This begins with cultivating an environment where team members feel safe being vulnerable.

We took several steps to begin creating a trusted space where we can learn together, both internally and with others. Instead of a separate Measurement, Evaluation, and Learning team, we asked individuals within our existing teams to take on the role of learning facilitators, providing teams with the autonomy to adjust the learning processes to fit their own contexts and needs As a result, within each of the teams, learning has unfolded in different ways—from “learning calls” in Africa, to a “learning log” in the US; and “system-sensing roundtables” in Brazil.

More broadly as an organization, we intentionally shifted our grant approval meetings to be more learning-focused — enabling our investment leads to name their questions, what they don’t yet know, and what potentially might never be known. Likewise, we updated our year-end individual evaluation process to emphasize learning and growth.

Conclusion

Ultimately, embracing all types of data, staying flexible and creating a trusted space for learning can serve as the basis for a more effective understanding of impact. By redefining what impact means, leaning into our values, and deeply listening, we are better able to understand the transformation that happens beyond the surface. We also know that this work will continue to evolve, and new ways to look at impact will arise. As curious learners, we look forward to sharing our learnings with all of you.