Learning by Listening to Those We Serve

Jessica and I are both parents of young children, and as we go into each day, we do our best to listen to our kids to help guide them to learn and grow. What they are thinking, how they are feeling, and how we interpret that communication back to us goes into how we might modify our behaviors in our role as parents.

How is this related to our day jobs, you may ask? Deeply listening to others — from our children, to our colleagues, to our investments, to those “customers” our portfolio serves — is something that we believe is a critical skill today, and one we wish was more widely practiced in our field. Listening to our customers and working in collaboration with them to improve and find solutions is something our Education team at Omidyar Network truly values.

Move Fast but Measure Impact

With a “100-year gap” for poor children to catch up to the educational levels of today’s wealthy children (Brookings Institute), Omidyar Network’s Education team approaches our work with a sense of urgency and willingness to take more bold bets when it comes to investing in emerging education innovations. Why? Because taking a chance that a solution might leapfrog current approaches means more students are closer to reaching their potential. However, this also means first and foremost, we want to be able to ensure those innovations are leading to more quality learning.

But how can we get more regular signals that can help us understand if our bets are succeeding — or even failing — to learn if we are on the right track? How can we surface common challenges and promising solutions to learn from and share with others? Like many other impact investors and philanthropic funders, we have limited data at the portfolio-level that can be used to meaningfully track progress toward impact and inform future work. However, we are exploring methods to support our portfolio, gain a deeper understanding for ourselves, and share with our collaborators and the broader field.

Thus, it seemed obvious that beneficiary survey data could provide a terrific opportunity to glean insights into the relative performance of our portfolio. It always starts with listening to the customer. But how could we hear from thousands of customers in 14 countries across four continents in a way that was customized, meaningful, and would give useful results? For this work, we enlisted the help of Acumen’s Lean Data team. Lean Data is an approach that helps social enterprises quickly and easily hear from customers about their behavior, feedback, and social performance.

Our Lean Data Education Sprint

We have a diverse portfolio, even at a sector-level, which makes it challenging to present one survey that would be useful to all. Our Education portfolio includes more than 90 nonprofit and for-profit organizations across the globe who provide a wide set of services to a wide range of ages. We needed to ensure participation in this survey would be seen as beneficial to our investees, and the data collected as useful for their own internal purposes.

As an entrepreneur-first funder, we are very careful not to enforce our standard measures on our portfolio. We were thrilled when 71 percent of our eligible portfolio opted to participate.

Between July and October 2018, Lean Data conducted projects with 24 Education investees across 14 countries in 11 languages. These surveys, a mix of voice and email, were customized depending on business model (skilling, school, product) and audience (students, teachers, parents), and through them we heard from more than 4,800 students, parents, teachers, and administrators. Our collective teams were all excited to jump into this work to gather rich, comparable social impact data on our Education portfolio for the first time.

The resulting report provided us with a valuable source of information and we have summarized some of these highlights to share with others outside our portfolio. For example, we gained insight into key value drivers for sub-segments of our portfolio. As expected, we found that freemium products are more inclusive than premium products. That said, we were surprised to learn that freemium products were perceived as having only slightly lower reported value than their paid counterparts. We also generated new insights by combining our own internal data with survey data. For example, the value constituents placed on services validated the high, medium, and low depth definitions that we had used for our retrospective impact analysis of our first 10 years.

Each participating investee received detailed decks, providing customer feedback on where they are strong and where they could improve. This has provided us with an opportunity to engage meaningfully with our investees and support their journey for ever more impact. And what was nice to hear back was that our investees shared this offered them a way to improve engagement and communication with their own staff, donors, and beneficiaries.

Jessica Sager, Chief Executive Officer at All Our Kin, shared how she used the report as a conversation tool:

Other investees used the data to validate their business plans. Caroline Hu Flexer, CEO and Co-Founder of DuckDuckMoose and Khan Academy Kids, shared:

This first portfolio-level Education survey is just a starting point. We will continue to explore methods to meaningfully assess progress toward our impact objectives so as to support our portfolio, gain a deeper understanding for ourselves, and share with our collaborators and the broader field. We plan to refine these measures and how this information is used over time. We are hungry for these learnings and believe this will help us become increasingly confident in our results. It will also ensure we always stay focused, listening to our investees and to the people we are working to serve, and acting based on insights grounded in data and their voices.

We hope you will find highlights from our learnings from this report helpful and welcome ideas for further improvement. Most importantly, thank you for listening!