Overview

The inaugural Learning for Action Fund was launched as a pilot initiative by the GitLab Foundation to increase the impact measurement and feedback practice capacity of its grantees.

Strong demand for feedback and measurement support: After announcing that GitLab Foundation grantees were eligible for up to $50,000 in additional capacity-building funds, nearly 70% of its grantees applied. This strong response demonstrates the demand for impact measurement and client feedback by the nonprofit sector.

Split demand between external and internal support: There were limited restrictions on how the funds could be used, and grantees were encouraged to use the funds for learning activities that would most likely lead to programmatic improvements or validate their impact to attract additional funding. Of the 27 applications, approximately 40% focused on using the funds for external vendors to support their projects, and 60% focused the funds on increasing internal staff capacity.

Split demand between improving program design and impact validation: About 60% of projects were focused on learning projects that informed the organizations how to best improve their programs, and 40% were primarily focused on projects that validated the impact of their programs.

Research activities were diverse across our grantee portfolio:

And a diverse set of goals:

Ultimately, 12 $50,000 proposals were selected for grant awards, totaling $600,000 in funding. Six learning grants were to organizations in the U.S. and six outside the U.S., in Colombia and Kenya.

1. Key Challenges Cited

  1. Longitudinal income tracking and low survey response rates

Many organizations have low response rates to their surveys from alums who have left their programs more than 3 months prior and have significantly lower response rates for those who last received services one year prior. Several organizations (CodePath, FreeWorld, CAEL, Industrial Commons, Tech Impact, Generation, BuildHer) have applied for Learning for Action Fund grants focused on improving how they track income over time - with some proposing to use tax reporting systems, some using incentives to increase response rates, and others proposing to scrape LinkedIn profiles.

  1. In-house versus external focus

Some grantee partners struggle to know whether to invest in improving their internal impact measurement capabilities or to hire a third-party company with more experience. We have guided organizations to realize that both can be important and, if facing this tough choice, to start with a simple outcome survey designed by a third party (60 Decibels) to learn best practices in creating their own surveys going forward.

  1. Information silos

Many grantee partners explain that their challenge isn’t a lack of impact-related data, but rather a lack of connected information and siloed team structures. For example, teams in charge of recruitment do not have visibility to alumni data, which makes it difficult to learn which baseline student data is most associated with success or challenges after the program ends. To help our grantees, we have introduced them to data platforms and advisors. Some of our grantees plan to use Learning for Action Fund grants to implement new unified data platforms such as Poket, FluentCRM, Sopact, and to solicit expert advice from leading advisory groups such as Innovation Poverty Action’s Right-Fit Evidence group, Project Evident, and Feedback Labs.

  1. Measuring intangible outcomes

Several applications mentioned the need to learn about social impacts beyond income and employment changes. For example, Immigration Policy Lab is interested in better understanding how newly resettled refugees describe what is most important to them when joining a new community. Industrial Commons seeks to better understand how their work impacts civic participation and community engagement. Generation aims to learn about personal barriers that alums face when looking for employment, such as family responsibilities and pressures.

  1. Snapshot of technologies cited
    1. Qualtrics, SurveyMonkey, Google Forms, HubSpot, Poket
    2. Salesforce, Tableau, Looker, Alteryx, PowerBI
    3. OpenAI

2. Types of Applications

  1. 68% of eligible grantees applied
    1. 40 grantees were eligible for additional capacity-building funds, and 27 applied
  2. Applications by type of project:
    1. External vendor compared to internal staff support
      1. 37% (10/27) focused on partnering with an external vendor
      2. 63% focused on projects that were primarily conducted by internal staff
    2. Validating impact or improving impact
      1. 59% (16/27) focused on learning projects that would lead to improved programs or services
      2. 41% focused on learning projects that validated the impact of their programs. Although the majority of these validation projects would also lead to some improvements

3. Winning Grants

  1. BuildHer: “Empowering Construction Tradeswomen: Harnessing Data Insights for Greater Impact and Income Growth”
    1. Description: The BuildHer team is launching a new alumni data collection system with an innovative data collection app, Poket. This will enable the team to collect robust feedback and impact data from alumni to track and analyze impact outcomes across key stages of their program.
    2. Impact: This will be instrumental in refining their training curriculum, improving job placement rates, and ensuring that alumni are well-prepared for higher-value jobs as their careers progress.
    3. Why We are Excited: Not only does this project level up BuildHer’s ability to track impact, but it uses a novel approach that could be replicated across other GitLab Foundation grantees. Poket provides incentives for users to complete surveys and shares the community responses back to those who completed their survey, thus increasing response rates and engagement among BuildHer alumni.
  2. CareerVillage.org: “Building Consistent Feedback Loops for Enhanced User Engagement and Impact”
    1. Description: Improving user feedback culture and consistency among CareerVillage staff by partnering with Listen4Good.
    2. Impact: This project will ensure that CV’s data, partner success, and fundraising teams are grounded in real user experiences to build trust and engagement with their users. This will ensure user voices are at the forefront of their program design and iteration processes. This may lead to an improved product design, resulting in higher job placements and program completion outcomes.
    3. Why We Are Excited: This project will be our first collaboration with a leading feedback organization, Listen4Good, spun out of the donor-collaborative Fund for Shared Insight.
  3. CodePath: “Enable Robust Measurement of CodePath’s Impact on Technical Employment Rates for Underrepresented Computing Students by Improving Access to University Employment Outcomes Using LinkedIn Career Data”
    1. Description: This project will scrape LinkedIn profiles of university computer science students as well as CodePath’s students to create a valid comparison analysis.
    2. Impact: This is an innovative solution to evaluate technical employment outcomes. Successfully designing and implementing a prototype will contribute to CodePath’s organizational goals as well as support relationship-building throughout the tech education community through sharing relevant data about technical employment outcomes.
    3. Why We are Excited: If this solution is developed and effectively implemented, it can be instrumental for the workforce development sector. This project has the potential to collect and analyze data for longitudinal comparison impact studies which will not only be helpful for CodePath to demonstrate its program’s impact but also create systems-level change in the sector.
  4. Digital Green Foundation: “Bridging Gaps: Farmer.Chat Engagement and Outcomes for Kenyan Small-Scale Farmers”
    1. Description: Partnering with an external impact measurement vendor, 60 decibels, to measure outcomes and collect farmer feedback.
    2. Impact: They have trained 6,200 lead farmers in Kenya to use Farmer.Chat. They will use a 3rd party vendor to randomly interview 400 farmers. The insights and evidence gathered through this assessment will be crucial for helping them to refine our approach, improve Farmer.Chat’s effectiveness in terms of meeting farmers’ information needs, ensuring that the information farmers receive meets high quality standards, and formulating a scale-up plan in Kenya.
    3. Why We Are Excited: This project will be our first collaboration with a leading impact measurement organization, 60 Decibels. It is also a comparison study to a similar study they did for Farmer.Chat in India. These reports are also fantastic for sharing with our board and a potential blog article. Lastly, because thousands of farmers use the app, any change to their program will have a considerable impact due to the reach alone.
  5. FreeWorld: “Building the Infrastructure for a New Gold Standard in Longitudinal Income and Recidivism Data”
    1. Description: Partnering with an external vendor, Tax Guard, to collect tax returns for program participants and alumni to validate wage and employment outcomes and conduct continuous recidivism checks through a partnership with Checkr.
    2. Impact: These data collection projects will help FreeWorld develop impact evidence, inform service delivery, and improve job placement outcomes. At a systems level, this will lead to the development of a dynamic, open-sourced national data platform to provide the evidence base to accelerate criminal justice reform efforts.
    3. Why We Are Excited: Currently most organizations conduct surveys at different time intervals to collect employment and wage data which have really low response rates. By accessing tax transcripts and analyzing them, organizations can get direct insights into the effectiveness of their programs, employment and wage outcomes after the participants have graduated. If designed and implemented effectively, this can be a pioneering solution for the workforce development sector.
  6. Generation: “Empowering Livelihoods: Mapping the Path to a Living Wage in Kenya”
    1. Description: Conducting focus groups and surveys with alumni to learn about key barriers preventing employment and financial health outcomes.
    2. Impact: Improve durability outcomes in Generation’s lower middle-income countries by understanding the barriers that are restricting the economic mobility for our alumni in LMICs, and causing long-term unemployment, employment breaks, and issues in seeking promotions.
    3. Why We Are Excited: Generation will be incorporating stakeholders’ voices by directly engaging with its participants to understand their challenges. The insights from this study will result in program modifications, improve data collection, and advocate to address systemic barriers.
  7. Human-I-T: “Amplifying Impact: Enhancing Digital Equity through Strategic Learning and Data-Driven Insights”
    1. Description: Partnering with an external impact measurement vendor, 60 decibels, to measure outcomes and collect participant feedback.
    2. Impact: They will randomly interview 450 recipients of Human-I-T services going back to 2021. The results from this in-depth survey will help Human-I-T to
      1. Fine-tune Human-I-T’s program offerings.
      2. Increase the efficiency and scalability of its services.
      3. Identify and address service gaps.
      4. Fundraise - With clear, evidence-based results, Human-I-T can more compellingly demonstrate the value of its work, attracting new collaborators and resources.
      5. Improve long-term feedback systems - The data collected will serve as a foundation for ongoing evaluation, helping the organization regularly assess and enhance its programs to meet the evolving needs of the communities it serves.
    3. Why We Are Excited: Again, this is the first project to partner with a leading outcome measurement organization, 60 decibels. This also directly improves the quality of results data we get regarding our current grant with Human-I-T, and lastly, it will set them up to continue these surveys in-house.
  8. Immigration Policy Lab, GeoMatch: “GeoMatch: Measuring Impact and Improving Refugee Integration Outcomes”
    1. Description: Conducting qualitative interviews with resettled refugees to better understand employment and quality of life experiences beyond 90 days to improve GeoMatch’s recommendation algorithm. With this grant, they will supplement that data collection in three key ways: 1) collecting outcomes for refugees at 240 days post-arrival, 2) collecting additional outcome data beyond employment, and 3) conducting qualitative interviews with refugees for a more holistic understanding of their resettlement experience.
    2. Impact: This helps GeoMatch better articulate the impact of their tool, build a cost-effective model, and leads to higher quality data that will update their matching algorithm.
    3. Why We Are Excited: This project lays the foundation for improving the required reporting to the State Department by resettlement agencies. It also provides a qualitative picture of the experiences of refugee families.
  9. Kuepa Edutech: “Gender Equality in Action”
  10. Laboratoria: “Empowering Women for the Future: Adapting Skills and Learning Pathways for the Digital Economy”
  11. Makaia: “Trust, Measure, Improve: A Community-Based Approach For Enhanced Program Impact”
  12. Research Improving People's Lives (RIPL): “Impact of the “Launch for Jobseekers” and “Launch for Employers” Reemployment Services Portals”

4. ROI Analysis

The GitLab Foundation conducts a return on investment analysis as a core part of its due diligence process to estimate the potential outcomes that would be generated from its investments. These learning grants will not have known outcomes, but the ROI analysis can estimate the programmatic improvements needed to generate a return of 100x ($100 dollars generated in lifetime earnings per $1 dollar spent).

These learning projects will reveal insights that will directly inform programmatic or organizational policy changes, which, if implemented, may increase the effectiveness of those programs. The GitLab Foundation used the existing income increase outcomes for each organization and calculated how much additional increase in annual income their programs would need to generate for their program participants to get to a 100x return. This process revealed that five of the 12 learning projects would only need to implement programmatic changes that increased their participants’ incomes by 2% or less to surpass the 100x threshold. If all projects increased incomes by 5% due to the key learnings uncovered through these projects, this cohort would generate a combined ROI of more than 300x, demonstrating the powerful impact potential that investments in learning can have.