Overview

The inaugural Learning for Action Fund was launched as a pilot initiative by the GitLab Foundation to increase the impact measurement and feedback practice capacity of its grantees.

Strong demand for feedback and measurement support: After announcing that GitLab Foundation grantees were eligible for up to $50,000 in additional capacity-building funds, nearly 70% of its grantees applied. This strong response demonstrates the demand for impact measurement and client feedback by the nonprofit sector.

Split demand between external and internal support: There were limited restrictions on how the funds could be used, and grantees were encouraged to use the funds for learning activities that would most likely lead to programmatic improvements or validate their impact to attract additional funding. Of the 27 applications, approximately 40% focused on using the funds for external vendors to support their projects, and 60% focused the funds on increasing internal staff capacity.

Split demand between improving program design and impact validation: About 60% of projects were focused on learning projects that informed the organizations how to best improve their programs, and 40% were primarily focused on projects that validated the impact of their programs.

Research activities were diverse across our grantee portfolio:

And a diverse set of goals:

Ultimately, 12 $50,000 proposals were selected for grant awards, totaling $600,000 in funding. Six learning grants were to organizations in the U.S. and six outside the U.S., in Colombia and Kenya.

1. Key Challenges Cited

  1. Longitudinal income tracking and low survey response rates

Many organizations have low response rates to their surveys from alums who have left their programs more than 3 months prior and have significantly lower response rates for those who last received services one year prior. Several organizations (CodePath, FreeWorld, CAEL, Industrial Commons, Tech Impact, Generation, BuildHer) have applied for Learning for Action Fund grants focused on improving how they track income over time - with some proposing to use tax reporting systems, some using incentives to increase response rates, and others proposing to scrape LinkedIn profiles.

  1. In-house versus external focus

Some grantee partners struggle to know whether to invest in improving their internal impact measurement capabilities or to hire a third-party company with more experience. We have guided organizations to realize that both can be important and, if facing this tough choice, to start with a simple outcome survey designed by a third party (60 Decibels) to learn best practices in creating their own surveys going forward.

  1. Information silos

Many grantee partners explain that their challenge isn’t a lack of impact-related data, but rather a lack of connected information and siloed team structures. For example, teams in charge of recruitment do not have visibility to alumni data, which makes it difficult to learn which baseline student data is most associated with success or challenges after the program ends. To help our grantees, we have introduced them to data platforms and advisors. Some of our grantees plan to use Learning for Action Fund grants to implement new unified data platforms such as Poket, FluentCRM, Sopact, and to solicit expert advice from leading advisory groups such as Innovation Poverty Action’s Right-Fit Evidence group, Project Evident, and Feedback Labs.

  1. Measuring intangible outcomes

Several applications mentioned the need to learn about social impacts beyond income and employment changes. For example, Immigration Policy Lab is interested in better understanding how newly resettled refugees describe what is most important to them when joining a new community. Industrial Commons seeks to better understand how their work impacts civic participation and community engagement. Generation aims to learn about personal barriers that alums face when looking for employment, such as family responsibilities and pressures.

  1. Snapshot of technologies cited
    1. Qualtrics, SurveyMonkey, Google Forms, HubSpot, Poket
    2. Salesforce, Tableau, Looker, Alteryx, PowerBI
    3. OpenAI

2. Types of Applications

  1. 68% of eligible grantees applied
    1. 40 grantees were eligible for additional capacity-building funds, and 27 applied
  2. Applications by type of project:
    1. External vendor compared to internal staff support
      1. 37% (10/27) focused on partnering with an external vendor
      2. 63% focused on projects that were primarily conducted by internal staff
    2. Validating impact or improving impact
      1. 59% (16/27) focused on learning projects that would lead to improved programs or services
      2. 41% focused on learning projects that validated the impact of their programs. Although the majority of these validation projects would also lead to some improvements

3. Winning Grants

https://lh7-rt.googleusercontent.com/docsz/AD_4nXdik1XTaO9Jult2SdzQuAYPWrRvnsx2zI7Ni-hxo5J3PQZMyYhcyGV2Lp-fNVp1Bcce64xr6lYW9m2h9lLyCtOGPkKi8YZERbkJD9ZneCqnIhUwqeYbDIzEnIlvXpRL0aTNYdlBOZJgCn7IWUwTcwkR6Siz?key=tdIQTx5eAdUsD40PGLz9rw

  1. BuildHer: “Empowering Construction Tradeswomen: Harnessing Data Insights for Greater Impact and Income Growth”
    1. Description: The BuildHer team is launching a new alumni data collection system with an innovative data collection app, Poket. This will enable the team to collect robust feedback and impact data from alumni to track and analyze impact outcomes across key stages of their program.
    2. Impact: This will be instrumental in refining their training curriculum, improving job placement rates, and ensuring that alumni are well-prepared for higher-value jobs as their careers progress.
    3. Why We are Excited: Not only does this project level up BuildHer’s ability to track impact, but it uses a novel approach that could be replicated across other GitLab Foundation grantees. Poket provides incentives for users to complete surveys and shares the community responses back to those who completed their survey, thus increasing response rates and engagement among BuildHer alumni.
  2. CareerVillage.org: “Building Consistent Feedback Loops for Enhanced User Engagement and Impact”
    1. Description: Improving user feedback culture and consistency among CareerVillage staff by partnering with Listen4Good.
    2. Impact: This project will ensure that CV’s data, partner success, and fundraising teams are grounded in real user experiences to build trust and engagement with their users. This will ensure user voices are at the forefront of their program design and iteration processes. This may lead to an improved product design, resulting in higher job placements and program completion outcomes.
    3. Why We Are Excited: This project will be our first collaboration with a leading feedback organization, Listen4Good, spun out of the donor-collaborative Fund for Shared Insight.
  3. CodePath: “Enable Robust Measurement of CodePath’s Impact on Technical Employment Rates for Underrepresented Computing Students by Improving Access to University Employment Outcomes Using LinkedIn Career Data”
    1. Description: This project will scrape LinkedIn profiles of university computer science students as well as CodePath’s students to create a valid comparison analysis.
    2. Impact: This is an innovative solution to evaluate technical employment outcomes. Successfully designing and implementing a prototype will contribute to CodePath’s organizational goals as well as support relationship-building throughout the tech education community through sharing relevant data about technical employment outcomes.
    3. Why We are Excited: If this solution is developed and effectively implemented, it can be instrumental for the workforce development sector. This project has the potential to collect and analyze data for longitudinal comparison impact studies which will not only be helpful for CodePath to demonstrate its program’s impact but also create systems-level change in the sector.
  4. Digital Green Foundation: “Bridging Gaps: Farmer.Chat Engagement and Outcomes for Kenyan Small-Scale Farmers”
    1. Description: Partnering with an external impact measurement vendor, 60 decibels, to measure outcomes and collect farmer feedback.
    2. Impact: They have trained 6,200 lead farmers in Kenya to use Farmer.Chat. They will use a 3rd party vendor to randomly interview 400 farmers. The insights and evidence gathered through this assessment will be crucial for helping them to refine our approach, improve Farmer.Chat’s effectiveness in terms of meeting farmers’ information needs, ensuring that the information farmers receive meets high quality standards, and formulating a scale-up plan in Kenya.
    3. Why We Are Excited: This project will be our first collaboration with a leading impact measurement organization, 60 Decibels. It is also a comparison study to a similar study they did for Farmer.Chat in India. These reports are also fantastic for sharing with our board and a potential blog article. Lastly, because thousands of farmers use the app, any change to their program will have a considerable impact due to the reach alone.
  5. FreeWorld: “Building the Infrastructure for a New Gold Standard in Longitudinal Income and Recidivism Data”
    1. Description: Partnering with an external vendor, Tax Guard, to collect tax returns for program participants and alumni to validate wage and employment outcomes and conduct continuous recidivism checks through a partnership with Checkr.
    2. Impact: These data collection projects will help FreeWorld develop impact evidence, inform service delivery, and improve job placement outcomes. At a systems level, this will lead to the development of a dynamic, open-sourced national data platform to provide the evidence base to accelerate criminal justice reform efforts.
    3. Why We Are Excited: Currently most organizations conduct surveys at different time intervals to collect employment and wage data which have really low response rates. By accessing tax transcripts and analyzing them, organizations can get direct insights into the effectiveness of their programs, employment and wage outcomes after the participants have graduated. If designed and implemented effectively, this can be a pioneering solution for the workforce development sector.
  6. Generation: “Empowering Livelihoods: Mapping the Path to a Living Wage in Kenya”
    1. Description: Conducting focus groups and surveys with alumni to learn about key barriers preventing employment and financial health outcomes.
    2. Impact: Improve durability outcomes in Generation’s lower middle-income countries by understanding the barriers that are restricting the economic mobility for our alumni in LMICs, and causing long-term unemployment, employment breaks, and issues in seeking promotions.
    3. Why We Are Excited: Generation will be incorporating stakeholders’ voices by directly engaging with its participants to understand their challenges. The insights from this study will result in program modifications, improve data collection, and advocate to address systemic barriers.
  7. Human-I-T: “Amplifying Impact: Enhancing Digital Equity through Strategic Learning and Data-Driven Insights”
    1. Description: Partnering with an external impact measurement vendor, 60 decibels, to measure outcomes and collect participant feedback.
    2. Impact: They will randomly interview 450 recipients of Human-I-T services going back to 2021. The results from this in-depth survey will help Human-I-T to
      1. Fine-tune Human-I-T’s program offerings.
      2. Increase the efficiency and scalability of its services.
      3. Identify and address service gaps.
      4. Fundraise - With clear, evidence-based results, Human-I-T can more compellingly demonstrate the value of its work, attracting new collaborators and resources.
      5. Improve long-term feedback systems - The data collected will serve as a foundation for ongoing evaluation, helping the organization regularly assess and enhance its programs to meet the evolving needs of the communities it serves.
    3. Why We Are Excited: Again, this is the first project to partner with a leading outcome measurement organization, 60 decibels. This also directly improves the quality of results data we get regarding our current grant with Human-I-T, and lastly, it will set them up to continue these surveys in-house.
  8. Immigration Policy Lab, GeoMatch: “GeoMatch: Measuring Impact and Improving Refugee Integration Outcomes”
    1. Description: Conducting qualitative interviews with resettled refugees to better understand employment and quality of life experiences beyond 90 days to improve GeoMatch’s recommendation algorithm. With this grant, they will supplement that data collection in three key ways: 1) collecting outcomes for refugees at 240 days post-arrival, 2) collecting additional outcome data beyond employment, and 3) conducting qualitative interviews with refugees for a more holistic understanding of their resettlement experience.
    2. Impact: This helps GeoMatch better articulate the impact of their tool, build a cost-effective model, and leads to higher quality data that will update their matching algorithm.
    3. Why We Are Excited: This project lays the foundation for improving the required reporting to the State Department by resettlement agencies. It also provides a qualitative picture of the experiences of refugee families.
  9. Kuepa Edutech: “Gender Equality in Action”
    1. Description: Interview women participants and investigate barriers preventing women from entering the tech sector in Colombia.
    2. Impact: Gather insights on gender and age equity in its secondary and tertiary education programs and analyze economic, social, ethnic, and other factors to enable Kuepa to optimize its strategies and develop more inclusive programs and support a higher number of participants.
    3. Why We Are Excited: One of the projects that has all the data and is focused on developing its internal team’s capacity. By gathering insights and analyzing the trends, the Kuepa team will be able to improve its programs and reduce employment inclusion barriers which will, in turn, result in formal employment and improved economic income for these individuals.
  10. Laboratoria: “Empowering Women for the Future: Adapting Skills and Learning Pathways for the Digital Economy”
    1. Description: Building new processes and tools including JobsPikr (a talent intelligence tool) to better understand emerging opportunities in the broader digital economy to improve job placement and career success outcomes.
    2. Impact: Update learning content and delivery methods, better preparing the women in its community to access income growth, living-wage jobs, and financial stability.
    3. Why We Are Excited: This comprehensive study will enable Laboratoria to identify emerging opportunities to tailor its programs to the current needs of the market. They will not only be able to have high job placement rates back but also empower more women to find and thrive in careers that align with their potential and circumstances.
  11. Makaia: “Trust, Measure, Improve: A Community-Based Approach For Enhanced Program Impact”
    1. Description: Creating a systematic process for collecting feedback and building a trust-based community of participants and alumni to collect higher quality feedback data to inform future program design.
    2. Impact: Upgrades their feedback systems using FluentCRM and aligns staff on feedback best practices. This will directly improve their STEM Incubator and Bootcamp programs by increasing completion rates.
    3. Why We Are Excited: This is rebuilding and retraining Makaia’s entire approach to collecting feedback. While this was more broad than most applicants, it has the power to change their system.
  12. Research Improving People's Lives (RIPL): “Impact of the “Launch for Jobseekers” and “Launch for Employers” Reemployment Services Portals”
    1. Description: Leveraging Unemployment Insurance (UI) data to determine which aspects of their “Launch For Jobseekers” (LFJ) web portal are most associated with successful and unsuccessful job placement and wage increases among portal users. Their core analysis will involve linking web user data from LFJ to individual-level UI administrative quarterly earnings records provided by TSS. They will use these linked records to identify workers who use LFJ, match them to workers with similar pre-unemployment characteristics and labor market trajectories, and compare these treatment and comparison groups' labor market outcomes (i.e. employment, earnings). They have a data use agreement with Arkansas that will grant us access to the UI earnings data.
    2. Impact: This will likely lead to more funding as they tell a better story of the impact of LAUNCH in Arkansas, which will be the first state to implement both the job-seeker and employer-facing tools. With the ability to communicate about specific impacts on earnings and employment, we believe we will be able to secure funding to implement the employer-facing tool in states that already have the job-seeker facing tool such as Colorado and Hawaii.
    3. Why We Are Excited: They are the only grantee to-date who has access to unemployment insurance quarterly data. This is a uniquely rigorous approach to measuring impact. It also sets the stage for national expansion.

4. ROI Analysis

The GitLab Foundation conducts a return on investment analysis as a core part of its due diligence process to estimate the potential outcomes that would be generated from its investments. These learning grants will not have known outcomes, but the ROI analysis can estimate the programmatic improvements needed to generate a return of 100x ($100 dollars generated in lifetime earnings per $1 dollar spent).

These learning projects will reveal insights that will directly inform programmatic or organizational policy changes, which, if implemented, may increase the effectiveness of those programs. The GitLab Foundation used the existing income increase outcomes for each organization and calculated how much additional increase in annual income their programs would need to generate for their program participants to get to a 100x return. This process revealed that five of the 12 learning projects would only need to implement programmatic changes that increased their participants’ incomes by 2% or less to surpass the 100x threshold. If all projects increased incomes by 5% due to the key learnings uncovered through these projects, this cohort would generate a combined ROI of more than 300x, demonstrating the powerful impact potential that investments in learning can have.

ROI Analysis Here

5. Grant Approval Rate Statistics