As an organization that values iteration, we very much welcome feedback and comments on our impact model and impact approach - Please email us with any feedback, comments, or questions at: [email protected]

1. North Star ROI Methodology

North Star ROI: We define benefits as the total income increase generated by our grant for all impacted people over 30 years, and our costs are the grant amount multiplied by our Foundation’s overhead rate.

A simplified version of our North Star ROI formula is here:

2. North Star ROI Formula

Below is the formula, glossary of terms, important considerations, and an example of the model in action.

Formulas:

1. Present Value (PV) Income Gain per Person ->

1. Aggregate Present Value (APV) Income Gain →

1. North Star Return on Investment (NSROI) →

Where

APV = aggregate present value, which is the present value (PV) of the lifetime earnings increase per person multiplied by the total number of people impacted (P)

PV = is the present value of the lifetime earnings increase

P = number of people impacted

W = inflation-adjusted annual wage increase per person

i = discount rate

n = number of years

C = total cost of the grant, including GitLab Foundation’s overhead rate

3. Key Variables Explained

4. Calculating the Number of People Impacted

1. Direct Service Grants
   1. When it is a direct service intervention, the number of people is the number of participants receiving the direct service or product.
2. Unrestricted Grants
   1. When we provide an unrestricted grant, we calculate what proportion of our grant amount accounts for their overall budget to determine the number of people impacted. For example, if the organization reaches 10,000 people and has an operating budget of $10 million, a $1 million grant will be estimated to reach 1,000 people - a 10% attribution.
3. Specific Program Grants
   1. If the grant is for a specific program, we ask the grantee to estimate the number of people impacted due to the grant amount (e.g., “a $500,000 grant will enable us to put 350 students through our training program”).
4. Systems Level Grants
   1. For large systems-level grants, it can be harder to estimate the number of people impacted. This is done through a series of research and assumption steps that provide evidence-backed estimates for the number of people that would be impacted. For example, for a grant that addresses short-term credential quality, we conduct research on the number of short-term credential graduates per year.
5. Incorporating ‘Flywheel’ Effects Leveraging Reinvestments
   1. When a program earns revenue back from participants or external stakeholders, and if that revenue directly supports more participants in the future, the ROI model will count the additional people affected by the re-use of generated revenues for five years after the grant ends.
   2. For example, an apprenticeship program costs $5,000 per participant and the apprenticeship earns $2,500 in revenues for the organization that is reinvested back into growing the apprenticeship program. If the Foundation's grant supports 20 students, $50,000 will be generated in revenue to support another ten apprenticeships the following year, and our impact model will take these ten into account.
6. Investing in Scaling Grants
   1. When the Foundation makes a grant to help an organization scale a program, the model counts the future number of people that will be reached through scaling the program for five years. For example, if the grant enables an organization to reach 100 participants in the first year, the model will also count the additional participants reached in years 2, 3, 4, and 5, given the program would not have been able to scale without the Foundation’s grant support.

6. Calculating Annual Income Gain per Person (Generating the Counterfactual)

1. One of the most critical inputs to the ROI model is the estimated earnings increase generated from the products or services of the grantee. We are not measuring earnings increase as the difference between their participants’ income before and after working with the grantee (a “pre-/post- methodology”); instead, we attempt to estimate the difference in earnings that the participant earns after working with the grantee compared to what they would have made had they not had access to the product or service. This is also known as estimating the “counterfactual” or “deadweight” in SROI methodology.

   1. For example, when estimating the counterfactual for a program that prevents dropout of Black and Latinx (Hispanic, Latino, Latina) computer science students in college, we referred to the lifetime earnings of graduates by college major and educational attainment from the Hamilton Project. We also looked at computer science dropout rates. We estimated the counterfactual dropout rates and lifetime earnings based on the grantee's own internal outcome data on starting salary level and computer science graduation rates for their participants. Ultimately, our analysis estimated program participants would have starting salaries of $62,000 compared to estimated counterfactual salaries of $51,000.

2. We take the available impact evidence and data that the grantee provides in their concept note and final application, and our Impact Measurement and Analytics team estimates the counterfactual using a mix of data:

   1. Grantee Pre and Post: Most grantees provide pre- and post- income gains
   2. Grantee Counterfactual: Some grantees can also demonstrate counterfactual (control) group income gain estimates
   3. Publicly Available Benchmarks: We use publicly available benchmark data (Bureau of Labor Statistics, Census Bureau, etc) to find comparable outcomes and incomes given the characteristics of program participants
   4. Literature Reviews: We also estimate the counterfactual based on published research papers that use rigorous causal methods (experimental or quasi-experimental methods).

3. Some of our counterfactual estimates are based on assumptions that are difficult to validate. We work with our grantees to collect impact data during the grant period to help validate some of the assumptions made during the ROI estimation process. We also continually look for rigorous external datasets that can assist us in making these counterfactual estimates.

4. This chart demonstrates why just measuring the pre- and post- incomes may overstate the impact of an intervention:

   

6. Putting the North Star ROI in Context of the Broader Due Diligence Process

The ROI calculation is based on the absolute estimated lifetime earnings gains generated by the grant. While this number is vital to our investment decision-making process, it is not the only input used to select grantees:

1. Balancing ROI with Other Values: Using only one number can lend itself to the false precision fallacy. In addition to the ROI, we look at other factors during the grantee due diligence process, including the organization’s leadership team, its track record in delivering impact, organizational financials, and participant demographics (including income, gender, race, ethnicity, age, disability, refugee status, location, and other categories). In addition, we consider alternative scenarios of our investments and examine both positive and negative unintended consequences. Lastly, we consider our Foundation's potential contribution in affecting change for the grantee: small, less-funded, grassroots organizations may benefit more from GitLab Foundation's support than large, nationally-known, well-funded organizations.
2. Priority Populations: The GitLab Foundation has an explicit goal of supporting individuals living below the living wage in their local context. As a result, we look at the percentage of program participants living below living wage thresholds and seek to work with grantees supporting those communities. Read more about our living wage analysis methodology at the link below:
3. Relative Income Gain: The core goal of the ROI model is estimating absolute income gains. However, the Foundation also calculates relative income gains, as populations with lower baseline incomes will need to earn many times more relative income to have the same absolute income gain as populations with a higher starting income level. Relative income change is meaningful, as the model compares grantees across global regions where average incomes vary immensely. A supplementary relative income gain calculation also factors into our decision-making.
   1. For example, these three programs all score a 100 ROI (assuming the grant is $500,000 and the baseline incomes are at the living wage thresholds in each country):
      1. In the United States, providing a $500,000 grant to a program that increases incomes by 30% for 575 people (+$11,400 USD annual income increase per person)
      2. In Kenya, providing a $500,000 grant to a program that increases incomes by 30% for 6,500 people (+$994 USD annual income increase per person)lo
      3. In Kenya, providing a $500,000 grant to a program that increases incomes by 340% for 575 people (+$11,400 USD annual income increase per person)
   2. In those examples above, making a $500k grant to a similar organization in Kenya will impact 11 times more people or increase relative annual income per person 11x more than in the United States.

7. North Star Absolute Income ROI Model Example

Grantee Intervention and Grant Amount: A $1 million investment in a job training and placement program.

Step 1

1. GitLab Foundation Grant: $1,000,000
2. GitLab Foundation Grant with Overhead Rate (23%) Included: 1.23*$1,000,000 = $1,230,000

Step 2

Model Assumptions:

1. Discount rate = 10%
2. Years of lifetime earnings = 30 years
3. Real earnings increase growth, year-over-year = 0%
   1. The initial income increase remains constant over time
4. Number of people reached, change year-over-year = 0%
   1. The number of people reached by the grant does not increase over time

Step 3

1. Estimated participant counterfactual annual earnings: $25,000
2. Participant post-program annual earnings: $35,000
3. Additional annual earnings due to grantee intervention (compared to counterfactual): $10,000 ($35,000 - $25,000) (40% increase in annual income)

Step 4

1. Number of people reached: 1,000 people completing the training program

Step 5

1. Present Value Lifetime Income Increase per Person: $94,269
   1. (30 years of $10,000 additional dollars = $300,000 -> at 10% annual discount rate = $94,269)
2. Aggregate Present Value Lifetime Income: $94.27 million
   1. ($94,270 Present Value Lifetime Income Increase per Person * 1,000 people)
3. Return on Investment (North Star ROI): 77
   1. ($94.27 million Aggregate Present Value / $1,230,000 total grant amount)