Field building in universities for AI policy careers in the US

Project summary

The project aims to improve AI policy talent pipelines by educating AI risk-aware students and young professionals about career impacts and decisions in US policy. Founders Coby Joseph and Aishwarya Saxena will develop and implement a pilot program with support from Rethink Priorities’ fiscal sponsorship and strategic advice.
This grant will cover pilot expenses like venue and logistics. Salaries will be funded separately by another funder.
Risks include insufficient funding from other sources, ineffective pilot design, and founder collaboration issues. Mitigation strategies:
- The project has backup funding from Rethink Priorities and seeks additional sources.
- Governance processes will support founder collaboration and project management.

What are this project's goals and how they be achieved?

This project aims to improve talent pipelines from universities to AI policy roles in the US. Specifically, this project aims to increase the number of talented artificial intelligence (AI) risk-minded students and young professionals planning to enter a US policy career by helping them understand arguments about catastrophic risk from AI, how these career tracks can be impactful, and how to navigate related career decisions successfully.

Over 4-6 months from Jan 2024, the founding team (Coby Joseph and Aishwarya Saxena) will develop a pilot concept and plan, backed by initial research from Rethink Priorities’ Existential Security team (XST), and implement a pilot program that demonstrates a high-impact way of achieving the above goal.

How will this funding be used?

This funding will be used to cover project expenses related to the pilot (e.g. venue booking, logistics, contractors, etc.) Salaries will be covered by another funder.

Who is on the team and what's their track record on similar projects?

The team is Coby Joseph and Aishwarya Saxena. Coby is a mid-career professional with entrepreneurial and project management experience, including founding a university-based nonprofit, Music Matters, which fundraised $300k over 3 years for charity. Aishwarya is a lawyer (PhD in Law) with a background in climate and nuclear law, who also has entrepreneurial experience, as Head of Legal & Partnerships, at SpeedLegal, a law-software start-up. These founders have been chosen via a competitive hiring process conducted by Rethink Priorities that screened for general entrepreneurial qualities (e.g. critical thinking and judgment, motivation to reduce x-risk) and specific subject matter expertise (e.g. knowledge of university-based programs or US policy frames).

Supporting this team are two researchers from the Institute for AI Policy and Strategy (IAPS), Renan Araujo (me) and Jam Kraprayoon, who will be providing strategic advice and research support. Renan and Jam’s previous team (XST) worked as an incubator generating project ideas and finding suitable founders to lead them, and has incubated field-building projects like Condor Camp.

Operational support will be provided by the Rethink Priorities Special Projects team under a fiscal sponsorship arrangement. The Special Projects team has supported similar projects, such as the Existential Risk Alliance (ERA). Overall governance of the project will be ultimately overseen by Rethink’s Board of Directors, and in practice, by the co-CEO Peter Wildeford, Special Projects manager Matthew Fargo, and me (in my capacity as IAPS research manager).

What are the most likely causes and outcomes if this project fails? (premortem)

The three key failure modes of this project are (1) insufficient funding, particularly to support an impactful version of a pilot, and (2) the chosen pilot version having minimal or negative impact.

For (1), the turnaround time for funding to cover the implementation phase might be too short and risk having the founding team’s momentum halted while they wait for news about fundraising. Proactively fundraising to cover (at partially), project expenses, and working on fundraising early do help with this – which is the role this grant is mainly playing.

For (2), much of this risk comes from the design of the pilot and the overall theory of change. Built into the 4-6 month timeline is a period for pilot design and exploration and support from advisors and subsequent feedback from funders should address this failure mode.

For (3), there could be issues with the founders underperforming or being unable to work together productively, as they don’t have experience working together. To mitigate this, RP is setting up governance processes to monitor and support productive collaboration between the founders.

What other funding is this person or project getting?

This project’s funding is backstopped by Rethink Priorities up to 50k to guarantee salaries and support for the founding team for up to 4 months. The plan is that this funding will be reimbursed by additional fundraising from non-Manifund sources (currently RP is also seeking funding from OP to cover salaries for the founding team for 6 months and operational support costs from SP). If this project receives funding from OP or another donor, this Manifund grant will cover project expenses related to the pilot (venue booking, logistics, contractors, etc.)

Main points in favor of this grant

This project’s targets a relatively neglected area and has a compelling theory of change (that received feedback from various other experienced groups in the field during XST’s research process).
The team seems well-placed to plan and execute this project (selected based on a systematic hiring process) and the support system to be provided by RP has a strong track record in similar projects (Condor Camp, ERA)
Since I (Renan) will have governance responsibilities for this project, I will be able to more closely oversee and steer the project so that it is more likely to be impactful than most grantees.
Separate from the potential direct impact, the information value from this project should be high. The x-risk space does not have other examples of founder search and pairing via formal hiring + dedicated support and this could inform future efforts at active grantmaking in the space.
This grant seems time-sensitive, since the project is due to start in early Jan and other grantmakers are not able to turn around funding by then. RP has limited capacity to fund the entirety of this project by itself at the moment, and this grant would fill a relevant gap.

Process for deciding amount

$25K is an estimate of the funding necessary to cover a pilot event (including venue booking, logistics, contractors, etc.)

Conflicts of interest

Please disclose e.g. any romantic, professional, financial, housemate, or familial relationships you have with the grant recipient(s).

1. The project is being incubated and fiscally sponsored by Rethink Priorities, which fiscally sponsors my employer, IAPS. I will play a role in the governance of the project along with my manager (Peter Wildeford) and RP’s Special Project team.

2. The project idea came from my previous team’s (XST) research.

However, the founding team is independent, will own the IP associated with this project, and is only employed as contractors through RP’s fiscal sponsorship program. Ultimately, if successful, this project will likely spin off as an independent project. My involvement is likely only to occur during this initial pilot stage in an advisory capacity.

Additionally, despite RP receiving the funding due to the fiscal sponsorship scheme, this funding will be completely restricted to this project and Coby and Aishwarya, and can’t be used for any other purpose (i.e., can’t be used by RP for other RP-related expenses).