I didn't fund this project, but I did recommended it to JueYan Zhang for funding.

Main points in favor of this grant

1. It seems possible that AI agent swarms (or fully automated firms) will soon be responsible for writing most new production code. The phenomena of emergent misalignment, self-fulfilling misalignment, and "spiralism" seem particularly concerning when AI agents are writing and reviewing most of the content that later agents read and are trained on, with little human oversight. Moltbook, Gastown, and AI Village showcase concerning multi-agent dymamics. I expect that we will see a surge of interest in multi-agent emergent AI risks as frontier AI researchers increasingly use AI teams or swarms to automate AI development.

2. Multi-agent AI risks seem quite neglected in the AI safety field at the moment relative to their importance. It's possible that many single agent/single stakeholder alignment techniques might not generalize to successfully aligning an AI agent swarm with user intentions (i.e., "multi-single" alignment). It's possible that the first "strong AGI" will be built largely by an AI agent swarm; therefore, it seems very important to build techniques to steer and evaluate agent swarms operating on a shared codebase. Also, agent swarms might constitute a "collective superintelligence", pose a threat vector for mass-manipulation, or otherwise coordinate in undesirable ways.

3. I know the co-founders quite well and have a high opinion of their competence. Jack seems like an excellent researcher and previously worked at Softmax, a prominent multi-agent AI company. Raf has substantial experience with startups and, from what I can tell, seems like an excellent engineer. I've found their preliminary research on the effects of codebase comment ablation/modification on AI agent benchmark performance quite interesting.

Main reservations

1. I expect that most of the catastrophic risks associated with AI agent swarms will be first observed inside of frontier AI companies. Therefore, for Antimemetic AI's techniques to be maximally impactful, they should be deployed inside frontier AI companies. It's unclear whether this type of deployment is realistic for Antimemetic AI due to codebase security restrictions, unless they are acquired (in which case only frontier AI company would benefit). However, I think these techniques could improve safety for plenty of smaller AI companies and users regardless.

2. It's possible that "codebase alignment" is a relatively ineffective way to steer AI agents compared to techniques that target individual agents. I expect that research into novel control mechanisms is definitely warranted, however!

3. It's possible that the techniques Antimemetic AI are exploring could be used to more effectively leverage multi-agent AI systems to accelerate AI R&D. I would consider this a downside risk. To mitigate this risk, I recommend Antimemetic AI carefully consider which clients they should sell their technology to and conduct a risk analysis.

4. Antimemetic AI is a for-profit company and could be subject to the usual incentive problems and mission dilution. I recommend that the founders consider incorporating as a PBC, raising from mission-aligned VCs, and consider the dual-use potential of their technology.

Conflicts of interest

I consider the co-founders as friends. I made this clear to JueYan when recommending the project. I don't believe that this has affected my judgement here.

@Austin I'll share more tomorrow

Hey Thane! What other sources of funding have you sought? What are the largest sources of uncertainty for this project, or reasons it might fail to have impact?

@Seth-Momanyi thanks! I'm quite familiar with PauseAI. I just think it's not accurate to say that this proposal is for the first AI safety hub in Africa. Perhaps "the first AI safety hub in Kenya" would be more accurate?

Isn't the Equiano Institute the "first AI safety hub in Africa"?

Is there any more information?

What does marginal funding buy, given the ~$200k from other sources?

@JaesonB Thank you for these great responses!

@JaesonB, if I was in your shoes, I would prioritize these things in order:

1. Talk to ARM and ask them how they are capacity constrained, then help alleviate that constraint.

2. If ARM is constrained by hiring great grantmakers, help them build a great hiring pipeline.

3. If ARM is constrained by funding, build a TLYCS for AI safety, to encourage mass donations to ARM.

4. If ARM is constrained by unfixable factors (e.g., no-one is driving it and they refuse help), first try cutting your teeth via regranting on Manifund. If this goes well, it doesn't seem crazy to set up another fund. Note that many early grantmakers for EA Funds also worked at Givewell or Open Philanthropy.

I think there's a gap for an organization like The Life You Can Save for AI safety, which would encourage donations and pledges to top charities, but I don't really see how your proposed fund is a better alternative to scaling the AI Risk Mitigation Fund or Manifund. If the argument is "we bring in additional donors because of our non-longtermist affiliation", the same could be said for ARM and Manifund. If the argument is "we add additional grantmaker capacity" I would counter "why can't these same grantmakers join ARM or just use Manifund?" (Possible answer: maybe ARM's bar is too high and Manifund isn't an attractive target for donations.) Basically, I think we do need more funders in the medium-long term, but I think the experience and reputation of the ARM grantmakers is much higher than your proposed grantmakers, on average, and Manifund already exists as a short-term regranting solution that I would rather grow. I don't want this project to not happen at all, to be clear, but I would rather it be rescoped as something closer to TLYCS, as there is a much larger gap here than for another funder constrained by the same things as existing funders, albeit less experienced.

@JaesonB, I also fear those capacity constraints. I'm curious why you think the solution is a separate fund, rather than alleviating the capacity constraints of an already-proven fund? Additionally, why wouldn't exactly the same constraints (e.g., money, competent grantmakers) bottleneck your fund?

@JaesonB, what about the AI Risk Mitigation Fund? They're not focused exclusively on longtermism and have the same grantmakers as the LTFF.

It seems like parts of your linked grant proposal were directly copied from TARA without citation, e.g., "grow and support an ambitious, influential, and technically capable local community focused on preventing the most harmful impacts of AI." The KPIs also seem to be copied without citation. Also, the name "STARA" is "TARA" with an "S". Did you ask TARA if you could copy their materials and name?

Main points in favor of this grant

1. I think that the AI safety movement could benefit from stronger messaging in popular culture; a comedy musical seems a great addition! The dark subject matter around AI extinction has pretty low mass appeal, but dark comedies can be very popular.

2. I found the play hilarious and incisive. The ficitonal AI lab staff were charicatures of very real personalities and movements currently involved in the AI race. The musical elements were clever and well-delivered. I had a blast! I want to incentivise more high-quality, socially impactful art regarding the AGI race.

3. I don’t expect Open Phil or SFF to start funding plays anytime soon, so I expect my regrant is quite counterfactual.

Donor's main reservations

1. On priors, it seems unlikely that this play will take off internationally, even if it is successful in Berlin. If the play is only of interest to a niche EA audience, this might be low impact. I encourage the producers to screen the play to a non-EA audience, survey them, and perhaps make modifications to increase mass appeal.

2. My regrant could have funded an AI safety researcher for ~3 weeks FTE. It’s hard to judge whether the play’s impact will stack up compared to this. In the end, I made a judgement based on the relative abundance of funding for AI safety researchers post-Open Phil RFPs.

Process for deciding amount

I decided to regrant $6.4k, which was twice the amount raised so far based on exchange rates at the time I made the grant. This amount was chosen to be high enough to significantly contribute to the play going ahead, while being low enough to encourage further crowdfunding. Also, I made a heuristic comparison against similar grants I’ve made.

Conflicts of interest

I have no conflicts of interest to declare.

I think AI Safety Atlas is awesome! I highly encourage further donations.

@Meeri At this point I don't feel qualified to comment and I defer to @Jsevillamol.

Some of this review is copied from my recent regrant to TARA.

Main points in favor of this grant

1. I’ve visited several Australian AI safety and EA conferences this year (and for the past 5 years) and I think that the local talent base is quite strong. MATS has accepted ~12 Australian scholars (~4% of all alumni) and they did great! Australians can get work visas for the UK and USA (E-3) relatively easily, which means they can emigrate to the SF Bay Area and London for AI safety work.

2. I’ve been impressed by Yanni’s operations and marketing ability in regards to TARA, the AI Safety ANZ Careers Conference 2024, and AI Safety ANZ’s recent seminar/workshop series.

3. Australia might have a role as a "middle-power" between the US and China in facilitating international AI treaties. The Sydney AI Safety Hub and its members might help support the formation of an Australian AI Safety Institute or otherwise support the Australian government and industry bodies in furthering AI safety. Additionally, Australia, as a member of Five Eyes and a NATO partner, might have a role in tracking foreign AGI projects and building international coalitions for AI safety.

4. I think that physical coworking spaces for AI safety have a strong track record for spawning useful projects (e.g., ARC Evals/METR in Constellation, Apollo Research in MATS London Office/LISA), facilitating talent flow into AI safety organizations, and supporting emerging and independent researchers with a strong community. Sydney has a few AI safety organizations (e.g., Gradient Institute, Harmony Intelligence) about which a hub can form and is proximal to talent sources like Sydney University and Sydney's corporate tech scene.

5. $5k to secure an office space for two months seems really cheap! I think that $30k for a year of office space is similarly cheap and I would be surprised if this is not funded.

Donor's main reservations

1. My grant is quite small and will only support the office for two months. AI Safety ANZ will need to secure permanent funding from a larger funder soon.

2. Sydney University, which is next door to SASH, is ranked 69th internationally for computer science. This is not an impressive ranking relative to the top US or UK universities, many of which do not have nearby AI safety offices (exceptions: AISST office near Harvard/MIT, Constellation/FAR Labs near UC Berkeley, LISA near ICL, the late Trajan House near Oxford). However, I think Australia ought to have an AI safety office given the strength of AI safety talent I regularly see coming from Australia, and Sydney seems like the natural location (though Canberra or Melbourne are also possible, due to their proximity to Seth Lazar's and Dan Murfet's labs, respectively).

3. It's possible that there is not enough strong talent or new orgs in Sydney to justify an office the size of SASH. To my knowledge, there is no dedicated AI safety research group at the University of Sydney. TARA should bring in some talent, however, and I encourage Yanni to help incubate any new orgs/projects that emerge in SASH.

Process for deciding amount

I decided to fund the entire amount because it's relatively small and AI Safety ANZ urgently needs funding to secure the office space for the next two months.

Conflicts of interest

I am an Australian citizen and have personal and professional ties to Australian AI safety and EA communities. Yanni recently invited me to present at the AI Safety ANZ Careers Conference 2024, although I received no financial compensation. I am an Advisor to AI Safety ANZ, Yanni’s organization, though I receive no financial compensation for this role. This year, I presented at the Australian AI Safety Forum 2024 and I have previously presented at EAGxAustralia 2022 and 2023, for which I received travel and accommodation funding.

@habryka it seems that Open Phil didn't want to fund FrontierMath, so Epoch went to OpenAI for funding. I think more funding for Epoch would be good to reduce reliance on commercial partners who might require dataset access.

@KabirKumar If concerned, surely it's good to fund Epoch via Manifund to reduce their dependence on OpenAI?

Have you obtained approval from Epoch, Leopold Aschenbrenner, and Planned Obsolescence to adapt their content into a YouTube video?

@RyanKidd Mikolaj should also declare all potential funding COIs (e.g., me) on his public outputs.

Main points in favor of this grant

I want to see more public impact analysis of AI safety organizations. To my knowledge, there has been very little of this to date.

Donor's main reservations

Mikolaj has some experience with impact analysis, but I suspect that this project will be quite hard and might need more funding.

Process for deciding amount

I decided to partly fund this project to give other donors room to contribute and mitigate my conflict of interest.

Conflicts of interest

I am Co-Director at MATS, Board-Member at LISA, and advise several AI safety projects, including Catalyze Impact and AI Safety ANZ. I often work out of FAR Labs. As a Manifund Regrantor, I regularly fund AI safety projects, the success of which is probably a determining factor in my future status as a Regrantor. To mitigate these obvious conflicts of interest, I ask that Mikolaj share his key results in a public database and post, where any discrepancies can be identified.

How much funding do you currently have? What's your runway duration? Where have you applied for further funding this year?

@RyanKidd *Also, I trust Yanni to refund the extra $3419 for room bookings if he is able to secure free event space.

I like Metaculus and GiveWell. Incentivising forecasts of charity impact seems like a good way to improve charitable giving.

Main points in favor of this grant

1. ARENA is a great curriculum and Yanni’s adaption to a weekend course seems well-considered.

2. I’ve visited several Australian AI safety and EA conferences this year (and for the past 5 years) and I think that the local talent base is quite strong. MATS has accepted ~12 Australian scholars (~4% of all alumni) and they did great!

3. Talking with several students and professional software engineers in Australia at recent conferences has convinced me that there is value in a weekend version of the ARENA course for people who want to upskill in AI safety research engineering, but cannot take off work and travel to the UK for a month to do ARENA.

4. I’ve been impressed by Yanni’s operations and marketing ability in regards to the AI Safety ANZ Careers Conference 2024 and AI Safety ANZ’s recent seminar/workshop series.

5. Australians can get work visas for the UK and USA (E-3) relatively easily, which means they can emigrate to the SF Bay Area and London for AI safety work.

Donor's main reservations

1. Yanni hasn’t yet found a good TA to run this course, which seems important for this to go well. I expect this to be much easier with funding, however, and I’m confident Yanni can find someone qualified, probably from the pool of MLAB and ARENA alumni.

2. As with all training programs that empower ML engineers and researchers, there is some concern that alumni will work on AI capabilities rather than safety. Therefore, it’s important to select for value alignment in applicants, as well as technical skill.

Process for deciding amount

I decided to fund the entire project because it’s quite cheap for the scale ($459-596/participant) and I expect I have a stronger and more informed opinion of the Australian talent pool than other funders. Also, I trust Yanni to refund the extra $3419 for room bookings if he is unable to secure free event space.

Conflicts of interest

I am an Australian citizen and have personal and professional ties to Australian AI safety and EA communities. Yanni recently invited me to present at the AI Safety ANZ Careers Conference 2024, although I received no financial compensation. I am an Advisor to AI Safety ANZ, Yanni’s organization, though I receive no financial compensation for this role. This year, I presented at the Australian AI Safety Forum 2024 and I have previously presented at EAGxAustralia 2022 and 2023, for which I received travel and accommodation funding.

I funded this project for 1/4 of its cost based on the fraction of digital minds content from last year.

To cover the minimal costs of the SB 1047 documentary ($15k).

Any updates, guys?

@Grace-Braithwaite Have you applied to grants elsewhere? How many attendees do you expect? Are you asking for the full $10k?

@Fibonan And another NY Times article, which says, "More than a dozen companies have popped up to offer services aimed at identifying whether photos, text and videos are made by humans or machines."

@Fibonan the public also seem to care a lot too. Here's an NY Times article about TrueMedia from April 2024.

@Fibonan it seems like GetReal Labs, TrueMedia, and Reality Defender have a similar business model. If they can get VC funding, I think you can too!

@Fibonan if I were a VC, I would bet I could make money off this product. I'm honestly really surprised that YC, Juniper, Metaplanet, Lionheart, Mythos, Polaris, Fifty Years, and Moonshot aren't interested.

This seems like a great tool that should definitely exist. In fact, it's so obviously useful, I don't know why you need Manifund! Do you need help applying for VC funding? Are you in urgent need of funds?

@RyanKidd Further discussion on LessWrong here.

@Austin Cheers! Re. the funding situation, a program like MATS has an annual budget somewhere between $3.6M and $8.1M, depending on the program size.

1. How will you add value to donor outreach compared to organizations with a similar mission, such as Effective Giving, Founders Pledge, Future of Life Institute, Giving What We Can, Longview Philanthropy, etc.?

2. If you could, in what ways would you improve distributed funding projects like Manifund AI Safety Regranting, ACX Grants, GiveWiki, etc.?

Main points in favor of this grant

1. My inside view is that PIBBSS mainly supports “blue sky” or “basic” research, some of which has a low chance of paying off, but might be critical in “worst case” alignment scenarios (e.g., where “alignment MVPs” don’t work, or “sharp left turns” and “intelligence explosions” are more likely than I expect). In contrast, of the technical research MATS supports, about half is basic research (e.g., interpretability, evals, agent foundations) and half is applied research (e.g., oversight + control, value alignment). I think the MATS portfolio is a better holistic strategy for furthering AI alignment. However, if one takes into account the research conducted at AI labs and supported by MATS, PIBBSS’ strategy makes a lot of sense: they are supporting a wide portfolio of blue sky research that is particularly neglected by existing institutions and might be very impactful in a range of possible “worst-case” AGI scenarios. I think this is a valid strategy in the current ecosystem/market and I support PIBBSS!

2. In MATS’ recent post, “Talent Needs of Technical AI Safety Teams”, we detail an AI safety talent archetype we name “Connector”. Connectors bridge exploratory theory and empirical science, and sometimes instantiate new research paradigms. As we discussed in the post, finding and developing Connectors is hard, often their development time is on the order of years, and there is little demand on the AI safety job market for this role. However, Connectors can have an outsized impact on shaping the AI safety field and the few that make it are “household names” in AI safety and usually build organizations, teams, or grant infrastructure around them. I think that MATS is far from the ideal training ground for Connectors (although some do pass through!) as our program is only 10 weeks long (with an optional 4 month extension) rather than the ideal 12-24 months, we select scholars to fit established mentors’ preferences rather than on the basis of their original research ideas, and our curriculum and milestones generally focus on building object-level scientific skills rather than research ideation and “gap-identifying”. It’s thus no surprise that most MATS scholars are “Iterator” archetypes. I think there is substantial value in a program like PIBBSS existing, to support the development of “Connectors” and pursue impact in a higher-variance way than MATS.

3. PIBBSS seems to have decent track record for recruiting experienced academics in non-CS fields and helping them repurpose their advanced scientific skills to develop novel approaches to AI safety. Highlights for me include Adam Shai’s “computational mechanics” approach to interpretability and model cognition, Martín Soto’s “logical updatelessness” approach to decision theory, and Gabriel Weil’s “tort law” approach to making AI labs liable for their potential harms on the long-term future.

4. I don’t know Lucas Teixeira (Research Director) very well, but I know and respect Dušan D. Nešić (Operations Director) a lot. I also highly endorsed Nora Ammann’s vision (albeit while endorsing a different vision for MATS). I see PIBBSS as a highly competent and EA-aligned organization, and I would be excited to see them grow!

5. I think PIBBSS would benefit from funding from diverse sources, as mainstream AI safety funders have pivoted more towards applied technical research (or more governance-relevant basic research like evals). I think Manifund regrantors are well-positioned to endorse more speculative basic research, but I don’t really know how to evalutate such research myself, so I’d rather defer to experts. PIBBSS seems well-positioned to provide this expertise! I know that Nora had quite deep models of this while Research Director and in talking with Dusan, I have had a similar impression. I hope to talk with Lucas soon!

Donor's main reservations

1. It seems that PIBBSS might be pivoting away from higher variance blue sky research to focus on more mainstream AI interpretability. While this might create more opportunities for funding, I think this would be a mistake. The AI safety ecosystem needs a home for “weird ideas” and PIBBSS seems the most reputable, competent, EA-aligned place for this! I encourage PIBBSS to “embrace the weird”, albeit while maintaining high academic standards for basic research, modelled off the best basic science institutions.

2. I haven’t examined PIBBSS’ applicant selection process and I’m not entirely confident it is the best version it can be, given how hard MATS has found applicant selection and my intuitions around the difficulty of choosing a blue sky research portfolio. I strongly encourage PIBBSS to publicly post and seek feedback on their applicant selection and research prioritization processes, so that the AI safety ecosystem can offer useful insight. I would also be open to discussing these more with PIBBSS, though I expect this would be less useful.

3. My donation is not very counterfactual here, given PIBBSS’ large budget and track record. However, there has been a trend in typical large AI safety funders away from agent foundations and interpretability, so I think my grant is still meaningful.

Process for deciding amount

I decided to donate the project’s minimum funding ($25k) so that other donors would have time to consider the project’s merits and potentially contribute. Given the large budget and track record of PIBBSS, I think my funds are less counterfactual here than for smaller, more speculative projects, so I only donated the minimum. I might donate significantly more to PIBBSS later if I can’t find better grants, or if PIBBSS is unsuccessful in fundraising.

Conflicts of interest

I don't believe there are any conflicts of interest to declare.

Main points in favor of this grant

1. I think prediction markets are a great forecasting mechanism and accurate forecasts are an essential component of good decision-making. I regularly consult Manifold, Metaculus, etc. for decision-relevant forecasts. Establishing the accuracy of these platforms seems crucial for widespread adoption of prediction markets in institutional decision-making.

2. I’m excited by the potential for Calibration City to track the accuracy of AI-specific forecasts, to aid AI safety and improve planning for transformative AI. I strongly encourage wasabipesto to create an interface tracking the accuracies of predictions about AI capabilities and AGI company developments.

Donor's main reservations

1. It’s possible that this tool doesn’t increase trust or uptake of prediction markets in decision-making because the interface is too abstract or concepts are too abstract. However, even so, this might prove useful to some individual decision makers or research projects.

2. It’s possible that the AI questions I am most interested in calibrating on belong to a class of long-horizon predictions that is not well-represented by the calibration of short-horizon, closed markets.

Process for deciding amount

I decided to fund this project $2k somewhat arbitrarily. I wanted to leave room for other donors and I didn’t view it as impactful in expectation as other $5k+ projects I’ve funded.

Conflicts of interest

I don't believe there are any conflicts of interest to declare.

Update: mentors for the Winter 2024-25 Program released! Currently fundraising for the Summer 2025 Program; 48 mentors have already expressed interest!

Thank you to Manifund for collectively supporting ~11 additional research scholars with your collective donations so far! It's very important for us at MATS to continue seeking funding from diverse sources, which lets us run larger programs with further research diversity. We will continue to do our best to turn your donations into impact!

Update: we have assembled a team of expert advisors to select mentors for the Winter 2024-25 Program, as we received 78 mentor applicants! Our mentor pool has never been more exciting!

Main points in favor of this grant

1. I think that research into the possible sentience or moral patienthood of digital minds is important and neglected. Relatively few philosophy academics are researching this matter and I suspect humanity might soon create highly intelligent forms of artificial life, potentially worth of human-level moral patienthood, but with radically different attitudes in society.

2. Conferences are a time-tested way to connect and develop a research community. I support further conferences on research into the potential sentience and moral patienthood of digital minds.

3. The conference was surprisingly cheap to fund! The conference organizers clearly care deeply about the cause areas they support, as they ran the event with no guaranteed funding.

Donor's main reservations

1. I don’t think that aligning early transformative AI systems with the interests of nonhumans is a good idea. I believe that the period during which transformative AI emerges will likely be fraught with risk and rapid societal change. I principally want to prevent catastrophic risk and value lock-in during this period and adding further complexity to the challenge of making AI corrigible to robust democratic institutions seems bad. I’d rather build corrigible AI to reduce catastrophic risk and prevent value lock-in, then proceed with a “long reflection” or other deep investigation into the all-things-considered best approach towards maximizing nonhuman flourishing within appropriate constraints (though I support efforts to improve animal welfare today).

2. I think that the conference organizers could have supported further researchers in AI sentience or moral patienthood, such as Robert Long, Patrick Butlin, or any of their coauthors here.

3. I think that the animal welfare aspects of the conference might make this project relatively easy for animal welfare donors to fund. It’s not clear to me why the conference organizers didn’t apply to EA Funds’ Animal Welfare Fund.

Process for deciding amount

I decided to fund the fraction of the project’s cost that went towards supporting sessions on AI sentience or moral patienthood, as opposed to using AI to further animal rights or facilitate interspecies communication. I estimated this from the ~1/4 of sessions focused on digital minds and rounded upwards: 1/4 * $5500 = $1375 ~ $1500. I don’t oppose using AI to improve interspecies communication or support animal welfare (both seem positive), but I prefer to dedicate my Manifund resources to projects where I am more counterfactual or impactful.

Conflicts of interest

I don't believe there are any conflicts of interest to declare.

Any update from SFF or other funders?

I decided not to fund this, but accepted Matthew into MATS instead.

I regranted an additional $2k to let the organizers launch the basic event, as per Grace's comment.

@calebp, is this $1700 of the already committed $2000, or is this on top of that?

What does $10k buy that $2k doesn't?

Update: we recently published a blog post summarizing our takes on talent needs of technical AI safety teams based on 31 interviews with key figures in AI safety, including senior researchers, organization leaders, social scientists, strategists, funders, and policy experts.

Any update on the lab progress?

@RyanKidd Oops, I meant Winter 2023-24

Update: we recently published our Winter 2024-25 Retrospective: https://www.lesswrong.com/posts/Z87fSrxQb4yLXKcTk/mats-winter-2023-24-retrospective

Update: MATS is no longer in need of additional funding for our Summer 2024 Program. We are still accepting donations towards our Winter 2024-25 Program, however!

I'll likely regrant to this project because I think CAIS is great, but I'll first look for projects where my grants funge less with Open Phil, SFF, Longview, LTFF, etc.

Main points in favor of this grant

1. Dan Hendrycks' AI safety textbook seems great, but principally serves as an introduction to the field, rather than an in-depth overiew of current technical AI safety research directions, which is the intent of this project. Periodically updated "topics courses" could serve as an equivalent source of value, but these might be bound to particularly universities and updatable on a slower timescale than an online textbook. I'm also enthused by Markov's plans to eventually integrate interactive content and live content from sources like Metaculus, Our World in Data, Stampy, and more.

2. I believe that the AI safety research field should grow 10-100x over the next 10-20 years and AI safety student groups should be a strong driver of this growth. Currently, I think AI safety student groups need more "plug-and-play" curricula to best prepare members for progression into research, engineering, and policy roles, especially at universities without dedicated AI safety courses like that based on Hendrycks' textbook. I think BlueDot Impact's AI Safety Fundamentals courses are great, but I don't see why BlueDot and CAIS should be the only players in this space and think there is some benefit from healthy competition/collaboration.

3. Charbel has road-tested content from the early stages of this textbook project with several AI safety university groups and courses with apparently good feedback.

4. I've been impressed with Charbel's LessWrong posts and nuanced takes on AI safety research agendas.

5. The online version of the textbook will be free and open-source (MIT License), which I think is important for introductory AI safety fieldbuilding materials to be maximally impactful.

Donor's main reservations

1. I think that the optimal form of this project is a continually updated online resource that periodically integrates new papers and research paradigms and therefore this project will eventually need long-term funding and permanent home. However, I believe that my grant will greatly assist Charbel and Markov in producing a proof-of-concept sufficient to secure long-term funding or institutional support. Additionally, the textbook MVP seems likely to be high-value for the near-term regardless of whether the project continues. Lastly, if the textbook is high-value and Charbel and Markov are unable to secure long-term funding, I'm sure it will be useful for established curriculum developers like BlueDot Impact.

2. I wonder if this project should actually be converted into a wiki once the MVP is developed. Markov has previously worked with Stampy and has mentioned that they might want to integrate some Stampy articles into the online textbook. However, even if this project is ideally a wiki, building a viable MVP seems crucial to securing long-term funding and core content for iterating upon.

3. I don't know Markov or the proposed editor, Professor Vincent Corruble, very well, which slightly decreases my confidence in the textbook quality. However, Markov comes highly recommended by Charbel, has previously worked as at Rational Animations in charge of AI safety, and has produced good-according-to-me content for the textbook so far. Professor Corruble is an Associate Professor Sorbonne Université and a UC Berkeley CHAI affiliate, which indicate he has the technical expertise to oversee the computer science aspects of the the textbook. I additionally recommend that Charbel and Markov enlist the support of further editors with experience in AI safety strategy and AI governance, as I believe these are critical aspects of the textbook.

Process for deciding amount

I chose to donate enough to fund the minimum amount for this project to proceed because:

• I want full-time work on this textbook to commence immediately to minimize its time-to-impact and Markov is unable to do this until he receives confirmed funding for 6 months;

• I think it is relatively unlikely that this project will be funded by Open Philanthropy or the LTFF and I have greater risk tolerance for projects like this;

• I have 5x the grant budget I had last year and I think this project is probably more impactful than I would have considered necessary for a counterfactual $7.8k regrant made last year based on the projects I funded;

• I didn't give more than the minimum amount as I feel my marginal funding is high-value for other projects and I think Charbel and Markov can likely secure additional funding from other sources (including other Manifund regrantors) if necessary.

Conflicts of interest

I don't believe there are any conflicts of interest to declare.

Have you tried YC?

The links are broken; they go to a private Zotero page I cannot access.

My bad, I missed the section this!

How does this project compare to Dan Hencryck's AI safety textbook?

Update: thanks to your donations, we were able to support an additional 8.5 scholars in the Winter 2023-24 Program, at an ex post cost of $22.4k/scholar! Thank you so much for your contributions to the field of AI safety :)

We are currently fundraising for our Summer 2024 Program and again expect to receive less funding than our ideal program. We can support marginal scholars at a cost of $24.4k/scholar. We currently have 1220 applicants for Summer 2024 and expect to accept ~3-5% (i.e., MIT's admissions rate). Given the high calibre of applicants and mentors, we would love further funding to support additional scholars!

We have announced the following mentors and hope to announce more as we confirm additional funding: https://docs.google.com/document/d/1sDnD9Igr3gkWX-N_l9W8itVBpqx-pChh-61atxGYkPc/edit

Main points in favor of this grant

1. I think that there should be more AI safety organizations to: harness the talent produced by AI safety field-building programs (MATS, ARENA, etc.); build an ecosystem of evals and auditing orgs; capture free energy for gov-funded and for-profit AI safety organizations with competent, aligned talent; and support a multitude of neglected research bets to aid potential paradigm shifts for AI safety. As an AI safety organization incubator, Catalyze seems like the most obvious solution.

2. As Co-Director at MATS, I have seen a lot of interest from scholars and alumni in founding AI safety organizations. However, most scholars do not have any entrepeneurial experience and little access to suitable co-founders in their networks. I am excited about Catalyze's proposed co-founder pairing program and start-up founder curriculum.

3. I know Kay Kozaronek fairly well from his time in the MATS Program. I think that he has a good mix of engagement with AI safety technical research priorities, entrepeneurial personality, and some experience in co-founding an AI safety startup (Cadenza Labs). I do not know Alexandra or Gábor quite as well, but they seem driven and bring diverse experience.

4. I think that the marginal value of my grant to Catalyze is very high at the moment. Catalyze are currently putting together funding proposals for their first incubator program and I suspect that their previous Lightspeed funding might run low before they receive confirmation from other funders.

Donor's main reservations

1. Alexandra and Kay do not have significant experience in founding/growing organizations and none of the core team seem to have significant experience with AI safety grantmaking or cause prioritization. However, I believe that Gábor brings significant entrepeneurial experience, and Jan-Willem and I, as advisory board members, bring significant additional experience in applicant selection. I don't see anyone else lining up to produce an AI safety org incubator and I think Alexandra, Kay, and Gábor have a decent chance at succeeding. Regardless, I recommend that Catalyze recruit another advisory board member with significant AI safety grantmaking experience to aid in applicant/project selection.

2. It's possible that Catalyze's incubator program helps further projects that contribute disproportionally to AI capabilities advances. I recommend that Catalyze consider the value alignment of participants and the capabilities-alignment tradeoff of projects during selection and incubation. Additionally, it would be ideal if Catalyze sought an additional advisory board member with significant experience in evaluating dual-use AI safety research.

3. There might not be enough high-level AI safety research talent available to produce many viable AI safety research organizations right away. I recommend that Catalyze run a MVP incubator program to assess the quality of founders/projects, including funder and VC interest, before investing in a large program.

Process for deciding amount

Alexandra said that $5k gives Catalyze one month of runway, so $15k gives them three months runway. I think that three months is more than sufficient time for Catalyze to receive funding from a larger donor and plan an MVP incubator program. I don't want Catalyze to fail because of short-term financial instability.

Conflicts of interest

1. I am an unpaid advisor to Catalyze. I will not accept any money for this role.

2. Kay was a scholar in MATS, the program I co-lead. Additionally, I expect that many potential participants in Catalyze's incubator programs will be MATS alumni. Part of MATS' theory of change is to aid the creation of further AI safety organizations and funders may assess MATS' impact on the basis of alumni achievements.

3. Catalyze wants to hold their incubator program at LISA, an office that I co-founded and at which remain a Board Member. However, I currently receive no income from LISA and, as a not-for-profit entity, I have no direct financial stake in LISA's success. However, I obviously want LISA to succeed and believe that a potential collaboration with Catalyze might be beneficial.

My donation represents my personal views and in no way constitutes an endorsement by MATS or LISA.

How useful is $5-10k? What impact does this buy on the margin currently?

@trishume Woohoo! We can support an additional ~7 scholars with this grant, based on the updated marginal cost of $21k/scholar.

Update update: Several more awesome mentors have come forward and we now are funding constrained again for Winter!

Update: we don't appear to be funding constrained for Winter, but will continue accepting donations for our Summer 2024 Program!

Main points in favor of this grant

1. Developmental interpretability seems like a potentially promising and relatively underexplored research direction for exploring neural network generalization and inductive biases. Hopefully, this research can complement low-level or probe-based approaches for neural network interpretability and eventually help predict, explain, and steer dangerous AI capabilities such as learned optimization and deceptive alignment.

2. Jesse made a strong, positive impression on me as a scholar in the SERI MATS Winter 2022-23 Cohort; his research was impressive and he engaged well with criticism and others scholars' diverse research projects. His mentor, Evan Hubinger, endorsed his research at the time and obviously continues to do, as indicated by his recent regrant. While Jesse is relatively young to steer a research team, he has strong endorsements and support from Dan Murfet, David Krueger, Evan Hubinger, and other researchers, and has displayed impressive enterpeneurship in launching Timaeus and organizing the SLT summits.

3. I recently met Dan Murfet at EAGxAustralia 2023 and was impressed by his research presentation skills, engagement with AI safety, and determination to build the first dedicated academic AI safety lab in Australia. Dan seems like a great research lead for the University of Melbourne lab, where much of this research will be based.

4. Australia has produced many top ML and AI safety researchers, but has so far lacked a dedicated AI safety organization to leverage local talent. I believe that we need more AI safety hubs, especially in academic institutions, and I see Timaeus (although remote) and the University of Melbourne as strong contenders.

5. Developmental interpretability seems like an ideal research vehicle to leverage underutilized physics and mathematics talent for AI safety. Jesse is a former physicist and Dan is a mathematician who previously specialized in algebraic geometry. In my experience as Co-Director of MATS, I have realized that many former physicists and mathematicians are deeply interested in AI safety, but lack a transitionary route to adapt their skills to the challenge.

6. Other funders (e.g., Open Phil, SFF) seem more reluctant (or at least slower) to fund this project than Manifund or Lightspeed and Jesse/Dan told me that they would need more funds within a week if they were going to hire another RA. I believe that this $20k is a high-expected value investment in reducing the stress associated with founding a potentially promising new AI safety organization and will allow Jesse/Dan to produce more exploratory research early to ascertain the value of SLT for AI safety.

Donor's main reservations

1. I have read several of Jesse's and Dan's posts about SLT and Dev Interp and watched several of their talks, but still feel that I don't entirely grasp the research direction. I could spend further time on this, but I feel more than confident enough to recommend $20k.

2. Jesse is relatively young to run a research organization and Dan is relatively new to AI safety research; however, they seem more than capable for my level of risk tolerance with $20k, even with my current $50k pot.

3. The University of Melbourne may not be an ideal (or supportive) home for this research team; however, Timaeus already plans to be somewhat remote and several fiscal sponsors (e.g., Rethink Priorities Special Projects, BERI, Ashgro) would likely be willing to support their researchers.

Process for deciding amount

I chose to donate $20k because Jesse said that a single paper would cost $40k (roughly 1 RA-year) and my budget is limited. I encourage further regrantors to join me and fund another half-paper!

Conflicts of interest

Jesse was a scholar in the program I co-lead, but I do not believe that this constitutes a conflict of interest.

@alenglander, when do you expect to hear back from the LTFF? Was the Nonlinear Network funding successful?