@geneparmezan Thank you so much! Really appreciate the support

@sheikheddy Thank you for this comment, I've just made a small donation in support as well.

to support rigorous safety analysis and clear communication of AI risks.

I've funded this up to 1 semesters worth. On my inside view, I'm not particularly excited about this kind of theory heavy non-empirical alignment work. on the outside view it sure seems like enough people are into this kind of thing that it should be taken seriously and given at least some resources. And my sense is that AI safety is a lot more constrained by plausibly reasonable places to spend money, than money. So this seems like a pretty reasonable thing to fund. I don't feel compelled enough to fully fund this, but would be happy to see someone else who feels excited about this do so.

Good luck! I hope it goes well.

Anthony has done great work on other OS libraries for AI safety (SAELens, neuronpedia, and SAEDashboard) and I'm excited that he wants to explore building tools for evals.

I've talked to him about his plans and think it's an obviously good funding opportunity.

I think the topic is important. I have neither done a check on the person running it nor read a more detailed project description than is visible in the grant itself.

If there are interesting results from this funding round, I might be willing to donate a larger follow-up round.

Description of subprojects and results, including major changes from the original proposal

Spending breakdown

Fully funded (I asked David to apply). I think Decode is great and serve a valuable role providing infrastructure for the interpretability community and I want to see them succeed. The co-founder Johnny is an absolutely cracked engineer and has made some amazing interfaces and surrounding infra, but I was sad to hear that Curt, the main one with a research background, has decided to move on, as I think it's very helpful to have someone with a research background around. I'm super excited that David is now interested in getting involved and happy to support this, I really liked some of David's past research like the feature absorption paper, and I think he'll be able to provide a lot of valuable research expertise to complement Johnny's skillset.

@NeelNanda Thanks for asking, Neel! As I say I mostly engage with John Wentworth and Sahil (I meet each weekly), and with the "Digital Minds" reading group run by Patrick Butlin, Rob Long, and Nick Bostrom (biweekly).

Here's a forthcoming paper on my AIT account of abstraction. Here are recordings of three talks I gave at ILIAD last summer. And here's a slide summary and draft link, from TAIS, of my most recent work, discussed above; there's also a talk version I gave at Hong Kong University. Let me know if you have questions!

Could you share a few examples of your work (in the AI alignment space) that you feel proudest of? (Blog posts are fine, no need to try to stick to published papers - just whatever you consider your best work to be).Or, if I wanted to ask some alignment researchers who've engaged a bunch with your research and found it valuable, who would you recommend?

Many new projects still use https://github.com/TransformerLensOrg/TransformerLens as a core dependency. Over 500 public code repositories on Github rely upon the transformer-lens package, including ones created by leading organizations such as Meta Research, Redwood Research, Model Evaluation & Threat Research, Apollo Research, and Decode Research.

Bryce has a proven track record of consistent contributions to the library and is the best possible owner for ensuring the stability and growth of TransformerLens moving forward (as well as compatibility across other packages for doing mechinterp). He brings a deep expert understanding of the existing feature set and has demonstrated rapid iteration speed and skill to adapt the framework for recently launched models.

He also actively volunteers to answer community questions on Slack and integrates user feedback into the development roadmap with detailed progress updates to relevant stakeholders. I would also be excited to see Fabian's contributions going into TransformerLens 3.0 and beyond. I'd endorse and highly recommend donating to this manifund project to help it reach its full funding goal through the rest of 2025.

Great project! Super interesting

@Austin Thanks!

I should clarify: I don't think "presenting at The Curve" slowed things down per se.

I think in a lot of ways it was positive (eg. in having a clear intermediary deadline, and I also met people there who introduced me to people who ended up donating a significant amount).

What slowed things down a little bit was 1) not starting to work with people I would want to work on the final product with from the start 2) "software decisions" where we worked in not-very modular ways before the conference, which implies we had to duplicate a lot of the work we did there.

I think there would have been ways to work in a more modular / long-term way from the start, and I take responsibility there.

Also, having a "MVP" (or "rough cut" in the movie world) was also instrumental in contracting more experienced folks, so having that "conference cut" from the Curve was maybe a necessary step in hiring more senior people after all.

@michaeltrazzi I really appreciate the thoughtful and in-depth retrospective! And I'm sorry if the presentation at The Curve slowed down the time to present this; I hope you still found value in presenting it there~

SB-1047 Documentary Post-mortem

This documentary took 27 weeks and $157k instead of my planned 6 weeks and $55k. Here's what I learned about documentary production

Total funding received: ~$143k ($119k from this grant, $4k from Ryan Kidd's regrant on another project, and $20k from the Future of Life Institute).

Total money spent: $157k

In terms of timeline, here is the rough breakdown month-per-month:
- Sep / October (production): Filming of the Documentary. Manifund project is created.
- November (rough cut): I work with one editor to go through our entire footage and get a first rough cut of the documentary that was presented at The Curve.
- December-January (final cut - one editor): I interview multiple potential editors that would work on the final cut, and decide on one candidate who would be the one that does most of the editing (from December to February).
- February-March (final cut - 7 Full-Time Equivalents): I work with a total of 7 seasoned professionals (working full-time) to have a finished documentary by the end of March. This is the most capital intensive period of the post-production phase.
- April: We wait back to hear from multiple distributors about whether they would be interested in publishing the documentary on their platform. Multiple outlets show strong interest (New York Times Op-Docs or Wired) but the content of the documentary doesn't fit publication policies.
- May: The documentary is published on May 5th.

Breaking down how the money was spent:
- Editing was the largest part of the expenses, since I ended up paying for a total of 4 different editors, that have worked from November to March included. From February to March I had multiple editors working on the documentary in parallel.
- Motion graphics was the second largest item, with two people working on motion graphics in February and March
- In terms of music & sound, the documentary used custom music made by a composer, with some of the songs played by real instruments, but also required the work of a seasoned sound mixer, which is why this is the third most expensive item.
- The director salary ended up representing only ~9% of the total expenses, since I had originally planned to pay myself $15k for 10 weeks, but the project ended up taking ~27 weeks instead.

But why did the project end up taking 27 weeks instead of 6 weeks?

- Short answer: I ended up getting more funding than I originally had asked for on Manifund, and had to hire many different professionals with that funding. Having to present something intermediary at the conference "The Curve" potentially slowed us down. And a lot of the steps had to happen one after the order, including all of the fundraising, hiring, multiple steps of post-production, on top of the distribution of phase where we had to wait to hear back from potential distributors. All of this considered made that the movie take ~5 months to be ready (and 6 months to be out) instead of 6 weeks.

- Long answer (breaking out month by month):
--> In November I was offered to do a first screening of the documentary at the AI conference The Curve, which I imagined to be a great way to present a first draft of the movie as we were working on it. However, in order to get this first draft done on time (for the conference), I had to hire an editor with whom I did not end up working with throughout the project, and most of the work was rushed using a software that we did not use in the future, meaning a lot of the work we did in November was not directly re-usable in later months.
--> In December a lot of time was spent trying to find a film editor that would be willing to work full-time with us on the project. In the end I was really satisfied with the editor we ended up with, but then came the end of year Holidays, so not a lot of editing happened then. On top of that, our funding was constrained enough that I would not have been able to hire more people to work on this, which would make my only editor much less productive compared to other environments where he would have an assistant editor, an archival producer or motion graphics person to help him.
--> In January, an important bottleneck was still not being able to hire more people because of funding constraints. But also the wildfires happening in LA, which directly impacted our main editor. When funding was secured, a lot of time was spent finding the right people that would join in February, including a composer, an archival producer, two motion graphic designers, and two editors. Some other things that slowed us down here was that the software and organisation decisions we had taken in earlier months were starting to slow us down, so had to do some large refactoring, and transition software.
--> In February most of the editing for the documentary happened, with 7 people working full-time on it. By the end of the month, we were able to have something that was almost done in terms of story and bytes.
--> In March most of the music, motion graphics and sound happened, because all of this work required the edit to be locked-in (also called "picture lock").
--> In April the movie was ready to be published, but we spent most of the month waiting for the potential distributors (NYT, Wired, LA Times, etc.) to get back to us. Other work involved upscaling and marketing.
--> In May the documentary was published on Youtube at the beginning of the month.

Impact:
- The documentary achieved 2,500 hours of watch time (25% retention rate) across 20k YouTube views, and 100k views on X
- The documentary got presented at AI Conference "The Curve" and I'm in talks to present it at the UK Parliament (depending on funding).
- I've heard from multiple news outlets (including Wired, NYT) or filmmakers that the film was very well edited and deserved to be on streaming platforms.
- As a consequence of that, I've recently started submitting the documentary to movie festivals.
- There is still potential work I want to be doing involving directly sending / showing this documentary to lawmakers in the UK / US (depending on funding).

What I would do differently next-time:
- I would try to work with a fixed timeline and realistic corresponding budget. A lot of the problems I've had with this project was that I wanted to finish a documentary quickly with not enough staff, and had to wait to fundraise more before hiring more people. Once I had enough staff, getting people to work quickly and finish the project was much easier.
- I would dedicate more funding to marketing / distribution, and go through the movie festival route first. A lot of the issues I was having towards the end for distribution is that getting potential distributors to watch your documentary is a process that takes time, and a lot more time needs to be allocated to send your movie through the movie festival pipeline, to get increased coverage down the line. I think one of the reasons the movie got less views on Youtube than expected is that people do not expect to watch a movie on Youtube and the content would have performed better on a streaming platform where they'd expect a documentary.
- In terms of the movie itself, I would spend more time at the start of the movie discussing exactly what is AI Safety, why AI is a big deal, what AI regulation is, talking to a wider audience. Given that the documentary was published on Youtube, it is necessary to explain those terms in a much more pedagogical way so that most people that had not heard of SB-1047 would have been able to understand what was happening and why they should care.
- I would focus on figuring out distribution first. I am already now in touch with distributors that would be happy to work with me from the beginning of the project next time, instead of having to convince later to distribute the project. This guarantees funding and distribution from the start.
- I would start with all of my team already figured out. Now that I have already talked to >50 seasoned professionals and contracted 16 of them, I would be able to get started with professionals I trust from day one for my next project, which would have saved me about 2 months on this project.
- I would also work in person instead of remotely, which I think would have saved me one month.

Steve Petersen provides an important bridge between mainstream academic philosophy and AI safety work. I have had many valuable discussions with him. He has work in algorithmic information theory, establishing an important theory of abstraction, which is a central concern of several AI safety research agendas (John Wentworth and Sam Eisenstat come to mind). This demonstrates his technical chops & ability to craft deep theories.

@Tebycimtime yes, sorry for the confusion in our interface -- the number $202 indicates your total donated at this point in time, summing up the $100 and $102 donations. You can also check out your profile for a log of transactions which display the two separate donations!

I want to assure that I'm communicating clearly. I intentionally donated $100 a month ago, and (intentionally) initiated a $100 donation on 7/29, which cleared my account on 7/30. All that's fine. Yet for some odd reason, the second $100 donation with $2 in tips didn't post to the AIFP roster. After communicating my concern to Jonas at AIFP, I formed the impression that perhaps the second donation was in processing limbo, as other subsequent donors' contributions had promptly posted. So I clicked on a pink button again, thinking that action was necessary to post the $102 amount. At that time, my impression was that I had increased my second donation by an unintended additional $100, totaling $202 (for the second donation) rather than the intended $102. Perhaps the $202 amount showing was the tally instead of the second line item. So, my AIFP donation of $102 this week is correct. I hope we're clear. Thank you, and I apologize for taking so much time with this... @Tebycimtime

Yes, Austin, but only because it's not in my budget at this time. Thank you. @Austin

Hi Michael -- are you saying that the second $100 donation was an accident and you want to undo it? If so, I can process that for you!

I'm embarrassed that I've navigated what should be a simple donation via Manifold so poorly. The second, unintended $100, if and when submitted, will be refused, only because my budget cannot accommodate it. @Tebycimtime

@geneparmezan Thank you so much! Really appreciate the support

@sheikheddy Thank you for this comment, I've just made a small donation in support as well.

to support rigorous safety analysis and clear communication of AI risks.

I've funded this up to 1 semesters worth. On my inside view, I'm not particularly excited about this kind of theory heavy non-empirical alignment work. on the outside view it sure seems like enough people are into this kind of thing that it should be taken seriously and given at least some resources. And my sense is that AI safety is a lot more constrained by plausibly reasonable places to spend money, than money. So this seems like a pretty reasonable thing to fund. I don't feel compelled enough to fully fund this, but would be happy to see someone else who feels excited about this do so.

Good luck! I hope it goes well.

Anthony has done great work on other OS libraries for AI safety (SAELens, neuronpedia, and SAEDashboard) and I'm excited that he wants to explore building tools for evals.

I've talked to him about his plans and think it's an obviously good funding opportunity.

I think the topic is important. I have neither done a check on the person running it nor read a more detailed project description than is visible in the grant itself.

If there are interesting results from this funding round, I might be willing to donate a larger follow-up round.

Description of subprojects and results, including major changes from the original proposal

Spending breakdown

Fully funded (I asked David to apply). I think Decode is great and serve a valuable role providing infrastructure for the interpretability community and I want to see them succeed. The co-founder Johnny is an absolutely cracked engineer and has made some amazing interfaces and surrounding infra, but I was sad to hear that Curt, the main one with a research background, has decided to move on, as I think it's very helpful to have someone with a research background around. I'm super excited that David is now interested in getting involved and happy to support this, I really liked some of David's past research like the feature absorption paper, and I think he'll be able to provide a lot of valuable research expertise to complement Johnny's skillset.

@NeelNanda Thanks for asking, Neel! As I say I mostly engage with John Wentworth and Sahil (I meet each weekly), and with the "Digital Minds" reading group run by Patrick Butlin, Rob Long, and Nick Bostrom (biweekly).

Here's a forthcoming paper on my AIT account of abstraction. Here are recordings of three talks I gave at ILIAD last summer. And here's a slide summary and draft link, from TAIS, of my most recent work, discussed above; there's also a talk version I gave at Hong Kong University. Let me know if you have questions!

Could you share a few examples of your work (in the AI alignment space) that you feel proudest of? (Blog posts are fine, no need to try to stick to published papers - just whatever you consider your best work to be).Or, if I wanted to ask some alignment researchers who've engaged a bunch with your research and found it valuable, who would you recommend?

Many new projects still use https://github.com/TransformerLensOrg/TransformerLens as a core dependency. Over 500 public code repositories on Github rely upon the transformer-lens package, including ones created by leading organizations such as Meta Research, Redwood Research, Model Evaluation & Threat Research, Apollo Research, and Decode Research.

Bryce has a proven track record of consistent contributions to the library and is the best possible owner for ensuring the stability and growth of TransformerLens moving forward (as well as compatibility across other packages for doing mechinterp). He brings a deep expert understanding of the existing feature set and has demonstrated rapid iteration speed and skill to adapt the framework for recently launched models.

He also actively volunteers to answer community questions on Slack and integrates user feedback into the development roadmap with detailed progress updates to relevant stakeholders. I would also be excited to see Fabian's contributions going into TransformerLens 3.0 and beyond. I'd endorse and highly recommend donating to this manifund project to help it reach its full funding goal through the rest of 2025.

Great project! Super interesting

@Austin Thanks!

I should clarify: I don't think "presenting at The Curve" slowed things down per se.

I think in a lot of ways it was positive (eg. in having a clear intermediary deadline, and I also met people there who introduced me to people who ended up donating a significant amount).

What slowed things down a little bit was 1) not starting to work with people I would want to work on the final product with from the start 2) "software decisions" where we worked in not-very modular ways before the conference, which implies we had to duplicate a lot of the work we did there.

I think there would have been ways to work in a more modular / long-term way from the start, and I take responsibility there.

Also, having a "MVP" (or "rough cut" in the movie world) was also instrumental in contracting more experienced folks, so having that "conference cut" from the Curve was maybe a necessary step in hiring more senior people after all.

@michaeltrazzi I really appreciate the thoughtful and in-depth retrospective! And I'm sorry if the presentation at The Curve slowed down the time to present this; I hope you still found value in presenting it there~

SB-1047 Documentary Post-mortem

This documentary took 27 weeks and $157k instead of my planned 6 weeks and $55k. Here's what I learned about documentary production

Total funding received: ~$143k ($119k from this grant, $4k from Ryan Kidd's regrant on another project, and $20k from the Future of Life Institute).

Total money spent: $157k

In terms of timeline, here is the rough breakdown month-per-month:
- Sep / October (production): Filming of the Documentary. Manifund project is created.
- November (rough cut): I work with one editor to go through our entire footage and get a first rough cut of the documentary that was presented at The Curve.
- December-January (final cut - one editor): I interview multiple potential editors that would work on the final cut, and decide on one candidate who would be the one that does most of the editing (from December to February).
- February-March (final cut - 7 Full-Time Equivalents): I work with a total of 7 seasoned professionals (working full-time) to have a finished documentary by the end of March. This is the most capital intensive period of the post-production phase.
- April: We wait back to hear from multiple distributors about whether they would be interested in publishing the documentary on their platform. Multiple outlets show strong interest (New York Times Op-Docs or Wired) but the content of the documentary doesn't fit publication policies.
- May: The documentary is published on May 5th.

Breaking down how the money was spent:
- Editing was the largest part of the expenses, since I ended up paying for a total of 4 different editors, that have worked from November to March included. From February to March I had multiple editors working on the documentary in parallel.
- Motion graphics was the second largest item, with two people working on motion graphics in February and March
- In terms of music & sound, the documentary used custom music made by a composer, with some of the songs played by real instruments, but also required the work of a seasoned sound mixer, which is why this is the third most expensive item.
- The director salary ended up representing only ~9% of the total expenses, since I had originally planned to pay myself $15k for 10 weeks, but the project ended up taking ~27 weeks instead.

But why did the project end up taking 27 weeks instead of 6 weeks?

- Short answer: I ended up getting more funding than I originally had asked for on Manifund, and had to hire many different professionals with that funding. Having to present something intermediary at the conference "The Curve" potentially slowed us down. And a lot of the steps had to happen one after the order, including all of the fundraising, hiring, multiple steps of post-production, on top of the distribution of phase where we had to wait to hear back from potential distributors. All of this considered made that the movie take ~5 months to be ready (and 6 months to be out) instead of 6 weeks.

- Long answer (breaking out month by month):
--> In November I was offered to do a first screening of the documentary at the AI conference The Curve, which I imagined to be a great way to present a first draft of the movie as we were working on it. However, in order to get this first draft done on time (for the conference), I had to hire an editor with whom I did not end up working with throughout the project, and most of the work was rushed using a software that we did not use in the future, meaning a lot of the work we did in November was not directly re-usable in later months.
--> In December a lot of time was spent trying to find a film editor that would be willing to work full-time with us on the project. In the end I was really satisfied with the editor we ended up with, but then came the end of year Holidays, so not a lot of editing happened then. On top of that, our funding was constrained enough that I would not have been able to hire more people to work on this, which would make my only editor much less productive compared to other environments where he would have an assistant editor, an archival producer or motion graphics person to help him.
--> In January, an important bottleneck was still not being able to hire more people because of funding constraints. But also the wildfires happening in LA, which directly impacted our main editor. When funding was secured, a lot of time was spent finding the right people that would join in February, including a composer, an archival producer, two motion graphic designers, and two editors. Some other things that slowed us down here was that the software and organisation decisions we had taken in earlier months were starting to slow us down, so had to do some large refactoring, and transition software.
--> In February most of the editing for the documentary happened, with 7 people working full-time on it. By the end of the month, we were able to have something that was almost done in terms of story and bytes.
--> In March most of the music, motion graphics and sound happened, because all of this work required the edit to be locked-in (also called "picture lock").
--> In April the movie was ready to be published, but we spent most of the month waiting for the potential distributors (NYT, Wired, LA Times, etc.) to get back to us. Other work involved upscaling and marketing.
--> In May the documentary was published on Youtube at the beginning of the month.

Impact:
- The documentary achieved 2,500 hours of watch time (25% retention rate) across 20k YouTube views, and 100k views on X
- The documentary got presented at AI Conference "The Curve" and I'm in talks to present it at the UK Parliament (depending on funding).
- I've heard from multiple news outlets (including Wired, NYT) or filmmakers that the film was very well edited and deserved to be on streaming platforms.
- As a consequence of that, I've recently started submitting the documentary to movie festivals.
- There is still potential work I want to be doing involving directly sending / showing this documentary to lawmakers in the UK / US (depending on funding).

What I would do differently next-time:
- I would try to work with a fixed timeline and realistic corresponding budget. A lot of the problems I've had with this project was that I wanted to finish a documentary quickly with not enough staff, and had to wait to fundraise more before hiring more people. Once I had enough staff, getting people to work quickly and finish the project was much easier.
- I would dedicate more funding to marketing / distribution, and go through the movie festival route first. A lot of the issues I was having towards the end for distribution is that getting potential distributors to watch your documentary is a process that takes time, and a lot more time needs to be allocated to send your movie through the movie festival pipeline, to get increased coverage down the line. I think one of the reasons the movie got less views on Youtube than expected is that people do not expect to watch a movie on Youtube and the content would have performed better on a streaming platform where they'd expect a documentary.
- In terms of the movie itself, I would spend more time at the start of the movie discussing exactly what is AI Safety, why AI is a big deal, what AI regulation is, talking to a wider audience. Given that the documentary was published on Youtube, it is necessary to explain those terms in a much more pedagogical way so that most people that had not heard of SB-1047 would have been able to understand what was happening and why they should care.
- I would focus on figuring out distribution first. I am already now in touch with distributors that would be happy to work with me from the beginning of the project next time, instead of having to convince later to distribute the project. This guarantees funding and distribution from the start.
- I would start with all of my team already figured out. Now that I have already talked to >50 seasoned professionals and contracted 16 of them, I would be able to get started with professionals I trust from day one for my next project, which would have saved me about 2 months on this project.
- I would also work in person instead of remotely, which I think would have saved me one month.

Steve Petersen provides an important bridge between mainstream academic philosophy and AI safety work. I have had many valuable discussions with him. He has work in algorithmic information theory, establishing an important theory of abstraction, which is a central concern of several AI safety research agendas (John Wentworth and Sam Eisenstat come to mind). This demonstrates his technical chops & ability to craft deep theories.

@Tebycimtime yes, sorry for the confusion in our interface -- the number $202 indicates your total donated at this point in time, summing up the $100 and $102 donations. You can also check out your profile for a log of transactions which display the two separate donations!

I want to assure that I'm communicating clearly. I intentionally donated $100 a month ago, and (intentionally) initiated a $100 donation on 7/29, which cleared my account on 7/30. All that's fine. Yet for some odd reason, the second $100 donation with $2 in tips didn't post to the AIFP roster. After communicating my concern to Jonas at AIFP, I formed the impression that perhaps the second donation was in processing limbo, as other subsequent donors' contributions had promptly posted. So I clicked on a pink button again, thinking that action was necessary to post the $102 amount. At that time, my impression was that I had increased my second donation by an unintended additional $100, totaling $202 (for the second donation) rather than the intended $102. Perhaps the $202 amount showing was the tally instead of the second line item. So, my AIFP donation of $102 this week is correct. I hope we're clear. Thank you, and I apologize for taking so much time with this... @Tebycimtime

Yes, Austin, but only because it's not in my budget at this time. Thank you. @Austin

Hi Michael -- are you saying that the second $100 donation was an accident and you want to undo it? If so, I can process that for you!

I'm embarrassed that I've navigated what should be a simple donation via Manifold so poorly. The second, unintended $100, if and when submitted, will be refused, only because my budget cannot accommodate it. @Tebycimtime