In my evals field-building efforts, I recently asked people who want to build something in evals to fill out a simple form (https://www.lesswrong.com/posts/uoAAGzn4hPAHLeG2Y/which-evals-resources-would-be-good)

This project was, in my opinion, the second best (after James project: https://manifund.org/projects/more-detailed-cyber-kill-chain-for-ai-control?tab=comments). My thinking is roughly:
1. The project itself sounds very reasonable to me. I find it very obvious that modeling capabilities with latent variable models is something clearly worth trying. I think that, in the best case, their efforts will be as impactful as observational scaling laws (https://arxiv.org/abs/2405.10938). In the worst case, the method is too hard to get working in practice or doesn't provide any clear benefit over other simple analysis tools. In expectation, I think this research will not be as impactful as observational scaling laws (but it has the chance to, which is already a high bar) but will be useful to a decent number of evaluators like Apollo, METR, AISIs, etc. Furthermore, I think that the project will be conceptually valuable because it partially forces you to think about which components this Bayesian model should have and partially because I expect the findings to be insightful. Concretely, I expect the analysis of the latent variables to be quite interesting.
2. I don't know Laurence or Desi personally but their work so far seems reasonable at a first glance and they are clearly more experienced researchers than the average MATS scholar.
3. The project salaries are already covered. This grant is merely for compute. Thus, it seems more impactful on the margins. I expect that a lot of the most interesting results come from runs with lots of compute, so this grant might unlock "the most interesting stuff".
4. Finally, I think science of evals is perfectly suited for academics. It often doesn't require access to the biggest models; many of the classic academic research skills directly transfer, and the results are useful for the entire field. Thus, I have a general intuition that we should try harder to fund academics to do more work on the science of evals and I'm surprised that this isn't happening more yet.

In my evals field-building efforts, I recently asked people who want to build something in evals to fill out a simple form (https://www.lesswrong.com/posts/uoAAGzn4hPAHLeG2Y/which-evals-resources-would-be-good)

James' project was, in my opinion, the best of all applications. This is because:
1. I have talked to James before and seen some of his work, and I think he is competent at executing the project.
2. I think the project itself is really important. Cybersec seems very important in itself but is also crucial for control and scheming research.
3. The cost of this project seems very low. I think a similar project description could have easily justified $50k+. In contrast, James only asks for $9k (split across two projects).

For context, I think this project is about as impactful as the average MATS extension but ~5x cheaper. If James has more room for funding, I'd encourage a funder to commit to a conditional follow-up grant, e.g. check in with James after 2 months to evaluate progress and potentially extend the funding for 6 months or so.

We wrote a 1-year update which includes the projects we used the Manifund funding for: https://www.apolloresearch.ai/blog/the-first-year-of-apollo-research

Thanks again for the support!

Thanks for the compliments. We quickly want to contextualize some of the reservations.
1. We understand that we're more expensive on a per-person basis than independent researchers. However, On top of salaries, our costs also include compute, pensions, insurance, travel cost, visas, office, etc. which are usually not factored into the applications for independent researchers. Furthermore, as discussed in the response to Austin Chen's comment, we're currently significantly cheaper than other AI safety labs. For example, we calculate with 200k/year total cost while many technical positions in Bay area AI safety orgs have >200k/year starter salaries (not yet including pensions, insurance, compute, etc).
2. Given that we don't train any models, it would require a lot of effort for us to pivot to becoming an AI accelerator. Also, as you can see in our sharing and security policy (https://www.apolloresearch.ai/blog/security), we're very aware of the potential to accidentally accelerate AI research and thus engage in various security measures such as differential publishing.
3. Obviously, our spending target is not infinite. The reason why we are asking for more money at the moment is that we have many obvious ways to use it effectively right now--primarily by hiring more researchers and engineers. As suggested in another comment, a short runway makes it both harder to plan and may make it impossible to hire very senior staff, despite them being interested in working for us.

Some people have been concerned with us scaling too fast and we think this is good advice in general. However, there are a couple of considerations that make us think that the risk of growing too slowly is bigger than the risk of growing too fast for our particular situation.
1. We have an actionable interpretability and evals agenda. In both cases, we're only limited by the number of experiments we can run. If we were bottlenecked by ideas we wouldn't try to scale.
2. We already form a coherent team. Many people in the team have worked together before and it currently feels like our team dynamic is very good. Otherwise, we would focus more on building cohesion first.
3. We have the management capacity. Multiple people in the team have supervised others before. This really doesn't feel like a problem right now.
4. The talent is amazing. There are some really great people in the AI safety space. They are motivated, capable, and nice to work with. It's really not that hard to onboard such candidates.
5. Time is ticking. AI alignment isn't gonna solve itself and there will be lots of evals organizations popping up that don't care about catastrophic risk. Making sure that there are multiple leading organizations that care about these risks seems important.
6. We can always stop scaling for a while. If we realize after the first hiring round that we're scaling too fast, we can always postpone the next round and focus on consolidation. On the other hand, if we realize that we need more people it's hard to hire fast. Most rounds take 3 months from job ad to onboarding.

Thank you!

Thanks for the high praise, we will try to live up to that expectation. These are totally reasonable questions and we're happy to answer them in more detail.
1. That mostly depends on your definition of "a lot". We have received enough funding to work for ~one year with the current team but we don't have enough funding to hire the number of people we think we can productively onboard. We have checked with our biggest funders and are allowed to disclose the amounts (which is why we didn't respond immediately). We have received ~$1.5M from OpenPhil and ~$500k from SFF speculation grants (the round has not closed yet, so we don't know how much we'll receive in total). We have a small number of other funders that add up to <$200k in addition to the two big institutional funders.
While this is a substantial amount of money, running an AI safety research lab is unfortunately quite expensive with salaries and compute being the biggest cost points. For comparison, Redwood started with $10M (and a $10M top-up after one year), and another alignment non-profit (anonymous on OpenPhil's website but not hard to deduce) started with $14.5M. Both of these organizations aim to pay competitively with FAANG salaries which are ~$200-500k for ML talent before equity.
2. We mostly agree. We hope to be fully funded by larger institutional funders or aligned investors in the near future. However, right now, we could easily onboard multiple new people but just don't have the funds to do so. Therefore, Manifund funding would directly make a big difference for us. Furthermore, the fact that people are enthusiastic about Apollo Research has meant multiple funders expected someone else to have already covered our entire funding gap or wanted to give less on the expectation that others would give more. Lastly, we're specifically asking for 3 positions because these have not been covered by other funders and we want to fill them as soon as possible.
3. We estimate that we could productively spend $5-7M in the first year, $10-20M in the second, and even more later. Furthermore, having more than 10 months of runway makes planning much easier and allows us to hire people with more experience who are often less willing to join an organization with a short runway. We don't intend to compete with Anthropic, OpenAI and DeepMind on salaries but it is a drawback if our salaries are 2-10x lower (which they currently are). For reference, a SWE at Anthropic London earns GPB 250k-420k before equity. Therefore, we would like to raise salaries to retain and attract talent in the long run (maybe so that they are only 1.5-5x lower).
Regarding talent: We are well connected in the AI safety scene and think that there is a lot of talent around that is currently looking for jobs but unable to find any. We think it is really bad for the ecosystem and for AI alignment that this talent is unable to fully contribute or has to jump from independent grant to independent grant without stability or ability to plan (this also heavily selects against people with families who are often more experienced). Our agenda is fairly straightforward and we have the management capacity to onboard new people, so we would really like to hire more people fairly soon. We think it is likely that we will be 15-30 people by next June if we have the funds to do so.

We're happy to answer more questions or clarify any of the above if needed!