@AntonMakiievskyi

TLDR; no big funders have retracted funding, we have undersold our impact in grant applications by not focusing on the metrics that we now know work best, our network within SF isn't as strong as other orgs, and the AIS funding ecosystem face general problems. See a longer response to this question here.

View our new application here: https://manifund.org/projects/keep-apart-research-going-global-ai-safety-research--talent-pipeline

Apart Research Progress Update - September 2024

Key Achievements

1. Expanded Participation: Our Sprints have gained significant traction in 2024, with over 1000 signups and approximately 100 team submissions at our events.

2. Growth in Lab Fellows and Projects: The Apart Lab has seen rapid expansion throughout the year. We've progressed from 17 fellows working on 7 projects in Q1 to our current cohort of 37 fellows collaborating on 12 projects in Q3, bringing the total to over 75 fellows (incl. Q2). We believe that this steady growth showcases the increasing demand for our program and our capacity to support more ambitious research endeavors.

3. Team Expansion:

   • Welcomed two experienced core team members:

     • Natalia Pérez-Campanero Antolín (former program manager for the Royal Society) - Leading research project management at Apart Lab

     • Archana Vaidheeswaran (former board member of Women Who Code) - Managing community and events for hackathons

   • Expanded our in-house mentorship capacity to support the growing number of fellows

4. Research Impact: Several research papers from our fellows are currently under review at AI and machine learning venues. A selection of these papers:

   • Classifies whether a model is cheating on evaluations, i.e. if the model has been trained on a benchmark

   • Identifies deceptive patterns in fine-tuned LLM conversation, inspired by dark patterns in UI design, finding that Meta's models are manipulative while Anthropic's are not

   • Evaluates cyber offense capabilities from LLM performance on realistic challenges conforming to industry-standard cyber offense risk classifications, showing that models can autonomously perform complex cyber offense operations

   • Examines how to mitigate infectious jailbreaks in realistic multi-agent systems

Ongoing Initiatives

1. Increasing participation of high-potential talent in hackathons, with a focus on underrepresented communities including senior tech professionals, cybersecurity experts, and individuals with hard science backgrounds

2. Enhancing the impact of lab research projects by supporting teams in pursuing ambitious and potentially groundbreaking research ideas

3. Developing an AI safety incubator to facilitate the transition of high-impact research ideas into real-world solutions

4. Actively recruiting a research engineer to further support our vision of producing impactful research projects

5. Onboarded a research communications specialist that can help us make all of our above progress more visible to followers of Apart

If you are interested in supporting us on our journey, feel free to use this page, or contact finn@apartresearch.com.

Project update

Since the submission on December 18 2023, we have...

• Reshaped our leadership with a renewed focus on strategy and organizational development, with Fazl Barez changing position from Co-director to Research Advisor and Jason Hoelscher-Obermaier stepping into the Co-director position.

• Onboarded 16 new fellows working on 7 different research projects. This new cohort stems from 9 different countries (Germany, the UK, Denmark, Sweden, the US, Spain, France, India, and Vietnam) and spans a broad spectrum of experience levels:

  • 2 PhDs (Deep Learning, Astrophysics)

  • 2 PhD students (ML, Statistics)

  • 5 with Master's degrees (Computer Science, Computational Modeling, Complex Systems, Social Science, and Logic, Philosophy and History of Science)

  • 2 Master's students (AI & Data Science, Human-Centered AI)

  • 2 Bachelor's graduates (Mechatronics, Epidemiology)

  • 3 Bachelor's students (Computer Science, Data Science, Mathematics)

• Had more research coming out of Apart Lab:

  • Published on Neuroplasticity in LLMs (still under review; see X)

  • Understanding Addition in Transformers accepted at ICLR 2024

• Collaborated on the Sleeper Agents paper published by Anthropic

Thank you for the kind words and the generous evaluation request, @Austin!

Thank you for the review and the positive words. We'll address your points below. Please let us know if anything remains unclear. We are more than happy to provide more details.

I don't understand why other people haven't funded this yet

Apart has, for much of 2023, been a very small team (2 FTE) fully focused on experimenting with and improving the research sprints and lab fellowship. We simply did not have the capacity to scale up the program and the corresponding fundraising so far. Now, with an additional leadership team member and two half-time assistants on board (4 FTE), we finally have the capacity to tackle scaling up and funding seriously.

Maybe this application is exaggerating stuff?

We have been quite careful not to overstate our case in this proposal and needless to say, all numbers presented are correct. Let us know if you need any more details or evidence.

Maybe the organization adds another step in the chain to impact, and it would be more efficient to fund individual people instead?

We think that a peer-reviewed publication (particularly a first-author publication) is close to a necessity for most research careers and that aspiring AI safety researchers are much less likely to achieve this on the same swift time scale without the right structure and guidance.

Another added benefit of funding us versus funding individual people directly is that, by directly interacting with a very large number of individuals through our research sprints, we can identify and carefully select talented people who would not be obvious targets for other funders, and thereby uncover talent that would otherwise be overlooked.

Maybe the biggest one: how do I know the success is counterfactual? Say that someone participated in a hackathon/fellowship/etc, and then later got a research position in some Oxford lab. How do I know that the person wouldn't have gotten something similarly impressive in the absence of your organization?

As mentioned above, we believe that (a) having a paper (especially as first author) accepted/submitted for peer-reviewed AI safety publication will often be critical to our target demographic's career trajectory and (b) the people we support are less likely to convert a hackathon project into a paper worthy of submission/acceptance at a top venue without our support. Evidence for (a) comes from our fellows in interview processes who report that most of their time in interviews have been discussions on their Apart Lab research project (with one interviewer remarking it as "a strong part of [the] resume" to go through the program). Evidence for (b) is harder to come by but self-reports of our fellows suggest that they believe they could not have achieved a similar output without the help of Apart Lab.

@NeelNanda That is correct, thank you for the clarification, Neel! The text has been updated and we're also happy to say that there are now 7 papers in review, so we're crossing fingers for the research teams :D