Project summary

• Luthien is a non-profit developing AI Control for immediate real-world deployments, based on Redwood Research's AI Control agenda. https://luthienresearch.org/

What are this project's goals? How will you achieve them?

• Luthien's ultimate goal is to increase the probability that effective AI Control systems will be deployed to mitigate catastrophic risks from frontier AI systems when they are developed.

• We think getting real-world feedback ASAP on how AI Control systems perform in real-world situations and iterating aggressively to develop practical, effective Control systems significantly increases the odds that effective AI Control systems will be deployed to mitigate otherwise-catastrophic risks.

• To that end we're developing AI Control systems targeting prosaic, lower-stakes scenarios, like occasionally-misbehaving coding assistants. By doing so, we hope to discover and solve the types of unforeseen problems that emerge when any new type of system is deployed, develop a playbook for effective AI Control deployments, establish standards and best practices for effective AI Control systems, and test and develop effective Control strategies in real-world situations. Additionally, we want to see how far we can push automated red/blue-teaming to develop effetive strategies quickly.

• Luthien's secondary goal is to establish an AI Safety presence in Seattle, where there is currently a great deal of latent talent but very few opportunities to onboard into the AI Safety space.

How will this funding be used?

• Funding extends runway. The longer I can support my family doing this, the more I can get done.

• Funding will be used for salary, API credits, other compute infrastructure, and org logistics like attending ControlConf

• These amounts will in practice fluctuate as development continues (we'll need less additional runway to get to the point of iterating on feedback)

Who is on your team? What's your track record on similar projects?

• Luthien currently consists of me, Jai Dhyani. I'm an ex-ML Software Engineer at Meta, MATS alum, and a co-author on "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts".

What are the most likely causes and outcomes if this project fails?

• We're unable to develop Control solutions that deliver enough benefit to justify the requisite costs in money, latency, and complexity for real-world use cases.

• Immediate real-world use cases are sufficiently different across all relevant dimensions such that there's little to no payoff for developing and deploying effective Control systems for high-stakes scenarios.

How much money have you raised in the last 12 months, and from where?

~$200. $150k of this is a one-time seed grant from the AI Safety Tactical Opportunities Fund meant to get us off the ground and spur further donations. We received a $30k speculative grant from the Survival and Flourishing Fund. The remainder is from individuals donating ~$5k-$20k each.