Principled Agents

Rubi Hudson

created Sep 8, 2026

Project summary

Principled Agents is a research nonprofit working on defining goals that are safe for powerful AI to pursue. We are looking to cover operational expenses for up to one year, starting from December 2026.

What are this project's goals? How will you achieve them?

The Principled Agents agenda is formally specifying safety properties that collectively make a utility function robust to optimization. The target we have in mind is unmanipulated and fully informed human approval, pursued corrigibly and conservatively.

We take a full-stack approach, starting with conceptual desiderata, then mathematically defining them alongside the utility functions and setups that incentivize them, and finally testing them empirically as a reward signal. Individual projects we run tend to focus on a single safety property (e.g. corrigibility), and involve reviewing existing proposals, identifying where they fail, and proposing new mechanisms that address at least some of the issues.

How will this funding be used?

This funding would be used to cover operational expenses at our current level, allowing us to continue making progress in our research.

Our monthly expenses are just under $40,000. Approximately 70% of this is allocated to the salaries of three researchers. The remaining 30% covers compute for experiments (10%), office space (8%), and operational overhead along with miscellaneous expenses (12%).

Who is on your team? What's your track record on similar projects?

Rubi Hudson is the founder of Principled Agents and the research lead. He has a track record of producing research that makes progress on important theoretical problems in AI safety. Notable outputs include Joint Scoring Rules (AAAI 2025), which showed how to elicit predictions from agents without incentivizing them to affect outcomes, and Corrigibility Transformation (ICML 2026), which introduced a method for modifying goals to remove instrumental incentives for avoiding updates and shutdown. Rubi is wrapping up a PhD at the University of Toronto, previously participated in the MATS program, and has mentored for the SPAR and PRISM programs.

Baram Sosis and Artem Petrov are researchers at Principled Agents. Baram has a PhD in Mathematics from the University of Pittsburgh, and previously participated in PIBBSS and the Anthropic Fellows program, where he investigated the effects of helpful-only training on emergent misalignment. Artem has three years of experience as a software engineer, including two years at Palisade Research where he led a team evaluating AI cyber capabilities.

What are the most likely causes and outcomes if this project fails?

Principled Agents has a number of projects to pursue that we believe could lead to a positive impact on making AI safer. It is likely that at least some individual projects fail due to misguided approaches or because the underlying problem is particularly intractable. We aim to determine this early on, by focusing on building formal models to capture the important intuitions or reveal inconsistencies.

The broader project can fail overall if there exist crucial safety properties that are very resistant to formalization. If neither we, nor other groups pursuing similar research, can successfully characterize a target, then aligning superintelligence will rely on ad hoc measures. However, partial progress can still reduce the number of issues that need to be addressed without a principled solution.

The organization can fail if employees leave to pursue other opportunities, or if we run out of funding. The impact of this would depend on whether current employees are able to pursue similar research in their subsequent work.

How much money have you raised in the last 12 months, and from where?

Principled Agents started with a grant of £132,000 from UK AISI through The Alignment Project, and has since received an additional $20,000 from the Corrigibility Research Fund. Together, these grants cover our expenses until the end of November 2026.

Approved--this sounds like exciting and neglected work!

I currently work on economic theory and alignment at the UK AI Security Institute (AISI), where we funded Principled Agents through the Alignment Project. I'm writing here in a personal capacity.

The alignment problem needs formal work (linked with empirical evidence) to progress. Economic theory is currently an under-researched avenue for understanding incentives and model behaviour, particularly in a way that will stay relevant for future, more capable models.

Rubi is one of a small number of people making progress in this area. He has been working seriously on alignment for a long time, and combines rigorous conceptual thinking in economics with a strong understanding of and engagement with the AI safety community. High-context work at this intersection is rare. I know Rubi well and rate his research and his agenda very highly. I would like to see this work fully funded, and a lot more investment in this kind of research.

I'm a former coauthor of Rubi's and I overall recommend donating to his org. I also plan to make a donation to Rubi myself in 2027.

I think Rubi does a great job autonomously pushing forward his research agenda and he keeps coming up with interesting results related to incentives and oversight.

I think the goals of his org, such as fully informed human approval, are important. E.g., it would be great to be able to train a coding model to be good at coding but also provide honest explanations about the effects of a proposed Pull Request.

One recommendation I have for Rubi is to try out algorithmic ideas early in practice, using frontier models and realistic tasks. Rubi has expressed to me that he is interested in doing so. (It might be nice to be able to increase the compute budget to this end.)

Overall, I think this work is very neglected, and there is a small chance that it could turn out to be very impactful.

Rubi was our mentor during the PRISM fellowship program when we were investigating AI preference drift during training. He gave good direction without micromanaging, which meant we could move fast and weren't dependent on him to make progress. He was also great at guiding us where to focus when needed. I have no doubt he would be great at managing a team at Principled Agents and hope to work with him again in the future.

I like Rubi's work, I think he made non-zero progress on corrigibility. It's plausible he and his team will have more good ideas. I ultimately don't think the approach of directly applying it to RL experiments makes much sense, but maybe it'll spark more ideas.

I am excited about Rubi's work, and Principled Agents more broadly. They're doing real alignment work, which we desperately need more of. As manager of the Corrigibility Research Fund I awarded Rubi's team a $20k grant as a vote-of-confidence, partly to give them a little more runway, but mostly to nudge other, bigger funders to send more substantial support to their newborn research org.