Project summary

Hiring a 3rd researcher for the Computational Rational Agents Laboratory (CORAL). The latter is a group working on creating a mathematical theory of agents, via the learning-theoretic agenda (LTA). The theory we are building will be ultimately applicable to rigorous analysis of threat models related to unaligned AI agents, and to formally proving the efficacy of AI alignment protocols under clear, well-defined assumptions. One candidate approach for such a protocol is Physicalist Superimitation. Within the LTA, our current main focus areas are:

• Compositional learning theory. This theory would explain how learning algorithms can exploit compositionality in the regularities of the real-world to exponentially improve sample complexity and computational complexity. It may explain the generalization properties and inductive biases of deep learning, or lead to alternative competitive algorithms with strong theoretical guarantees.

• Metacognitive agents. This extension of reinforcement learning theory can explain important aspects of cognition, such as logical uncertainty, symbolic reasoning, metacognition and the navigation of traps (dangerous irreversible events). It would also lead to a formalization of metalearning and recursive self-improvement.

• Multi-agency. This includes multiple approaches that attempt to use infra-Bayesianism (robust learning) to avoid the "grain of truth problem" which prevents classical learning theory from analyzing multi-agent scenarios, while also improving the prescriptive theory of rationality for multiple agents using well-founded notions of counterfactuals.

What are this project's goals? How will you achieve them?

The projects goals are:

• Finding and hiring a suitable new researcher.

• Training the new researcher

• Working with them to produce new papers on our research focus areas.
  

How will this funding be used?

Paying salary, costs of employment, administrative overhead and travel expenses for conferences, for a period of about 1 year.

Who is on your team? What's your track record on similar projects?

Our team is Vanessa Kosoy and Alex "Diffractor" Appel. We produced much relevant research over the years, both ourselves and collaborating with others.

What are the most likely causes and outcomes if this project fails?

1. We fail to find a suitable researcher.

2. We hire a researcher, but they turn out to be insufficiently skilled.

3. The specific research direction chosen turns out to be largely unsuccessful.
   

How much money have you raised in the last 12 months, and from where?

In the past year we spent about 400,000 USD. The money came from the Advanced Research+Invention Agency (ARIA), a donor lottery managed by Effective Ventures, the Long-Term Future Fund (LTFF), the Machine Intelligence Research Institute (MIRI) and the Survival and Flourishing Fund (SFF). Out of that, the funding from MIRI and the LTFF was granted directly to Vanessa rather than to CORAL as a group.