Update.

Thanks to the funding, we spent three periods in London working with Tom Everitt, Jonathan Richens and Victoria Krakovna at Google DeepMind. We also joined the larger Causal Incentives Working Group and interacted with the AI safety community in London. The project continued to gain traction through the extention of MATS and the involvement of interested people. We also visited each other twice to work together.

As anticipated, we conducted a literature review on agency and empowerment, interacting with academics both in information theory and AI and in social sciences such as community psychology. We proposed and analysed different formalisations, relating the concept to safety-critical concepts such as harm, benefit, goal-directedness and control. We are currently working on a submission to a high-profile ML conference.