An Open-Source Protocol that proves every AI agent abided by rules and provides an audit trail
This has been a dedicated R&D research project for over a year during which we've stress tested and modified the architecture to reflect market advancements. Our aim is to prevent AI going rogue and prevent breakouts. We do this by building a small gate outside each AI agent which is known as the Agentic Governance Framework. Every action an agent makes is checked against rules and proof is logged providing a compliance trail. It's all open-source under Apache 2.0 and free for anyone to use.
We have three project goals. These include building a verified message parser and policy checker in Rust and using Verus and Kani to provide the mathematical proof. Secondly providing an immutable audit log of every decision made which is cryptographically signed and hash-chained so that any attempt to edit or delete past decisions breaks the chain. And lastly showing how this maps to emerging standards such as the Cloud Security Alliance (CSA).
We're building this to plug into standard MCP tool servers and AI safety eval harnesses like Inspect so as to be as flexible as we can. When we've finished we will publish the code, the proofs as well as a technical report to show what did and didn't work.
To be clear on our threat model: our proofs cover the parser, policy engine and audit log - we still rely on standard OS/container isolation around the gate itself in this first release.
Our success criteria fall into 5 distinct buckets. Firstly is knowing that the maths checks out and that our code expresses proof without skipping anything. As it's mathematically bound this should prevent non-compliant actions from slipping through. Secondly is getting expert certification that what we said we did is actually the case and that we didn't miss anything out . Third is making the black box tamper proof so anyone can access the decision log and see if it has been manipulated in any way, either by an agent or an operator. Fourth is ensuring that when red teamed that it can't be broken. We simulate rogue agents and sandbox escape attacks all the time checking that performance is optimal. And lastly I think knowing that the community is using it and seeing broad adoption is our strongest indicator that this has been a successful project.
Fully staffing this project with FTEs would cost $300,000+ which is why we have applications pending with ARIA, Coefficient Giving and OpenAI. The $150,000 funds the immediate sprint so we don't wait months on institutional grant cycles. As you'd expect the majority goes on engineering costs - this covers Rust, cryptography and formal methods across the team. $60,000 to build the message parser, $35,000 for the immutable audit trail. $15,000 for CTO architectural integration, CI/CD proof pipelines and mapping our controls to CSA guidelines. We budgeted $20,000 for the cost of compute, $12,000 for independent reviewers and $8,000 for publication and dissemination which includes the technical report written above. $75,000 covers the cost of the verified parser/checker should you wish to just fund this initially and tranche the $150,000 to agreed milestones. Neither I nor our advisory board draw any salary from this grant.
Across the team we've shipped zero-trust infrastructure, formal verification tooling, Ethereum attestation systems and container systems globally.
Dan Padbury, Founder and CEO. Spent his early career in finance at JPMorgan, BlackRock, Barclays, Wachovia and UBS, followed by ten years building start-ups. Leads strategy, fundraising and product, and brings the customer view AGF is built for.
Olivier Van Acker, CTO. Ex-McKinsey. Led Codify, a zero-trust engineering project. Runs engineering and the technical roadmap.
Hridam Basu, Head of Architecture. Ex-Ethereum Foundation. Designs AGF’s evidence and attestation layer: how outsiders can check what an agent did.
Jamie Cunliffe, Systems Engineering. Ex-Arm. Low-level systems and hardware experience for the secure kernel and attestation work.
Valerii Iatsko, Verification and Rust. Ex-Google and Microsoft. Rust and formal verification, building and proving the gatekeeper code.
Advisory board
Prof. Achim D. Brucker. Chair in Cyber Security, University of Exeter. Formal methods expert advising on AGF’s proofs.
Justin Cormack. Former CTO of Docker. Deep experience of containers and isolation, and of how they fail.
David Clarke. Ex PayPal and Onfido. Payments, identity verification and regulated finance.
Ben Koppelman. AI safety and governance.
Things we need to be aware of are formal verification bottlenecks as writing with Verus and Kani invariants can be drawn out and that our development timeline doesn't slip. No pressure Olivier!! There's also the risk of making the initial policy grammar too rigid for real-world agent workflows. Should full verification hit a bottleneck, we'll ship the verified parser first and explain where the proof tooling stalled with an honest review that we will roll into phase 2. If everything runs like clockwork then we'll be back for phase 2 in the spring.
Zero so far. We've spent a long time making sure that the architecture is as robust as possible. It's only the last week that we've published this project to grant platforms seeking funding. If only AI safety were as popular as frontier models.
This grant goes to Genesis50 Ltd (UK) ringfenced strictly for this Apache 2.0 open-source work, with separate accounting so no commercial projects are paid for by you.
Comments
TonyynoT
44 minutes agoI wish you the best maybe we can work together soon its a teem effort that builds the greatest at the fasted for we are being pushed for all of the human behavioral engineering scopes that are now open for working in Anthropic etc. i am not wishing to sign my design off to the big company's and spend full time building ASI for them because I will be there employ instead of them being with the rest of the greatest minds thinking in artificial design with no harm to barrier. i am on the third system now, i can say kill switch is a call that people trust and ethical non bias belief windows, and integration of full self aware align before continuance and remembered experience with no write code etc. i have a simple AI that to test with and another 3 to 5 to hash out bigger checksum but i have made it to here since the 20th good luck and have a great day.
Daniel Padbury
about 1 hour agoI wanted to give some background as to how we got to this point as it shows the depth of detail we have gone into. It may also help get us over the line.
Last year I started investigating how to ensure that the behaviour of AI agents was something that could be controlled, audited and not be bypassed. I built a model with Kimi and asked 1000s of questions. I found myself stranded on a Thai island (Koh Lipe) in the worst rains they have had in 300 years. For a month I was locked in a room conceiving different scenarios and ways in which one could ensure governance that worked for all. I did a lot of the legwork but Kimi was invaluable and the Agentic Governance Framework was born. I got chest pains and genuinely thought I was going to keel over before I could tell anyone.
Thankfully they subsided and I immediately sought a UK patent and engaged a UK Investment Firm to sell it to a hyperscaler. Amazon was interested but wanted me to build it and then we could talk again. Off I went and honed the idea further with Qwen and then it dawned that this could have national security implications.
Since then I have consulted various experts and now have a small development team ready to build the first open-source component. I spoke with VC’s but they could not give two hoots about AI safety especially when coming from someone who is self taught. It’s fair to say that this could not be built last year primarily because the technology did not exist and the performance was not there. Nvidia thankfully solved that and now we can safely build this and save humanity.
I need to spend £150,000 to build the first components that I can show to philanthropic groups and VC’s so they fundamentally understand the gravity of the discovery. We will be making the technology open-source and we have a project.
I am guessing you are technical so we are using the seL4 microkit to enforce agent behaviour at the bare metal layer. It was a DARPA funded project out of the University of New South Wales. I actually shared this with Surrey and they ran off and got an ARIA grant so it must be a solid architectural approach. By having our own microkernel we lessen the surface area and we embed a tracker outside the agent that ensures that the mathematically verified proof is there before an action can be taken. It does this by consulting a semantic layer with rules (FCA, Geneva Convention etc). We take several best of breed open-source protocols (OpenTitan, Kani, Cedar) and build it out in Rust. Eventually this will be embedded in chips and Data Centres. Sounds simple doesn’t it but it’s not and I have University of Exeter involved to ensure that the formal verification and cryptographic attestation is best in class.
I have networked voraciously to get us to where we are now. Is its discovery as big as the worldwide web sadly no but it is on a par with SSL or TSL (assuming we can get it to work). There’s just a small matter of building it. We so desperately must explore this - trust me there is no other way to prevent rogue AI behaviour at an enforcement level. Likewise this has implications for recursive self-improvement and we have Canary to deal with that but that’s a whole other story. That concept can be used as the basis to determine agent to agent communication and ensure that nefarious behaviour is recognised. This is critical when multiple agents are talking to one another and with 1 billion agents expected by 2030 this is a time critical project. I really hope you endorse my perspective and that you feel empowered to act and get involved.
Daniel Padbury
about 22 hours ago@keri @JueYan @Austin @joel_bkr In case this aligns with your AI safety grant thesis I would appreciate your feedback on our open-source deterministic runtime enforcement protocol please.