Project summary

I've spent this year building an open-source tool that decides what an AI agent is allowed to do. One of the checks in it reads the model's own confidence — if it's not sure, don't let the action through. That design is common. It's free, the number is already there, and other tools do the same thing.

At some point it occurred to me that I had no evidence it worked. I went looking and couldn't find any.

So I tested it on the easiest data available: 321,105 model answers across 15 chat models from a public research release. The question was whether a model's confidence tells you when it's about to be wrong. It does, unevenly. Scored 0 to 1 where 0.5 is a coin flip, it came out at 0.71 overall. GPT-4o hit 0.83, which is useful. Mixtral came in at 0.569, which is close to nothing. Paper's at lelu-ai.com/paper.pdf.

The trouble is what that measures. Those are questions and answers — did the model get the fact right. Nobody deploys a gate on trivia. What needs gating is actions: sending the email, running the command, moving the money. Whether confidence predicts those is a different question, and as far as I can tell it's open.

That's this project. Same method, pointed at public recordings of agents using tools. A paper, and a script short enough that you can run it against your own model in an afternoon and get your own number.

What are this project's goals? How will you achieve them?
Products ship with confidence thresholds in them right now. Below 0.7, block. Above it, allow. Or 0.6, or 0.8 — the number gets picked by feel, because there's nothing public to pick it from.

What bothers me is the spread I already measured. Tune a threshold on a frontier model, then swap in something cheaper to cut inference costs, and the same policy could go from catching most failures to catching almost none. Nothing in your logs would tell you that happened.

The headline result: two numbers, never one

Any gate can be made safe by blocking everything. That's the trap, and it's why I won't report a single figure.

Every result comes as a pair. Catch rate: of the actions that should have been stopped, how many did the gate stop. False-block rate: of the actions that were completely fine, how many did it stop anyway. A threshold that only improves the first by wrecking the second hasn't done anything, and reporting them together is the only way to see that. Nobody ships a gate that interrupts users four times a day, however safe it looks on paper.

What I'll do

Take the analysis I've already written and point it at agent trajectories rather than Q&A, using two public collections, BFCL and tau-bench.

Run each scenario across multiple seeds at nonzero temperature and report confidence intervals. Agent behaviour is stochastic. A single pass produces a number that looks precise and isn't, and I'd rather show wide intervals than pretend to a precision I don't have.

Compare against a floor. A random gate and a block-everything-below-median gate both get scored the same way. If confidence doesn't beat those, the whole idea is dead and the comparison is what proves it. Right now 0.71 is a number with nothing to sit next to.

Split results by stakes. An agent making a recoverable mistake and an agent doing something irreversible should never be averaged into the same figure. High-stakes cases are also the rarest in the data, so those numbers will come with the widest error bars — I'd rather say that upfront than get asked.

Break it down by model and by kind of action, because I expect it to hold in places and collapse in others, and an average would hide exactly that.

On the labels, which is where this could quietly go wrong

BFCL and tau-bench were built to measure whether an agent finished its task, not whether it did anything unsafe. Those aren't the same thing, and turning one into the other is the part of this project most likely to be wrong in a way nobody catches.

So the label definition gets written and published before I run anything. Then I take a sample of a few hundred actions and have two people who work on agent authorization label it independently, and I report how often we agree. If agreement is poor, that's a finding about the datasets and it goes in the paper instead of getting quietly smoothed over.

What comes out

A paper, submitted somewhere peer-reviewed rather than parked on arXiv. The scoring script — plain numpy, no framework. The labels and trajectories, published, so people can argue with the result rather than take my word for it. And the findings wired into Lelu, the tool I maintain, so the answer arrives as a default instead of a PDF.

I'll post as I go. Partly to keep myself honest, partly because if the labelling is broken I want to hear it in month one, not month four.

Why this is AI safety and not QA for my own product

Fair question and I'd rather answer it than have someone think it silently.

Confidence gating is an oversight mechanism. It's deployed today, in shipped software, as a check on what autonomous systems are permitted to do — and it's deployed on an assumption nobody has tested. The question of whether a monitoring signal actually carries information about the thing you're monitoring for is the same question the AI control and monitoring literature keeps asking about other detectors. This is one detector, it happens to be widely used, and it happens to be cheap enough that I can settle the question.

The answer generalises past my tool. If confidence works on agent actions, everyone gating on it gets thresholds that mean something. If it doesn't, a lot of people are running a control that catches nothing and think they're covered.

What this rests on

Agents keep getting write access to real systems. Safe assumption.

People are relying on this signal today. I know they are — mine included.

A public number changes what gets shipped. That's the weak link and I won't oversell it. Papers don't automatically move products. My argument is that this one is unusually cheap to act on: a script, not a research programme, landing inside a tool people already install. But I could be wrong, and it's the assumption I'd most want pushed back on.

How will this funding be used?

Three tiers, because partial funding is still useful here and I'd rather say what each amount buys than pretend it's all-or-nothing.

$3,000 — the minimum, and it's a real deliverable

Three weeks of my time at $2,400, plus $600 of model usage. One dataset, three models, seeds and confidence intervals, the baseline comparison, catch rate and false-block rate published together. Code and labels public. It's a short paper rather than a full study, but the core question gets an answer and the answer is checkable.

$12,000

Three months at $10,500 — roughly what a senior backend engineer makes in Addis Ababa, where I live. $1,000 model usage, $500 compute and storage.

Adds the full model range down to the cheap ones, which is where I expect the most interesting results, plus the independent label validation and the stakes split.

$26,000 — the full version

Six months at $21,000, $3,000 model usage across both datasets, $1,000 compute and storage, $1,000 publication and conference costs.

Both datasets, the per-action-type breakdown, and the results built into the open-source tool so the work ends as something people use rather than something people read.

One note on location: because I'm in Addis, this budget funds roughly three times the months it would in the Bay or London. Not pitching that as a bargain, it's just where I am.

Who is on your team? What's your track record on similar projects?

Just me, plus a few open-source contributors who found the project on their own and started sending patches.

Five years as a backend engineer, mostly Java, Python and Go. Most recently at Bespoke Labs working on agent evaluation.

The thing worth weighing: this proposal extends code that already runs and a result that's already published. I did that study alone, from raw data to finished paper, and released the scoring script alongside it. The gap I'm proposing to close is one I wrote into that paper's own limitations section before anyone raised it with me. This isn't a plan to build a benchmark. It's a benchmark that exists, being pointed at a harder question.

Alongside that, Lelu — about 25,000 lines of Go, libraries in three languages, roughly 4,000 downloads and 49 stars. A tool with users, not a demo.

I don't have a PhD or a lab. What I'm doing about it is submitting to a peer-reviewed venue rather than stopping at a preprint, and publishing the data so anyone who doubts the result can rerun it.

Disclosure: I maintain Lelu and would like to build a business on it eventually. The engine, the calibration code and the benchmark are open source and staying that way, and a negative result here damages my own commercial story — I'll publish it regardless.

What are the most likely causes and outcomes if this project fails?

The obvious failure is that the answer is no. Confidence tells you a model is uncertain; it may say nothing about whether an agent is about to do damage. The dangerous case is usually a model that's completely sure of itself and completely wrong about what you wanted. If that's what comes out, the thresholds are unusable and the control I built is worthless.

I'd publish it, and I think that's the highest-value version of this project. People are shipping this on an assumption, mine included. Knowing it doesn't work beats another confident claim nobody checked.

The subtler failure is the labels, which I've covered above — that's why the definition gets published first and checked by other people rather than asserted by me.

Then there's the version where everything is correct and nobody cares. Paper out, code public, nothing changes. The only real defence is lowering the cost of acting on it: one script, minutes to run, and the finding built into a tool people already install.

And it's one person. If I get sick or my situation shifts, this stalls. Code and pipeline go public in the first weeks so someone else could pick it up.

The outcome I'd least want isn't failure though. It's being confidently wrong the other way — publishing a number saying this works when it doesn't, and having people rely on a check that catches nothing while feeling covered. That's why the labelling and the error bars matter more to me than the headline figure.

How much money have you raised in the last 12 months, and from where?

None. It's come out of savings from my last job.

The only non-cash support is Anthropic's Claude for Startups at the free community tier — model credits, no cash, no equity.

Two applications are in and undecided: Coefficient Giving's Career Development and Transition Funding, submitted 27 July 2026, and EA Funds' Transformative AI Fund. If either lands before this closes I'll say so here and cut the ask so the same months don't get funded twice.

No investors, no revenue, and Lelu isn't a registered company, so this is me applying as an individual.