You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
I'm running a small independent project to answer one question: are the AI evaluations people lean on actually as stable as their headline scores suggest? I don't assume the answer. I measure it.
I've already done a version of this once and published it — an MMLU robustness audit. It re-runs the benchmark while cyclically reordering the answer choices under fixed seeds, freezes the results, and reports how far the score drifts when nothing about the underlying question has changed. Alongside it I've built evaluation items that are graded by deterministic code rather than an LLM judge, and a reproducibility-first workflow that keeps every run auditable.
This 90-day pilot takes that one-off audit and makes it reusable, runs it against two more public evaluations, and publishes the whole trail: the code, the frozen artifacts, the cases that broke, and a short technical report.
The point is narrow and, I think, important. If a score shifts when you change things that shouldn't matter — option order, formatting, the random seed, evaluator settings — how much weight should the field really put on that number?
Existing work:
https://github.com/GrobeStreet/mmlu-robustness-audit
https://github.com/GrobeStreet/ai-eval-work-sample
https://github.com/GrobeStreet/bobby-research-os
Four concrete goals for the 90 days:
Turn my MMLU robustness audit into a reusable framework for perturbation and reproducibility testing — not a one-off script.
Run full robustness audits on two more public AI evaluations.
Build at least one audit so it plugs into Inspect AI (or another framework people actually use).
Publish an Evaluation Reliability Report: code, frozen results, the null findings, the limitations, and step-by-step reproduction instructions.
Every audit starts the same way — I write down the research question and the perturbation plan and freeze them before I look at any results. Then I reproduce the reference evaluation to confirm I match it, and only then start changing one thing at a time: option order, formatting, seeds, scoring configuration, calibration, and the other implementation choices that aren't supposed to affect the answer.
I'm not trying to publish another leaderboard. I'm testing whether the measurement itself holds still.
Most of it buys protected time, so I can work on this full-time for the three months instead of around the edges.
Rough split at full funding:
$24,000 — three months of research and engineering time
$7,000 — model API usage and compute
$4,000 — independent replication and statistical/technical review
$3,000 — research workstation / local compute
$2,000 — storage, software, hosting, and reproducibility infrastructure
Where I can use free credits, open-source tools, or my own local compute instead of spending grant money, I will. The project still works at lower funding — with a partial grant I'd audit fewer evaluations and concentrate on shipping the framework plus one solid audit rather than three.
It's me — Robert "Bobby" Morong, an independent research engineer focused on AI evaluation, statistical stress-testing, and reproducibility. I work solo right now.
What's already public:
MMLU robustness audit: https://github.com/GrobeStreet/mmlu-robustness-audit
Machine-verifiable AI evaluation work sample: https://github.com/GrobeStreet/ai-eval-work-sample
Research/reproducibility operating system: https://github.com/GrobeStreet/bobby-research-os
GitHub: https://github.com/GrobeStreet
I'll say this plainly, because it matters for how you read the rest: I use AI heavily — for orchestrating the research, generating hypotheses, reviewing literature, writing code, and arguing against my own conclusions. What keeps that honest is that the empirical claims bottom out in deterministic code, tests, frozen artifacts, and experiments anyone can rerun. The AI helps me move fast; the checkers decide what's true.
A few ways it could go sideways.
The evaluations I test might turn out to be sturdier than I expect. The failure modes I find might not generalize past one benchmark. Or I might build something technically clean that other evaluators find too fussy to adopt.
None of those are wasted. I plan to publish the null results and the failed experiments the same way I'd publish a positive finding — the whole point of the project is honest measurement, and "this eval was actually robust" is a real result.
The outcome I'd count as real failure is a complicated framework nobody uses. To keep that from happening I'm keeping the tooling small, tying it to real evaluations from day one, and treating adoption — not feature count — as the bar.
None dedicated to this project in the last 12 months. I've submitted a separate application to Lightcone Commons for a larger, nine-month version of the Open Evaluation Robustness Lab, which is currently pending. If there's any overlap I'll disclose it and adjust the budget and milestones so nothing gets double-funded.
There are no bids on this project.