Project summary

kernelbench.com evaluates frontier coding agents (Claude, GPT, Gemini, GLM, Kimi, DeepSeek, and others) on writing and optimizing real GPU kernels, scored against each GPU's hardware roofline rather than a soft baseline. Three tracks: Hard (six per-op CUDA/Triton problems across RTX PRO 6000, H100, B200), Mega (fully fused megakernels - Claude Fable 5 recently shipped the first genuine single-launch megakernel at 18.7x over reference, covered in Import AI 464), and Multi (forthcoming: NVLink multi-GPU kernel optimization on 8xH100 nodes). Every published run includes full agent transcripts on Hugging Face and a judge-assisted reward-hacking audit before it counts; several prior "wins" have been disqualified by the authenticity gate.

Kernel engineering is one of the concrete bottlenecks of AI research itself. This benchmark measures whether models can genuinely automate it - with roofline grounding, anti-gaming audits, and public artifacts - independent of any lab. It also tracks token spend per result, giving a capability-per-dollar frontier across models.

What are this project's goals? How will you achieve them?

Goals over the 6-month grant period:

1. Full benchmark sweeps within days of each major frontier model release, including API-priced models currently skipped on cost (e.g. GPT-5.5 Pro).

2. Launch the Multi track publicly: NVLink multi-GPU kernel optimization on rented 8xH100 nodes.

3. Continue publishing full agent transcripts and per-run reward-hacking audits for every result on the board.

The infrastructure already exists and runs (harness, roofline machinery, audit pipeline, transcript publication). To date it has run on coding-plan subscriptions plus my own hardware, which no longer scales: Fable-class models at high effort are a different cost tier, some models are API-priced only, pricing fluctuates release to release, and the Multi track requires multi-GPU nodes I don't own. Funding converts an existing, working benchmark from cost-limited to cost-covered.

How will this funding be used?

$15,000 over 6 months, indicative split:

- $6,000 - frontier model credits/API for full sweeps on each major release, including API-only models currently excluded

- $4,000 - multi-GPU cloud rentals for KernelBench-Multi (8xH100; a full sweep at the 3h/run cap across the model matrix is roughly 100-150 node-hours)

- $5,000 - maintainer time: running sweeps, per-run reward-hacking audits (where most hours go), harness and site maintenance

The split is indicative; model pricing is volatile enough release to release that allocation will shift with actual costs, and any substantial deviation gets flagged to the funder.

Who is on your team? What's your track record on similar projects?

Solo project: I built and run kernelbench.com end to end (harness, problem decks, roofline scoring, audit pipeline, site). Day job is inference/training optimization at an AI avatar company; kernelbench is maintained entirely outside work hours.

Track record: the benchmark was covered in Import AI 464; VentureBeat cited it as an independent authority when its run logs contradicted a lab's benchmark claims; the Mega track hosted the first verified single-launch megakernel result. Related public work: CUDA/Triton kernel engineering across NVIDIA, AMD MI300X, and TPU, RL-trained kernel optimization research, and AI education content with 2M+ learners (FreeCodeCamp LLM and CUDA courses). GitHub: github.com/Infatoshi.

What are the most likely causes and outcomes if this project fails?

Most likely failure mode is cost overrun, not technical failure: frontier model pricing spikes beyond the credits budget, or 8xH100 availability/pricing makes Multi sweeps more expensive than estimated. The outcome would be reduced sweep cadence (fewer models or fewer releases covered) or a delayed Multi launch, not a dead project - the site and existing boards stay up and maintained regardless. Secondary risk is my time, since this runs outside a full-time job; the time line item exists to keep it sustainable.

How much money have you raised in the last 12 months, and from where?

$2,000 in hardware credits from Nvidia. Everything else self-funded (coding-plan subscriptions and my own hardware).