Energy-Efficient LLM Fine-Tuning via Rank-Adaptive Attention Dynamics

Project summary

Transformer models consume far too much GPU energy. My goal in this 1-year project is to fix that. Based on my recent IEEE TIT paper ("Rank-Adaptive Local Empirical Processes in Low-Rank Attention", (under review)DOI: 10.5281/zenodo.23147548), (The Goal)I build lightweight training algorithms that cut GPU power and FLOPs by up to 90% without sacrificing accuracy.

During training, Self-Attention weights naturally collapse into low-rank subvarieties M_{≦s}. I exploit this geometry to dynamically allocate rank and prune weights on the fly. The math is already done in my paper. Now I am turning it into working PyTorch code for LLaMA and Mistral.

10/15

I have submitted a new paper, which serves as the foundation for my applied research, to JMLR.

“Metric Entropy and Generalization Upper Bounds for

Low-Rank Symmetric-Score Polynomial Self-Attention”(under review) DOI:https://doi.org/10.5281/zenodo.23387738

10/19

Clarifying My Numbers for Investors: How the "90%–99% Theoretical Potential" Connects to Our GPT-2 Benchmark

In my papers and video overview, I reference up to a 90%–99.22% reduction in parameter degrees of freedom alongside my new OpenAI GPT-2 empirical benchmarks (-15.83% MSE distortion). Here is how to read these numbers accurately without hype:

1. The Theoretical Ceiling ("The Destination" — Up to 99.22% DoF Cut at Zero Distortion):

My IEEE TIT study proves that if and when an attention projection layer undergoes complete rank collapse onto a rank-s manifold (e.g., s=16 in a 4096 X 4096 layer), effective degrees of freedom drop by 99.22% with 0% expected output distortion (verified down to the 10^{-7}FP32 precision floor in Cells 1–2).

2. The Real-World Bottleneck ("The Terrain" — Partial Decay in Pre-Trained LLMs):

In pre-trained off-the-shelf LLMs like GPT-2, attention heads do not collapse uniformly on their own. Some heads condense rapidly Head #6), while others retain heavy high-rank tails Head #9). Applying naive uniform compression across all heads breaks the model.

3. What Our New GPT-2 Benchmark Proves ("The Navigation System" — 96.8% Prediction & -15.83% Error):

To safely push real LLMs toward massive energy savings, you first need an accurate sensor to tell you which circuits can be pruned and which must be protected. Cells 3 prove on real GPT-2 weights that my coupled math (JMLR Lemma 3.1 + IEEE TIT Theorem 5.1) predicts actual post-softmax distortion with 96.8% correlation (r = +0.9680) and cuts full-layer MSE distortion by 15.83% under a 73.5% parameter cut—at zero extra budget.

Why I Am Raising Funds: I am not claiming a magic 90% cut out-of-the-box without conditions. I have built and verified the mathematical navigation system AGEPAttentionHookV2). Manifund funding allows me to combine this allocator with rank-collapsing regularization during live LLM fine-tuning—systematically driving real models toward those theoretical efficiency limits.

10/20

Addendum: Native HuggingFace GPT-2 End-to-End Verification (-21.65% / -20.18% MSE Reduction)

Interactive Google Colab Notebook (Reproducible Demo): https://colab.research.google.com/drive/1ueLY8TygEimHR1ulUx23t5pyUhiS_V_x?usp=sharing

What are this project's goals? How will you achieve them?

I have three concrete technical deliverables for this 1-year project:

1. Sub-Root Adaptive LoRA

Dynamic rank allocation during fine-tuning based on my sub-root fast rate bound

r^* = O(s(d_out + d_in) log s)/(λ_min n).

Cuts GPU power during fine-tuning.

2. Provable Zero-Shot Pruning

One-shot model compression using my SVD prediction-distortion bound

E_P [1/2 ||WX - ∏_s(W)X||_2^2] ≤ (λ_max(Σ_X)/2) (Σ_{j=s+1} σ_{j}(W)^2).

Guaranteed zero distortion under exact rank collapse.

3. Rank-Collapsing Regularization

A new objective function using nuclear-norm penalties to force early rank collapse during training, lowering GPU load as training proceeds.

Months 1–4: Extend my TIT theorems to multi-head PyTorch attention layers and design the adaptive rank selection algorithm.

Months 5–8: Implement in PyTorch. Measure actual GPU power draw (Joules/token, W/epoch) on LLaMA/Mistral models using cloud GPUs.

Months 9–12: Release the open-source PyTorch library on GitHub and publish the findings.

How will this funding be used?

PI Stipend ($45,000): Covers my living expenses for 1 year so I can work 100% full-time on this research without taking outside contracting work.

GPU Cloud Compute ($8,000): RunPod / Lambda Labs H100/A100 instances for GPU power and rank dynamics benchmarks.

Workstation & Tooling ($3,000): Development environment, API costs, and verification tooling.

Publication & Outreach ($4,000): Open-access publication fees and documentation for the GitHub repository.

Total of $60,000 per year.

Deliverables:

◾️$20,000: Focused on Deliverable 1 ("Sub-Root Adaptive LoRA" for small-scale models/single attention layers).

(1) Technical documentation outlining implementation prerequisites.

(2) Proof-of-Concept (PoC) code in PyTorch.

(3) Benchmark comparisons against standard LoRA (evaluating task accuracy, FLOPs, and peak memory usage).

(4) GitHub repository and public report.

◾️$60,000: Full scope of the project for the year.

(Includes Deliverable 2, "Provable Zero-Shot Pruning," and Deliverable 3, "Rank-Collapsing Regularization.")

(1) Comprehensive benchmarking on LLaMA/Mistral models using cloud-based GPUs (measuring actual energy consumption per token in Joules).

(2) Release of a PyTorch library and publication of research findings via open access.

What's your track record on similar projects?

Im Hideki Ishiyama, an independent researcher specializing in mathematical learning theory and algebraic geometry. I design the core theory, math, and algorithms myself. I use AI tools (like Gemini Notebook) as interactive assistants for coding and verification.

In my paper for IEEE TIT and the corresponding preprint on Zenodo, I have mathematically demonstrated that by constraining the rank to s=16 for a 4096 X 4096 attention projection layer, the degrees of freedom of the underlying manifold are reduced by 99.22% (decreasing the number of parameters from 16.77 million to an effective dimension of 130.8K). This, in turn, guarantees a reduction in local Rademacher complexity.

What are the most likely causes and outcomes if this project fails?

<Risk>

In extremely deep models or long-context inputs, the minimum eigenvalue of the input covariance might degenerate (λ_{min}->0). This expands the local radius ρ_r = \sqrt{2r / λ_min } and could slightly degrade zero-shot pruning accuracy.

My rank-collapsing regularization explicitly prevents spectral degeneration. Even if extreme compression fails on certain huge models, I will release the open-source PyTorch profiling tools and attention spectrum datasets. This provides value to the AI Safety and DevInterp research communities regardless.DevInterp, and Green AI (energy-efficient AI).

How much money have you raised in the last 12 months, and from where?

$0 USD. I self-funded 100% of my research so far. This grant is my first external funding application.