Projects

I presented empirical benchmark results using OpenAI’s GPT-2 (MSE distortion: -15.83%).

https://colab.research.google.com/drive/1ueLY8TygEimHR1ulUx23t5pyUhiS_V_x?usp=sharing

https://youtu.be/FL7O-Kw4oY4?si=E8COvH78wsQ6ZWQp

Technical Progress Update: AGEP PyTorch Pilot for Multi-Head Attention Diagnostics

Status: Initial implementation and synthetic-case diagnostics completed; end-to-end LLM fine-tuning speedups and energy-efficiency claims remain to be tested.

I have implemented an audit-ready pilot PyTorch hook API, ⁠AGEPAttentionHookV2⁠, that attaches to standard ⁠nn.MultiheadAttention⁠ layers and computes algebraic-geometric and empirical-process diagnostics for rank-adaptive attention.

The interactive Google Colab notebook provides a reproducible starting point for examining per-head rank structure, local Schur complements, and truncation distortion bounds in a controlled multi-head attention experiment.

Reproducible pilot (Google Colab):

https://colab.research.google.com/drive/1ueLY8TygEimHR1ulUx23t5pyUhiS_V_x?usp=sharing

What the pilot currently demonstrates

Direct Parameter & Covariance Inspection: A working prototype hook that inspects internal per-head projection blocks (\bm{W_Q^{(h)}, W_K^{(h)}, W_V^{(h)} \in \mathbb{R}^{d_k \times E}}) from ⁠module.in_proj_weight⁠ alongside empirical token covariance spectra (\bm{\hat{\Sigma}_X}) from forward activations.

Explicit Separation of Capacity Baselines:

1. JMLR Polynomial Score Surrogate (\bm{d_k \times d_k} subspace, \bm{d_k=4, r=1}): Confirms machine-precision vanishing of the skew-symmetric quadratic form (\bm{\max \vert{}x^\top \mathrm{skew}(A) x\vert{} \sim 10^{-8}}), tightening the low-rank score+value capacity exponent from \bm{D_r^{\mathrm{gen}} = r(2d_k-r)+d_k^2 = 23} to \bm{D_r^{\mathrm{sym}} = \frac{r(2d_k-r+1)}{2}+d_k^2 = 20} (13.04% cut vs. generic low-rank; 37.50% cut vs. unconstrained \bm{2d_k^2 = 32}). 

2. IEEE TIT Linear Projection Manifold (\bm{d_{\mathrm{out}} \times d_{\mathrm{in}}} projection, \bm{4 \times 16, s=1}): Quantifies the manifold effective dimension \bm{p_{\mathrm{eff}}(s) = s(d_{\mathrm{out}}+d_{\mathrm{in}}-s) = 19} against the ambient projection dimension \bm{d_{\mathrm{out}} d_{\mathrm{in}} = 64} (70.3125% reduction in ambient degrees of freedom). 

Controlled Pruning Verification (Collapsed vs. Full-Rank Control):

When ⁠Head #0⁠ is synthetically restricted to rank 1 (\bm{\sum_{j>1}\sigma_j(W)^2 \sim 10^{-16}}, \bm{K_{\mathrm{NF}} \sim 10^{-14}}), rank-1 SVD truncation yields a floating-point output deviation at machine precision (\bm{\Vert{}Y_{\mathrm{before}} - Y_{\mathrm{after}}\Vert{}_F / \Vert{}Y_{\mathrm{before}}\Vert{}_F \sim 10^{-7}} in FP32).

Conversely, applying the same rank-1 truncation to full-rank ⁠Head #1⁠ (\bm{K_{\mathrm{NF}} \approx 0.14}) produces significant output distortion (relative error \bm{> 10^{-1}}), confirming that the diagnostic correctly distinguishes prunable heads from active full-rank heads.

Delineation of Conditional Theoretical RLCT: Clarifies that the Real Log Canonical Threshold \bm{\lambda_{\mathrm{NF}} = \frac{1+d_k^2}{2} = 8.50} (\bm{d_k=4}) is an analytical closed-form value derived under monoidal desingularization of the determinantal normal form \bm{K_{\mathrm{NF}}(x,y,z,w,v) = (xy-zw)^2 + \Vert{}v\Vert{}_2^2}, which governs local asymptotics conditionally when a head approaches the singular locus (\bm{K_{\mathrm{NF}} \to 0}). 

What remains to be established

The current pilot is a synthetic-case verification of the mathematical diagnostics; it is not yet evidence of general lossless pruning in pre-trained LLMs, end-to-end training speedups, or measured GPU energy savings. The behavior of approximate rank decay (\bm{\sigma_{j>s} > 0}) during real gradient-based fine-tuning requires empirical validation on standard benchmarks.

Next milestone (Months 1–4 Target with Initial Funding)

With initial funding, I will:

1. Validate the hook diagnostics against independent numerical reference checks across FP32/FP64 precision and varying condition numbers.

2. Quantify the accuracy–distortion trade-off on pre-trained small-scale Transformer layers where singular values decay gradually rather than vanishing exactly.

3. Implement a controlled benchmark comparing rank-adaptive updates against standard fixed-rank LoRA, tracking task performance, FLOPs, wall-clock time, peak VRAM, and hardware power draw (⁠nvidia-smi⁠ / pyNVML telemetry).

The goal of this project is to build a bridge between algebraic-geometric theory and practical methods for energy-efficient LLM fine-tuning.

Technical Progress Update: Months 1–4 Pilot Hook API & Lossless Pruning Verified in PyTorch

To demonstrate the immediate feasibility of our Months 1–4 milestone (*extending the single-head theory to multi-head PyTorch attention layers and designing adaptive rank-selection algorithms*), we have completed and verified a working pilot of the AGEP (Algebraic-Geometric Empirical Process) PyTorch Hook API.

Interactive Google Colab Notebook (Reproducible Demo): [https://colab.research.google.com/drive/1ueLY8TygEimHR1ulUx23t5pyUhiS_V_x?usp=sharing]

1. What I Have Built & Verified

I implemented a non-invasive register_forward_hook module AGEPAttentionHook) that attaches to standard PyTorch nn.MultiheadAttention layers in a single line of code. It bridges our IEEE TIT framework (local empirical processes, effective manifold dimension $p_{\text{eff}}(s)$, and truncated SVD distortion bounds) with our JMLR companion framework ($S$-Reduction, local Schur complements $H(S)$, determinantal normal form $K_{\text{NF}}$, and exact local learning coefficients $\lambda_{\text{NF}}$ for DevInterp).

In our multi-head verification benchmark ($E=16, H=4, d_k=4$, target rank $r=1$, anchor rank $r_*=1$, where Head #0 undergoes rank-1 collapse while Heads #1–#3 remain full-rank), the live PyTorch telemetry produces the following exact diagnostics:

=================================================================================

[AGEP Hook API] Multi-Head Attention Algebraic Geometry & Capacity Diagnostic

=================================================================================

■ Head #0 (Head Dim d=4, Target Rank r=1, Anchor r*=1) <-- [Rank-Collapsed Head]

├─ [S-Reduction] Skew-Symmetric Error |x^T skew(A)x| : 2.14e-08 (Zero Functional Weight)

├─ [Capacity Dr] S-Reduced Dr = 20 vs Generic Matrix = 23 (Eliminated 3 redundant DoF)

├─ [Schur Chart] ||H(S)||_F = 1.940950 | Effective Residual Rank = 1

├─ [Normal Form] Generator g(x,y,z,w) = 0.000001 | Local K_NF = 0.000000

├─ [Exact RLCT ] Local Learning Coefficient lambda_NF = 8.50 (Score Pole = 0.5)

└─ [Pruning ROI] Manifold Dim p_eff = 19 (70.31% Cut) | SVD Distortion Bound <= 0.000000

----------------------------------------------------------------------------------

■ Head #1 (Head Dim d=4, Target Rank r=1, Anchor r*=1) <-- [Full-Rank Head]

├─ [S-Reduction] Skew-Symmetric Error |x^T skew(A)x| : 1.49e-08 (Zero Functional Weight)

├─ [Capacity Dr] S-Reduced Dr = 20 vs Generic Matrix = 23 (Eliminated 3 redundant DoF)

├─ [Schur Chart] ||H(S)||_F = 0.188800 | Effective Residual Rank = 3

├─ [Normal Form] Generator g(x,y,z,w) = -0.000971 | Local K_NF = 0.140849

├─ [Exact RLCT ] Local Learning Coefficient lambda_NF = 8.50 (Score Pole = 0.5)

└─ [Pruning ROI] Manifold Dim p_eff = 19 (70.31% Cut) | SVD Distortion Bound <= 0.050772

=================================================================================

I have submitted a new paper, which serves as the foundation for my applied research, to JMLR.

“Metric Entropy and Generalization Upper Bounds for

Low-Rank Symmetric-Score Polynomial Self-Attention”(under review) DOI:https://doi.org/10.5281/zenodo.23387738

For your reference, we have uploaded AGEP v3.14—which is scheduled for peer-review submission to JMLR next month—to Zenodo.

https://doi.org/10.5281/zenodo.22716189

Rapid Rigor Upgrade: Preprint v1.1 is Out! (Complete Proof of Theorem 4.1 & Scheme Correction)

Hi everyone,

In the spirit of rigorous, open-source AI safety research, I want to share a rapid and exciting upgrade to our AGEP preprint on Zenodo, moving from **v1 to v1.1**.

A sharp commenter recently pointed out that in our $2 \times 2$ toy model, the determinantal ideal $I = \langle xy - zw \rangle$ is prime (since $xy - zw$ is irreducible in a UFD), which mathematically makes the coordinate ring an integral domain. Thus, the rank-1 collapse subscheme $S_{\text{collapse}} = \text{Spec}(R/I)$ is technically a **reduced scheme** containing no non-zero nilpotents.

We have addressed this feedback immediately! Describing the PoC model as containing nilpotents in v1 was indeed a technical mischaracterization, and we have fully corrected the wording in **v1.1**.

However, this feedback has actually provided a beautiful mathematical bridge to our upcoming **Work Package 1 (WP1)**:
- In real-world deep learning loss landscapes, we *do* encounter true non-reduced structures (learning plateaus with severe flatness).
- To model these plateaus, we must extend our determinantal ideals to non-reduced forms, such as $I_{\text{non-reduced}} = \langle (xy - zw)^2 \rangle$ (representing quadratic flatness near the singularity) or $I_{\text{non-reduced}} = \langle xy - zw, x^2 \rangle$ (representing isolated nilpotent "spikes" when query weights completely die). This non-reduced framework will be the core of our v2.

#### 🎓 Presenting the Complete Proof of Theorem 4.1 (Appendix)
To satisfy the rigorous standards of both algebraic geometers and mathematical statisticians, we have added a comprehensive, **step-by-step mathematical proof of Theorem 4.1** in the Appendix of v1.1.

This 4-page mathematical appendice showcases the precise calculus behind how:
1. The monoidal blow-up at the origin of $\mathbb{A}^4$ pulls back and factors the singular cone into a smooth exceptional divisor $E$ and a resolved hypersurface.
2. The local Kullback-Leibler (KL) divergence and volume Jacobian are simultaneously monomialized into $K(u) = u_1^4 (u_2')^2$ and $|g'(u)| = u_1^3$.
3. The Real Log Canonical Threshold (RLCT) is algebraically derived as $\lambda = 0.5$, bounding the local bracketing entropy to $\log N_{[]} \le C \log(1/\varepsilon)$.
4. Dudley's bracketing entropy integral converges to a finite value as the scale parameter $\delta \to 0$ via a rigorous change of variables ($t = \sqrt{\log(1/\varepsilon)}$) and integration by parts (utilizing Mill's ratio bound for the Gaussian tail).

#### 🐎 Speed & Capital Efficiency
The transition from receiving technical feedback to compiling a revised 17-page mathematical draft with a complete proof took us **less than 24 hours**.

This is the power of the **AI-Co-PI paradigm**. By leveraging highly-synchronized interactive AI pipelines, we can bypass the slow administrative overhead of traditional academic research and deliver world-class mathematical safety foundations at lightning speed.

The revised preprint v1.1 is now available for download on Zenodo. We hope you enjoy reading the beauty of resolved singularities and uniform weak convergence.

As always, we are eager to hear your thoughts, comments, and questions. Let's keep the momentum going!

Best regards,
Hideki Ishiyama
PI, AGEP Lab

Quick Update: Correcting a Mathematical Term in v1 (Reduced vs. Non-reduced Schemes) & The Road to v2

I want to address a very sharp and helpful feedback I received regarding the algebraic properties of our $2 \times 2$ linear attention singularity model in the pre-print.

In the draft, I described the rank-1 collapse subscheme $S_{\text{collapse}} = \text{Spec}(\mathbb{R}[x, y, z, w] / \langle xy - z w \rangle)$ as a "non-reduced" scheme containing nilpotent elements.

As a math-rigor check, the commenter is 100% correct: since $xy - zw$ is an irreducible polynomial in a UFD, the generated ideal $I = \langle xy - zw \rangle$ is prime. Therefore, the coordinate ring is an integral domain, making the scheme reduced (meaning it contains no nilpotents). Describing the $2 \times 2$ toy model as non-reduced was a technical misnomer on my part, and I will correct this wording in the next revision (v2) on Zenodo.

Does this affect the core claims of the AGEP Theorem? Absolutely not. The core mathematical proof of Theorem 4.1 (the algebraic restoration of the Donsker property) and our PyTorch simulation results remain completely untouched and robust. The monoidal blow-up still successfully regularizes the KL divergence and the Jacobian into normal crossing monomials, and the Dudley entropy integral still beautifully converges on the exceptional divisor.

Why this feedback actually accelerates our roadmap: This correction does not weaken our theory; rather, it provides the perfect mathematical bridge to our next generalized phase.

In real-world deep learning, we do encounter true non-reduced structures. The severe "learning plateaus" (flat regions where gradients collapse) are best modeled as nilpotents. As suggested, by extending our ideal to non-reduced forms such as $I = \langle (xy - zw)^2 \rangle$ (representing quadratic flatness near the singularity) or $I = \langle xy - zw, x^2 \rangle$ (representing isolated nilpotent "spikes" when certain weights die), we can mathematically describe the dynamic of SGD getting stuck in plateaus.

I am extremely grateful for this high-level feedback. It has already sharpened our focus for Work Package 1 (WP1), and we are now formulating the "Non-Reduced AGEP Theorem" for v2 to directly capture these flat plateaus.

Thank you for watching and supporting our journey! I will keep you posted as we refine the draft.

Thank you for the flag! Yes, Pangram is 100% correct in detecting AI assistance in the prose, and this is actually by design. 

As an independent researcher without the backing of a major US academic institution or a native English-speaking PR team, I utilized my customized LLM pipeline (Gemini Notebook) as an interactive "Co-PI" to draft and polish the English prose of this proposal to meet international standards. This is explicitly disclosed and detailed in "Part 5: The 'AI-Co-PI' Research Paradigm" of this proposal.

I drive 100% of the mathematical vision, topological intuition, and hands-on PyTorch coding. However, delegating the heavy lifting of English translation and formatting to AI allowed me to produce a citable 14-page LaTeX pre-print and submit this grant in less than a week. 

This flag actually proves the exact efficiency of the human-AI co-working model that I am proposing to fund. I invite reviewers to look past the "perfect" AI-assisted prose and evaluate the core mathematical novelty of the AGEP Theorem and the verified PyTorch simulation data hosted on my Zenodo DOI.