Gating Enables Curvature: A Geometric Expressivity Gap in Attention

View PDF HTML (experimental) Abstract:Multiplicative gating is widely used in neural architectures and has recently been applied to attention layers to improve performance and training stability in large language models. Despite the success of gated attention, the mathematical implications of gated attention mechanisms remain poorly understood. We study attention through the geometry of its representations by modeling outputs as mean parameters of Gaussian distributions and analyzing the induced Fisher--Rao geometry. We show that ungated attention operator is restricted to intrinsically flat statistical manifolds due to its affine structure, while multiplicative gating enables non-flat geometries, including positively curved manifolds that are unattainable in the ungated setting. These results establish a geometric expressivity gap between ungated and gated attention. Empirically, we show that gated models exhibit higher representation curvature and improved performance on tasks requiring nonlinear decision boundaries whereas they provide no consistent advantage on tasks with linear decision boundaries. Furthermore, we identify a structured regime in which curvature accumulates under composition, yielding a systematic depth amplification effect. Comments: 41 pages, 9 figures Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML) Cite as: arXiv:2604.14702 [cs.LG] (or arXiv:2604.14702v1 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2604.14702 arXiv-issued DOI via DataCite (pending registration) Submission history From: Satwik Bathula [view email] [v1] Thu, 16 Apr 2026 07:09:11 UTC (2,452 KB)