arxivJul 3bullish

A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

arXiv:2607.02303v1 Announce Type: new Abstract: Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall

HOTRGD3 models #language models #attention mechanisms

arxivJun 15bullish

Exact Linear Attention

arXiv:2605.18848v4 Announce Type: replace-cross Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation erro

TRYO2 models #machine learning #transformer #attention mechanisms Read on arxiv →

arxivMay 5bullish

FG$^2$-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control

arXiv:2604.19021v2 Announce Type: replace Abstract: Linear attention mechanisms have emerged as promising alternatives to softmax attention, offering linear-time complexity during inference. Recent advances such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) have demonstrated that the delta

GAKIFG4 models · +1 #machine learning #attention mechanisms #optimization Read on arxiv →

arxivApr 16bullish

FAST: A Synergistic Framework of Attention and State-space Models for Spatiotemporal Traffic Prediction

arXiv:2604.13453v1 Announce Type: new Abstract: Traffic forecasting requires modeling complex temporal dynamics and long-range spatial dependencies over large sensor networks. Existing methods typically face a trade-off between expressiveness and efficiency: Transformer-based models capture global d

TRFAMA4 models · +1 #traffic forecasting #spatiotemporal modeling #state-space models Read on arxiv →