ICLR paper highlights

A pair of widely shared ICLR 2026 paper picks pointed to the same theme: simple ideas are still beating expensive machinery in parts of machine learning. (openreview.net 1) (openreview.net 2) One paper, “FlashAttention on a Napkin,” was accepted by Transactions on Machine Learning Research on March 6, 2025. Vincent Abbott and Gioele Zardini say they can explain hardware-speed tricks for transformer attention with diagrams that derive streaming and tiling steps, instead of hand-tuning kernels by trial and error. (openreview.net 1) (openreview.net 2) Attention is the part of a transformer that compares each token with many others, and that comparison grows roughly with the square of sequence length. The ICLR 2026 FlashAttention overview notes that a 2,048-token attention matrix takes 16 megabytes, while 16,384 tokens push that to about 1 gigabyte per layer. (iclr-blogposts.github.io) FlashAttention’s core trick is to move less data in and out of slow memory, because modern graphics processors are often bottlenecked by memory traffic rather than arithmetic. Abbott and Zardini write that FlashAttention reached about a 6x speedup over native PyTorch by avoiding unnecessary transfers, and their paper tries to turn that intuition into a reusable method. (openreview.net 1) (openreview.net 2) The second paper came from time-series forecasting, where models try to predict the next values in sequences such as weather, heartbeats, or sensor readings. In “Context parroting,” Yuanzhao Zhang and William Gilpin report that a naive method that simply copies matching patterns from the prompt can outperform leading time-series foundation models on low-dimensional chaos, turbulence, coupled oscillators, and electrocardiograms. (openreview.net) That result lands in a field that has spent the last two years adapting language-model ideas to numerical sequences. Another ICLR 2026 paper, “A Closer Look at Transformers for Time Series Forecasting,” says simpler transformer designs often beat more elaborate ones on standard benchmarks, and that much of the predictive signal comes from within each variable over time rather than across variables. (openreview.net) Chaotic systems are a hard test because tiny changes in starting conditions can produce diverging futures, even when the underlying rules are fixed. Gilpin’s earlier benchmark paper assembled 131 chaotic dynamical systems with precomputed time series specifically to test forecasting models under those conditions. (openreview.net) Zhang and Gilpin argue that many foundation models succeed here by “parroting” context rather than learning deeper dynamics, and they tie forecast accuracy to the fractal dimension of the underlying attractor. Their paper says the cheap baseline wins at “a tiny fraction of the computational cost,” while the learned models often drift toward the mean when parroting fails. (openreview.net) Neither result says large models are finished. The same ICLR 2026 program includes papers that try to make transformers work better on chaotic systems, including architectures tuned for large-scale dynamics and pretrained models built specifically for chaotic forecasting. (openreview.net) (openreview.net) What these papers put on the table is narrower and more concrete: one asks whether speed tricks can be derived systematically, and the other asks whether a forecast is really learned or just copied. At ICLR 2026, those were enough to make two of the most circulated takeaways. (openreview.net) (openreview.net)