Looper transformer research

A transformer usually gets smarter by adding more layers; these new papers test a different idea: run the same layer stack again and again instead. (arxiv.org) In a standard transformer, each layer has its own weights, so deeper models usually mean more parameters and more memory. In a looped or depth-recurrent model, the network reuses one block of layers for multiple passes, like rereading the same paragraph instead of printing a longer book. (sandyresearch.github.io) The catch has been training stability. The Parcae paper, posted to arXiv on April 14, 2026 by researchers at the University of California San Diego and Together AI, says earlier looped models often suffered “residual explosion” and loss spikes during training. (arxiv.org) Parcae tries to fix that by constraining how much each loop can amplify the model’s internal state. The authors describe the loop as a dynamical system and say their design keeps the recurrent update stable enough to train at scale. (arxiv.org) On results, the paper reports up to 6.3% lower validation perplexity than prior large-scale looped models. Perplexity is the standard next-token prediction score in language modeling, and lower is better. (arxiv.org) The team also says a 770 million-parameter Parcae model matched the quality of a 1.3 billion-parameter transformer trained on the same data, or roughly the same performance with about half the parameters. In the paper’s larger-scale comparison, a 1.3 billion-parameter Parcae model improved CORE and Core-Extended scores by 2.99 and 1.18 points over transformer baselines under a fixed parameter and data budget. (sandyresearch.github.io; arxiv.org) The paper’s other claim is about scaling laws, the empirical rules labs use to predict how model quality changes as they add compute, data, or parameters. Parcae says looping can become its own scaling axis, with training gains when recurrence and data rise together and test-time gains that taper off in a predictable curve. (arxiv.org; github.com) That matters for labs chasing smaller models that can run on limited hardware. The Together AI and UC San Diego write-up says the approach could help “memory-constrained on-device models,” because extra passes through the same block add compute without doubling the stored weights. (together.ai; sandyresearch.github.io) A separate January 2026 paper points in a similar direction, though with a different architecture. “Depth-Recurrent Attention Mixtures,” from researchers at Aleph Alpha Research, Technical University of Munich, TU Darmstadt and collaborators, describes a fully depth-recurrent single-layer setup that reuses parameters across depth for language reasoning tasks. (arxiv.org) That paper reports its models needed 2 to 8 times fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched baselines, and outperformed models about twice as large on language reasoning benchmarks. The authors frame the tradeoff as latent reasoning in continuous hidden states rather than longer chains of printed words. (arxiv.org) Not every question is settled. Parcae is an arXiv preprint rather than a peer-reviewed conference paper, and another 2025 analysis of a depth-recurrent transformer found only limited evidence that these models carry out an interpretable hidden “chain of thought” inside the loop. (arxiv.org; arxiv.org) What the new papers do show is a concrete alternative to the usual “make it bigger” recipe. Instead of buying depth with new weights every time, they try to buy it by sending the same circuitry around the track one more lap. (arxiv.org; arxiv.org)