LoRA enables tiny fine‑tunes
- Open-source model makers showed LoRA adapters can add task-specific behavior to giant image and video generators while training only tiny weight deltas instead of full models. - Meituan’s LongCat-Video says its LoRA adapters add under 1% of parameters to a 13.6B video transformer, while Qwen-Image distillation cut generation from 40 steps to 15. - The setup lets teams swap small adapters onto frozen base models, echoing QLoRA’s memory-saving playbook. (arxiv.org)
LoRA is a way to fine-tune a model by adding small side weights instead of rewriting the whole network. The base model stays frozen, and only those small additions learn. (arxiv.org) (huggingface.co) That matters because the base models now being adapted are huge. Meituan’s LongCat-Video documentation describes a 13.6 billion-parameter video transformer, and its LoRA adapters add less than 1% extra parameters. (deepwiki.com) LongCat-Video uses two separate adapters for two separate jobs. One distills generation so inference falls from 50 steps to 16, and another handles spatial and temporal upscaling from 480p to 720p. (deepwiki.com) A similar pattern is showing up in image generation. DiffSynth-Studio published a distilled LoRA for Qwen-Image that keeps the base model but replaces full fine-tuning with adapter training. (huggingface.co 1) (huggingface.co 2) Its model card says the distilled Qwen-Image LoRA was trained on 16,000 images generated from DiffusionDB prompts, ran for about one day on eight AMD MI308X graphics processors, and targets 15 inference steps instead of 40. (huggingface.co) The practical appeal is storage and serving. Hugging Face’s LoRA documentation says these adapters are often only a few hundred megabytes, which makes them easier to store, share, and swap in for different tasks. (huggingface.co) That swap-in model also helps operators keep one heavyweight base model online while loading different customer or task adapters as needed. LongCat-Video’s docs describe loading adapters once, enabling them for a generation stage, then disabling them to avoid interference. (deepwiki.com) The idea comes from older language-model work. The original LoRA paper said the method could cut trainable parameters by 10,000 times and reduce GPU memory needs by three times versus full fine-tuning on GPT-3-scale systems. (arxiv.org) QLoRA pushed the same logic further by combining frozen quantized base models with LoRA adapters. Its authors said that setup let a 65 billion-parameter model be fine-tuned on a single 48GB graphics card while preserving full 16-bit fine-tuning performance. (arxiv.org) What is new in these image and video examples is not the LoRA idea itself, but where it is landing. The same low-parameter trick now shows up on diffusion transformers and multimodal generators, where full retraining is even harder to ship. (deepwiki.com) (huggingface.co) The result is a simple production pattern: keep the giant model fixed, move the small deltas around, and pay the training bill on the adapter instead of the whole system. (huggingface.co) (arxiv.org)
