LoRA vs QLoRA vs Unsloth tested

Training a language model usually means rewriting billions of numbers, and that is why small teams hit a wall on graphics cards with limited video memory. Fixstars tested three ways around that wall on an NVIDIA RTX PRO 6000 Blackwell Max-Q card instead of treating fine-tuning as something only data centers can do. (blog.us.fixstars.com) Low-Rank Adaptation is the oldest trick in this comparison. The original 2021 LoRA paper freezes the big model and trains small extra matrices instead, which cuts the number of trainable parameters by orders of magnitude. (arxiv.org) You can picture Low-Rank Adaptation like adding a thin set of sticky notes to a textbook instead of rewriting every page. Unsloth’s documentation says this usually means changing about 1 percent of weights rather than the whole model. (unsloth.ai) Quantized Low-Rank Adaptation goes one step further by shrinking the frozen base model itself before training starts. The 2023 QLoRA paper stores base weights in 4-bit form with NormalFloat 4, adds double quantization, and uses paged optimizers to control memory spikes. (arxiv.org) That trade changes the hardware math. Hugging Face’s Transformers docs describe QLoRA as 4-bit quantization plus Low-Rank Adaptation adapters, which is why teams use it when a 16-bit base model will not fit on a single workstation card. (huggingface.co) Unsloth is not a third tuning method in the same sense as Low-Rank Adaptation or Quantized Low-Rank Adaptation. Unsloth is a training library that says it speeds up Low-Rank Adaptation, Quantized Low-Rank Adaptation, full fine-tuning, and reinforcement learning with custom Triton kernels and lower video memory use. (github.com) Fixstars set up the comparison around real size limits instead of toy examples. Their February 4, 2026 write-up says they benchmarked Qwen3-8B, Qwen3-32B, and GPT-OSS-120B, then focused the main comparison on Qwen3-32B and GPT-OSS-120B because Qwen3-8B finished too quickly to be very informative. (blog.us.fixstars.com) The headline result was not subtle. Fixstars reports up to a 3x speedup and about an 80 percent memory reduction with Unsloth compared with standard Low-Rank Adaptation on that workstation setup. (blog.us.fixstars.com) They also say they fine-tuned GPT-OSS-120B on a single GPU workstation, which is the kind of sentence that used to sound unrealistic for a 120 billion parameter model. In the same post, Fixstars says the RTX PRO 6000 Blackwell Max-Q beat an H100 SXM5 on GPT-OSS-120B fine-tuning, while the H100 SXM5 stayed faster on Qwen3-32B. (blog.us.fixstars.com) That split hints at what teams are really choosing between. Standard Low-Rank Adaptation is simpler when you have enough video memory, Quantized Low-Rank Adaptation is the escape hatch when the model does not fit, and Unsloth is the layer you add when you want either path to waste less time and memory. (arxiv.org 1) (arxiv.org 2) (github.com) Fixstars framed the whole exercise around overnight jobs and local model updates, not giant research clusters. If your team has one expensive card on a desk instead of a rack of servers, this comparison is basically a map of which compromise buys you the most room. (blog.us.fixstars.com)