Free SGLang inference course

A new DeepLearning.AI course tries to teach one of the least glamorous parts of artificial intelligence: making large models cheaper to run after they are already trained. The course is called “Efficient Inference with SGLang: Text and Image Generation,” and DeepLearning.AI describes it as a hands-on class built with LMSYS and RadixArk, taught by RadixArk engineer Richard Chen. (learn.deeplearning.ai) The course is short by machine learning standards. DeepLearning.AI lists it at 1 hour and 20 minutes, which puts it closer to a practical workshop than a semester-style lecture series. (learn.deeplearning.ai) To understand why that matters, start with what “inference” means. Training is when a model learns from huge datasets, but inference is the everyday job of taking a prompt, reading it token by token, and generating an answer one token at a time. (docs.sglang.io) That second phase is where many companies actually spend their money. A model that already exists still has to repeatedly process prompts, conversation history, and system instructions every time a user sends a request. (learn.deeplearning.ai) One of the central ideas in the course is the key-value cache, usually shortened to KV cache. In transformer models, that cache stores intermediate attention data from earlier tokens so the model does not have to recompute the same internal work from scratch for every new generated token. (lmsys.org) A simple way to picture the key-value cache is autocomplete that remembers the part of the sentence it already read. If a model has already processed a 2,000-token prompt, the cache lets it build on that work instead of rereading all 2,000 tokens every time it produces the next word. (lmsys.org) That helps within a single request, but it does not solve the whole problem. Many production workloads contain repeated prefixes, such as the same system prompt, the same retrieved documents, or the same earlier turns in a chat, and ordinary serving systems often recompute those shared sections again and again. (lmsys.org) This is where RadixAttention comes in. The SGLang team describes it as a method that keeps KV cache entries in a radix tree so different requests with the same token prefix can automatically share cached work at runtime. (arxiv.org, lmsys.org) A radix tree is basically a filing system for strings that start the same way. Instead of storing ten nearly identical prompts as ten separate full copies, the system stores the shared beginning once and branches only where the prompts diverge. (mintlify.com) SGLang itself is the serving framework around these ideas. Its documentation describes it as a high-performance framework for large language models and multimodal models, built for low-latency and high-throughput inference from a single graphics processing unit to distributed clusters. (docs.sglang.io) The course leans into that practical layer rather than pure theory. DeepLearning.AI says learners will build a mental model of inference cost, implement RadixAttention to extend caching across users and requests, and measure the speedups directly, then apply similar ideas to diffusion-based image generation. (learn.deeplearning.ai, youtube.com) That makes the release notable for a specific audience: engineers who do not want to redesign model architectures or write a research paper just to cut latency. The pitch is closer to “use better serving tricks on the models you already have” than “invent a new model from scratch.” (learn.deeplearning.ai) The underlying technology already has a research pedigree. The original SGLang paper reported up to 6.4 times higher throughput than comparison systems on a range of large language and multimodal workloads, while the project paper and blog both frame RadixAttention as a core mechanism for reusing KV cache across generation calls. (arxiv.org, lmsys.org) DeepLearning.AI’s contribution is packaging those ideas into a free, compact format at a moment when inference costs have become a frontline engineering problem. For teams already serving chatbots, agents, or image models, an 80-minute course on caching and prefix reuse is really a course about getting more output from the same hardware. (learn.deeplearning.ai)