Users run 35B models offline

X users on Friday described a wider range of local AI setups than was common even a year ago, with one post saying quantized models up to 35 billion parameters were running offline on a 16GB GPU and on repurposed “ex-government servers.” The posts pointed to a maturing ecosystem of open-weight models, quantization formats and one-command tools that let users trade some model quality for lower memory use. Separate community guides and course listings show the same pattern: local inference is moving from hobbyist tinkering toward a more standardized workflow built around Ollama, llama.cpp and similar tools. UCLA Extension has scheduled a one-day live online course on July 10 called “Running a Local LLM: AI on Your Own Computer,” describing local models as a way to gain privacy, control and offline capability. ### How are people fitting bigger models onto smaller machines? Quantization is the main reason larger models can run on modest hardware. Community documentation on GitHub describes local inference as “privacy-focused” and “cost-effective” and lists 8GB of GPU memory as a workable floor for smaller setups, with 16GB or more recommended for larger ones. The same guide highlights llama.cpp as a memory-efficient path because it is optimized for CPU and GPU inference and supports quantized model files. (uclaextension.edu) The Friday social posts did not amount to a benchmark suite, and they should be read as user reports rather than lab-tested comparisons. But they are consistent with the broader local-inference playbook in which users compress weights, accept slower output or partial offloading, and run models that would otherwise exceed the memory budget of a laptop or midrange desktop GPU. (github.com) ### Which models are people recommending for 8GB-class setups? Community recommendations cited Friday centered on smaller open-weight models rather than the 35B-class systems. The list circulating in social posts included Phi-4-mini at 3.8B parameters, Qwen3 4B and 8B, Gemma 3 4B and Llama 3.2 3B, with Ollama named as the easiest entry point for many users. Those sizes line up with the hardware guidance in public local-LLM tutorials that frame 1B-to-7B models as the practical range for consumer devices with tight memory limits. (github.com) GitHub guides also point new users to multiple front ends and runtimes, including Ollama, Hugging Face Transformers, vLLM, LM Studio and llama.cpp. That menu matters because the model choice is only part of the setup; users also need a runtime that matches their hardware and the quantized format they downloaded. ### Why are users bothering to run models offline at all? (mljourney.com) UCLA Extension’s course listing answers that directly, describing local language models as a way to keep work on a user’s own computer while providing privacy, control and offline capability. GitHub documentation aimed at practitioners uses similar language, describing local inference as a way to avoid cloud dependence while lowering recurring costs. (github.com) Those benefits help explain why the discussion has spread beyond enthusiasts building custom rigs. A one-day UCLA Extension course scheduled for July 10 says it will cover how to install, configure and run lightweight LLMs locally for teaching, research and business workflows. (uclaextension.edu) ### Where are newcomers finding practical instructions? GitHub repositories and short-form social posts are doing much of the onboarding. One public repository surfaced in search describes itself as a “comprehensive guide” to running LLMs locally and includes setup steps, hardware requirements and framework comparisons. UCLA Extension is also packaging the topic into a live online class, suggesting demand now extends to continuing education rather than only developer forums. (uclaextension.edu) The next public milestone is July 10, when UCLA Extension’s MGMT 781.4 course is scheduled to run live online from 12:00 p.m. to 1:00 p.m. Pacific time, according to the course page. (uclaextension.edu) (github.com)