RTX 5090 AI demo
Despite the scam headlines, an enthusiast built a powerful local AI search engine running entirely on a single GeForce RTX 5090, showcasing the card’s raw compute for demanding AI workloads (en.gamegpu.com). The demo underlines why top‑end cards remain attractive to builders who want to run heavy local models or inference tasks on one GPU (en.gamegpu.com).
The developer released the project as "SoyLM" with the entire backend in a single app.py FastAPI file on GitHub. (github.com(github.com)) (github.com) The stack pairs NVIDIA’s Nemotron Nano 9B v2 model served via the vLLM library with SQLite FTS5 for retrieval, an inline FastAPI UI, and parallel DuckDuckGo web searches for source discovery. (media.patentllm.org(media.patentllm.org) / github.com(github.com)) (media.patentllm.org) Independent benchmarks reported roughly 80–120 tokens per second for single requests, a batched throughput near 630 tok/s, and a Time‑To‑First‑Token around 45–60 ms when running Nemotron Nano 9B v2 on the setup used in the write‑ups. (dev.to(dev.to) / media.patentllm.org(media.patentllm.org)) (dev.to) The tool implements a two‑step "Extract → Execute" workflow that extracts bilingual (English+Japanese) keywords, runs OR‑joined FTS5 and DuckDuckGo searches in parallel, then prompts the user to select sources before generating the final answer to avoid overloading model context. (media.patentllm.org(media.patentllm.org)) (media.patentllm.org) The author "soy‑tuber" documents multiple related projects (SoyLM, PatentLLM, SubsidyDB) and describes using an RTX 5090-equipped desktop alongside a lightweight always‑on server to process millions of patents for local search experiments. (github.com(github.com) / dev.to(dev.to)) (github.com) NVIDIA’s Nemotron model family and the Nano 9B v2 release notes confirm the model’s design for high‑throughput local inference and tool calling, and the project notes reference using NVIDIA parser plugins and NIM‑compatible model packaging. (huggingface.co(huggingface.co) / docs.api.nvidia.com(docs.api.nvidia.com)) (huggingface.co)
