← BASELOAD English

RTX 5090 Local LLM — How Many Billion Parameters Can It Run? VRAM Guide

July 8, 2026 · 32 min read 한국어 원문

If you’ve bought an RTX 5090 — or you’re weighing whether to — one question sits near the top of the list: “How many billion parameters of local LLM will this thing actually run?” The short answer: with its 32GB of VRAM, the RTX 5090 runs a 32B model very comfortably, and a 70B model only if you crank the quantization hard or offload part of it to the CPU. This piece walks through why that’s the answer — the VRAM math, feasibility and speed by model size, and hands-on Ollama tips — all in one place. If local LLMs are new to you, start with the Local LLM Beginner’s Guide [Part 1].

The Verdict First — What the RTX 5090 Can Do

The RTX 5090 ships with 32GB of GDDR7 VRAM. For local LLMs, the thing that matters most isn’t raw GPU compute — it’s this VRAM capacity. When a model fits entirely in VRAM it runs fast; when it overflows, it slows to a crawl or won’t run at all. Here’s the one-line summary at 32GB:

• 7B–14B: Roomy and very fast. Nothing to think about.
• 32B: The sweet spot. Loads comfortably at Q4–Q8 quantization, and the speed is practical (40–55 tok/s).
• 70B: Q4 (about 40GB) won’t fit in 32GB. Even Q3 (about 34GB) slightly overshoots 32GB, so you need offloading — pushing part of the model to the CPU (force it on without offloading and you’ll get around 20 tok/s; spill into system RAM and it drops to single digits).
• 100B and up: Too much for a single 5090. You’d need multiple GPUs or aggressive quantization.

Photo by Matheus Bertelli on Pexels

The VRAM Math — How Many B Is How Many GB?

“How many B will run” ultimately comes down to does the model fit in VRAM. You estimate the memory you need roughly like this:

Required VRAM ≈ (parameter count) × (bytes per parameter) + headroom for context

Bytes per parameter are set by the level of quantization. Quantization is the technique of compressing a model’s weights into fewer bits to shrink its memory footprint.

In other words, the same model can need several times more or less memory depending on its quantization. The most widely used level for local work is Q4 (a balance of quality and size), dropping to Q3 when VRAM is tight. On top of that, the KV cache eats additional VRAM in proportion to context length — meaning the longer the document you feed in, the more memory you need — so you should leave headroom above the numbers above.

The Full Breakdown by Model Size — On an RTX 5090 (32GB)

Plug the math above into the 5090’s 32GB and you get this table. Figures are for Q4, and speeds are approximate.

To sum up, the 5090’s realistic ceiling is “32B at Q4, 70B at Q3.” A 32B model is about 20GB at Q4, so it leaves generous context headroom — and if you want, you can bump it up to Q6–Q8 for even higher quality. A 70B model, by contrast, blows well past 32GB at Q4, and even the quality-sacrificing Q3 (about 34GB) slightly overshoots 32GB, so offloading some layers to CPU memory becomes unavoidable. And that costs you a lot of speed.

Ollama in Practice — How to Actually Run It

Run it through Ollama and most of this is automatic. When you pull a model, you pick the quantization with a tag.

# 32B model (the 5090's sweet spot)
ollama run qwen2.5:32b

# For 70B, tighten the quantization tag (Q3 family)
ollama run llama3.3:70b-instruct-q3_K_M

Three practical points worth remembering:

• Ollama offloads automatically. If a model is bigger than your VRAM, it doesn’t die — it automatically pushes the overflowing layers to the CPU/RAM. But it slows down accordingly. Force a 70B onto a 5090 and it will run, but you have to accept the speed hit.
• Mind your context length. The longer the document you feed in, the more VRAM the KV cache eats. Even a 32B you’d been running with room to spare can get tight if you set a large context.
• Check the tag (Q4_K_M, etc.). The same model differs in size and quality depending on the tag. Make sure the default fits your VRAM before you pull it.

5090 vs 4090 vs Mac — a Quick Comparison

• RTX 4090 (24GB): With 8GB less VRAM than the 5090, 70B is even tighter and 32B is effectively the ceiling. The 5090’s 32GB is what opened the door to “you can at least touch a 70B, conditionally.”
• Mac (unified memory): A Mac’s structure has the GPU share system RAM, so a 64GB or 128GB Mac can, from a pure VRAM standpoint, “load” larger models than a 5090. That said, the 5090 leads on bandwidth and compute speed, so for the same model the 5090 is generally faster. (Details by Mac model in the next post.)

So What — “Enough for the Job” Beats “How Many B”

The real answer to “does a 5090 run 70B?” is closer to “why bother?” In many cases, running a 32B comfortably at high-quality Q6–Q8 gives more real-world satisfaction than grinding a 70B out at Q3. The performance of recent 32B-class models has climbed so much that most practical work — coding, summarization, document Q&A — is well served by 32B.

So here’s the conclusion: it’s most accurate to see the RTX 5090 as a “local LLM machine that runs 32B smoothly.” The 70B is a “possible-but-with-compromises bonus.” Rather than getting hung up on the hardware-spec number (“how many B”), decide first what model size is enough for your work — and the 5090 will comfortably handle most of an individual’s local-AI needs.

Local LLM Series

• Part 1 — Running AI on Your Own Computer
• Part 2 — Running a Local LLM on Your iPhone
• Hermes AI × LM Studio Local Integration

Sources

• ModelFit — RTX 5090 Local LLM Benchmarks (32GB, tok/s by model)
• InsiderLLM — VRAM Requirements Cheat Sheet (required VRAM by quantization)
• ToolHalla — Best Local LLMs for RTX 5090 (2026) (model recommendations & quantization)