What LLM can I run on an RTX 4090?
The RTX 4090 has 24 GB of GDDR6X. After the driver and display reserve (we assume ~90% usable), about 21.6 GB is left for weights, KV cache and runtime buffers. That is the dividing line in the tables below.
Each row shows the highest-quality GGUF quantization that fits completely on the GPU. Models that do not fit can still run with partial CPU offload, which the table marks with the estimated number of GPU layers.
Estimates with default assumptions: 10% runtime overhead, ~90% usable VRAM (~72% on Macs), FP16 KV cache, one sequence. Model data synced from Hugging Face on 2026-10-18. Methodology.
Models on NVIDIA GeForce RTX 4090 (24 GB)
36 of 44 models fit fully at 8K context. Memory shown is the total estimate for the best fitting quantization. “Offload” assumes 64 GB system RAM.
| Model | Params | 8K context | 32K context |
|---|---|---|---|
| Mixtral 8x7B Instruct (MoE) | 46.7B | Q2_K (17.0 GB, big quality loss) | Q2_K (20.0 GB, big quality loss) |
| Qwen3.6 35B-A3B (MoE) | 35.5B | IQ4_XS (20.4 GB) | IQ4_XS (20.9 GB) |
| DeepSeek R1 Distill Qwen 32B | 32.8B | Q3_K_M (18.3 GB) | Q2_K (20.6 GB, big quality loss) |
| Qwen3 32B | 32.8B | IQ4_XS (20.1 GB) | Q2_K (20.6 GB, big quality loss) |
| Qwen2.5 Coder 32B Instruct | 32.8B | IQ4_XS (20.1 GB) | Q2_K (20.6 GB, big quality loss) |
| Gemma 4 31B | 30.7B | Q4_K_M (20.2 GB) | IQ4_XS (20.1 GB) |
| Qwen3 Coder 30B-A3B (MoE) | 30.5B | Q4_K_M (19.8 GB) | IQ4_XS (19.8 GB) |
| Qwen3 30B-A3B Instruct 2507 (MoE) | 30.5B | Q4_K_M (19.8 GB) | IQ4_XS (19.8 GB) |
| GLM-4.7 Flash (MoE) | 30.0B | Q4_K_M (19.2 GB) | Q4_K_M (20.4 GB) |
| Qwen3.8 27B | 27.3B | Q5_K_M (20.9 GB) | Q4_K_M (19.0 GB) |
| Gemma 3 27B | 27.0B | Q5_K_M (20.8 GB) | Q4_K_M (19.9 GB) |
| Qwen3.6 27B | 26.9B | Q5_K_M (20.6 GB) | Q4_K_M (19.4 GB) |
| Qwen3.5 27B | 26.9B | Q5_K_M (20.7 GB) | Q4_K_M (19.3 GB) |
| Gemma 4 26B-A4B (MoE) | 25.3B | Q5_K_M (20.1 GB) | Q5_K_M (20.6 GB) |
| Devstral Small 2 24B | 23.6B | Q6_K (21.1 GB) | Q4_K_M (19.7 GB) |
| Mistral Small 3.2 24B | 23.6B | Q6_K (21.1 GB) | Q4_K_M (19.7 GB) |
| gpt-oss 20B (MoE) | 21.0B | MXFP4 (12.6 GB) | MXFP4 (13.2 GB) |
| Qwen2.5 Coder 14B Instruct | 14.8B | Q8_0 (17.6 GB) | Q6_K (18.4 GB) |
| DeepSeek R1 Distill Qwen 14B | 14.8B | Q8_0 (17.6 GB) | Q6_K (18.4 GB) |
| Qwen3 14B | 14.8B | Q8_0 (17.3 GB) | Q8_0 (21.1 GB) |
| Phi-4 14B | 14.7B | Q8_0 (17.5 GB) | Q8_0 (19.1 GB) |
| Mistral Nemo 12B Instruct | 12.3B | Q8_0 (14.6 GB) | Q8_0 (18.3 GB) |
| Gemma 4 12B | 11.9B | Q8_0 (13.4 GB) | Q8_0 (13.8 GB) |
| Gemma 3 12B | 11.8B | Q8_0 (13.6 GB) | Q8_0 (15.1 GB) |
| Qwen3.5 9B | 9.0B | F16 (18.7 GB) | F16 (19.4 GB) |
| Qwen3 8B | 8.2B | F16 (17.9 GB) | F16 (21.3 GB) |
| Llama 3.1 8B Instruct | 8.0B | F16 (17.5 GB) | F16 (20.5 GB) |
| Qwen2.5 7B Instruct | 7.6B | F16 (16.0 GB) | F16 (17.4 GB) |
| Mistral 7B Instruct v0.3 | 7.2B | Q8_0 (8.89 GB) | Q8_0 (11.9 GB) |
| Qwen3.5 4B | 4.2B | F16 (8.93 GB) | F16 (9.68 GB) |
| Qwen3 4B Instruct 2507 | 4.0B | F16 (9.37 GB) | F16 (12.7 GB) |
| Gemma 3 4B | 3.9B | F16 (8.23 GB) | F16 (8.70 GB) |
| Phi-4 mini 3.8B | 3.8B | F16 (8.87 GB) | F16 (11.9 GB) |
| Llama 3.2 3B Instruct | 3.2B | F16 (7.47 GB) | F16 (10.1 GB) |
| Qwen2.5 3B Instruct | 3.1B | F16 (6.61 GB) | F16 (7.45 GB) |
| Llama 3.2 1B Instruct | 1.2B | F16 (2.79 GB) | F16 (3.54 GB) |
| DeepSeek R1 0528 671B (MoE) | 671B | Doesn’t fit | Doesn’t fit |
| Qwen3 235B-A22B Instruct 2507 (MoE) | 235B | Doesn’t fit | Doesn’t fit |
| Qwen3.8 Flash Next | 177B | Doesn’t fit | Doesn’t fit |
| gpt-oss 120B (MoE) | 117B | Offload MXFP4 (~11 layers on GPU) | Offload MXFP4 (~11 layers on GPU) |
| GLM-4.5 Air 106B (MoE) | 110B | Offload Q2_K (~20 layers on GPU) | Offload Q2_K (~19 layers on GPU) |
| Llama 4 Scout 17B-16E (MoE) | 108B | Offload Q2_K (~24 layers on GPU) | Offload Q2_K (~24 layers on GPU) |
| Qwen2.5 72B Instruct | 72.7B | Offload Q2_K (~52 layers on GPU) | Offload Q2_K (~42 layers on GPU) |
| Llama 3.3 70B Instruct | 70.6B | Offload Q2_K (~58 layers on GPU) | Offload Q2_K (~46 layers on GPU) |
Try your own setup
Loading model list…
Questions
What is the biggest model that runs well on a 4090?
Dense 32B models fit at about 4-bit (IQ4_XS) with an 8K context, and 24–27B models fit at Q5_K_M or Q6_K. Mixture-of-experts models such as Qwen3 30B-A3B fit at Q4_K_M, and gpt-oss 20B fits in its native MXFP4 format with plenty of room; both run fast because only a few billion parameters are active per token. 70B models need offload or a second GPU.
Is a used RTX 3090 just as good for LLMs?
For capacity, yes: both have 24 GB, so the same models fit. The 4090 is faster at prompt processing; generation speed is mostly limited by memory bandwidth, which is similar (about 1 TB/s on both).
Should I quantize the KV cache?
If you want long context on 24 GB, a Q8_0 KV cache halves its size with little quality loss in most reports. It needs flash attention enabled (llama.cpp -fa on, or OLLAMA_FLASH_ATTENTION=1).