Can I run Llama 3.3 70B locally?
Llama 3.3 70B is a dense model: all 70 billion parameters are read for every token, so it needs far more memory than a mixture-of-experts model of similar quality. It uses grouped-query attention with 8 KV heads across 80 layers, which keeps the KV cache moderate for its size.
The tables below use the real GGUF file sizes from Hugging Face and add the KV cache and runtime overhead. They answer the usual question: does it fit on one 24 GB card, on two, or do you need a Mac with lots of unified memory or a rented GPU?
Estimates with default assumptions: 10% runtime overhead, ~90% usable VRAM (~72% on Macs), FP16 KV cache, one sequence. Model data synced from Hugging Face on 2026-10-18. Methodology.
Llama 3.3 70B Instruct: memory by quantization
70.6B parameters, 80 layers. Weights are real GGUF file sizes from unsloth/Llama-3.3-70B-Instruct-GGUF.
| Quantization | Weights | Total at 8K context | Total at 32K context |
|---|---|---|---|
| F16 | 131.4 GB | 147.1 GB | 154.6 GB |
| Q8_0 | 69.8 GB | 79.3 GB | 86.8 GB |
| Q6_K | 53.9 GB | 61.8 GB | 69.3 GB |
| Q5_K_M | 46.5 GB | 53.7 GB | 61.2 GB |
| Q4_K_M | 39.6 GB | 46.1 GB | 53.6 GB |
| IQ4_XS | 35.3 GB | 41.4 GB | 48.9 GB |
| Q3_K_M | 31.9 GB | 37.6 GB | 45.1 GB |
| Q2_K | 24.6 GB | 29.5 GB | 37.0 GB |
Which hardware runs Llama 3.3 70B Instruct?
Highest-quality quantization that fits fully. “Offload” means it runs only with part of the model in system RAM (64 GB assumed).
| Hardware | 8K context | 32K context |
|---|---|---|
| NVIDIA GeForce RTX 4090 | Offload Q2_K (~58 layers on GPU) | Offload Q2_K (~46 layers on GPU) |
| 2× NVIDIA GeForce RTX 4090 | Q3_K_M (37.6 GB) | Q2_K (37.0 GB, big quality loss) |
| 2× NVIDIA GeForce RTX 5090 | Q5_K_M (53.7 GB) | Q4_K_M (53.6 GB) |
| NVIDIA RTX A6000 (48 GB) | IQ4_XS (41.4 GB) | Q2_K (37.0 GB, big quality loss) |
| Mac with M4 Max (64 GB) | Q4_K_M (46.1 GB) | Q3_K_M (45.1 GB) |
| Mac Studio M3 Ultra (96 GB) | Q6_K (61.8 GB) | Q5_K_M (61.2 GB) |
| Mac with M4 Max (128 GB) | Q8_0 (79.3 GB) | Q8_0 (86.8 GB) |
| NVIDIA H100 SXM (80 GB) | Q6_K (61.8 GB) | Q6_K (69.3 GB) |
Try your own setup
Result
Fits with partial CPU offload
Llama 3.3 70B Instruct · Q4_K_M · 8K → 2× NVIDIA GeForce RTX 4090. ~71 of 80 layers on GPU, ~5.18 GB in system RAM. Expect much slower generation than a full fit.
| Component | Memory |
|---|---|
| Weightsactual GGUF file: Q4_K_M | 39.6 GB |
| KV cache8K tokens × 1 seq, F16 | 2.50 GB |
| Runtime overhead10% of weights | 3.96 GB |
| Total needed | 46.1 GB |
| Usable on your hardware90% of 48.0 GB | 41.0 GB |
What would make it fit
- Use Q3_K_M at this context (37.6 GB).
- Even 512 tokens of context do not fit with Q4_K_M.
- Smallest Mac memory tier: Mac mini M5 Pro (64 GB).
- Cloud: NVIDIA A100 (80 GB) (rent by the hour).
GPU cloud providers (not sponsored): RunPod · Vast.ai · DigitalOcean GPU Droplets · Lambda
Run it
Starting point, not a guarantee. Downloads the GGUF from Hugging Face.
llama-server -hf unsloth/Llama-3.3-70B-Instruct-GGUF:Q4_K_M -c 8192 -ngl 71OLLAMA_CONTEXT_LENGTH=8192 ollama serve
ollama run hf.co/unsloth/Llama-3.3-70B-Instruct-GGUF:Q4_K_MOllama picks GPU layers automatically; to force the estimate, run /set parameter num_gpu 71 in the chat.
Model details
- Parameters
- 70.6BDerived from the BF16 GGUF size (141117917856 bytes / 2 bytes per weight), text model only
- Layers
- 80
- Attention
- Full attention (GQA)80× full (8 KV × 128)
- Max context
- 128K
- License
- llama3.3
- Config source
- unsloth/Llama-3.3-70B-Instructsynced 2026-10-18
Questions
Can Llama 3.3 70B run on a single RTX 4090 or 3090?
Not fully on the GPU at any usable quantization: even Q2_K weights are larger than the ~21.6 GB usable on a 24 GB card once the KV cache and overhead are added. It runs with partial CPU offload if you have enough system RAM, but generation is much slower. See the hardware table above for the exact estimate.
Is two 24 GB GPUs enough for Llama 3.3 70B?
Two 24 GB GPUs give about 41 GB usable after the default reserve and split loss. In our estimate that fits Q3_K_M at 8K context; IQ4_XS lands just over the limit, and Q4_K_M needs about 46 GB at 8K. At 32K context only Q2_K fits fully.
Which Mac can run Llama 3.3 70B?
With the default 72% of unified memory usable, a 64 GB Mac holds Q4_K_M at 8K context right at the limit and drops to Q3_K_M at 32K. 96 GB fits Q6_K at 8K and 128 GB fits Q8_0. Memory bandwidth, not capacity, limits the speed.
How much does the context length add?
With 80 layers × 8 KV heads × 128 dimensions, each token stores 320 KB at FP16. That is 2.5 GiB at 8K tokens and 10 GiB at 32K. A Q8_0 KV cache roughly halves this.