Ollama Extremely Slow (1-2 tok/s)? Stopping CPU Offloading
How to Keep 100% of LLM Layers in Fast GPU VRAM
🧮 Root Cause & Mathematical Reality
If a model's weights plus context buffer exceed GPU VRAM by even 500MB, Ollama automatically spills layers into system RAM. System DDR5 RAM bandwidth (~60 GB/s) is 16x slower than GPU VRAM (~1,000 GB/s), causing token speeds to collapse from 35 tok/s to 1.5 tok/s.
- GPU GDDR6X bandwidth is ~1,008 GB/s; dual-channel DDR5 system RAM is only ~60–80 GB/s.
- Offloading just 4 out of 64 transformer layers to CPU bottlenecks the entire pipeline to system bus speeds.
- Ollama defaults to running multiple parallel request slots unless OLLAMA_NUM_PARALLEL=1 is specified.
🛠️ Actionable Fix & Tuning Steps
- Check Ollama server logs: Look for 'offloaded X/Y layers to GPU'. If X < Y, CPU offload is active.
- Reduce context size in Modelfile or CLI: Set 'num_ctx 8192' instead of 32768 or 65536.
- Force single-stream allocation: Set environment variable 'OLLAMA_NUM_PARALLEL=1'.
- Enable FlashAttention: Set environment variable 'OLLAMA_FLASH_ATTENTION=1'.
- Switch to a slightly smaller quantization: Use Q4_K_S or Q3_K_M instead of Q5_K_M or Q8_0.
⚖️ The 3-Tier Honesty Funnel
Enable FP8/Q8_0 KV-cache or clamp context tokens to fit 100% within your local GPU memory.
If model weights or 128k context exceed physical VRAM, rent an 80GB GPU by the hour.
❓ Frequently Asked Questions
How can I tell if Ollama is using my GPU?
Run 'ollama ps' while generating text. It will display the model name, size, and the percentage loaded in VRAM (e.g. '100% GPU').
Why does Ollama use so much RAM on Apple Silicon Macs?
Apple Silicon uses unified memory shared between CPU and GPU. macOS allocates up to ~75% of total system RAM to Metal by default.
Explore the Full OOM Troubleshooting Hub
Master VRAM requirements for LLaMA 3.3 70B, DeepSeek V3, Qwen 2.5, and vLLM concurrency.
View All 12 GPU & VRAM Fixers →